system

Abstract

The system according to this embodiment aims to facilitate communication with the person receiving care and to support health management. [Solution] The system according to the embodiment comprises a conversation unit, a video unit, an observation unit, and a notification unit. The conversation unit engages in conversation with the person receiving care. The video unit displays images based on the information obtained by the conversation unit. The observation unit observes changes in the physical condition of the person receiving care and provides advice. The notification unit provides notifications in emergencies based on the information obtained by the observation unit.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the conventional technology, communication with care recipients and physical condition management are not sufficiently carried out, and there is room for improvement.

[0005] The system according to the embodiment aims to smooth communication with care recipients and support physical condition management.

Means for Solving the Problems

[0006] The system according to the embodiment includes a conversation unit, a video unit, an observation unit, and a notification unit. The conversation unit conducts conversations with care recipients. The video unit projects videos based on the information obtained by the conversation unit. The observation unit observes changes in the physical condition of care recipients and provides advice. The notification unit issues notifications in case of emergencies based on the information obtained by the observation unit. [Effects of the Invention]

[0007] The system according to this embodiment can facilitate communication with the person receiving care and support health management. [Brief explanation of the drawing]

[0008] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Modes for carrying out the invention]

[0009] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0010] First, let's explain the terminology used in the following explanation.

[0011] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), or TPU (Tensor Processing Unit).

[0012] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0013] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0014] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0015] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B". That is, "A and / or B" means that it may be only A, only B, or a combination of A and B. Also, in this specification, when expressing three or more matters connected by "and / or", the same concept as "A and / or B" is applied.

[0016] [First Embodiment] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0017] As shown in FIG. 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device is a server.

[0018] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. Also, the database 24 and the communication I / F 26 are connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0019] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. Also, the reception device 38, the output device 40, and the camera 42 are connected to the bus 52.

[0020] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, and accepts user input. The touch panel 38A accepts user input via touch by detecting contact with an object (e.g., a pen or finger). The microphone 38B accepts user input via voice by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 (see Figure 2) acquires the data indicating the user input.

[0021] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user by outputting the data in a form perceptible to the user (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0022] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0023] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0024] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0025] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0026] In the smart device 14, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The specific processing program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 operating as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart device 14 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0027] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device (e.g., a generation server) may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device having the data generation model 58. The data processing device 12 may also be a server device or a terminal device owned by a user (e.g., a mobile phone, robot, home appliance, etc.). Next, an example of processing by the data processing system 10 according to the first embodiment will be described.

[0028] (Example of form 1) The caregiving communication robot system according to an embodiment of the present invention is a communication robot system for people who require care, such as those who are paralyzed on one side due to a stroke, the elderly, and those with dementia. This system has the function of being able to converse normally with the person being cared for and to consult about anything. Not only does it answer questions, but it can also sometimes scold or encourage and initiate conversations itself. It functions as a monitoring robot like a family member or friend. Furthermore, this system can project images onto walls and other surfaces. For example, by projecting family photos, natural landscapes, World Heritage sites, ruins, or buildings, the person being cared for, who has difficulty traveling freely, can experience the feeling of being on a trip. The voice and tone of voice are not robotic, and multiple patterns can be selected. For example, it can be selected according to the preferences of the person being cared for, such as an adult woman, a child of elementary school age, or an elderly person. The robot's size ranges from palm-sized to about the size of a cat, and its shape is that of a soft, warm, pet-like plush toy. It can also be dressed up with clothes. It also has an notification function, monitoring the person being cared for, observing changes in their physical condition throughout the day, and providing advice. The system also has functions to notify users of medication times and other regularly needed information, and regular notifications are sent to guardians (family members, etc.) about the user's condition. In emergencies, it also has a function to notify users by phone and email. In this way, the caregiving communication robot system can support the lives of those receiving care and provide a sense of security.

[0029] The caregiving communication robot system according to this embodiment comprises a conversation unit, a video unit, an observation unit, and a notification unit. The conversation unit engages in conversation with the person receiving care. The conversation unit recognizes the person receiving care's statements using, for example, speech recognition technology and generates an appropriate response. The conversation unit can also provide appropriate answers to the person receiving care's questions using natural language processing technology. Furthermore, the conversation unit can estimate the person receiving care's emotions using emotion recognition technology and adjust the tone and content of its responses. For example, if the person receiving care is sad, the conversation unit will converse in a comforting and gentle tone. If the person receiving care is agitated, the conversation unit can converse in a calming and gentle tone. If the person receiving care is tired, the conversation unit can converse in a relaxing and gentle tone. The video unit displays images based on the information obtained by the conversation unit. The video unit can project images onto a wall using, for example, a projector. The video unit can also display images using a display. Furthermore, the video unit can select the content of the images according to the preferences of the person receiving care. For example, the video unit can display family photos, natural landscapes, World Heritage sites, ruins, and buildings. The observation unit observes changes in the care recipient's physical condition and provides advice. The observation unit measures vital signs such as the care recipient's heart rate, body temperature, and blood pressure using sensors, for example. It can also observe the care recipient's facial expressions and movements using a camera. Furthermore, the observation unit can monitor changes in the care recipient's physical condition in real time and issue an alert if an abnormality is detected. The notification unit provides notifications in emergencies based on the information obtained by the observation unit. The notification unit can send emergency notifications to the care recipient's family or medical institutions, for example, by phone or email. It can also send notifications using a smartphone app. Furthermore, the notification unit can adjust the content and timing of notifications according to changes in the care recipient's physical condition. As a result, the care communication robot system according to this embodiment can support the care recipient's life and provide a sense of security.

[0030] The conversation unit engages in conversations with the person receiving care. For example, the conversation unit uses speech recognition technology to recognize what the person receiving care says and generates appropriate responses. Specifically, speech recognition technology converts the person receiving care's statements into text data, and then uses natural language processing technology to generate responses based on that text data. Natural language processing technology uses algorithms to analyze the intent of the person receiving care's questions and statements and provide appropriate answers. For example, if the person receiving care asks, "What day is it today?", the conversation unit refers to calendar information and replies, "Today is the day of ○○." The conversation unit can also use emotion recognition technology to estimate the person receiving care's emotions and adjust the tone and content of the response. Emotion recognition technology analyzes data such as tone and speed of speech and facial expressions to estimate the person receiving care's emotional state. For example, if the person receiving care is sad, the conversation unit will respond in a comforting, gentle tone, "What's wrong? Please tell me what happened." If the person receiving care is agitated, the conversation unit can respond in a calming, gentle tone, "It's okay, let's talk slowly." Furthermore, if the person being cared for is tired, the conversational unit can suggest, in a calm and relaxing tone, "Shall we take a short break?" This allows the conversational unit to communicate appropriately according to the feelings and condition of the person being cared for, providing them with a sense of psychological security.

[0031] The video unit displays images based on information obtained from the conversation unit. For example, the video unit can project images onto a wall using a projector. Specifically, the projector projects high-resolution images, allowing the care recipient to enjoy them visually. The video unit can also display images using a display screen. The display screen adjusts the angle to the care recipient's line of sight and position, improving visibility. Furthermore, the video unit can select the content of the images according to the care recipient's preferences. For example, the video unit can display family photos, natural landscapes, World Heritage sites, ruins, and buildings. This allows the care recipient to relax while watching their favorite images. The video unit can also work in conjunction with the conversation unit to switch images in response to the care recipient's statements and requests. For example, if the care recipient says, "I want to see family photos," the video unit will immediately display family photos. If the care recipient says, "I want to see natural landscapes," the video unit will display beautiful natural scenery. In this way, the video unit can provide images that meet the care recipient's needs and preferences, improving their quality of life.

[0032] The observation unit monitors changes in the care recipient's physical condition and provides advice. For example, the observation unit measures vital signs such as heart rate, body temperature, and blood pressure using sensors. Specifically, the heart rate sensor measures the care recipient's pulse in real time, and the body temperature sensor measures skin temperature. The blood pressure sensor periodically measures the care recipient's blood pressure to check for abnormalities. The observation unit can also observe the care recipient's facial expressions and movements using a camera. The camera captures high-resolution video and analyzes the care recipient's facial expressions and movements. For example, it can determine whether the care recipient is smiling, tired, or in pain. Furthermore, the observation unit can monitor changes in the care recipient's physical condition in real time and issue alerts if abnormalities are detected. For example, if the heart rate suddenly increases or the body temperature becomes abnormally high, the observation unit immediately issues an alert and notifies the caregiver or medical institution. This allows the observation unit to constantly monitor the care recipient's health status, detect abnormalities early, and take appropriate action. The observation department can support the health management of those receiving care and provide a sense of security.

[0033] The notification unit provides notifications in emergencies based on information obtained by the observation unit. For example, the notification unit can send emergency notifications to the care recipient's family or medical institutions via telephone or email. Specifically, if the observation unit detects an abnormality, the notification unit automatically calls pre-registered contacts and explains the abnormality verbally. It can also send emails with detailed information about the abnormality in text. Furthermore, the notification unit can also send notifications using a smartphone app. The smartphone app allows for real-time notifications, enabling the care recipient's family or medical institutions to respond quickly. The notification unit can also adjust the content and timing of notifications according to changes in the care recipient's condition. For example, if a minor abnormality is detected, it will notify the family; if a serious abnormality is detected, it will notify the medical institution directly. The notification unit can also set notification priorities and adjust the notification method and frequency according to importance. This allows the notification unit to provide quick and appropriate notifications in emergencies, ensuring the safety of the care recipient. The notification unit can also strengthen collaboration with the care recipient's family and medical institutions, providing the care recipient with a sense of security.

[0034] The voice selection unit allows you to choose a voice and tone that suits the preferences of the person receiving care. For example, you can choose from multiple voice patterns, such as an adult woman's voice, a child's voice (around elementary school age), or an elderly person's voice, according to the care recipient's preferences. The voice selection unit also allows you to adjust the tone and speed of the voice. For example, you can adjust the tone and speed to suit the care recipient's emotions and situation, such as a gentle tone or an energetic tone. This allows you to select a voice and tone that matches the care recipient's preferences.

[0035] The shape setting unit allows you to configure the robot's size and shape to suit the preferences of the person receiving care. For example, the shape setting unit can adjust the robot's size from palm-sized to cat-sized. It can also set the robot's shape to resemble a soft, warm, pet-like plush toy. Furthermore, the shape setting unit can change the robot's clothing. This allows you to customize the robot's size and shape to suit the preferences of the person receiving care.

[0036] The notification unit can inform the person receiving care about medication times and other regularly needed items. For example, it can use alarms to remind people about medication times. It also has a function to remind people about regularly needed items, such as exercise, meals, and hydration, which are essential for the person receiving care's health management. This ensures that the person receiving care is informed about medication times and other regularly needed items.

[0037] The conversation function can analyze the care recipient's past conversation history and select the most appropriate conversation topics. For example, the conversation function can provide topics of interest based on what the care recipient has said in the past. It can also provide relevant information based on what the care recipient has asked in the past. Furthermore, the conversation function can provide ongoing topics based on what the care recipient has said in the past. In this way, it can provide optimal conversation content based on the care recipient's past conversation history.

[0038] The conversation unit can monitor the health status of the person receiving care during conversation and take appropriate action if an abnormality is detected. For example, the conversation unit can analyze the tone and speed of the person receiving care's voice during conversation to detect abnormalities. It can also monitor the person receiving care's breathing sounds during conversation to detect abnormalities. Furthermore, the conversation unit can analyze the person receiving care's facial expressions during conversation to detect abnormalities. This allows for monitoring the health status of the person receiving care during conversation and taking appropriate action if an abnormality is detected.

[0039] The conversational component can provide relevant topics based on the care recipient's hobbies and interests during a conversation. For example, the conversational component can talk about the care recipient's favorite music, travel destinations they are interested in, or their hobby of handicrafts. This allows for the provision of relevant topics based on the care recipient's hobbies and interests.

[0040] The conversational unit can refer to information about the care recipient's family and friends during the conversation and provide relevant topics. For example, the conversational unit can talk about the care recipient's family's current situation. It can also talk about memories with the care recipient's friends. Furthermore, the conversational unit can provide topics for celebrating the birthdays of the care recipient's family and friends. This allows the unit to provide relevant topics based on information about the care recipient's family and friends.

[0041] The video unit can adjust the color tone and brightness of the video based on the visual preferences of the person receiving care. For example, if the person receiving care prefers bright colors, the video unit will brighten the color tone. It can also darken the color tone if the person receiving care prefers dark colors. Furthermore, if the person receiving care prefers softer colors, the video unit can soften the color tone. This allows the video's color tone and brightness to be adjusted based on the visual preferences of the person receiving care.

[0042] The video unit can select the most suitable video by referring to the care recipient's past viewing history when displaying video. For example, the video unit can display related videos based on videos the care recipient has watched in the past. It can also select the most suitable video based on the genre of videos the care recipient has watched in the past. Furthermore, it can select the most suitable video based on the time period of videos the care recipient has watched in the past. This allows the system to provide the most suitable video based on the care recipient's past viewing history.

[0043] The video unit can prioritize displaying relevant videos based on the geographical location information of the person receiving care. For example, the video unit can display videos of scenery in the area where the person receiving care lives. It can also display videos of places the person receiving care has visited in the past. Furthermore, it can display videos of areas that the person receiving care is interested in. This allows the system to provide relevant videos based on the geographical location information of the person receiving care.

[0044] The video unit can select background music based on the care recipient's musical preferences when displaying video. For example, the video unit can set music that the care recipient likes as background music. It can also set music that helps the care recipient relax as background music. Furthermore, it can set music that excites the care recipient as background music. This allows the system to provide the most suitable background music based on the care recipient's musical preferences.

[0045] The observation unit can detect abnormalities early by referring to the care recipient's past health data during observation. For example, the observation unit can detect abnormalities by referring to the care recipient's past blood pressure data. It can also detect abnormalities by referring to the care recipient's past heart rate data. Furthermore, it can detect abnormalities by referring to the care recipient's past body temperature data. This allows for early detection of abnormalities based on the care recipient's past health data.

[0046] The observation unit can select the optimal observation timing based on the care recipient's daily rhythm. For example, the observation unit can conduct observations in accordance with the care recipient's waking time. It can also conduct observations in accordance with the care recipient's meal times. Furthermore, it can conduct observations in accordance with the care recipient's bedtime. This allows the unit to provide the optimal observation timing based on the care recipient's daily rhythm.

[0047] The observation unit can evaluate the overall health status of the person receiving care by referring to their diet and exercise data during observation. For example, the observation unit can evaluate the health status by referring to the person receiving care's diet data. It can also evaluate the health status by referring to the person receiving care's exercise data. Furthermore, the observation unit can evaluate the health status by comprehensively referring to the person receiving care's diet and exercise data. This allows for an overall assessment of the health status of the person receiving care based on their diet and exercise data.

[0048] The observation unit can evaluate the health status of the person receiving care by referring to their sleep data during observation. For example, the observation unit can evaluate the health status by referring to the person's sleep duration. It can also evaluate the health status by referring to the quality of the person receiving care. Furthermore, it can evaluate the health status by referring to the person's sleep patterns. This allows for the evaluation of the health status of the person receiving care based on their sleep data.

[0049] The notification unit can select the optimal notification method by referring to the care recipient's past notification history when sending a notification. For example, the notification unit can select the optimal notification method based on the care recipient's preferred notification method in the past. The notification unit can also select the optimal notification timing by referring to the care recipient's past notification history. Furthermore, the notification unit can analyze the care recipient's past notification history to select the optimal notification content. This allows the system to provide the most suitable notification method based on the care recipient's past notification history.

[0050] The notification unit can automatically update the emergency contact information of the person receiving care when a notification is sent. For example, the notification unit periodically checks the emergency contact information of the person receiving care and updates it to the latest information. The notification unit can also automatically obtain contact information of the person receiving care's family and friends and set them as emergency contacts. Furthermore, the notification unit can save the person receiving care's emergency contact information to the cloud and update it as needed. This allows the emergency contact information of the person receiving care to be automatically updated.

[0051] The notification unit can select the most appropriate notification method based on the geographical location information of the person receiving care when a notification is sent. For example, if the person receiving care is at home, the notification unit will prioritize voice notifications. It can also prioritize notifications to a smartphone if the person receiving care is out. Furthermore, if the person receiving care is in a specific location, the notification unit can select a notification method appropriate for that location. This allows the system to provide the most suitable notification method based on the geographical location information of the person receiving care.

[0052] The notification unit can customize notification content by referencing the contact information of the care recipient's family and friends when sending a notification. For example, the notification unit can refer to the contact information of the care recipient's family and provide notification content appropriate for the family. It can also refer to the contact information of the care recipient's friends and provide notification content appropriate for the friends. Furthermore, the notification unit can comprehensively refer to the contact information of the care recipient's family and friends to provide the most optimal notification content. This allows for the customization of notification content based on the contact information of the care recipient's family and friends.

[0053] The voice selection unit can select the optimal voice by referring to the care recipient's past voice selection history when selecting a voice. For example, the voice selection unit can provide the optimal voice based on the voice the care recipient has selected in the past. The voice selection unit can also select the optimal voice tone by referring to the care recipient's past voice selection history. Furthermore, the voice selection unit can analyze the care recipient's past voice selection history and select the optimal voice speed. This allows the system to provide the optimal voice based on the care recipient's past voice selection history.

[0054] The voice selection unit can select the most suitable voice based on the care recipient's language and dialect during voice selection. For example, the voice selection unit can provide the most suitable voice based on the language used by the care recipient. It can also provide the most suitable voice based on the dialect used by the care recipient. Furthermore, the voice selection unit can analyze the characteristics of the care recipient's language and dialect to select the most suitable voice. This allows for the provision of the most suitable voice based on the care recipient's language and dialect.

[0055] The shape setting unit can select the optimal shape by referring to the care recipient's past shape setting history when setting the shape. For example, the shape setting unit provides the optimal shape based on the shape the care recipient has previously selected. The shape setting unit can also select the optimal size by referring to the care recipient's past shape setting history. Furthermore, the shape setting unit can analyze the care recipient's past shape setting history to select the optimal shape and size. This allows the unit to provide the optimal shape based on the care recipient's past shape setting history.

[0056] The shape setting unit can select the optimal shape based on the living environment of the person receiving care during the shape setting process. For example, the shape setting unit can provide a robot with the optimal shape to suit the living environment of the person receiving care. Furthermore, the shape setting unit can also provide a robot of the optimal size based on the living environment of the person receiving care. Additionally, the shape setting unit can analyze the living environment of the person receiving care and select the optimal shape and size. This allows the unit to provide the optimal shape based on the living environment of the person receiving care.

[0057] The notification system can select the most appropriate notification method by referring to the care recipient's past notification history. For example, it can select the most appropriate notification method based on the care recipient's preferred notification methods in the past. The notification system can also select the most appropriate notification timing by referring to the care recipient's past notification history. Furthermore, the notification system can analyze the care recipient's past notification history to select the most appropriate notification content. This allows the system to provide the most suitable notification method based on the care recipient's past notification history.

[0058] The notification system can select the most appropriate notification method based on the geographical location of the person receiving care. For example, if the person receiving care is at home, the system will prioritize voice notifications. If the person receiving care is out, the system can also prioritize notifications to their smartphone. Furthermore, if the person receiving care is in a specific location, the system can select a notification method appropriate for that location. This allows the system to provide the most suitable notification method based on the geographical location of the person receiving care.

[0059] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0060] The caregiving communication robot system can also be equipped with a hobby support unit. This unit can suggest activities tailored to the care recipient's hobbies and interests. For example, if the care recipient enjoys handicrafts, it can suggest a new handicraft project. Similarly, if the care recipient enjoys reading, it can suggest books of interest. Furthermore, the hobby support unit can refer to the care recipient's past hobby activity history to suggest the most suitable activities. This allows for the provision of activity support tailored to the care recipient's hobbies and interests.

[0061] The caregiving communication robot system can also be equipped with a learning support unit. This unit can provide educational materials and programs to enhance the learning motivation of those receiving care. For example, if a care recipient wants to learn a new language, a language learning program can be provided. Similarly, if a care recipient is interested in history, historical materials can be provided. Furthermore, the learning support unit can refer to the care recipient's past learning history and suggest the most suitable educational materials and programs. This allows for support that enhances the care recipient's motivation to learn.

[0062] The caregiving communication robot system can also be equipped with a social participation support unit. This unit can provide information and opportunities for care recipients to participate in social activities. For example, if a care recipient wants to participate in a local event, it can provide information about that event. It can also suggest appropriate volunteer activities if a care recipient wants to participate in volunteer work. Furthermore, the social participation support unit can refer to the care recipient's past social participation history and suggest the most suitable activities. This allows for support of the care recipient's social participation.

[0063] The caregiving communication robot system can also be equipped with a pet imitation unit. This unit can provide the care recipient with an experience similar to that of owning a pet. For example, it can offer interactive play so that the care recipient can play with a pet. It can also suggest virtual pet care so that the care recipient can care for a pet. Furthermore, the pet imitation unit can refer to the care recipient's past pet ownership history to provide an optimal pet imitation experience. This allows the care recipient to experience what it's like to own a pet.

[0064] The caregiving communication robot system can also be equipped with a travel experience section. This section can offer various travel experiences, allowing the person receiving care to virtually enjoy traveling. For example, the travel experience section can provide a virtual tour of a World Heritage site. It can also offer a virtual trip to enjoy natural scenery. Furthermore, the travel experience section can refer to the person receiving care's past travel history and suggest the most suitable travel experience. This allows the person receiving care to virtually enjoy traveling.

[0065] The following briefly describes the processing flow for example form 1.

[0066] Step 1: The conversation unit engages in conversation with the person receiving care. The conversation unit uses speech recognition technology to recognize what the person receiving care says and generates appropriate responses. It can also use natural language processing technology to provide appropriate answers to the person receiving care's questions. Furthermore, it can use emotion recognition technology to estimate the person receiving care's emotions and adjust the tone and content of the responses accordingly. Step 2: The video unit displays images based on the information obtained by the conversation unit. The video unit can display images using a projector or display. In addition, it can select and display images such as family photos, natural landscapes, or World Heritage sites according to the preferences of the person being cared for. Step 3: The observation unit observes changes in the care recipient's physical condition and provides advice. The observation unit can measure vital signs such as heart rate, body temperature, and blood pressure using sensors, and observe facial expressions and movements using a camera. Furthermore, it can monitor changes in physical condition in real time and issue alerts if abnormalities are detected. Step 4: The notification unit will send notifications in emergencies based on the information obtained by the observation unit. The notification unit can send emergency notifications to the care recipient's family or medical institutions via telephone, email, or smartphone app. It can also adjust the content and timing of notifications according to changes in the person's condition.

[0067] (Example of form 2) The caregiving communication robot system according to an embodiment of the present invention is a communication robot system for people who require care, such as those who are paralyzed on one side due to a stroke, the elderly, and those with dementia. This system has the function of being able to converse normally with the person being cared for and to consult about anything. Not only does it answer questions, but it can also sometimes scold or encourage and initiate conversations itself. It functions as a monitoring robot like a family member or friend. Furthermore, this system can project images onto walls and other surfaces. For example, by projecting family photos, natural landscapes, World Heritage sites, ruins, or buildings, the person being cared for, who has difficulty traveling freely, can experience the feeling of being on a trip. The voice and tone of voice are not robotic, and multiple patterns can be selected. For example, it can be selected according to the preferences of the person being cared for, such as an adult woman, a child of elementary school age, or an elderly person. The robot's size ranges from palm-sized to about the size of a cat, and its shape is that of a soft, warm, pet-like plush toy. It can also be dressed up with clothes. It also has an notification function, monitoring the person being cared for, observing changes in their physical condition throughout the day, and providing advice. The system also has functions to notify users of medication times and other regularly needed information, and regular notifications are sent to guardians (family members, etc.) about the user's condition. In emergencies, it also has a function to notify users by phone and email. In this way, the caregiving communication robot system can support the lives of those receiving care and provide a sense of security.

[0068] The caregiving communication robot system according to this embodiment comprises a conversation unit, a video unit, an observation unit, and a notification unit. The conversation unit engages in conversation with the person receiving care. The conversation unit recognizes the person receiving care's statements using, for example, speech recognition technology and generates an appropriate response. The conversation unit can also provide appropriate answers to the person receiving care's questions using natural language processing technology. Furthermore, the conversation unit can estimate the person receiving care's emotions using emotion recognition technology and adjust the tone and content of its responses. For example, if the person receiving care is sad, the conversation unit will converse in a comforting and gentle tone. If the person receiving care is agitated, the conversation unit can converse in a calming and gentle tone. If the person receiving care is tired, the conversation unit can converse in a relaxing and gentle tone. The video unit displays images based on the information obtained by the conversation unit. The video unit can project images onto a wall using, for example, a projector. The video unit can also display images using a display. Furthermore, the video unit can select the content of the images according to the preferences of the person receiving care. For example, the video unit can display family photos, natural landscapes, World Heritage sites, ruins, and buildings. The observation unit observes changes in the care recipient's physical condition and provides advice. The observation unit measures vital signs such as the care recipient's heart rate, body temperature, and blood pressure using sensors, for example. It can also observe the care recipient's facial expressions and movements using a camera. Furthermore, the observation unit can monitor changes in the care recipient's physical condition in real time and issue an alert if an abnormality is detected. The notification unit provides notifications in emergencies based on the information obtained by the observation unit. The notification unit can send emergency notifications to the care recipient's family or medical institutions, for example, by phone or email. It can also send notifications using a smartphone app. Furthermore, the notification unit can adjust the content and timing of notifications according to changes in the care recipient's physical condition. As a result, the care communication robot system according to this embodiment can support the care recipient's life and provide a sense of security.

[0069] The conversation unit engages in conversations with the person receiving care. For example, the conversation unit uses speech recognition technology to recognize what the person receiving care says and generates appropriate responses. Specifically, speech recognition technology converts the person receiving care's statements into text data, and then uses natural language processing technology to generate responses based on that text data. Natural language processing technology uses algorithms to analyze the intent of the person receiving care's questions and statements and provide appropriate answers. For example, if the person receiving care asks, "What day is it today?", the conversation unit refers to calendar information and replies, "Today is the day of ○○." The conversation unit can also use emotion recognition technology to estimate the person receiving care's emotions and adjust the tone and content of the response. Emotion recognition technology analyzes data such as tone and speed of speech and facial expressions to estimate the person receiving care's emotional state. For example, if the person receiving care is sad, the conversation unit will respond in a comforting, gentle tone, "What's wrong? Please tell me what happened." If the person receiving care is agitated, the conversation unit can respond in a calming, gentle tone, "It's okay, let's talk slowly." Furthermore, if the person being cared for is tired, the conversational unit can suggest, in a calm and relaxing tone, "Shall we take a short break?" This allows the conversational unit to communicate appropriately according to the feelings and condition of the person being cared for, providing them with a sense of psychological security.

[0070] The video unit displays images based on information obtained from the conversation unit. For example, the video unit can project images onto a wall using a projector. Specifically, the projector projects high-resolution images, allowing the care recipient to enjoy them visually. The video unit can also display images using a display screen. The display screen adjusts the angle to the care recipient's line of sight and position, improving visibility. Furthermore, the video unit can select the content of the images according to the care recipient's preferences. For example, the video unit can display family photos, natural landscapes, World Heritage sites, ruins, and buildings. This allows the care recipient to relax while watching their favorite images. The video unit can also work in conjunction with the conversation unit to switch images in response to the care recipient's statements and requests. For example, if the care recipient says, "I want to see family photos," the video unit will immediately display family photos. If the care recipient says, "I want to see natural landscapes," the video unit will display beautiful natural scenery. In this way, the video unit can provide images that meet the care recipient's needs and preferences, improving their quality of life.

[0071] The observation unit monitors changes in the care recipient's physical condition and provides advice. For example, the observation unit measures vital signs such as heart rate, body temperature, and blood pressure using sensors. Specifically, the heart rate sensor measures the care recipient's pulse in real time, and the body temperature sensor measures skin temperature. The blood pressure sensor periodically measures the care recipient's blood pressure to check for abnormalities. The observation unit can also observe the care recipient's facial expressions and movements using a camera. The camera captures high-resolution video and analyzes the care recipient's facial expressions and movements. For example, it can determine whether the care recipient is smiling, tired, or in pain. Furthermore, the observation unit can monitor changes in the care recipient's physical condition in real time and issue alerts if abnormalities are detected. For example, if the heart rate suddenly increases or the body temperature becomes abnormally high, the observation unit immediately issues an alert and notifies the caregiver or medical institution. This allows the observation unit to constantly monitor the care recipient's health status, detect abnormalities early, and take appropriate action. The observation department can support the health management of those receiving care and provide a sense of security.

[0072] The notification unit provides notifications in emergencies based on information obtained by the observation unit. For example, the notification unit can send emergency notifications to the care recipient's family or medical institutions via telephone or email. Specifically, if the observation unit detects an abnormality, the notification unit automatically calls pre-registered contacts and explains the abnormality verbally. It can also send emails with detailed information about the abnormality in text. Furthermore, the notification unit can also send notifications using a smartphone app. The smartphone app allows for real-time notifications, enabling the care recipient's family or medical institutions to respond quickly. The notification unit can also adjust the content and timing of notifications according to changes in the care recipient's condition. For example, if a minor abnormality is detected, it will notify the family; if a serious abnormality is detected, it will notify the medical institution directly. The notification unit can also set notification priorities and adjust the notification method and frequency according to importance. This allows the notification unit to provide quick and appropriate notifications in emergencies, ensuring the safety of the care recipient. The notification unit can also strengthen collaboration with the care recipient's family and medical institutions, providing the care recipient with a sense of security.

[0073] The voice selection unit allows you to choose a voice and tone that suits the preferences of the person receiving care. For example, you can choose from multiple voice patterns, such as an adult woman's voice, a child's voice (around elementary school age), or an elderly person's voice, according to the care recipient's preferences. The voice selection unit also allows you to adjust the tone and speed of the voice. For example, you can adjust the tone and speed to suit the care recipient's emotions and situation, such as a gentle tone or an energetic tone. This allows you to select a voice and tone that matches the care recipient's preferences.

[0074] The shape setting unit allows you to configure the robot's size and shape to suit the preferences of the person receiving care. For example, the shape setting unit can adjust the robot's size from palm-sized to cat-sized. It can also set the robot's shape to resemble a soft, warm, pet-like plush toy. Furthermore, the shape setting unit can change the robot's clothing. This allows you to customize the robot's size and shape to suit the preferences of the person receiving care.

[0075] The notification unit can inform the person receiving care about medication times and other regularly needed items. For example, it can use alarms to remind people about medication times. It also has a function to remind people about regularly needed items, such as exercise, meals, and hydration, which are essential for the person receiving care's health management. This ensures that the person receiving care is informed about medication times and other regularly needed items.

[0076] The conversational unit can estimate the emotions of the person being cared for and adjust the content and tone of the conversation based on the estimated emotions. For example, if the person being cared for is sad, the conversational unit will speak in a comforting and gentle tone. If the person being cared for is agitated, the conversational unit can speak in a calming and gentle tone. If the person being cared for is tired, the conversational unit can speak in a relaxing and gentle tone. This allows the conversational unit to provide content and tone that are appropriate to the emotions of the person being cared for. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI may be, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0077] The conversation function can analyze the care recipient's past conversation history and select the most appropriate conversation topics. For example, the conversation function can provide topics of interest based on what the care recipient has said in the past. It can also provide relevant information based on what the care recipient has asked in the past. Furthermore, the conversation function can provide ongoing topics based on what the care recipient has said in the past. In this way, it can provide optimal conversation content based on the care recipient's past conversation history.

[0078] The conversation unit can monitor the health status of the person receiving care during conversation and take appropriate action if an abnormality is detected. For example, the conversation unit can analyze the tone and speed of the person receiving care's voice during conversation to detect abnormalities. It can also monitor the person receiving care's breathing sounds during conversation to detect abnormalities. Furthermore, the conversation unit can analyze the person receiving care's facial expressions during conversation to detect abnormalities. This allows for monitoring the health status of the person receiving care during conversation and taking appropriate action if an abnormality is detected.

[0079] The conversational unit can estimate the emotions of the person being cared for and adjust the frequency of conversation based on the estimated emotions. For example, if the person being cared for is feeling lonely, the conversational unit will talk to them more frequently. Conversely, if the person being cared for is tired, the conversational unit can reduce the frequency of conversation. Also, if the person being cared for is agitated, the conversational unit can adjust the frequency of conversation to calm them down. In this way, the frequency of conversation can be adjusted according to the emotions of the person being cared for. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0080] The conversational component can provide relevant topics based on the care recipient's hobbies and interests during a conversation. For example, the conversational component can talk about the care recipient's favorite music, travel destinations they are interested in, or their hobby of handicrafts. This allows for the provision of relevant topics based on the care recipient's hobbies and interests.

[0081] The conversational unit can refer to information about the care recipient's family and friends during the conversation and provide relevant topics. For example, the conversational unit can talk about the care recipient's family's current situation. It can also talk about memories with the care recipient's friends. Furthermore, the conversational unit can provide topics for celebrating the birthdays of the care recipient's family and friends. This allows the unit to provide relevant topics based on information about the care recipient's family and friends.

[0082] The video unit can estimate the emotions of the person receiving care and adjust the video content based on the estimated emotions. For example, if the person receiving care is sad, the video unit can display comforting scenery. It can also display calming nature footage if the person receiving care is agitated. Furthermore, it can display relaxing footage if the person receiving care is tired. This allows the system to provide video content that matches the emotions of the person receiving care. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI may include, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0083] The video unit can adjust the color tone and brightness of the video based on the visual preferences of the person receiving care. For example, if the person receiving care prefers bright colors, the video unit will brighten the color tone. It can also darken the color tone if the person receiving care prefers dark colors. Furthermore, if the person receiving care prefers softer colors, the video unit can soften the color tone. This allows the video's color tone and brightness to be adjusted based on the visual preferences of the person receiving care.

[0084] The video unit can select the most suitable video by referring to the care recipient's past viewing history when displaying video. For example, the video unit can display related videos based on videos the care recipient has watched in the past. It can also select the most suitable video based on the genre of videos the care recipient has watched in the past. Furthermore, it can select the most suitable video based on the time period of videos the care recipient has watched in the past. This allows the system to provide the most suitable video based on the care recipient's past viewing history.

[0085] The video unit can estimate the emotions of the person receiving care and adjust the video display time based on the estimated emotions. For example, if the person receiving care is agitated, the video unit can display a short video. Conversely, if the person receiving care is relaxed, the video unit can display a longer video. Furthermore, if the person receiving care is tired, the video unit can display a short video. This allows the video display time to be adjusted according to the emotions of the person receiving care. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI may include, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0086] The video unit can prioritize displaying relevant videos based on the geographical location information of the person receiving care. For example, the video unit can display videos of scenery in the area where the person receiving care lives. It can also display videos of places the person receiving care has visited in the past. Furthermore, it can display videos of areas that the person receiving care is interested in. This allows the system to provide relevant videos based on the geographical location information of the person receiving care.

[0087] The video unit can select background music based on the care recipient's musical preferences when displaying video. For example, the video unit can set music that the care recipient likes as background music. It can also set music that helps the care recipient relax as background music. Furthermore, it can set music that excites the care recipient as background music. This allows the system to provide the most suitable background music based on the care recipient's musical preferences.

[0088] The observation unit can estimate the emotions of the person being cared for and adjust the frequency and method of observation based on the estimated emotions. For example, if the person being cared for is feeling anxious, the observation unit will observe more frequently. Conversely, if the person being cared for is relaxed, the observation unit can reduce the frequency of observation. Furthermore, if the person being cared for is agitated, the observation unit can adjust the method of observation to calm them down. In this way, the frequency and method of observation can be adjusted according to the emotions of the person being cared for. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0089] The observation unit can detect abnormalities early by referring to the care recipient's past health data during observation. For example, the observation unit can detect abnormalities by referring to the care recipient's past blood pressure data. It can also detect abnormalities by referring to the care recipient's past heart rate data. Furthermore, it can detect abnormalities by referring to the care recipient's past body temperature data. This allows for early detection of abnormalities based on the care recipient's past health data.

[0090] The observation unit can select the optimal observation timing based on the care recipient's daily rhythm. For example, the observation unit can conduct observations in accordance with the care recipient's waking time. It can also conduct observations in accordance with the care recipient's meal times. Furthermore, it can conduct observations in accordance with the care recipient's bedtime. This allows the unit to provide the optimal observation timing based on the care recipient's daily rhythm.

[0091] The observation unit can estimate the emotions of the person being cared for and adjust the display method of the observation results based on the estimated emotions. For example, if the person being cared for is feeling anxious, the observation unit provides a simple and highly visible display method. If the person being cared for is relaxed, the observation unit can also provide a display method that includes detailed information. If the person being cared for is agitated, the observation unit can also provide a display method that focuses on the key points. This allows the display method of the observation results to be adjusted according to the emotions of the person being cared for. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI may be, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0092] The observation unit can evaluate the overall health status of the person receiving care by referring to their diet and exercise data during observation. For example, the observation unit can evaluate the health status by referring to the person receiving care's diet data. It can also evaluate the health status by referring to the person receiving care's exercise data. Furthermore, the observation unit can evaluate the health status by comprehensively referring to the person receiving care's diet and exercise data. This allows for an overall assessment of the health status of the person receiving care based on their diet and exercise data.

[0093] The observation unit can evaluate the health status of the person receiving care by referring to their sleep data during observation. For example, the observation unit can evaluate the health status by referring to the person's sleep duration. It can also evaluate the health status by referring to the quality of the person receiving care. Furthermore, it can evaluate the health status by referring to the person's sleep patterns. This allows for the evaluation of the health status of the person receiving care based on their sleep data.

[0094] The notification unit can estimate the emotions of the person receiving care and adjust the content and timing of notifications based on the estimated emotions. For example, if the person receiving care is feeling anxious, the notification unit will send a notification with reassuring content. The notification unit can also adjust the timing of notifications if the person receiving care is relaxed. Furthermore, if the person receiving care is agitated, the notification unit can send a notification with calming content. In this way, the content and timing of notifications can be adjusted according to the emotions of the person receiving care. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI is a text generation AI (e.g., LLM) or a multimodal generation AI, but is not limited to these examples.

[0095] The notification unit can select the optimal notification method by referring to the care recipient's past notification history when sending a notification. For example, the notification unit can select the optimal notification method based on the care recipient's preferred notification method in the past. The notification unit can also select the optimal notification timing by referring to the care recipient's past notification history. Furthermore, the notification unit can analyze the care recipient's past notification history to select the optimal notification content. This allows the system to provide the most suitable notification method based on the care recipient's past notification history.

[0096] The notification unit can automatically update the emergency contact information of the person receiving care when a notification is sent. For example, the notification unit periodically checks the emergency contact information of the person receiving care and updates it to the latest information. The notification unit can also automatically obtain contact information of the person receiving care's family and friends and set them as emergency contacts. Furthermore, the notification unit can save the person receiving care's emergency contact information to the cloud and update it as needed. This allows the emergency contact information of the person receiving care to be automatically updated.

[0097] The notification unit can estimate the emotions of the person receiving care and determine the priority of notifications based on the estimated emotions. For example, if the person receiving care is feeling anxious, the notification unit will prioritize important notifications. It can also adjust the priority of notifications if the person receiving care is relaxed. Furthermore, if the person receiving care is agitated, the notification unit can prioritize urgent notifications. This allows for the prioritization of notifications according to the emotions of the person receiving care. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI may include, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0098] The notification unit can select the most appropriate notification method based on the geographical location information of the person receiving care when a notification is sent. For example, if the person receiving care is at home, the notification unit will prioritize voice notifications. It can also prioritize notifications to a smartphone if the person receiving care is out. Furthermore, if the person receiving care is in a specific location, the notification unit can select a notification method appropriate for that location. This allows the system to provide the most suitable notification method based on the geographical location information of the person receiving care.

[0099] The notification unit can customize notification content by referencing the contact information of the care recipient's family and friends when sending a notification. For example, the notification unit can refer to the contact information of the care recipient's family and provide notification content appropriate for the family. It can also refer to the contact information of the care recipient's friends and provide notification content appropriate for the friends. Furthermore, the notification unit can comprehensively refer to the contact information of the care recipient's family and friends to provide the most optimal notification content. This allows for the customization of notification content based on the contact information of the care recipient's family and friends.

[0100] The voice selection unit can estimate the emotions of the person being cared for and adjust the tone and speed of the voice based on the estimated emotions. For example, if the person being cared for is relaxed, the voice selection unit will provide a relaxed tone of voice. It can also provide a rapid tone of voice if the person being cared for is in a hurry. Furthermore, it can provide a calm tone of voice if the person being cared for is agitated. This allows the voice tone and speed to be adjusted according to the emotions of the person being cared for. Emotion estimation is achieved using an emotion estimation function, for example, with an emotion engine or generative AI. Generative AI may include, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0101] The voice selection unit can select the optimal voice by referring to the care recipient's past voice selection history when selecting a voice. For example, the voice selection unit can provide the optimal voice based on the voice the care recipient has selected in the past. The voice selection unit can also select the optimal voice tone by referring to the care recipient's past voice selection history. Furthermore, the voice selection unit can analyze the care recipient's past voice selection history and select the optimal voice speed. This allows the system to provide the optimal voice based on the care recipient's past voice selection history.

[0102] The voice selection unit can estimate the emotions of the person being cared for and adjust the timing of voice switching based on the estimated emotions. For example, if the person being cared for is relaxed, the voice selection unit will switch voices slowly. It can also switch voices quickly if the person being cared for is in a hurry. Furthermore, if the person being cared for is agitated, the voice selection unit can switch voices at a calmer timing. This allows the timing of voice switching to be adjusted according to the emotions of the person being cared for. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI may include, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0103] The voice selection unit can select the most suitable voice based on the care recipient's language and dialect during voice selection. For example, the voice selection unit can provide the most suitable voice based on the language used by the care recipient. It can also provide the most suitable voice based on the dialect used by the care recipient. Furthermore, the voice selection unit can analyze the characteristics of the care recipient's language and dialect to select the most suitable voice. This allows for the provision of the most suitable voice based on the care recipient's language and dialect.

[0104] The shape setting unit can estimate the emotions of the person being cared for and adjust the shape and size of the robot based on the estimated emotions. For example, if the person being cared for is relaxed, the shape setting unit can provide a robot with a soft shape. If the person being cared for is agitated, the shape setting unit can also provide a robot with a calm shape. Furthermore, if the person being cared for is feeling anxious, the shape setting unit can provide a robot with a reassuring shape. This allows the shape and size of the robot to be adjusted according to the emotions of the person being cared for. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI may be, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0105] The shape setting unit can select the optimal shape by referring to the care recipient's past shape setting history when setting the shape. For example, the shape setting unit provides the optimal shape based on the shape the care recipient has previously selected. The shape setting unit can also select the optimal size by referring to the care recipient's past shape setting history. Furthermore, the shape setting unit can analyze the care recipient's past shape setting history to select the optimal shape and size. This allows the unit to provide the optimal shape based on the care recipient's past shape setting history.

[0106] The shape setting unit can estimate the emotions of the person being cared for and adjust the timing of shape changes based on the estimated emotions. For example, if the person being cared for is relaxed, the shape setting unit will change the shape slowly. If the person being cared for is in a hurry, the shape setting unit can also change the shape quickly. If the person being cared for is agitated, the shape setting unit can also change the shape at a calmer pace. This allows the timing of shape changes to be adjusted according to the emotions of the person being cared for. Emotion estimation is achieved using an emotion estimation function, for example, using an emotion engine or generative AI. Generative AI may be, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0107] The shape setting unit can select the optimal shape based on the living environment of the person receiving care during the shape setting process. For example, the shape setting unit can provide a robot with the optimal shape to suit the living environment of the person receiving care. Furthermore, the shape setting unit can also provide a robot of the optimal size based on the living environment of the person receiving care. Additionally, the shape setting unit can analyze the living environment of the person receiving care and select the optimal shape and size. This allows the unit to provide the optimal shape based on the living environment of the person receiving care.

[0108] The notification unit can estimate the emotions of the person receiving care and adjust the content and timing of notifications based on the estimated emotions. For example, if the person receiving care is feeling anxious, the notification unit will send a reassuring notification. It can also adjust the timing of notifications if the person receiving care is relaxed. Furthermore, if the person receiving care is agitated, the notification unit can send a calming notification. This allows for adjustment of notification content and timing according to the emotions of the person receiving care. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI may include, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0109] The notification system can select the most appropriate notification method by referring to the care recipient's past notification history. For example, it can select the most appropriate notification method based on the care recipient's preferred notification methods in the past. The notification system can also select the most appropriate notification timing by referring to the care recipient's past notification history. Furthermore, the notification system can analyze the care recipient's past notification history to select the most appropriate notification content. This allows the system to provide the most suitable notification method based on the care recipient's past notification history.

[0110] The notification system can estimate the emotions of the person receiving care and determine the priority of notifications based on those estimated emotions. For example, if the person receiving care is feeling anxious, the notification system will prioritize important notifications. It can also adjust the priority of notifications if the person receiving care is relaxed. Furthermore, if the person receiving care is agitated, the notification system can prioritize urgent notifications. This allows for the prioritization of notifications according to the emotions of the person receiving care. Emotion estimation is achieved using an emotion estimation function, such as an emotion engine or generative AI. Generative AI may include, but is not limited to, text generation AI (e.g., LLM) or multimodal generation AI.

[0111] The notification system can select the most appropriate notification method based on the geographical location of the person receiving care. For example, if the person receiving care is at home, the system will prioritize voice notifications. If the person receiving care is out, the system can also prioritize notifications to their smartphone. Furthermore, if the person receiving care is in a specific location, the system can select a notification method appropriate for that location. This allows the system to provide the most suitable notification method based on the geographical location of the person receiving care.

[0112] The system according to the embodiment is not limited to the example described above, and various modifications are possible, for example, as follows.

[0113] The caregiving communication robot system can also be equipped with a music playback unit. This unit can play music tailored to the preferences of the person receiving care. For example, if the person wants to relax, it can play calming classical music. If the person wants to feel energized, it can play upbeat pop music. Furthermore, the music playback unit can estimate the emotions of the person receiving care and select music genres and songs based on those estimated emotions. This allows the system to provide music that matches the emotions of the person receiving care.

[0114] The caregiving communication robot system can also be equipped with a fragrance dispensing unit. This unit can provide fragrances tailored to the preferences of the person receiving care. For example, if the person wants to relax, it can provide a lavender scent. If the person wants to feel energized, it can provide a citrus scent. Furthermore, the fragrance dispensing unit can estimate the emotions of the person receiving care and adjust the type and intensity of the fragrance based on the estimated emotions. This allows the system to provide fragrances that match the emotions of the person receiving care.

[0115] The caregiving communication robot system can also be equipped with an exercise support unit. This unit can provide exercise programs tailored to the care recipient's physical condition and exercise abilities. For example, if the care recipient desires light exercise, it can suggest stretching or light exercises. If the care recipient is undergoing rehabilitation, it can suggest specialized rehabilitation exercises. Furthermore, the exercise support unit can estimate the care recipient's emotions and adjust the content and intensity of the exercise program based on these estimations. This allows for the provision of exercise support that is appropriate to the care recipient's emotional state.

[0116] The caregiving communication robot system can also be equipped with a meal management unit. This unit can manage the diet and nutritional balance of the person receiving care. For example, if the person needs to consume a specific nutrient, it can suggest a meal containing that nutrient. Furthermore, if the person needs to manage their food intake, it can suggest an appropriate amount. In addition, the meal management unit can estimate the person's emotions and adjust the content and timing of meals based on those emotions. This allows for meal management tailored to the person's emotional state.

[0117] The caregiving communication robot system can also be equipped with a sleep support unit. This unit can optimize the sleep environment for the person receiving care. For example, it can play calming music or nature sounds to help the person relax and fall asleep. It can also adjust room temperature and humidity to ensure comfortable sleep. Furthermore, the sleep support unit can estimate the person's emotions and adjust the sleep environment based on those emotions. This allows for sleep support tailored to the person's emotional state.

[0118] The caregiving communication robot system can also be equipped with a hobby support unit. This unit can suggest activities tailored to the care recipient's hobbies and interests. For example, if the care recipient enjoys handicrafts, it can suggest a new handicraft project. Similarly, if the care recipient enjoys reading, it can suggest books of interest. Furthermore, the hobby support unit can refer to the care recipient's past hobby activity history to suggest the most suitable activities. This allows for the provision of activity support tailored to the care recipient's hobbies and interests.

[0119] The caregiving communication robot system can also be equipped with a learning support unit. This unit can provide educational materials and programs to enhance the learning motivation of those receiving care. For example, if a care recipient wants to learn a new language, a language learning program can be provided. Similarly, if a care recipient is interested in history, historical materials can be provided. Furthermore, the learning support unit can refer to the care recipient's past learning history and suggest the most suitable educational materials and programs. This allows for support that enhances the care recipient's motivation to learn.

[0120] The caregiving communication robot system can also be equipped with a social participation support unit. This unit can provide information and opportunities for care recipients to participate in social activities. For example, if a care recipient wants to participate in a local event, it can provide information about that event. It can also suggest appropriate volunteer activities if a care recipient wants to participate in volunteer work. Furthermore, the social participation support unit can refer to the care recipient's past social participation history and suggest the most suitable activities. This allows for support of the care recipient's social participation.

[0121] The caregiving communication robot system can also be equipped with a pet imitation unit. This unit can provide the care recipient with an experience similar to that of owning a pet. For example, it can offer interactive play so that the care recipient can play with a pet. It can also suggest virtual pet care so that the care recipient can care for a pet. Furthermore, the pet imitation unit can refer to the care recipient's past pet ownership history to provide an optimal pet imitation experience. This allows the care recipient to experience what it's like to own a pet.

[0122] The caregiving communication robot system can also be equipped with a travel experience section. This section can offer various travel experiences, allowing the person receiving care to virtually enjoy traveling. For example, the travel experience section can provide a virtual tour of a World Heritage site. It can also offer a virtual trip to enjoy natural scenery. Furthermore, the travel experience section can refer to the person receiving care's past travel history and suggest the most suitable travel experience. This allows the person receiving care to virtually enjoy traveling.

[0123] The following briefly describes the processing flow for example form 2.

[0124] Step 1: The conversation unit engages in conversation with the person receiving care. The conversation unit uses speech recognition technology to recognize what the person receiving care says and generates appropriate responses. It can also use natural language processing technology to provide appropriate answers to the person receiving care's questions. Furthermore, it can use emotion recognition technology to estimate the person receiving care's emotions and adjust the tone and content of the responses accordingly. Step 2: The video unit displays images based on the information obtained by the conversation unit. The video unit can display images using a projector or display. In addition, it can select and display images such as family photos, natural landscapes, or World Heritage sites according to the preferences of the person being cared for. Step 3: The observation unit observes changes in the care recipient's physical condition and provides advice. The observation unit can measure vital signs such as heart rate, body temperature, and blood pressure using sensors, and observe facial expressions and movements using a camera. Furthermore, it can monitor changes in physical condition in real time and issue alerts if abnormalities are detected. Step 4: The notification unit will send notifications in emergencies based on the information obtained by the observation unit. The notification unit can send emergency notifications to the care recipient's family or medical institutions via telephone, email, or smartphone app. It can also adjust the content and timing of notifications according to changes in the person's condition.

[0125] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0126] Data generation model 58 is a form of so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> Examples of generative AI include text generation AI, image generation AI, and multimodal generation AI. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats from audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-means clustering, convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each of the above parts is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example.Furthermore, processing performed by AI, including generative AI, may be replaced with rule-based processing, and rule-based processing may be replaced with processing performed by AI, including generative AI.

[0127] Furthermore, the processing performed by the data processing system 10 described above is carried out by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart device 14, but it may also be carried out by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart device 14. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart device 14 or an external device, and the smart device 14 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0128] Each of the multiple elements described above, including the conversation unit, video unit, observation unit, and notification unit, is implemented in at least one of the smart device 14 and the data processing unit 12. For example, the conversation unit is implemented by the processor 46 of the smart device 14 and conducts conversations with the person receiving care using speech recognition technology and natural language processing technology. The video unit can display video using the output device 40 of the smart device 14. The observation unit observes the physical condition of the person receiving care using the camera 42 and sensors of the smart device 14. The notification unit can send notifications in emergencies using the communication I / F 44 of the smart device 14. The correspondence between each unit and the device or control unit is not limited to the example described above and can be modified in various ways.

[0129] [Second Embodiment] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0130] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0131] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0132] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0133] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0134] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0135] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0136] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing by the processor 28. The storage 32 stores the specific processing program 56.

[0137] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0138] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0139] In the smart glasses 214, specific processing is performed by the processor 46. The storage 50 stores a specific processing program 60. The processor 46 reads the specific processing program 60 from the storage 50 and executes the read specific processing program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific processing program 60 executed on the RAM 48. The smart glasses 214 also have a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0140] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0141] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0142] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0143] The data processing system 210 according to the second embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 210 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the smart glasses 214, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the smart glasses 214. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the smart glasses 214 or an external device, and the smart glasses 214 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0144] Each of the multiple elements described above, including the conversation unit, video unit, observation unit, and notification unit, is implemented in at least one of the smart glasses 214 and the data processing unit 12. For example, the conversation unit is implemented by the processor 46 of the smart glasses 214 and conducts conversations with the person receiving care using speech recognition technology and natural language processing technology. The video unit can display images using the display of the smart glasses 214. The observation unit observes the physical condition of the person receiving care using the camera 42 and sensors of the smart glasses 214. The notification unit can send notifications in emergencies using the communication I / F 44 of the smart glasses 214. The correspondence between each unit and the device or control unit is not limited to the example described above and can be modified in various ways.

[0145] [Third Embodiment] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0146] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0147] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0148] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0149] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0150] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0151] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0152] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0153] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0154] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0155] In the headset terminal 314, specific processing is performed by the processor 46. The storage 50 stores a specific program 60. The processor 46 reads the specific program 60 from the storage 50 and executes the read specific program 60 on the RAM 48. The specific processing is realized by the processor 46 acting as a control unit 46A according to the specific program 60 executed on the RAM 48. The headset terminal 314 also has a data generation model 58 and an emotion identification model 59, similar to the data generation model and emotion identification model 59, and can perform processing similar to that of the specific processing unit 290 using these models.

[0156] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0157] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0158] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0159] The data processing system 310 according to the third embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 310 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the headset terminal 314, but may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the headset terminal 314. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the headset terminal 314 or an external device, and the headset terminal 314 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0160] Each of the multiple elements described above, including the conversation unit, video unit, observation unit, and notification unit, is implemented in at least one of the headset terminal 314 and the data processing unit 12. For example, the conversation unit is implemented by the processor 46 of the headset terminal 314 and conducts conversations with the person receiving care using speech recognition technology and natural language processing technology. The video unit can display video using the display 343 of the headset terminal 314. The observation unit observes the physical condition of the person receiving care using the camera 42 and sensors of the headset terminal 314. The notification unit can send notifications in emergencies using the communication I / F 44 of the headset terminal 314. The correspondence between each unit and the device or control unit is not limited to the example described above and can be modified in various ways.

[0161] [Fourth Embodiment] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0162] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0163] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN and / or LAN.

[0164] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0165] The microphone 238 receives voice signals from the user and accepts instructions from the user. The microphone 238 captures the voice signals from the user, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0166] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS image sensor or CCD image sensor, which captures images of the area around the user (for example, an imaging range defined by a field of view equivalent to the field of vision of a typical healthy person).

[0167] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0168] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. The robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0169] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0170] The processor 28 reads a specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 acting as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0171] Storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290. The identification processing unit 290 can estimate the user's emotions using the emotion identification model 59 and perform identification processing using the user's emotions. The emotion estimation function (emotion identification function) using the emotion identification model 59 performs various estimations and predictions regarding the user's emotions, including but not limited to these examples. Furthermore, emotion estimation and prediction also include, for example, emotion analysis.

[0172] In robot 414, specific processing is performed by processor 46. A specific program 60 is stored in storage 50. Processor 46 reads the specific program 60 from storage 50 and executes it on RAM 48. The specific processing is achieved by processor 46 acting as a control unit 46A according to the specific program 60 executed on RAM 48. Robot 414 also has data generation model 58 and emotion identification model 59, similar to those of the robot, and can perform processing similar to that of the specific processing unit 290 using these models.

[0173] Furthermore, other devices besides the data processing device 12 may also have the data generation model 58. For example, a server device may have the data generation model 58. In this case, the data processing device 12 obtains processing results (such as prediction results) using the data generation model 58 by communicating with the server device that has the data generation model 58. Also, the data processing device 12 may be a server device or a terminal device owned by the user (for example, a mobile phone, robot, home appliance, etc.).

[0174] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0175] The data generation model 58 is a so-called generative AI. An example of a data generation model 58 is a generative AI such as ChatGPT. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and inference data such as audio data representing speech, text data representing text, and image data representing images (e.g., still image data or video data). The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference result in one or more data formats such as audio data, text data, and image data. The data generation model 58 includes, for example, text generation AI, image generation AI, and multimodal generation AI. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The specific processing unit 290 performs the specific processing described above using the data generation model 58. The data generation model 58 may be a fine-tuned model that outputs inference results from prompts that do not contain instructions, in which case the data generation model 58 can output inference results from prompts that do not contain instructions. In the data processing device 12, etc., there are multiple types of data generation models 58, and the data generation model 58 includes AI other than generative AI. AI other than generative AI includes, for example, linear regression, logistic regression, decision trees, random forests, support vector machines (SVM), k-means clustering, convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), or naive Bayes, and can perform various processes, but is not limited to these examples. Also, the AI ​​may be an AI agent. Furthermore, when the processing of each part described above is performed by the AI, the processing may be performed by the AI ​​in part or in whole, but is not limited to this example. Also, processing performed by an AI including a generative AI may be replaced by rule-based processing, and rule-based processing may be replaced by processing performed by an AI including a generative AI.

[0176] The data processing system 410 according to the fourth embodiment performs the same processing as the data processing system 10 according to the first embodiment. The processing by the data processing system 410 is performed by the specific processing unit 290 of the data processing device 12 or the control unit 46A of the robot 414, but it may also be performed by the specific processing unit 290 of the data processing device 12 and the control unit 46A of the robot 414. In addition, the specific processing unit 290 of the data processing device 12 acquires or collects information necessary for processing from the robot 414 or an external device, and the robot 414 acquires or collects information necessary for processing from the data processing device 12 or an external device.

[0177] Each of the multiple elements described above, including the conversation unit, video unit, observation unit, and notification unit, is implemented in at least one of the robot 414 and the data processing unit 12. For example, the conversation unit is implemented by the processor 46 of the robot 414 and engages in conversation with the person receiving care using speech recognition technology and natural language processing technology. The video unit can display video using the display of the robot 414. The observation unit observes the physical condition of the person receiving care using the camera 42 and sensors of the robot 414. The notification unit can send notifications in emergencies using the communication I / F 44 of the robot 414. The correspondence between each unit and the device or control unit is not limited to the example described above and can be modified in various ways.

[0178] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0179] Figure 9 shows the emotion map 400, in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0180] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0181] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0182] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, and motorcycles, emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated based, for example, on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0183] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0184] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0185] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing method for the specific process may be used, which includes computer 22 and multiple other computers.

[0186] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0187] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0188] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0189] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0190] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0191] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0192] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0193] Furthermore, although the above-described examples were divided into four embodiments, some or all of these embodiments may be combined. Also, the smart device 14, smart glasses 214, headset terminal 314, and robot 414 are just examples, and they may be combined, or other devices may be used. Also, although the above-described examples were divided into two embodiments, Embodiment 1 and Embodiment 2, these may be combined.

[0194] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and other things that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0195] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0196] (Note 1) The conversation department, which engages in conversations with those receiving care, A video unit that displays images based on the information obtained by the aforementioned conversation unit, The observation department observes changes in the health of those receiving care and provides advice, The system includes a notification unit that provides notification in an emergency based on information obtained by the observation unit. A system characterized by the following features. (Note 2) It features a voice selection unit that allows you to choose the voice and tone of voice. The system described in Appendix 1, characterized by the features described herein. (Note 3) It includes a shape setting unit for setting the size and shape of the robot. The system described in Appendix 1, characterized by the features described herein. (Note 4) It features a notification unit that reminds you of medication times and other regularly needed reminders. The system described in Appendix 1, characterized by the features described herein. (Note 5) The aforementioned conversation section is, The system estimates the emotions of the person receiving care and adjusts the content and tone of conversation based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 6) The aforementioned conversation section is, Analyze the past conversation history of the person receiving care and select the most appropriate conversation topics. The system described in Appendix 1, characterized by the features described herein. (Note 7) The aforementioned conversation section is, During conversations, monitor the health status of the person receiving care and take appropriate action if any abnormalities are detected. The system described in Appendix 1, characterized by the features described herein. (Note 8) The aforementioned conversation section is, The system estimates the emotions of the person receiving care and adjusts the frequency of conversation based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 9) The aforementioned conversation section is, During conversations, provide topics relevant to the care recipient's hobbies and interests. The system described in Appendix 1, characterized by the features described herein. (Note 10) The aforementioned conversation section is, During the conversation, refer to information about the care recipient's family and friends and provide relevant topics. The system described in Appendix 1, characterized by the features described herein. (Note 11) The aforementioned video unit is The system estimates the emotions of the person receiving care and adjusts the video content based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 12) The aforementioned video unit is When displaying video, the color tone and brightness of the video are adjusted based on the visual preferences of the person receiving care. The system described in Appendix 1, characterized by the features described herein. (Note 13) The aforementioned video unit is When displaying video, the system selects the most suitable video by referring to the care recipient's past viewing history. The system described in Appendix 1, characterized by the features described herein. (Note 14) The aforementioned video unit is The system estimates the emotions of the person receiving care and adjusts the video display time based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 15) The aforementioned video unit is When displaying videos, the system prioritizes showing relevant videos based on the geographical location information of the person receiving care. The system described in Appendix 1, characterized by the features described herein. (Note 16) The aforementioned video unit is When displaying video, background music is selected based on the musical preferences of the person receiving care. The system described in Appendix 1, characterized by the features described herein. (Note 17) The observation unit is, The emotional state of the person receiving care is estimated, and the frequency and method of observation are adjusted based on the estimated emotional state. The system described in Appendix 1, characterized by the features described herein. (Note 18) The observation unit is, During observation, abnormalities are detected early by referring to the care recipient's past health data. The system described in Appendix 1, characterized by the features described herein. (Note 19) The observation unit is, During observation, the optimal timing for observation is selected based on the care recipient's daily routine. The system described in Appendix 1, characterized by the features described herein. (Note 20) The observation unit is, The system estimates the emotions of the person receiving care and adjusts the display method of observation results based on the estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 21) The observation unit is, During observation, the overall health status of the person receiving care is assessed by referring to data on their diet and exercise. The system described in Appendix 1, characterized by the features described herein. (Note 22) The observation unit is, During observation, the health status of the person receiving care is assessed by referring to their sleep data. The system described in Appendix 1, characterized by the features described herein. (Note 23) The aforementioned notification unit, The system estimates the emotions of the person receiving care and adjusts the content and timing of notifications based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 24) The aforementioned notification unit, When sending a notification, the system will refer to the care recipient's past notification history to select the most suitable notification method. The system described in Appendix 1, characterized by the features described herein. (Note 25) The aforementioned notification unit, When a notification is sent, the emergency contact information of the person receiving care will be automatically updated. The system described in Appendix 1, characterized by the features described herein. (Note 26) The aforementioned notification unit, The system estimates the emotions of the person receiving care and prioritizes notifications based on those estimated emotions. The system described in Appendix 1, characterized by the features described herein. (Note 27) The aforementioned notification unit, When sending a notification, the most suitable notification method will be selected based on the geographical location information of the person receiving care. The system described in Appendix 1, characterized by the features described herein. (Note 28) The aforementioned notification unit, When sending notifications, the content of the notifications can be customized by referencing the contact information of the care recipient's family and friends. The system described in Appendix 1, characterized by the features described herein. (Note 29) The aforementioned voice selection unit is It estimates the emotions of the person receiving care and adjusts the tone and speed of the voice based on the estimated emotions. The system described in Appendix 2, characterized by the features described herein. (Note 30) The aforementioned voice selection unit is When selecting a voice, the system refers to the care recipient's past voice selection history to select the most suitable voice. The system described in Appendix 2, characterized by the features described herein. (Note 31) The aforementioned voice selection unit is The system estimates the emotions of the person receiving care and adjusts the timing of voice changes based on the estimated emotions. The system described in Appendix 2, characterized by the features described herein. (Note 32) The aforementioned voice selection unit is When selecting a voice, the system selects the most suitable voice based on the language and dialect of the person receiving care. The system described in Appendix 2, characterized by the features described herein. (Note 33) The shape setting unit is The robot estimates the emotions of the person receiving care and adjusts the robot's shape and size based on those estimated emotions. The system described in Appendix 3, characterized by the features described herein. (Note 34) The shape setting unit is When setting the shape, the system selects the optimal shape by referring to the care recipient's past shape setting history. The system described in Appendix 3, characterized by the features described herein. (Note 35) The shape setting unit is The system estimates the emotions of the person receiving care and adjusts the timing of shape changes based on those estimated emotions. The system described in Appendix 3, characterized by the features described herein. (Note 36) The shape setting unit is When setting the shape, the optimal shape is selected based on the living environment of the person receiving care. The system described in Appendix 3, characterized by the features described herein. (Note 37) The aforementioned notification section is, The system estimates the emotions of the person receiving care and adjusts the content and timing of notifications based on those estimated emotions. The system described in Appendix 4, characterized by the features described herein. (Note 38) The aforementioned notification section is, When sending a notification, the system will refer to the care recipient's past notification history to select the most appropriate notification method. The system described in Appendix 4, characterized by the features described herein. (Note 39) The aforementioned notification section is, The system estimates the emotions of the person receiving care and determines the priority of notifications based on those estimated emotions. The system described in Appendix 4, characterized by the features described herein. (Note 40) The aforementioned notification section is, When sending notifications, the most appropriate notification method will be selected based on the geographical location information of the person receiving care. The system described in Appendix 4, characterized by the features described herein. [Explanation of symbols]

[0197] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots

Claims

1. The conversation department, which engages in conversations with those receiving care, A video unit that displays images based on the information obtained by the aforementioned conversation unit, The observation department observes changes in the health of those receiving care and provides advice, The system includes a notification unit that provides notification in an emergency based on information obtained by the observation unit. A system characterized by the following features.

2. It features a voice selection unit that allows you to choose the voice and tone of voice. The system according to feature 1.

3. It includes a shape setting unit for setting the size and shape of the robot. The system according to feature 1.

4. It features a notification unit that reminds you of medication times and other regularly needed reminders. The system according to feature 1.

5. The aforementioned conversation section is, The system estimates the emotions of the person receiving care and adjusts the content and tone of conversation based on those estimated emotions. The system according to feature 1.

6. The aforementioned conversation section is, Analyze the past conversation history of the person receiving care and select the most appropriate conversation topics. The system according to feature 1.

7. The aforementioned conversation section is, During conversations, monitor the health status of the person receiving care and take appropriate action if any abnormalities are detected. The system according to feature 1.

8. The aforementioned conversation section is, The system estimates the emotions of the person receiving care and adjusts the frequency of conversation based on those estimated emotions. The system according to feature 1.

9. The aforementioned conversation section is, During conversations, provide topics relevant to the care recipient's hobbies and interests. The system according to feature 1.

10. The aforementioned conversation section is, During the conversation, refer to information about the care recipient's family and friends and provide relevant topics. The system according to feature 1.

11. The aforementioned video unit is The system estimates the emotions of the person receiving care and adjusts the video content based on those estimated emotions. The system according to feature 1.

Images