Information Systems

Abstract

By reflecting information from the physical space into the virtual space VS based on events that occurred in the physical space, the convenience of user 1 is improved, and user 1's safety and security in the residence HS are ensured. [Solution] The system comprises a control unit (terminal control unit 6y) that renders images of a virtual space VS, a detection unit that detects events in a physical space RS, an imaging unit that captures images of the physical space RS, and a display unit that displays at least one of the images of the virtual space VS and the images of the physical space RS. When a predetermined event is detected by the detection unit, the control unit displays the images of the physical space RS on the display unit in place of, or together with, the images of the virtual space VS.

Description

Technical Field

[0001] The present invention relates to an information system that monitors users using information related to residences and houses.

Background Art

[0002] With the progress of the aging of the population with low birthrates, the population of the elderly aged 65 years or older has been increasing year by year in Japan, and the number of households including the elderly is also on the rise. According to the 2025 White Paper on the Aging Society, the number of elderly people aged 65 years or older reached approximately 36.24 million as of 2024. Furthermore, as of 2023, households including the elderly account for approximately 49.5% of all households. Among these, solitary households have reached 8.55 million households, accounting for approximately 30% of elderly households, and the proportion of solitary households is particularly high among elderly women. These changes in the household structure are causing social problems such as the isolation of elderly people living alone and the decline in their will to live.

[0003] The difference between the average life expectancy and the healthy life expectancy is also a major problem. According to the Ministry of Health, Labour and Welfare's data for 2024 (regarding the value of healthy life expectancy in 2022), the average life expectancy of men is 81.05 years and that of women is 87.09 years, while the healthy life expectancy is 72.57 years for men and 75.45 years for women. The difference between the average life expectancy and the healthy life expectancy, so-called "unhealthy period", reaches approximately 8.5 years for men and approximately 11.6 years for women. Although the unhealthy period has been showing a tendency to shorten year by year, the fact that a state with restrictions in daily life remains is a factor increasing the burden of caregiving and medical treatment.

[0004] In addition, changes are also seen in the lifestyle of the elderly. Although the number of elderly people entering nursing facilities is increasing, entering a nursing facility itself does not necessarily lead to an extension of the healthy life expectancy, and cases have been reported where irregular sleep, a life spent in bed all day, and a decrease in activity level occur. Also, the number of elderly people living alone has increased due to midlife divorce or the death of a spouse, and problems such as social isolation, decline in cognitive function, increase in lonely deaths, and further, the chain deterioration of frailty (frailty domino) have become apparent.

[0005] To address these challenges and extend healthy life expectancy and improve quality of life (QOL), it is crucial for older adults to maintain social connections while living independently at home. Specifically, this requires gentle supervision in daily life, social participation through regular conversations and interactions, and maintenance and improvement of health conditions such as exercise, nutrition, and sleep. By continuously supporting these aspects, we can enhance the older adult's motivation to live, prevent cognitive decline and physical deterioration, and provide peace of mind to their families.

[0006] To meet these needs, the introduction of robots is progressing. Regarding monitoring robots for the elderly and those requiring care, the Ministry of Economy, Trade and Industry and the Ministry of Health, Labour and Welfare have designated "monitoring and communication" as a "priority area for the use of robot technology in elderly care," and have listed "platforms for devices using robot technology equipped with sensors and external communication functions for use in elderly care facilities" and "life support devices using robot technology for communication with the elderly, etc." as priority areas. The name of the "priority area for the use of robot technology in elderly care" has been changed to "priority area for the use of elderly care technology" in light of the recent progress in data utilization using ICT and IoT technologies and new social challenges in elderly care settings, with the aim of promoting the development and dissemination of innovative devices, and public and private sectors are promoting support for the independence of the elderly, etc.

[0007] By establishing a robot-centered platform (cloud service infrastructure), robots can collaborate with devices such as smart home appliances and environmental sensors to understand the living conditions of the elderly, provide appropriate guidance through conversational services with AI and experts, and offer integrated health advice. On the other hand, protecting privacy is a crucial issue when monitoring the elderly. Monitoring living conditions and collecting behavioral information must be done in a way that is acceptable and reassuring, without excessively accessing personal information. From this perspective, introducing a virtual space into monitoring is considered useful. It is expected that the introduction of a virtual space will allow for gentle monitoring while protecting the privacy of users.

[0008] Regarding technologies for recreating residences in virtual space, for example, there is a known housing-related information provision system that provides housing-related information based on a simulation of the actual feeling of living in a virtual reality world that constructs a pseudo-realistic virtual world related to residences. In this technology, for example, a housing information server equipped with a product information database containing information on housing-related products distributes housing information to clients via the internet. The client is provided with a virtual housing space creation means that creates a virtual housing space on a display, a product placement means that places housing-related products selected from the product information database into the virtual housing space, and a walkthrough means that allows an avatar to walk through the virtual housing space where the selected housing-related products are placed. (Patent Document 1)

[0009] According to Patent Document 1, based on digital camera images of the target house and data read from blueprints, the client uses application software to create a virtual three-dimensional image of the house on the client's display using a virtual living space creation means. The virtual three-dimensional image of the product is then placed within the virtual three-dimensional space of the house. A walkthrough means allows the user's avatar to move through the virtual three-dimensional space inside the house, enabling a three-dimensional simulated experience of interior effects such as the viewing angle of the television, the field of view when the door is open, the field of view through the windows, and whether the lighting effects of the lighting fixtures are appropriate.

[0010] Furthermore, information processing devices are known that facilitate communication between users in the physical space based on specific events occurring in the virtual space. Regarding this technology, for example, when user avatars gather in a predetermined location within the virtual space, communication is established between terminals used by users who intend to communicate, and at least one of images or audio is transmitted and received. (Patent Document 2)

[0011] According to Patent Document 2, it is possible to easily initiate real-time communication using images and audio. [Prior art documents] [Patent Documents]

[0012] [Patent Document 1] Japanese Patent Publication No. 2001-155179 [Patent Document 2] International Publication No. 2023 / 054838 [Disclosure of the Invention] [Problems that the invention aims to solve]

[0013] However, the technology described in Patent Document 1 addresses the problem that even when purchasing furniture that seems to match a newly acquired house's room from an internet site, no simulation is performed beforehand, as is the case when purchasing from a catalog, making it difficult for users to purchase furniture that fully satisfies them. Therefore, the aim is to provide housing-related information based on a simulation of the actual living sensation using virtual reality, which constructs a pseudo-realistic virtual world related to housing. Thus, the technology in Patent Document 1 merely simulates the physical space within a virtual space, and events that occur in the physical space are not reflected in the virtual space in real time.

[0014] Furthermore, according to the technology described in Patent Document 2, information from the physical space is reflected in the virtual space based on specific events that occur in the virtual space. However, it is not suggested that information from the physical space be reflected in the virtual space in real time based on events that occur in the physical space.

[0015] This invention was devised to solve the problems of the prior art, and its purpose is to provide an information system that improves user convenience and ensures the safety and security of users in their homes while respecting their privacy, by reflecting information from the physical space in a virtual space based on events that occur in the physical space. [Means for solving the problem]

[0016] The present invention, made to solve the aforementioned problems, is an information system comprising: a control unit for rendering images of a virtual space; a detection unit for detecting events in a physical space; an imaging unit for capturing images of the physical space; and a display unit for displaying at least one of the images of the virtual space and the images of the physical space. The control unit, when a predetermined event is detected by the detection unit, displays the images of the physical space on the display unit in place of, or together with, the images of the virtual space. This improves user convenience and ensures user safety and security in the event of an anomaly, while respecting user privacy during normal times.

[0017] Furthermore, in this invention, the detection unit is provided on a mobile body configured to move within the physical space. This makes it possible to monitor the user in any room or location within the residence.

[0018] Furthermore, in the present invention, when the control unit detects the movement of an object in an image based on the image captured by the imaging unit, it displays the image of the physical space on the display unit in place of, or together with, the image of the virtual space. This makes it possible for the user to understand the situation based on the actual image if an abnormality occurs in the residence.

[0019] Furthermore, in the present invention, when the control unit detects the movement of an object in an image based on the image captured by the imaging unit, it applies a predetermined processing to the virtual space image corresponding to the area in the physical space where movement was detected before displaying the image of the physical space. This prevents the indiscriminate display of the actual image of the residence and makes it possible to protect the privacy of the user.

[0020] Furthermore, in the present invention, when the control unit detects the movement of an object in an image based on the image captured by the imaging unit, it displays an image that has been processed in a predetermined manner from the image captured by the imaging unit before displaying the image of the physical space. This prevents the indiscriminate display of actual images of the residence and makes it possible to protect the privacy of the user.

[0021] Furthermore, the present invention further includes a sound detection unit, and when the sound pressure detected by the sound detection unit exceeds a predetermined range, the control unit displays an image of the physical space on the display unit in place of, or together with, the image of the virtual space. This makes it possible for users to understand the situation based on real images when an abnormality occurs in their residence.

[0022] Furthermore, in the virtual space, a first avatar corresponding to the first user who is to be monitored and a second avatar corresponding to the second user who monitors the first user are reproduced, and there are multiple second users, and the control unit changes the appearance of the second avatar according to each of the different second users. This makes it possible for the user who is to be monitored to easily recognize who they are communicating with. [Effects of the Invention]

[0023] According to the present invention, by reflecting information from the physical space into the virtual space based on events that occur in the physical space, it becomes possible to improve user convenience while ensuring the safety and security of users in their homes, while also considering their privacy.

Brief Description of the Drawings

[0024] [Figure 1] Block diagram showing a configuration example of an information system S1 according to the first embodiment of the present invention [Figure 2] Perspective view explaining the appearance and functions of the robot 2 [Figure 3] Block diagram showing the configuration of the robot 2 [Figure 4] Configuration diagram of a platform for realizing device cooperation [Figure 5] Explanatory diagram showing an overview of device cooperation [Figure 6] Explanatory diagram showing an overview of voice calling and conversation functions [Figure 7] Explanatory diagram showing an overview of health advice [Figure 8] Explanatory diagram showing a business model related to the smart kit 67 [Figure 9] Explanatory diagram showing an example of the smart kit 67 [Figure 10] Explanatory diagram showing the physical space RS in the second embodiment of the present invention [Figure 11] Explanatory diagram showing the video of the virtual space VS

Modes for Carrying Out the Invention

[0025] (First Embodiment) Hereinafter, the first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration example of an information system S1 according to the first embodiment of the present invention. The information system S1 includes a robot 2, an information terminal 6, and a server 30. These components are connected to a network 50 and exchange information with each other. The information system S1 may include any one of a smart meter 4, a smart remote control hub 5, and a smart tap 8, and these are also connected to the network 50. Further, the information system S1 may include a controlled device 3 and an electric device 7.

[0026] In the first embodiment, the user of the information system S1 is assumed to be, for example, an elderly person living alone in a residence HS (see Figure 10), designated as the first user 1a. Of course, the first user 1a does not have to be elderly, and furthermore, there may be multiple first users 1a (a married couple, or relatives living together, etc.). In addition, the users of the information system S1 include those who look after the first user 1a (including family members, relatives, doctors, caregivers, guardians, and others) (second user 1b). In the following description, when the first user 1a and the second user 1b are not distinguished, they may simply be referred to as user 1. Furthermore, user 1 also includes a developer 1c (see Figure 8) who creates content, including application software (hereinafter sometimes referred to as "apps") used in the information system S1.

[0027] The first cloud service platform 51 is a platform built by a person (first service provider) who provides a predetermined service using the information system S1. The second cloud service platform 52 and the third cloud service platform 53 are platforms built by a third party other than the first service provider (second service provider). Hereafter, the first to third cloud service platforms 51 to 53 may be collectively referred to as "cloud service platform 54". There is no limit to the number of cloud service platforms 54 built by the second service provider, and the information system S1 contains multiple cloud service platforms 54. Here, at least one of the cloud service platforms 54 may be a system platform, a so-called IoT platform, that aggregates, analyzes, visualizes, and controls data collected from devices including the second environmental sensor 55 and IoT devices 56. Furthermore, the cloud service platform 54 may provide the user 1 with multiple software (components, modules) that realize predetermined functions.

[0028] Here, the second environmental sensor 55 may include a temperature and humidity sensor for measuring the temperature (air temperature) and humidity of the living space of the first user 1a (e.g., at least one room of a residence HS), an illuminance sensor for detecting illuminance, a motion sensor for detecting infrared radiation emitted by people, an odor sensor for detecting odors, and a fire detection sensor (temperature sensor (e.g., thermocouple), smoke sensor (e.g., light scattering type), flame detection sensor (e.g., infrared sensor), gas detection sensor (e.g., CO, CO2, VOC (volatile organic compound) detection sensor)) (none of which are shown).

[0029] Furthermore, IoT devices 56 include so-called smart air conditioners, smart lighting, smart home appliances (refrigerators, washing machines, robotic vacuums, etc.), smart ovens, smart locks, intercoms, smart speakers, wearable devices, sensor modules with communication functions, smart jump ropes equipped with communication functions and training support functions for the first user 1a, etc. Some of the electrical equipment 7 and controlled devices 3 shown in Figure 1 may also be included in IoT devices 56. In a broader sense, IoT devices 56 may also include robots 2 and sensors (second environmental sensors 55), but in each embodiment, these may be distinguished from IoT devices 56 for the sake of clarity.

[0030] Applications used by robot 2 and information terminal 6 can access components provided by other cloud service platforms 54 besides the first cloud service platform 51 via APIs (Application Programming Interfaces) and remotely execute various functions. Conversely, devices receiving services from, for example, the second cloud service platform 52 can utilize components provided by the first cloud service platform 51 and data measured by robot 2. In other words, the first cloud service platform 51 and the other cloud service platforms 54 are linked via APIs.

[0031] Controlled device 3 refers to home appliances such as televisions, air conditioners, and lighting fixtures. Of course, controlled device 3 is not limited to these, and also includes home appliances that can be controlled, such as by turning them ON / OFF or setting their operation settings, using a remote controller (hereinafter sometimes referred to as "remote control"). Furthermore, home appliances that are connected to the network 50 or comply with wireless communication standards (Wi-Fi (Wireless Fidelity), BLE (Bluetooth® Low Energy)), etc., and can be remotely accessed or programmed via the cloud service platform 54 or local control are also included in controlled device 3. As will be described later, robot 2 has the function of a remote control. Robot 2 may also have the function of a so-called smart remote control or smart hub, which can operate multiple controlled devices 3 with a single unit.

[0032] The smart meter 4 includes a measurement unit, a communication module, a data processing unit, etc. (none of which are shown), and measures the energy consumption of electricity, gas, etc., of the residence HS, etc., where the first user 1a resides, in real time or periodically, and transmits it to the server 30 via the network 50. The energy consumption data stored in the server 30 can be accessed, for example, by a robot 2 or an information terminal 6.

[0033] The Smart Tap 8 is a device that interposes itself between a power supply outlet in a house or other building and a designated electrical device 7, supplying power to the electrical device 7, and also works in conjunction with an information terminal 6 or a robot 2 to remotely turn home appliances and electronic devices ON / OFF. Furthermore, the Smart Tap 8 is equipped with a power measurement unit (not shown) that measures the power supplied to the electrical device 7. The Smart Tap 8 also includes a communication module (not shown) that complies with wireless communication standards such as LTE (Long Term Evolution), LTE-M (Long Term Evolution - Machine, LTE Cat.M1), 4G, 5G, and BLE.

[0034] The smart tap 8 turns the electrical equipment 7 connected to it ON / OFF based on control signals input via the network 50, and also measures the amount of power (energy consumed) by the electrical equipment 7. Then, it transmits the measured power data to an external source (e.g., server 30) via the network 50. Here, the electrical equipment 7 may include, for example, a refrigerator, television, air conditioner, lighting fixture, and other home appliances. Thus, the electrical equipment 7 has aspects of a controlled device 3, and in the first embodiment, the smart tap 8 can control the electrical equipment 7 based on control signals transmitted from the robot 2. In the following description, the controlled device 3 and the electrical equipment 7 may be collectively referred to as "controlled device 3, etc." When the first user 1a gives instructions to the robot 2, the robot 2 receives the instructions from the first user 1a and outputs control signals to control the controlled device 3, etc. based on the instructions input by the first user 1a.

[0035] The smart remote control hub 5 has at least one of the functions of a so-called smart remote control or a smart hub. Here, a smart remote control refers to a device that enables the operation of an infrared remote control function from an information terminal 6 or a voice assistant, etc. A smart hub refers to a device that controls electrical equipment 7 via a network 50 including Wi-Fi, or controls electrical equipment 7 based on a communication standard such as BLE. Furthermore, the smart remote control hub 5 may also include a sensor (second environmental sensor 55) that measures temperature, humidity, and illuminance.

[0036] Server 30 is a known computer system and consists of a server control unit 30a and a server storage unit 30b. The server control unit 30a consists of a CPU (Central Processing Unit) and memory (not shown) and controls the components of Server 30. The server storage unit 30b has large-capacity storage consisting of ROM (Read Only Memory), RAM (Random access memory), and RAID (Redundant Arrays of Independent Disks), etc. There may be multiple Server 30s, and it may also include a group of storage devices (not shown), a group of network devices such as L2 / L3 switches and routers, a load balancer, etc. Together, the Server 30 etc. constitute the first cloud service infrastructure 51.

[0037] Information terminal 6 is, for example, a mobile information terminal such as a smartphone or tablet, a PC (Personal Computer), smart glasses, or VR (Virtual Reality) goggles, and comprises a display unit 6x, a terminal control unit 6y, a terminal storage unit (not shown), an input unit, etc. The terminal control unit 6y consists of a CPU and memory (not shown) and controls the components of information terminal 6. Here, the first user 1a uses the first information terminal 6a, and the second user 1b uses the second information terminal 6b. In the following description, when the first information terminal 6a and the second information terminal 6b are not distinguished, they may be referred to as information terminal 6.

[0038] Figure 2 is a perspective view illustrating the appearance and functions of robot 2. As shown in the figure, robot 2 is equipped with a head 2a, torso 2b, legs 2c, arms 2d, imaging unit 2e, distance measuring unit 2f, sound detection unit 2g, first notification unit 2h, second notification unit 2i, third notification unit 2j, SELECT / ENTER switch 2k, CANCEL switch 2m, control signal output unit 2n, USB socket 2p, and touch sensor 2r. Thus, robot 2 is equipped with various sensors (imaging unit 2e, distance measuring unit 2f, sound detection unit 2g) and can perform multiple functions depending on the environment, making it possible to provide various services to the first user 1a.

[0039] Robot 2 is configured to be mobile by a movement mechanism 16a (see Figure 3) provided on its legs 2c. Specifically, the legs 2c are equipped with drive wheels (not shown). The wheels are connected to a drive source 16 (see Figure 3), such as a motor. By controlling the orientation and rotation direction of the wheels relative to the floor, Robot 2 can be moved in any direction, forward or backward.

[0040] In the first embodiment, the size of the robot 2 is, for example, 40 cm to 60 cm in height. Of course, there are no particular restrictions on the size or form of the robot 2. The robot 2 may be a so-called humanoid that walks on two legs, a smaller robot toy, a quadrupedal animal type, or it may be equipped with wheels or tracks instead of two or four legs. Furthermore, the robot 2 may not have a locomotion mechanism 16a and may be in a stationary form, may not have arms, may be composed of only one housing part, or may be in the form of a remote control or a wearable device.

[0041] Figure 3 is a block diagram showing the configuration of robot 2 in the first embodiment. The configuration of robot 2 will be described below with reference to Figure 2. The first control unit 10 is composed of a CPU and operates according to a control program stored in a memory unit 11 composed of ROM, RAM, etc. The first control unit 10 is connected to other components by a bus 20, etc., and the first control unit 10 controls the other components via the bus 20, etc. The memory unit 11 also includes non-volatile memory (EEPROM (Electrically Erasable Programmable Read-Only Memory), etc.). The memory unit 11 stores pattern data of control signals output when controlling the controlled device 3.

[0042] The imaging unit 2e is equipped with an image sensor composed of a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). The imaging unit 2e outputs image data. The first control unit 10 may transmit the image data acquired by the imaging unit 2e to the server 30. The server control unit 30a, having received the image data, determines the first user 1a using the face recognition model built in the server storage unit 30b. The server control unit 30a then transmits the user ID of the determined first user 1a to the robot 2. The first control unit 10 can identify the first user 1a based on the user ID. Of course, the first control unit 10 may also perform the identification of the first user 1a.

[0043] The distance measuring unit 2f performs distance measurement using LiDAR. LiDAR measures the distance to an object, the shape of the object, etc., by emitting laser light and measuring the time of flight (TOF) of the reflected light. The first control unit 10 maps the house and generates a house map based on the image captured by the imaging unit 2e and / or the distance measurement information measured by the distance measuring unit 2f. Then, it moves the robot 2 using the generated house map. In other words, the robot 2 is capable of moving autonomously inside the house. Furthermore, the first control unit 10 can grasp the relative positional relationship between the first user 1a and the robot 2 based on the output of the distance measuring unit 2f, the user ID, and the attitude information of the first user 1a. If there are multiple first users 1a, the positional relationship between the robot 2 and each first user 1a is grasped. The outputs of the imaging unit 2e and the distance measuring unit 2f may be used to construct the virtual space VS described in the second embodiment.

[0044] The sound detection unit 2g consists of multiple (e.g., six) microphones and detects sound from all directions. The microphone outputs are converted into digital data by an A / D converter (not shown). The first control unit 10 recognizes the digitized voice of the first user 1a. In speech recognition, phonemes are extracted from the acquired voice information, converted into text, and then vocabulary information is identified. Based on this vocabulary information, the first control unit 10 recognizes the instructions of the first user 1a and the intent of the information input by the first user 1a.

[0045] The speech recognition processing may be performed by the server control unit 30a. The application for processing speech information can be implemented, for example, by combining a natural language processing (NLP) model and a machine learning model. Furthermore, a rule-based system may be used to convert abstract words into specific settings for the controlled device 3, etc. This allows the first control unit 10 to determine, based on abstract utterances from the first user 1a that do not specify the controlled device 3, etc., such as "It's kind of hot," or "The room is dark," that the temperature setting of the air conditioner should be changed to a lower temperature (cooling), or the luminous intensity of the lighting fixtures should be increased.

[0046] The fall detection unit 2s detects whether the first user 1a has fallen based on the outputs of the imaging unit 2e and the sound detection unit 2g. Specifically, the fall detection unit 2s detects the posture, position, and movement of the first user 1a based on the image data captured by the imaging unit 2e, and further detects falling sounds and abnormal sounds based on the sound data acquired by the sound detection unit 2g. If the degree or rate of decrease in the position of the first user 1a's waist or head is greater than a predetermined value based on skeletal estimation (posture estimation) using the image data, the first flag is turned ON. Also, if a falling sound or the like is detected based on the sound data, the second flag is turned ON. These two flags are transmitted to the first control unit 10, and if both flags are ON, the first control unit 10 determines that there is a high probability that the first user 1a has fallen. In addition, the second flag may be turned ON by referring to vibration data acquired by the inertial sensor 19, which will be described later, instead of or in conjunction with the output of the sound detection unit 2g.

[0047] The distance measurement data measured by the distance measuring unit 2f may also be referenced in fall detection. Specifically, if the distance to the first user 1a changes significantly within a predetermined time, the third flag is turned ON. The first control unit 10 may then determine that there is a high probability of a fall if all of the first to third flags are ON. Furthermore, if it determines that there is a high probability of a fall, it may call out to the first user 1a via the notification unit 14 (first notification unit 2h) described later, and if no response voice is detected by the sound detection unit 2g within a predetermined period, or if words such as "it hurts" or "I can't move" are detected, it may be determined that the first user 1a has fallen. Of course, the presence or absence of a fall may also be determined using a learning model that has been pre-learned to track the relationship between the time-series changes in the outputs of the imaging unit 2e, distance measuring unit 2f, sound detection unit 2g, and inertial sensor 19 and body movement. Alternatively, for example, an infrared sensor array with 16x16 pixels may be used, and the skeleton of the first user 1a may be estimated based on the ultra-low-resolution FIR (Far Infrared Rays) image output by the sensor, and the presence or absence of a fall may be detected based on the estimated state of the skeleton.

[0048] The notification unit 14 consists of a first notification unit 2h, a second notification unit 2i, and a third notification unit 2j, which serve as a user interface. The first notification unit 2h is composed of, for example, a speaker and provides sound information and voice information to the first user 1a. The second notification unit 2i is a display composed of an LCD (liquid crystal display), OLED (organic light emitting diode), etc., and provides various image information and text information to the first user 1a. The second notification unit 2i displays a menu for selecting the functions of the robot 2. Furthermore, when the first user 1a requests any information from the robot 2, that information is displayed on the second notification unit 2i.

[0049] The third notification unit 2j consists of 7-segment LEDs positioned at the eye locations along the curved surface of the part corresponding to the face of the robot 2. The first control unit 10 lights up / turns off / flashes the selected segment to inform the first user 1a of the state of the robot 2 (e.g., active / sleep state) and to express emotions, etc.

[0050] As described above, NLP may be built into the server 30. Alternatively, the server control unit 30a may access an external large language model (LLM) via the network 50. By using a large language model (LLM), natural communication (e.g., chat / conversation) becomes possible, where the utterance of the first user 1a detected by the sound detection unit 2g is recognized and the first notification unit 2h responds to the first user 1a. In other words, the robot 2 may have a "chat mode" for conversing with the first user 1a.

[0051] In this way, voice information is input to the sound detection unit 2g, and the first notification unit 2h outputs voice information. This enables voice communication between the robot 2 and the first user 1a. Of course, for information input, a touch panel or the like superimposed on the second notification unit 2i may be used instead of the sound detection unit 2g, or together with the sound detection unit 2g. This allows the first user 1a to communicate with the robot 2 via the touch panel or the like.

[0052] The control signal output unit 2n is composed of an infrared light-emitting diode or a transmitter that outputs radio waves. The first control unit 10 extracts pattern data from the storage unit 11 and drives the infrared light-emitting diode or generates a radio wave signal based on the pattern data. That is, the robot 2 functions as an infrared (IR) remote control or a radio wave (RF) remote control, and an IR or RF control signal is transmitted to the controlled device 3. The control signal output unit 2n may be composed of a communication module compliant with a short-range wireless standard such as BLE. When the control signal output unit 2n is composed of an infrared light-emitting diode, the control signal output unit 2n is provided on at least one side of the robot 2. The imaging unit 2e, the second notification unit 2i, and the control signal output unit 2n described above are integrated and provided on the front of the torso 2b in the form of a tablet suspended from the neck of the robot 2 by a strap (see Figure 2).

[0053] The SELECT / ENTER switch 2k and CANCEL switch 2m are used by the first user 1a to select the operating mode, functions, etc., of the robot 2. The first user 1a rotates the SELECT / ENTER switch 2k, which is shaped like an earpiece, in the circumferential direction to switch between menu selection items. Then, by pressing the SELECT / ENTER switch 2k (pushing it towards the head 2a), the menu selection is confirmed (if the menu has a hierarchical structure, it moves to a lower selection item). Also, by pressing the CANCEL switch 2m, the selection is canceled (if the menu has a hierarchical structure, it moves to a higher selection item). This process is displayed on the second notification unit 2i.

[0054] As described above, the first user 1a and the robot 2 can communicate by voice. However, if, for example, a malfunction occurs in the sound detection unit 2g or the first notification unit 2h, the first user 1a can give instructions to the robot 2 using the SELECT / ENTER switch 2k and the CANCEL switch 2m. Specifically, for example, based on the instructions input using the SELECT / ENTER switch 2k, the first control unit 10 can transmit a control signal to the controlled device 3, such as an air conditioner, via the control signal output unit 2n.

[0055] The USB socket 2p (for example, a USB Type-C female) is provided in a recessed area on the robot 2's head 2a (corresponding to the back of the head). The robot 2 and the information terminal 6 may be connected via the USB socket 2p, and data may be transmitted and received. In addition, power for charging devices such as smartphones may be supplied via the USB socket 2p.

[0056] The touch sensor 2r is positioned on the top of the robot 2's head. When the first user 1a's finger or other object touches the touch sensor 2r, the robot 2 starts / stops operation. In case of malfunctions in the robot 2's operation, the robot 2 may be reset by the first user 1a pressing and holding the touch sensor 2r.

[0057] The communication unit 12 is equipped with a communication module (not shown) that conforms to wireless communication standards such as LTE, LTE-M, 4G, and 5G, and connects to the network 50. The communication unit 12 may be made compliant with the Wi-Fi standard, and the robot 2 may connect to the network 50 via a wireless router or the like. The first control unit 10 sends and receives information between the smart meter 4, smart tap 8, information terminal 6, and server 30 via the communication unit 12. Of course, the communication unit 12 may be configured with a communication module (not shown) that conforms to a short-range wireless standard such as BLE. In this case, the first control unit 10 can send and receive information directly between the smart meter 4, smart tap 8, and information terminal 6, for example, without going through the network 50.

[0058] The first environmental sensor 17, like the second environmental sensor 55 described above, may include a temperature sensor, humidity sensor, illuminance sensor, motion sensor, odor sensor, fire detection sensor, smoke sensor, gas detection sensor, etc. Of course, the robot 2 does not have to include all of these sensors. Based on the output of the first environmental sensor 17, the first control unit 10 can determine the environment and conditions of the dwelling HS (living room), whether or not the first user 1a is in the room, the number of people, and further detect the occurrence of a fire or the like.

[0059] The inertial sensor 19 is composed of, for example, a three-axis accelerometer and / or a gyroscope. Here, the three-axis accelerometer outputs the acceleration of the robot 2 in the direction and by how much its velocity is changing for the three axes X, Y, and Z. The gyroscope also outputs the angular velocity of the robot 2 in the direction and at what speed for the three axes X, Y, and Z. Generally, the gyroscope is used to detect the direction of movement and the accelerometer is used to detect the distance traveled. In this way, the inertial sensor 19 outputs three-axis acceleration information and three-axis angular velocity information based on the movement of the robot 2. The first control unit 10 detects the posture or position of the robot 2 based on the output of the inertial sensor 19. In particular, if the robot 2 is bipedal, it controls the robot 2 to maintain balance and prevent it from falling over when moving forward, backward (walking), or standing on one leg. The inertial sensor 19 may also detect vibrations applied to the robot 2. By referring to the output of the inertial sensor 19, contact of the first user 1a with the robot 2 and the aforementioned falls of the first user 1a can be detected. In other words, the inertial sensor 19 can be one of the elements that make up the tipping detection unit 2s described above.

[0060] The drive source 16 is composed of, for example, a DC motor. Multiple drive sources 16 are provided on the robot 2. The first control unit 10 controls the drive sources 16 and the moving mechanism 16a to move the robot 2. The drive sources 16 may also be used to displace the arms 2d and head 2a of the robot 2. Specifically, the arms 2d can be moved forward to transport luggage or to make body contact with the first user 1a. The head 2a may also be rotated horizontally within a predetermined angular range to reproduce negative emotions (expressions such as "no" or "no"), or rotated vertically within a predetermined angular range to reproduce positive emotions (expressions such as "OK" or "yes").

[0061] The first control unit 10 operates the drive source 16 and the movement mechanism 16a based on the outputs of the imaging unit 2e and the distance measuring unit 2f, so that the robot 2 can move autonomously forward and backward, and further change its posture (including the posture of the head 2a) in the left and right and up and down directions. That is, the robot 2 can move forward, backward and rotate, and may also change the imaging range of the imaging unit 2e or the scanning range of the distance measuring unit 2f by substantial panning and tilting.

[0062] Robot 2 operates based on power supplied from a battery 23 (a secondary battery such as a lithium-ion battery) built into its torso 2b. The charging control unit 22 has the function of a BMS (Battery Management System). The BMS monitors the remaining capacity of the battery 23 and also monitors the voltage, charge / discharge current, temperature, etc., of the cells that make up the battery 23. The BMS also measures the power consumed by Robot 2. Based on the monitoring results of the BMS, if charging is required, the first control unit 10 moves Robot 2 to the charging dock 25 based on the house map described above. The charging dock 25 may be provided with a predetermined marker, and the first control unit 10 recognizes the marker based on the output of the imaging unit 2e and docks with the charging dock 25.

[0063] Figure 4 is a diagram of the platform configuration for realizing device collaboration. As described above, the server 30 and other components constitute the first cloud service infrastructure 51. As shown in the figure, developer 1c, as part of user 1, can create a program (develop an application) to control the operation of robot 2 using the resources provided by the first cloud service infrastructure 51. For example, Scratch can be used as a programming language.

[0064] Developer 1c can use APIs (Application Programming Interfaces) pre-prepared on the first cloud service platform 51 to link data and functions (API linkage) with other platforms such as the second cloud service platform 52 and the third cloud service platform 53. For example, Developer 1c can control IoT devices 56 such as home appliances and acquire the output of a predetermined sensor (second environmental sensor 55) via the platform provided by Tuya®.

[0065] An application running on server 30 can receive a response from a generation AI by sending text or other data (request) to the third cloud service platform 53, which provides a generation AI service, for example. Furthermore, an application running on server 30 can develop an application that utilizes the second environmental sensor 55 by using an API in the third cloud service platform 53 to acquire the output of the second environmental sensor 55. Additionally, applications running on robot 2 or IoT devices 56 can similarly utilize services such as generation AI provided by the third cloud service platform 53, and other services provided by the second cloud service platform 52 (for example, acquiring the output of the second environmental sensor 55) via the first cloud service platform 51.

[0066] In the services provided by the first cloud service platform 51, the developer 1c does not need to be aware of which cloud service platform 54 provides the function to acquire and refer to the sensor output. By selecting and specifying the sensor to be used (second environmental sensor 55) in the program development environment provided by the first cloud service platform 51, the developer 1c can easily incorporate the output of the second environmental sensor 55 into the program. In other words, the first cloud service platform 51 comprehensively manages and provides to the developer 1c the APIs used for data transmission and reception or functional cooperation with other cloud service platforms 54. Programs designed under such a development environment are downloaded to the robot 2, IoT device 56, and information terminal 6, and these operate standalone. In this way, by utilizing the first cloud service platform 51 and other cloud service platforms 54 with API integration, services such as device cooperation, voice guidance / conversation, and health advice can be easily realized.

[0067] Conversely, resources such as the first environmental sensor 17, inertial sensor 19, fall detection unit 2s, touch sensor 2r, imaging unit 2e, distance measuring unit 2f, sound detection unit 2g, and control signal output unit 2n (all of which are shown in Figure 3) provided on the robot 2 can be used in the development environments provided by the second cloud service platform 52 and the third cloud service platform 53, enabling the development and provision of services that span multiple cloud service platforms 54.

[0068] Figure 5 is an explanatory diagram illustrating the overview of device coordination. In Figure 5, use case [A] shows an example of turning an air conditioner ON / OFF and adjusting its temperature, and use case [B] shows an example of turning an LED light ON / OFF and dimming it. In use case [A], robot 2 refers to the output of a second environmental sensor 55 (here, a temperature and humidity sensor) placed in the room, and if the room temperature exceeds a predetermined value, it notifies the first user 1a via the first notification unit 2h (see Figure 3) with a message such as "It's hot, so I'll turn on the air conditioner," and controls the electrical device 7 (here, the air conditioner) via the smart remote control hub 5. In use case [B], robot 2 refers to the output of a second environmental sensor 55 (here, an illuminance sensor), and if the room illuminance falls below a predetermined value, it notifies the first user 1a with a message such as "It's dark, so I'll turn on the lights," and controls the controlled device 3 (here, the lights) via the control signal output unit 2n (see Figure 3).

[0069] Furthermore, device integration is not limited to these examples; it is possible to provide various services to the first user 1a, such as detecting lost items using anti-loss tags, security using door and window sensors, and ensuring safety and security using emergency contact buttons. In addition, device integration makes it possible to easily construct the virtual space VS shown in the second embodiment.

[0070] Figure 6 is an explanatory diagram illustrating the overview of the voice prompting and conversation functions. In the first embodiment, a conversation function using Retrieval-Augmented Generation (RAG) is provided. The conversation function is performed in the following steps. Step (1): The first user 1a inputs a question to the robot 2. The question is, for example, made by voice, and the robot 2 converts the voice into text and sends it to an external search system (for example, the third cloud service platform 53 described above). The robot 2 may also send attribute information of the first user 1a (age, gender, hobbies, etc.) along with the question. Step (2): The external search system accesses the database and searches for documents related to the received text. The database already contains various documents and other content, and this content is associated with the attribute information described above. Step (3): The search system extracts relevant documents and texts from the database and sends them to robot 2. Step (4): Robot 2 sends both the question from the first user 1a and the highly relevant documents and texts returned from the search system (the integrated question) to the large language model LLM. Step (5): The large-scale language model LLM generates an answer (text) to the integrated question and sends it to robot 2. Step (6): Robot 2 converts the received text into speech and presents it to the first user 1a via the first notification unit 2h (see Figure 3). At this time, the text may also be displayed on the second notification unit 2i (see Figure 3).

[0071] Figure 7 is an explanatory diagram illustrating the overview of health advice. In Figure 7, the second environmental sensor 55 is placed, for example, on the bedding of the first user 1a, such as a bed. Here, the second environmental sensor 55 includes, for example, a pressure-sensitive sensor sheet. The output of the pressure-sensitive sensor sheet while the first user 1a is sleeping is transmitted to the cloud service infrastructure 54 and analyzed by a server device (not shown) that constitutes the cloud service infrastructure 54. The analysis results are then transmitted, for example, as an SMS to the second information terminal 6b used by the second user 1b who is monitoring the first user 1a. Furthermore, the robot 2 may also acquire the analysis results and output health advice based on the analysis results.

[0072] Furthermore, the second environmental sensor 55 may include, for example, a motion detection sensor composed of a triaxial accelerometer and / or a gyroscope. The motion detection sensor is attached, for example, to the arm or leg of the first user 1a in the form of a wristband. The output of the motion detection sensor is aggregated to the cloud service platform 54 and also transmitted to the robot 2. Based on the output of the motion detection sensor, the robot 2 can mimic the movements of the first user 1a's limbs. This improves the first user 1a's motivation to exercise. The second environmental sensor 55 may also include, for example, a pulse wave sensor. The output of the pulse wave sensor is also aggregated to the cloud service platform 54 and transmitted to the robot 2. The robot 2 (first control unit 10 (see Figure 3)) estimates the exercise load of the first user 1a based on the output of the pulse wave sensor, and based on the estimated exercise load, it can take measures such as having the first user 1a rest. Furthermore, by processing the data aggregated to the cloud service platform 54, it becomes possible to provide health advice based on exercise history, etc.

[0073] Figure 8 is an explanatory diagram showing a business model related to the smart kit 67, and Figure 9 is an explanatory diagram showing an example of the smart kit 67. The concept of the smart kit 67 is to stimulate the intellectual curiosity of the developer 1c and drive innovation by allowing free customization of the robot 2 and IoT device 56. The smart kit 67 shown in Figure 9 is suitably used in the development of the aforementioned application. The smart kit 67 includes an information terminal 6, a robot 2, and a starter pack 66, and is provided to the developer 1c as a package.

[0074] The starter pack 66 includes a smart remote control hub 5, a second environmental sensor 55 (e.g., a temperature and humidity sensor, an illuminance sensor), an IoT device 56 (e.g., a smart jump rope), and a smart tap 8. Depending on the application that developer 1c intends to develop, variations of the starter pack 66 may be set, such as a sports pack, a health pack, an energy-saving pack, or a monitoring pack. Here, the sports pack may include a wristband-shaped motion detection sensor as the second environmental sensor 55, the health pack may include a pulse wave sensor, the energy-saving pack may include multiple smart taps 8 (see Figure 3), and the monitoring pack may include multiple motion sensors. Developer 1c can program robot 2 using a visual programming language such as Scratch with the smart kit 67. Then, within the information system S1, it becomes possible to easily create a customized smart system by linking robot 2 and IoT device 56.

[0075] The following explanation will continue using Figures 1, 2, and 3. By using the smart kit 67, it is possible to program the operation of robot 2 using the API linkage described above. In this case, developer 1c can indirectly control home appliances (electrical equipment 7, IoT devices 56) using robot 2. Robot 2 sends control commands for predetermined home appliances to smart remote control hub 5 via network 50, and the smart remote control hub 5 controls the home appliances by sending infrared (IR) signals or control signals based on communication standards such as BLE.

[0076] Of course, the robot 2 may also be equipped with an infrared remote control function (control signal output unit 2n). In this case, the robot 2 performs image recognition processing on the image data captured by the imaging unit 2e (this may also be performed on the server 30), moves the robot 2 to face the controlled device 3 that controls it, and then transmits an infrared signal, thereby eliminating the blind spots of so-called infrared remote controls. Furthermore, by using the smart kit 67, the usage status of the IoT device 56 by the first user 1a can also be transmitted to the second information terminal 6b used by the second user 1b.

[0077] As shown in Figure 8, the app created by developer 1c may be listed and traded on the Smart Kit Market (online trading market 65). Transaction history is managed by blockchain for transactions such as listing and purchasing apps. In addition, a Non-Fungible Token (NFT) is attached to the app to prove that the app is genuine. A user of the program (in this case, first user 1a) can purchase the listed app. When purchasing, first user 1a pays developer 1c points (consideration) used for transactions on the Smart Kit Market. These points may be cryptocurrency. In such transactions, a Web3 mechanism can be utilized, in which the operation of internet content such as data ownership, assurance of reliability, and payment processing is decentralized to the user.

[0078] (Second Embodiment) Figure 10 is an explanatory diagram showing the physical space RS in the second embodiment of the present invention, and Figure 11 is an explanatory diagram showing the image of the virtual space VS. Hereinafter, Figures 10 and 11 will be used in conjunction with Figures 1 to 3 to explain an application example of the API linkage described in the first embodiment. Figure 10 illustrates a state in which a first user 1a is present in one room, and two robots 2, an IoT device 56, a second environmental sensor 55, an electrical device 7, and a smart tap 8 are arranged in the same room, but of course the second embodiment is not limited to this example. For example, there may be multiple IoT devices 56, second environmental sensors 55, electrical devices 7, and smart taps 8 in one room, and the number of robots 2 may be arbitrary. Furthermore, at least one robot 2 may autonomously move between multiple rooms in the residence HS.

[0079] When the first cloud service platform 51 shown in Figure 1 and an IoT platform provided by, for example, Tuya (corresponding to the second cloud service platform 52) are linked via API, as shown in Figure 10, in a residential HS, it becomes possible to understand the status of each device (smart remote control hub 5, smart tap 8, second environmental sensor 55, IoT device 56), the lifestyle of the first user 1a, the environment of each room, and energy consumption status based on the floor plan of the residential HS. When robot 2 is added to the information system S1, robot 2 can converse with the first user 1a and understand information about the first user 1a, their preferred environment, and a more detailed lifestyle. Furthermore, since it can also understand the energy consumption status, such as electricity, based on the output of smart tap 8, it becomes possible to use robot 2 to eliminate wasteful energy consumption and provide advice on energy saving.

[0080] In the information system S1, robot 2 may control devices (electrical equipment 7, IoT devices 56) within the residence HS based on a conversation with first user 1a. For example, robot 2 may detect that the television is ON based on data output by smart tap 8, and further detect the gaze of first user 1a based on image data captured by imaging unit 2e. If robot 2 determines that first user 1a has not been watching the television for a predetermined period of time, it may say to first user 1a, "It looks like you haven't been watching the television, should I turn it off?" In this case, if first user 1a gives an affirmative response such as "yes" or "uh-huh," robot 2 controls the television to turn OFF.

[0081] Furthermore, if the temperature in the room is lower than a predetermined value based on the output of the temperature and humidity sensor (second environmental sensor 55) included in the smart remote control hub 5, and the system determines that the air conditioner is OFF based on the output of the smart tap 8, it may say "Are you cold?", and if the first user 1a gives an affirmative response, it may respond to the first user 1a by saying "I'll warm up the room" and then turn on the air conditioner in heating mode. In this way, by using the information system S1 according to the present invention, it becomes possible to promote the smartification of residential HS.

[0082] Based on information obtained in advance regarding the floor plan of the residence HS and objects such as furniture and home appliances placed in each room, the server control unit 30a digitally constructs the physical space RS of the residence HS as a corresponding virtual space VS, as shown in Figure 11. Figure 11 shows a state in which a specific room of the residence HS is reproduced as the virtual space VS, but of course, the entire residence HS may be reproduced as a two-dimensional virtual space VS, for example. The rendering of the virtual space VS is performed on the information terminal 6 (terminal control unit 6y) used by user 1 (first user 1a or second user 1b). Of course, the rendering process may also be performed by the server control unit 30a. In other words, both the server 30 and the information terminal 6 can function as control units.

[0083] A second environmental sensor 55 (e.g., a motion sensor, a temperature and humidity sensor) and a smart tap 8 placed in the physical space RS (real world) may detect some of the conditions in the physical space RS and construct a virtual space VS by referring to this information. Specifically, for example, if a specific electrical device 7 is connected to the smart tap 8 and the attributes of the electrical device 7 (air conditioner, television, lighting, etc.) are registered, an object corresponding to the attributes of the electrical device 7 will be added to the virtual space VS. Furthermore, the ON / OFF status of the electrical device 7, such as lighting, will be reflected in the virtual space VS.

[0084] In the second embodiment, a mobile body (robot 2) is introduced into the physical space RS. Figure 11 shows a state in which the interior furnishings such as furniture, the avatar of the first user 1a (first avatar 71), and the avatars of the two robots 2 (robot avatars 72) are reproduced in the virtual space VS. There are no particular restrictions on the viewpoint when rendering the virtual space VS, but it is preferable to render it using the imaging unit 2e of robot 2 as the viewpoint. By rendering the virtual space VS with the imaging unit 2e as the viewpoint, when at least a part of the virtual space VS is switched to a real image and displayed as described later, the position of the first user 1a or the position of objects etc. displayed on the display unit 6x (see Figure 1) does not change. Therefore, the second user 1b, who is watching over the first user 1a, can reliably track the first user 1a and further reliably grasp the area in the room where an anomaly has occurred.

[0085] If a new electrical device 7 is added to the physical space RS, and a corresponding new object is added to the virtual space VS, the robot 2 may move to the room where the electrical device 7 was added and, for example, take an image of the electrical device 7. This makes it possible to collect more detailed data about the shape of the electrical device 7 and to render the object more appropriately.

[0086] When robot 2, which moves in the physical space RS, is reproduced as an avatar in the virtual space VS, the robot avatar 72 moves within the virtual space VS in accordance with its movement in the physical space RS. Pre-created mapping data (house map) is used for movement. By autonomously moving robot 2, robot 2 can collect information about the physical space RS (residence HS or rooms). The collected information (for example, changes in the position of furniture and electrical equipment 7) is used to reconstruct the virtual space VS. The mapping data may also be updated based on the output of, for example, the rangefinder 2f (LiDAR) acquired during movement. Based on the updated mapping data, the virtual space VS can be reconstructed.

[0087] Furthermore, when robot 2 is not working for the first user 1a (for example, when it is not conversing with the first user 1a), it may autonomously move around within the residence HS. In addition, when robot 2 is moving around within the residence, if it finds the first user 1a based on the image acquired by the imaging unit 2e, it may initiate a conversation.

[0088] The first user 1a can use the first information terminal 6a to understand the state of the virtual space VS and, for example, confirm the location of robot 2 in the residence HS. Then, for example, the first user 1a can instruct robot 2 to move closer to them. Upon receiving this instruction, robot 2 moves to the vicinity of the first user 1a and, for example, can begin a conversation. When the first user 1a communicates with robot 2 using the information system S1, the virtual space VS reproduced on the first information terminal 6a reproduces the avatar of the first user 1a (first avatar 71) and the avatar of robot 2 (robot avatar 72).

[0089] In the information system S1 of the second embodiment, the robot 2 can be shared not only among the first user 1a but also among third parties such as family members, relatives, caregivers, and doctors (second user 1b). Here, the first user 1a is the person being monitored, and the second user 1b is the person who monitors the first user 1a using the information system S1. Hereafter, the person who operates the robot 2 may be referred to as the "operator," and the right to operate the robot 2 may be referred to as the "operating right." Here, "operation" includes at least one of the following: displacement / movement of the robot 2, or speech by the robot 2 (the robot 2 reproducing as speech a voice spoken or a text transmitted by the second user 1b). When the second user 1b is the operator, there may be multiple second users 1b. That is, the operating right of the robot 2 can be shared among multiple second users 1b. This corresponds to a scenario in which multiple second users 1b share one robot 2, and one of the second users 1b with the operating right communicates with the first user 1a.

[0090] At this time, multiple second users 1b are able to operate robot 2 at different times. That is, second users 1b share robot 2 in a time-share manner and operate it in the physical space RS. This allows second users 1b to check the status of the first avatar 71 (first user 1a) via the virtual space VS reproduced on the second information terminal 6b. Specifically, second users 1b can understand (monitor) the status of first user 1a, which is reproduced as an avatar based on images taken by the imaging unit 2e (see Figure 3).

[0091] Furthermore, the first user 1a and the second user 1b can communicate through conversation and other means via avatars. In this virtual space VS, as shown in Figure 11, the avatar of robot 2 (robot avatar 72) is switched to the operator's avatar (the avatar of second user 1b (second avatar 73)). In other words, the situation in the physical space RS where the first user 1a and robot 2 are facing each other is reproduced in the virtual space VS as the first avatar 71 (first user 1a) and the second avatar 73 (second user 1b) are facing each other and communicating.

[0092] Furthermore, if the second user 1b who has control rights is replaced by another second user 1b (for example, from a family member to a doctor), the second avatar 73 corresponding to the second user 1b in the virtual space VS may be switched to an appearance corresponding to each person who has control rights (i.e., the appearance of the second avatar 73 may be switched according to the role, such as doctor or caregiver). Also, if multiple second users 1b speak simultaneously, multiple avatars corresponding to each speaker may be displayed. This makes it easy for the first user 1a to recognize who they are communicating with.

[0093] Thus, in the second embodiment, the information system S1 reproduces a first avatar 71 corresponding to the first user 1a who is the subject of monitoring, and a second avatar 73 corresponding to the second user 1b who monitors the first user 1a, in the virtual space VS. There are multiple second users 1b, and the control unit (terminal control unit 6y of the information terminal 6) changes the appearance of the second avatar 73 according to each of the different second users 1b. As a result, the first user 1a who is the subject of monitoring can easily recognize who the person they are communicating with is.

[0094] The second user 1b communicates with the first user 1a via the virtual space VS. In the virtual space VS, the first user 1a is represented as a first avatar 71 corresponding to the first user 1a. On the other hand, in the physical space RS, the first user 1a communicates with the second user 1b via the robot 2. At this time, if the first user 1a makes a predetermined input via the sound detection unit 2g, SELECT / ENTER switch 2k, etc. of the robot 2, or if the fall of the first user 1a is detected by the fall detection unit 2s (see Figure 3) provided on the robot 2, an image of the physical space RS (hereinafter sometimes referred to as "real image") may be provided to the second user 1b. That is, the fall detection unit 2s functions as one of the detection units. The robot 2 is equipped with a movement mechanism 16a (see Figure 3) as described above. Thus, the detection unit is provided on a mobile body (robot 2) configured to move in the physical space RS. This makes it possible to monitor the first user 1a in any room or location in the residence HS.

[0095] When the second user 1b obtains the right to operate robot 2, it is preferable that a prescribed authentication procedure is performed. As an authentication procedure, biometric authentication such as facial recognition by the second information terminal 6b can be used. At this time, the appearance attributes of the avatar, such as its skin, may be changed according to the operator (i.e., the ID of the authenticated person). After the authentication procedure, the operator who has obtained the right to operate robot 2 can move the robot avatar 72 in the virtual space VS. Consequently, robot 2 moves in the physical space RS.

[0096] When the second user 1b, who has acquired control of robot 2, wishes to communicate with the first user 1a on a daily basis, the second user 1b can use the virtual space VS reproduced on the second information terminal 6b (see Figure 1). That is, in the virtual space VS, the second user 1b, who has acquired control of robot 2, communicates with the first user 1a's first avatar 71 via the second avatar 73. In the physical space RS, the second user 1b, who has acquired control of robot 2, communicates with the first user 1a via robot 2.

[0097] On the other hand, if any abnormality occurs with the first user 1a (for example, if the fall detection unit 2s (see Figure 3) detects that the first user 1a has fallen), the actual image acquired by the imaging unit 2e (see Figure 3) mounted on the robot 2 is transmitted to the second information terminal 6b, and the second user 1b can understand the abnormality via the second information terminal 6b. In other words, in the second embodiment, the system is configured to allow switching between the first avatar 71 corresponding to the first user 1a and the actual image of the first user 1a, depending on the situation.

[0098] It is also possible for the first user 1a to operate the robot 2. In this case, the first user 1a becomes the operator. The first user 1a operates the robot 2 avatar (robot avatar 72) in the virtual space VS via the first information terminal 6a (see Figure 1). For example, the robot avatar 72 can be moved to a specific room, and specific electrical equipment 7 can be controlled using the infrared remote control function, and further security and monitoring of the residence HS can be performed. For security and monitoring of the residence HS, image data from the imaging unit 2e mounted on the robot 2 is used. Based on the image data captured by the imaging unit 2e, the movement of objects in the image may be detected. For example, the first control unit 10 (see Figure 3) may capture multiple images in a specific room at predetermined time intervals (e.g., every few seconds), use the first captured image as background data, and detect movement by acquiring background differences between it and subsequently captured images. The control unit (here, the imaging unit 2e and the first control unit 10) that detects the movement of objects in the image based on background differences also functions as a detection unit.

[0099] If motion is detected in an object in the image using the background subtraction method, the first control unit 10 may transmit to the information terminal 6 a message indicating that an anomaly has been detected, along with an image captured by the imaging unit 2e at the time the anomaly was determined to have occurred. The information terminal 6 (terminal control unit 6y) that receives this information may switch between displaying the video of the virtual space VS and the actual image captured by the imaging unit 2e on the display unit 6x (see Figure 1), display the actual image together with the video of the virtual space VS, or overlay the video of the virtual space VS and the actual image. The video of the virtual space VS and the actual image may be displayed as, for example, wallpaper. This allows the first user 1a or the second user 1b to understand the situation remotely (for example, from a different room in the case of the first user 1a), for example, if an intruder enters the residence HS. The robot 2 may also be equipped with a lighting device (not shown), and the lighting device may output near-infrared light. In this case, it is preferable that the imaging unit 2e is composed of an image sensor that is sensitive to near-infrared light.

[0100] Thus, in the information system S1 of the second embodiment, when the control unit (first control unit 10, terminal control unit 6y) detects the movement of an object in an image based on the image captured by the imaging unit 2e, it displays an image of the physical space RS (real image) on the display unit 6x in place of, or together with, the image of the virtual space VS. As a result, if an abnormality occurs in the residence HS, the user 1 can understand the situation based on the real image.

[0101] Furthermore, as mentioned above, robot 2 is equipped with an inertial sensor 19 (accelerometer). The inertial sensor 19 also functions as one of the detection units. Even if the inertial sensor 19 detects an acceleration greater than a predetermined value (i.e., robot 2 is subjected to a large impact), a real image may be displayed instead of, or in conjunction with, the depiction of the virtual space VS. Additionally, a sound detection unit 2g (see Figure 3), such as a microphone, provided on robot 2 also functions as one of the detection units. Even if the sound detection unit 2g detects a sound pressure greater than a predetermined value, or if it detects a sound containing a specific frequency component (for example, the sound component of glass breaking), a real image may be displayed instead of, or in conjunction with, the depiction of the virtual space VS.

[0102] Thus, the information system of this embodiment further includes a sound detection unit 2g, and the control unit (first control unit 10, terminal control unit 6y) may, when the sound pressure detected by the sound detection unit 2g exceeds a predetermined range, display an image of the physical space RS on the display unit 6x in place of, or together with, the image of the virtual space VS. This makes it possible for user 1 to understand the situation based on the actual image if an abnormality occurs in the residence HS.

[0103] Furthermore, the information system S1 of this embodiment includes a control unit (terminal control unit 6y) that renders images of the virtual space VS, a detection unit (at least one of the fall detection unit 2s, imaging unit 2e, inertial sensor 19, and sound detection unit 2g) that detects events in the physical space RS (events that occur in real time or suddenly in the physical space RS), an imaging unit 2e that captures images of the physical space RS, and a display unit 6x that displays at least one of the images of the virtual space VS and the images of the physical space RS. When a predetermined event is detected by the detection unit, the control unit (terminal control unit 6y) displays the image of the physical space RS on the display unit 6x in place of, or together with, the image of the virtual space VS. In this way, by directly reflecting information from the physical space RS (in this case, actual images) in at least a part of the virtual space VS based on events that occur in the physical space RS, it is possible to improve the convenience of user 1 (first user 1a), while considering the privacy of first user 1a during normal times and ensuring the safety and security of first user 1a in the event of an abnormality.

[0104] Of course, the security and monitoring of the residence HS may be performed by a second user 1b other than the first user 1a. In this case, the control unit (here, the terminal control unit 6y of the second information terminal 6b) may, as a preliminary step before displaying the actual image on the display unit 6x of the information terminal 6, apply processing such as mosaic processing or noise addition to the actual image (image in physical space RS) based on the image captured by the imaging unit 2e of the robot 2, or apply processing such as changing the color tone to red to the image in virtual space VS. Alternatively, a specific pattern or avatar may be displayed instead of the actual image. This protects the privacy of the first user 1a. Furthermore, the second user 1b may, for example, contact the first user 1a by phone to check on their well-being, and if there is no response from the first user 1a, they may switch the processed image or specific pattern to the actual image. In other words, when there is a high possibility that an abnormal situation has occurred, ensuring safety takes priority, and for example, the second user 1b may be able to switch to the actual image using the second information terminal 6b.

[0105] Thus, in the information system S1 of the second embodiment, if the control unit (terminal control unit 6y) detects the movement of an object in an image based on the image captured by the imaging unit 2e, it may, before displaying the image in the physical space RS, apply a predetermined processing to the image of the virtual space VS corresponding to the area in the physical space RS where movement was detected and then display it. Alternatively, if the control unit detects the movement of an object in an image based on the image captured by the imaging unit 2e, it may, before displaying the image in the physical space RS, display an image that has been processed with a predetermined processing to the image captured by the imaging unit 2e. This prevents the indiscriminate display of the actual image of the residence HS and protects the privacy of the first user 1a.

[0106] Although specific embodiments of the information system S1 according to the present invention have been shown above, these are merely examples, and the present invention is not limited to these embodiments. For example, the division of processing performed by the first control unit 10, the server control unit 30a, and the terminal control unit 6y may be changed as appropriate. That is, the control unit can be composed of one or more of the first control unit 10, the server control unit 30a, and the terminal control unit 6y. Furthermore, when the control unit is composed of a combination of the first control unit 10, the server control unit 30a, and the terminal control unit 6y, the configuration of the combination may be changed as appropriate depending on the processing. [Industrial applicability]

[0107] The information system S1 according to the present invention reflects information from the physical space RS in the virtual space VS based on events that occur in the physical space RS, thereby improving the convenience of user 1 (first user 1a) and ensuring the safety and security of user 1 in their residence HS while respecting their privacy. Therefore, it can be suitably applied to monitoring the elderly and the like. [Explanation of symbols]

[0108] 1 User 1a 1st user 1b 2nd user 2 Robots 2e Imaging Unit 2f Ranging section 2g sound detection unit 2s Fall detection unit 6a First Information Terminal 6b Second Information Terminal 6x display 10 First Control Unit 17. First Environmental Sensor 19. Inertial Sensor 30 servers 30a Server Control Unit 50 Networks 54 Cloud Service Infrastructure 55 Second environmental sensor 56 IoT devices 71 First Avatar 72 Robot Avatars 73 Second Avatar S1 Information Systems HS housing VS Virtual Space RS physical space

Claims

1. A control unit that renders images in a virtual space, A detection unit that detects events in physical space, An imaging unit that captures an image in the aforementioned physical space, A display unit that displays at least one of the video of the virtual space and the image of the physical space, Equipped with, The control unit, An information system characterized in that, when a predetermined event is detected by the detection unit, an image of the physical space is displayed on the display unit in place of, or together with, the image of the virtual space.

2. The detection unit is The information system according to claim 1, characterized in that it is provided on a mobile body configured to be movable in the aforementioned physical space.

3. The control unit, The information system according to claim 1, characterized in that when the movement of an object in an image is detected based on an image captured by the imaging unit, the image of the physical space is displayed on the display unit in place of, or together with, the image of the virtual space.

4. The control unit, The information system according to claim 3, characterized in that, when motion of an object in an image is detected based on an image captured by the imaging unit, a predetermined process is applied to the image of the virtual space corresponding to the area in the physical space where motion was detected, before displaying the image of the physical space.

5. The control unit, The information system according to claim 3, characterized in that, when the movement of an object in an image is detected based on the image captured by the imaging unit, an image obtained by applying a predetermined processing to the image captured by the imaging unit is displayed before the image of the physical space is displayed.

6. Furthermore, it is equipped with a sound detection unit, The information system according to claim 1, characterized in that when the sound pressure detected by the sound detection unit exceeds a predetermined range, the control unit displays an image of the physical space on the display unit in place of, or together with, the image of the virtual space.

7. In the aforementioned virtual space, The first avatar corresponds to the first user who is the target of monitoring, A second avatar corresponding to the second user who monitors the first user, It was recreated, The aforementioned second user is multiple, The information system according to claim 1, characterized in that the control unit changes the appearance of the second avatar according to the second users, who are different from each other.

Images