Notification system

Abstract

This system provides a notification system that can appropriately deliver notifications based on the concentration level of users present in the target space. [Solution] The notification system comprises: a detection device provided in a target space for detecting information within the target space; a microphone provided in the target space for collecting sounds within the target space; a skeleton data generation unit that estimates the skeleton of a user present in the target space and generates skeleton data based on detection data indicating the detection results of the detection device; a first behavior estimation unit that estimates the user's actions based on the skeleton data; an environment estimation unit that estimates the environment of the target space from the sounds; a concentration level estimation unit that estimates the user's level of concentration on work based on the estimation results of the first behavior estimation unit and the estimation results of the environment estimation unit; and a notification unit that provides notifications based on the estimation results of the concentration level estimation unit.

Description

Technical Field

[0001] The present disclosure relates to a notification system.

Background Art

[0002] Patent Document 1 discloses a technique for extracting features of a user's movement from an image of the user captured according to a task performed by the user by image recognition and calculating the user's concentration.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure provides a notification system capable of appropriately performing a notification based on the concentration of a user present in a target space.

Means for Solving the Problems

[0005] The notification system in the present disclosure includes a detection device provided in a target space for detecting information in the target space, a microphone provided in the target space for collecting sound in the target space, a skeleton data generation unit for estimating a skeleton of a user present in the target space based on detection data indicating a detection result of the detection device and generating skeleton data, a first action estimation unit for estimating the action of the user based on the skeleton data, an environment estimation unit for estimating the environment of the target space from the sound, a concentration estimation unit for estimating the concentration of the user with respect to work based on an estimation result of the first action estimation unit and an estimation result of the environment estimation unit, and a notification unit for performing a notification based on an estimation result of the concentration estimation unit.

Effects of the Invention

[0006] The notification system described in this disclosure estimates the concentration level using multiple pieces of information obtained from the target space, thereby enabling accurate estimation of the concentration level of users in the target space. Therefore, it can appropriately deliver notifications based on the concentration level of users present in the target space. [Brief explanation of the drawing]

[0007] [Figure 1] Diagram showing the schematic configuration of the notification system in Embodiment 1. [Figure 2] An explanatory diagram illustrating user sensing in Embodiment 1. [Figure 3] This figure shows an example of the structure of a wireless outlet in Embodiment 1. [Figure 4] This figure shows an example of the configuration of a wireless outlet in Embodiment 1. [Figure 5] Diagram showing the configuration of the main components of the server control system in Embodiment 1. [Figure 6] Flowchart showing the operation of the server and detection unit in Embodiment 1 [Figure 7] Diagram showing the schematic configuration of the notification system in Embodiment 2. [Figure 8] Block diagram showing the main components of the computer control system in Embodiment 2 [Figure 9] Flowchart showing the operation of the computer in Embodiment 2 [Modes for carrying out the invention]

[0008] (Knowledge and other information that formed the basis of this disclosure) At the time the inventors conceived this disclosure, there was a technology, such as Patent Document 1, that extracted the characteristics of a user's movements from an image of the user using image recognition to estimate the degree of concentration on a task. However, this technology has a need to provide notifications based on the degree of concentration. However, the inventors discovered a problem in conventional technologies that use image recognition to extract movements, as they estimate the degree of concentration using only one image of the user obtained from the target space, and therefore may not be able to estimate the degree of concentration with high accuracy. To solve this problem, the subject matter of this disclosure was established. Therefore, this disclosure provides a notification system that can appropriately deliver notifications based on the concentration level of users present in the target space.

[0009] The embodiments will be described in detail below with reference to the drawings. However, some unnecessarily detailed explanations may be omitted. For example, detailed explanations of already well-known matters or redundant explanations of substantially identical configurations may be omitted. The attached drawings and the following description are provided to enable those skilled in the art to fully understand this disclosure and are not intended to limit the subject matter described in the claims.

[0010] (Embodiment 1) Embodiment 1 will be described below with reference to Figures 1 to 6. [1-1. Structure] [1-1-1. Notification System Configuration] Figure 1 shows the configuration of the notification system 1 in Embodiment 1. The notification system 1 of this embodiment detects the state of user P, who is the target of detection, present in a space (for example, a living room) using multiple electrical equipment 2 placed in that space, without using sensors.

[0011] Notification system 1 estimates the degree of user P's concentration on the task (hereinafter referred to as "concentration level").

[0012] In addition, the notification system 1 notifies the user P that they are not concentrating on the work according to the degree of concentration. Furthermore, the notification system 1 prompts the user P to interrupt the work, and by extension, prompts the user P to take a break to improve the degree of concentration.

[0013] The work performed by the user P in the first embodiment is studying. The user P uses the booklet N to perform the work. The booklet N is a book such as a textbook or a notebook.

[0014] The notification system 1 includes a terminal device 6 used by the user P. In FIG. 1, as the terminal device 6, a portable smartphone having a display means is illustrated. The terminal device 6 is not limited to this, and may be a desktop computer, a laptop computer, or other computers having a notification means used by the user P.

[0015] Hereinafter, the space to which the detection by the notification system 1 is applied is referred to as the target space 9. The target space 9 may be an enclosed space surrounded by walls and a ceiling, or may be a space with doors, windows, etc. open. Also, the use of the target space 9 is not limited. For example, the target space 9 may be a living space, a living area, a business space such as an office, a public space, and a space used for other purposes.

[0016] The target space 9 illustrated in FIG. 1 is an indoor space surrounded by a floor 91, a ceiling 92, and four walls 93, 94, 95, and 96.

[0017] The notification system 1 includes a plurality of electrical equipment materials 2 arranged in the target space 9 to which power is supplied, and a detection processing unit. In the notification system 1, among the plurality of electrical equipment materials 2, the state of the user P present in the space is detected by any two electrical equipment materials 2 having a function of transmitting and receiving radio waves (hereinafter also referred to as a radio function).

[0018] In this embodiment, as an example, the electrical equipment materials 2 having a radio function are the outlets 30A and 30B.

[0019] Furthermore, in this embodiment, the detection processing unit is provided in each of the two electrical equipment 2 having wireless functionality, namely outlets 30A and 30B. However, the detection processing unit may be provided in a device other than the electrical equipment 2. For example, the notification system 1 may include a relay device 40 and a server 41 that is communicably connected to outlets 30A and 30B via a network NW such as the Internet, and the server 41 may be configured to have the detection processing unit.

[0020] Electrical equipment 2 includes components, equipment, machinery, and devices that are connected to commercial power and permanently installed within a building.

[0021] For example, the target space 9 shown in Figure 1 is equipped with a lighting device 10, a switch 20, and outlets 30A and 30B. Lighting fixtures 10, switches 20, and outlets 30A and 30B can be installed on the floor 91, ceiling 92, and the four walls 93, 94, 95, and 96 of the target space 9. A distribution board 50 is also installed outside the target space 9. Hereinafter, the surfaces of the floor 91, ceiling 92, and the four walls 93, 94, 95, and 96 that define the target space 9 will be collectively referred to as the walls of the target space 9.

[0022] Power lines 5A, 5B, and 5C are drawn into the target space 9 from the distribution board 50. Power lines 5A, 5B, and 5C are, for example, wires laid concealed within the walls of the target space 9, and are power cables such as VVF (Vinyl insulated Vinyl sheathed Flat-type cable) cable and CV (cross-linked polyethylene insulated vinyl sheath) cable. Hereafter, power lines 5A, 5B, and 5C will be referred to as power line 5 when not distinguished. The distribution board 50 branches off the service drop line 58, which is connected to the commercial power supply, and connects it to power line 5.

[0023] The lighting device 10 is mounted on the ceiling 92. A ceiling socket 112 is fixed to the ceiling 92, and the lighting device 10 is mounted on the ceiling socket 112. The lighting device 10 corresponds to an example of "lighting means".

[0024] The ceiling socket 112 is connected to power line 5A, and the lighting device 10 is connected to power line 5A by being attached to the ceiling socket 112. The ceiling socket 112 may be configured in a form called a lighting rosette.

[0025] The air conditioning unit 8 is mounted on the wall. The air conditioning unit 8 is a so-called air conditioner and corresponds to an example of a "blowing means." The blowing means may be a blowing device such as a fan.

[0026] The switch 20 has a switch body 21 that is connected to power line 5A. The switch 20 connects and disconnects power line 5A between the lighting device 10 and the distribution board 50. When the switch 20 is on, power is supplied to the lighting device 10. When the switch 20 is off, the power supply to the lighting device 10 is cut off, and the lighting device 10 turns off.

[0027] Outlets 30A and 30B are outlets that can be connected to load equipment that operates on commercial power. Outlets can also be called power outlets. For example, a single-phase 100V load is connected to outlets 30A and 30B. Outlet 30A is connected to the distribution board 50 by power line 5B, and outlet 30B is connected to the distribution board 50 by power line 5C.

[0028] The notification system 1 includes, for example, outlets 30A and 30B as two electrical equipment 2 equipped with the function of transmitting and receiving radio waves. In this embodiment, outlets 30A and 30B are located on opposing walls 93 and 95 of the target space 9, respectively. Furthermore, the height H1 from the floor 91 at the installation position of outlet 30A and the height H2 from the floor 91 at the installation position of outlet 30B are different from each other. Hereafter, when outlets 30A and 30B are not distinguished, they will be referred to as outlet 30.

[0029] Each of the outlets 30 is configured to transmit and receive radio waves in a predetermined frequency band to each other, and the state of user P in the target space 9 is detected by analyzing the state of the received radio waves.

[0030] Figure 2 is an explanatory diagram illustrating the sensing of user P by notification system 1. In notification system 1, the radio waves transmitted by either outlet 30 (outlet 30A in the example in Figure 2) propagate within the target space 9, and after being reflected once or multiple times by the walls of the target space 9 or the surfaces of real objects such as the user P present within the target space 9, they reach the other outlet 30 (outlet 30B in the example in Figure 2), or they reach the other outlet 30 directly without undergoing any reflection.

[0031] In other words, the other outlet 30B receives a direct wave DW (dashed line in the figure), which is a radio wave transmitted from one outlet 30A and reaching the other outlet 30B without reflection in the target space 9, and one or more indirect waves IW (dotted lines in the figure), which are radio waves that arrive after one or more reflections in the target space 9. Although Figure 2 is shown in plan view, the radio wave transmitted from outlet 30A can also propagate in the direction of the floor 91 and ceiling 92, generating indirect waves IW.

[0032] Therefore, by analyzing the amplitude and / or phase of the received direct wave DW and indirect wave IW at another outlet 30B, the three-dimensional position of one outlet 30A, which is the source of the radio waves, and the three-dimensional position and three-dimensional surface shape of user P present in the target space 9 can be obtained. The data showing the three-dimensional position and three-dimensional surface shape of user P present in the target space 9 corresponds to an example of "detection data". When outlet 30B transmits radio waves and outlet 30A receives those radio waves, three-dimensional information of the target space 9 can be obtained in outlet 30A in the same manner as described above.

[0033] [1-1-2. Outlet Configuration] In this embodiment, the electrical equipment 2 having wireless functionality, namely outlets 30A and 30B, have the same structure and configuration as an example. However, outlets 30A and 30B may have different structures and / or configurations, as long as they have functional elements similar to those of the detection unit 33 described later. The structure and configuration of outlet 30 will be described below. Figure 3 is a front view showing an example of the structure of the outlet 30. Figure 4 is a diagram showing an example of the configuration of the outlet 30.

[0034] The outlet 30 includes an outlet block 301. The outlet block 301 is a box-shaped enclosure made of insulating material and houses the wiring connected to each of the two terminals 32. A cover 302 with a pair of openings is positioned on the front of the outlet block 301. The terminals 32 are exposed at the back of the openings in the cover 302, and power is supplied to the load device by inserting the plug of the load device into the openings in the cover 302.

[0035] The outlet block 301 has a wire connection section 305 for connecting the power line 5B. The wire connection section 305 is a terminal into which the conductors of the power line 5B can be inserted, and the wire connection section 305 is provided with two wire connection sections 305 corresponding to the two conductors that make up the power line 5B. The terminals 32 of the outlet body 31 are connected to the wire connection section 305.

[0036] The outlet block 301 is fitted into the mounting frame 311. For example, the outlet block 301 is fixed to the mounting frame 311 by engaging a projection on the mounting frame 311 with a groove (not shown) formed on the side of the outlet block 301.

[0037] A rectangular hole is drilled in the wall material to which the outlet 30 is fixed. The support member 312 is placed behind the wall material, such as a board or gypsum board, through this hole. The mounting frame 311 is fixed to the building material by being connected to the support member 312 with the building material in between.

[0038] A microphone 38 is incorporated into the outlet block 301. The microphone 38 receives power by being connected to the wire connection part 305. Alternatively, the microphone 38 may receive power by being connected to a different power connection part than the wire connection part 305. The outlet block 301 incorporates a detection unit 33. The detection unit 33 corresponds to an example of a "detection device". The detection unit 33 is connected to the wire connection part 305 to receive power. An antenna cover 303 is placed on the surface of the outlet block 301 that is exposed to the target space 9, and an array antenna 37 used by the transmitter 35 and receiver 36 of the detection unit 33, which will be described later, is built into a position overlapping the antenna cover 303. The array antenna 37 may be configured as a single element in which multiple antenna elements are arranged in a line or in a grid.

[0039] The antenna cover 303 is a plate-shaped member that covers the surface of the array antenna 37 and is made of a material (for example, resin) that does not obstruct the propagation of radio waves transmitted or received by the array antenna 37.

[0040] The configuration in which the detection unit 33 is installed on the outlet block 301 is just one example. For example, the detection unit 33 may be configured as a separate unit from the outlet block 301, and the wiring inside the outlet block 301 may be connected to the detection unit 33. In this case, the outlet block 301 is provided with an array antenna 37, and the main body of the detection unit 33 can be positioned on the mounting frame 311 away from the cover 302 or in the vicinity of the mounting frame 311.

[0041] Referring to Figure 4, the detection unit 33 includes a control device 34, a transmitter 35, a receiver 36, an array antenna 37, and a microphone 38. The transmitter 35 transmits radio waves of a predetermined frequency band to the target space 9 using the array antenna 37. The receiver 36 receives the radio waves of the predetermined frequency band transmitted to the target space 9 via the array antenna 37. In this embodiment, the radio waves of the predetermined frequency band may be radio waves of the frequency band used in wireless Wi-Fi (registered trademark).

[0042] As described above, the array antenna 37 can be configured as a single element in which multiple antenna elements are arranged in a row or in a grid. The antenna elements may be, for example, a chip antenna, a microstrip antenna, etc. The array antenna 37 allows the transmitter 35 to transmit directional radio waves to the target space 9 in various directions. The receiver 36 can receive radio waves from each antenna element of the array antenna 37.

[0043] The control unit 34 is a computer equipped with a processor 340 such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit) and memory 341.

[0044] Memory 341 is a memory that stores programs and data. Memory 341 stores program 342. Memory 341 has a non-volatile storage area. Memory 341 also has a volatile storage area and constitutes the work area of ​​processor 340. Memory 341 is composed of, for example, ROM (Read Only Memory) and RAM (Random Access Memory).

[0045] The processor 340 includes, as functional units, a detection communication control unit 343, a radio wave state acquisition unit 344, a detection processing unit 345, a radio wave separation unit 346, and an estimation unit 347. The acquisition unit 344 corresponds to an example of a "second acquisition unit". These functional elements of the processor 340 are realized, for example, by the processor 340 of the control device 34 (which is a computer) reading and executing a program 342 stored in memory 341. The program 342 can be stored in any storage medium readable by a computer. Alternatively, all or part of the functional elements of the processor 340 can be configured by hardware, each including one or more electronic circuit components.

[0046] The detection and communication control unit 343 instructs the transmitter 35 to transmit radio waves in one or more directions toward the target space 9 via the array antenna 37. The detection and communication control unit 343 also instructs the receiver 36 to receive radio waves from the array antenna 37. As described above, in this embodiment, the radio waves are radio waves in the frequency band used in wireless WiFi (hereinafter also referred to as WiFi radio waves). The detection communication control unit 343 performs WiFi communication using the transmitter 35 and receiver 36 in accordance with the WiFi communication standard. As a result, the outlet 30 can transmit the processing results calculated by the detection processing unit 345, etc. (described later), to other devices such as the server 41.

[0047] Furthermore, when a predetermined trigger occurs, the detection communication control unit 343 transmits transmission data, including detection data D1 and sound collection data D2, to the server 41 via the relay device 40 connected to the power outlet 30. In this embodiment, the predetermined trigger is the elapsed time (for example, 1 minute) since the transmission of the previous transmission data. The transmission data, including detection data D1 and sound collection data D2, transmitted by the detection communication control unit 343 includes an identification ID that identifies the target space 9. The identification ID is information that uniquely identifies the detection unit 33 installed in the target space 9.

[0048] The radio wave state acquisition unit 344 acquires radio wave state information, which is information about the state of the radio wave, based on the radio wave received by the receiver 36. The radio wave state information may include, for example, information about the amplitude and / or phase of the radio wave received by each antenna element of the array antenna 37. In this embodiment, the radio wave state information may be channel state information (CSI) acquired based on the WiFi radio wave, which is the radio wave received by the receiver 36. The radio wave condition acquisition unit 344 provides the acquired radio wave condition information to the detection processing unit 345.

[0049] Here, the radio wave condition acquisition unit 344 may store each acquired radio wave condition data in a memory 341 or the like in time series. The radio wave condition acquisition unit 344 may calculate a moving average value for each data of the radio wave condition information from the present to a predetermined time in the past, and provide the detection processing unit 345 with a dataset composed of the above moving average values ​​for each data as the current radio wave condition information. This reduces the noise contained in the radio wave condition information and enables more accurate detection of user P.

[0050] The detection processing unit 345 analyzes the radio waves received by the receiver 36 based on the radio wave state information acquired by the radio wave state acquisition unit 344 to estimate the state of user P in the target space 9. Specifically, the detection processing unit 345 analyzes the amplitude and / or phase state of the radio waves received by the receiver 36 based on the radio wave state information to estimate the state of user P in the target space 9. The estimated state of user P detected by the detection processing unit 345 corresponds to the 3D position and 3D surface shape of user P and corresponds to detection data D1. That is, the detection processing unit 345 acquires the detection data D1 of user P.

[0051] The detection processing unit 345 estimates the state of user P in the target space 9 using an estimation model that has been trained on the relationship between the radio wave state information provided by the radio wave state acquisition unit 344 and the state of user P in the target space 9. For example, deep learning techniques are used for the machine learning. This makes it possible to detect the state of user P with higher accuracy and ease using an estimation model that has been trained on the relationship between the information-rich radio wave state information and the detailed state of user P.

[0052] The radio wave separation unit 346 separates and extracts information on the direct wave DW and indirect wave IW contained in the radio waves received by the receiver 36 from the radio wave condition information provided by the radio wave condition acquisition unit 344.

[0053] The estimation unit 347 estimates the straight-line distance to the radio wave transmission source from the radio wave intensity of the direct wave DW, based on the information of the direct wave DW separated by the radio wave separation unit 346. The estimation unit 347 also estimates the incidence angle of the direct wave DW and indirect wave IW to the array antenna 37 from information such as the phase of the radio waves received at each antenna element of the array antenna 37. Then, the estimation unit 347 estimates the three-dimensional position of the radio wave transmission source in the target space 9 and the extent of the target space 9 from the estimated straight-line distance and incidence angle. Here, the extent of the target space 9 may be, for example, the three-dimensional position of each wall surface of the target space 9.

[0054] The sound detection processing unit 348 acquires sound collection data D2 via the microphone 38. Sound collection data D2 is data indicating the sound collected by the microphone 38.

[0055] In the notification system 1 having the above configuration, the state of user P present in the target space 9 within the building can be detected by electrical equipment 2 that can be distributed throughout the building.

[0056] [1-1-3. Server] Figure 5 is a block diagram showing the main components of the control system of server 41. Server 41 comprises a server control device 400 and a server communication device 401. The server control device 400 includes a server processor 410, which is a processor that executes programs such as a CPU and MPU, and a server memory 420. The server control device 400 controls various parts of the server 41 by having the server processor 410 read and execute the server program 421 stored in the server memory 420. By executing the server program 421 stored in the server memory 420, the server processor 410 functions as a server communication control unit 411, a management unit 412, a skeletal data generation unit 413, a first action estimation unit 414, a sound estimation unit 415, a first concentration level estimation unit 416, and a first notification unit 417. The sound estimation unit 415 corresponds to an example of the "second behavior estimation unit" and the "environment estimation unit". The first concentration level estimation unit 416 corresponds to an example of the "concentration level estimation unit". The first notification unit 417 corresponds to an example of the "notification unit".

[0057] The server memory 420 has memory for storing programs executed by the server processor 410 and data processed by the server processor 410. The server memory 420 stores the server program 421 executed by the server processor 410, the management DB (database) 422, and various other data. The server memory 420 has a non-volatile storage area for storing programs and data non-volatilely. The server memory 420 may also have a volatile storage area and constitute a work area for temporarily storing programs executed by the server processor 410 and data to be processed.

[0058] The management DB 422 is a database that stores data received from each target space 9, as well as data generated by the server control device 400, in chronological order from the past to the present. Each data entry stored in the management DB 422 is associated with an identification ID and communication information. The communication information is information for communicating with each part of the target space 9 and the terminal device 6 associated with the target space 9, with address information being an example. The communication information also includes a terminal identification ID that identifies the terminal device 6.

[0059] The server communication device 401 is a communication interface equipped with a wireless circuit, antenna, and other communication-related configurations in accordance with a predetermined communication standard, and communicates with each part of the target space 9 and the terminal device 6 associated with the target space 9 in accordance with the predetermined communication standard.

[0060] As described above, the server processor 410 functions as a server communication control unit 411, a management unit 412, a skeletal data generation unit 413, a first action estimation unit 414, a sound estimation unit 415, a first concentration level estimation unit 416, and a first notification unit 417.

[0061] The server communication control unit 411 communicates with each part of the target space 9 and the terminal device 6 associated with the target space 9 via the server communication device 401.

[0062] The management unit 412 performs operations on the management DB 422, such as storing data, identifying data, and retrieving data.

[0063] [1-1-3-1. Skeletal Data Generation Unit] The skeleton data generation unit 413 estimates the skeleton of user P based on the detected data D1 and generates skeleton data D3. In particular, the skeleton data generation unit 413 estimates that the skeleton data D3 should include information indicating the position of user P's spine BB.

[0064] The skeletal information of the spine BB particularly represents the movement of the user P's torso. In this embodiment, where the task is studying, the following is expected: that is, the movement of the user P's torso is expected to have a strong correlation with the user P's level of concentration.

[0065] Similar results can be expected even when the task is not studying. For example, if the task involves user P sitting in chair 97 and facing desk 98, then the skeletal information of the spine BB is expected to have a strong correlation with user P's level of concentration. Such tasks could include computer-based desk work, drawing, or reading. The task could also include sleeping, training, etc.

[0066] For example, if the spine BB is extended vertically, it is likely that user P is sitting upright in chair 97. For example, if the spine BB is tilted forward from user P's perspective, it is likely that user P is sitting slumped over desk 98 in chair 97.

[0067] The skeletal information in the spine BB corresponds to, for example, the information of the lines connecting key points near the neck and key points near the waist or buttocks. Key points refer to the major points of the skeleton in the estimation of the skeleton. Key points correspond to joints such as the neck, shoulders, and knees, or body parts such as the face and chest.

[0068] Furthermore, the estimated skeletal information for the spine BB may include three or more key points. In this case, the bending movement of the spine BB can be estimated with greater accuracy.

[0069] The skeletal data generation unit 413 generates time-series skeletal data D3 in response to the time-series detection data D1 and stores it in the management DB 422. Skeletal data D3 is data that shows the time-series changes in the three-dimensional position of each part of the user P's skeleton in the target space 9.

[0070] The skeleton data generation unit 413 generates the skeleton data D3 of user P in the target space 9 by using an estimation model that has been machine-learned to determine the relationship between the detection data D1 provided by the detection unit 33 and the state of user P in the target space 9. Machine learning can be performed using methods such as deep learning. This makes it possible to detect the skeleton as the state of user P with higher accuracy and ease by using an estimation model that has been machine-learned to determine the relationship between the information-rich detection data and the skeleton as the detailed state of user P.

[0071] [1-1-3-2. 1st behavior estimation section] The first behavior estimation unit 414 estimates the behavior of user P present in the target space 9. Based on the skeletal data D3, the first behavior estimation unit 414 generates first behavior estimation data D4.

[0072] The first behavior estimation data D4 is data that shows the estimated probability (e.g., percentage) that user P is performing a corresponding action for each type of user P action.

[0073] The types of behavior include, for example, User P sleeping, studying, holding a pen, deep in thought, stretching, relaxing, etc. The types of behavior may also include the types of User P's posture. The first behavior estimation unit 414 corresponds to an example of a "posture estimation unit".

[0074] The type of action is predetermined according to the task. For example, in this embodiment the task is studying, but if the task is desk work, the type of action corresponding to desk work (for example, answering the phone) is predetermined.

[0075] The first behavior estimation unit 414 estimates the type of behavior of user P present in the target space 9 using an estimation model that has been machine-learned to determine the relationship between the skeletal data D3 provided by the skeletal data generation unit 413 and the first behavior estimation data D4 (probability for each type of behavior). For example, as shown in Figure 5, the first behavior estimation unit 414 estimates, based on the skeletal data D3, that there is a 3% probability of sleeping and a 78% probability of studying as types of behavior. The first behavior estimation unit 414 uses an estimation model to calculate the probability for each type of behavior.

[0076] [1-1-3-3.Sound estimation section] The sound estimation unit 415 estimates the environment within the target space 9. It also estimates the actions of user P present within the target space 9. The sound estimation unit 415 generates sound estimation data D5 based on the sound collection data D2 provided by the microphone 38. The sound collection data D2 may include the direction of the voice, and may also include data showing the time-series changes in the voice waveform, separated by the direction of the voice.

[0077] The sound estimation data D5 is data that shows the estimated probability (e.g., percentage) of a sound being of a specific type for each type of sound. Sound types can be broadly categorized into two types: sound-related actions and the environment of the target space.

[0078] Examples of sound-related behaviors include user P snoring, fidgeting, or talking. The types of sound-related behaviors are predetermined.

[0079] The types of environments in target space 9 include the presence of noise, the presence of calming music, and the presence of loud music. The types of environments in target space 9 are predetermined.

[0080] The sound estimation unit 415 generates the sound estimation data D5 of user P present in the target space 9 by using an estimation model that has been machine-learned to determine the relationship between the sound collection data D2 provided by the microphone 38 and the sound estimation data D5 (probability for each type of sound).

[0081] For example, as shown in Figure 5, the sound estimation unit 415 estimates, based on the collected sound data D2, that there is a 98% probability that the person is snoring, an 8% probability that they are having a conversation, and a 9% probability that noise is occurring in the target space 9. The sound estimation unit 415 then uses an estimation model to calculate the probability for each type of sound.

[0082] [1-1-3-4. Concentration degree estimation section] Figure 5 illustrates the overview of the data flow for skeletal data D3, first behavior estimation data D4, and sound estimation data D5. Each of these data, along with the detection data D1 and sound collection data D2 received from the target space 9, are stored in the management DB 422.

[0083] The skeletal data D3 is generated by the skeletal data generation unit 413 based on the detection data D1 transmitted from the detection unit 33 of the target space 9. The skeletal data D3 is provided to the first action estimation unit 414.

[0084] Furthermore, the sound estimation data D5 is generated by the sound estimation unit 415 based on the sound collection data D2 transmitted from the detection unit 33 provided in the target space 9.

[0085] The first behavior estimation data D4 generated by the first behavior estimation unit 414 and the sound estimation data D5 generated by the sound estimation unit 415 are provided to the first concentration level estimation unit 416.

[0086] The first concentration level estimation unit 416 estimates the user P's level of concentration on the task based on the first behavior estimation data D4 and the sound estimation data D5.

[0087] The first concentration level estimation unit 416 estimates the concentration level of user P present in the target space 9 using the first behavior estimation data D4 provided by the first behavior estimation unit 414, the sound estimation data D5 provided by the sound estimation unit 415, and an estimation model that has been machine-learned to determine the relationship between the concentration level of user P and first behavior estimation data D4 provided by the first behavior estimation unit 414 and sound estimation data D5 provided by the sound estimation unit 415.

[0088] The first concentration level estimation unit 416 may also estimate the concentration level of user P based on a rule base that associates the types of first behavior estimation data D4 and sound estimation data D5 with the concentration level of user P.

[0089] In this embodiment, the first concentration estimation unit 416 calculates the concentration level as a percentage. If the percentage is below a predetermined threshold, the first concentration estimation unit 416 determines that user P is not concentrating on the work.

[0090] For example, suppose the first behavior estimation data D4 indicates that there is a 3% probability that the user is sleeping and a 78% probability that they are studying. Furthermore, suppose the sound estimation data D5 indicates that there is a 98% probability that user P is snoring and that there is a 9% probability that noise is present in the environment of the target space 9. In such a case, the level of concentration is likely to be low. In such a case, if the first concentration level estimation unit 416 calculates the level of concentration as a percentage below a predetermined threshold, it determines that user P is not concentrating on the task. The above example corresponds to a situation where user P is sleeping in a low-noise target space 9, sitting upright with their torso upright.

[0091] In this way, the first concentration level estimation unit 416 estimates the user P's level of concentration on the task based on the user P's behavior, the sounds the user P makes, and the environment within the target space 9. This improves the accuracy of the concentration level estimation.

[0092] [1-1-3-5. First Notification Department] The first notification unit 417 issues a notification based on the estimation result of the first concentration estimation unit 416. In this embodiment, the first notification unit 417 notifies user P that user P should interrupt their work if user P is not concentrating on their work. More specifically, the first notification unit 417 transmits information to the terminal device 6 used by user P, including a message urging user P to interrupt their work.

[0093] [1-2. Operation] Figure 6 is a flowchart showing the operation of the server 41 and the detection unit 33. Flowchart FA shows the operation of server 41, and flowchart FB shows the operation of detection unit 33. The flowchart shown in Figure 6 assumes that the task is studying.

[0094] As shown in the flowchart FB, the detection communication control unit 343 of the detection unit 33 determines whether a trigger has occurred to send transmission data, including detection data D1 and sound collection data D2, to the server 41 (step SB1). If the trigger occurs (step SB1: YES), the detection communication control unit 343 sends the transmission data to the server 41 (step SB2). After step SB2 is completed, and the detection communication control unit 343 determines that the trigger has not occurred (step SB1: NO), the detection unit 33 returns to step SB1.

[0095] As shown in flowchart FA, the server communication control unit 411 determines whether or not it has received new detection data D1 and sound collection data D2 (step SA1). When the server communication control unit 411 receives new detection data D1 and sound collection data D2, that is, when it receives new transmission data (step SA1: YES), the management unit 412 stores the new detection data D1 and sound collection data D2 in the management DB 422 (step SA2).

[0096] Next, the skeleton data generation unit 413 generates skeleton data D3 based on the detection data D1 received in step SA1 and stores it in the management DB 422 (step SA3). In step SA3, the skeleton data generation unit 413 may delete the detection data D1 stored in step SA2 from the management DB 422.

[0097] Next, the first behavior estimation unit 414 generates first behavior estimation data D4 based on the skeletal data D3 and stores it in the management DB 422 (step SA4).

[0098] Next, the sound estimation unit 415 generates sound estimation data D5 based on the sound collection data D2 and stores it in the management DB 422 (step SA5). In step SA5, the sound estimation unit 415 may delete the sound collection data D2 stored in step SA2 from the management DB 422.

[0099] Next, the first concentration level estimation unit 416 estimates the concentration level of user P based on the first behavior estimation data D4 and sound estimation data D5, and stores the estimation result in the management DB 422 (step SA6).

[0100] Next, the first concentration estimation unit 416 determines whether or not user P is concentrating on the work (step SA7). If the first concentration estimation unit 416 determines that user P is not concentrating on the work (step SA8: YES), the first notification unit 417 sends information to the terminal device 6 used by user P, including a request for user P to interrupt the work (step SA8). After step SA8 is completed, and if the first concentration estimation unit 416 determines that user P is concentrating on the task (step SA8: NO), the server 41 returns to step SA1.

[0101] [1-3. Effects] As described above, the notification system 1 includes a detection unit 33 provided in the target space 9 to detect information within the target space 9, a microphone 38 provided in the target space 9 to collect sounds within the target space 9, a skeleton data generation unit 413 that estimates the skeleton of user P present in the target space 9 based on detection data D1 indicating the detection result of the detection unit 33 and generates skeleton data D3, a first action estimation unit 414 that estimates user P's actions based on the skeleton data D3, a sound estimation unit 415 that estimates the environment of the target space 9 from sounds, a first concentration level estimation unit 416 that estimates user P's level of concentration on work based on the estimation results of the first action estimation unit 414 and the estimation results of the sound estimation unit 415, and a first notification unit 417 that provides notifications based on the estimation results of the first concentration level estimation unit 416. According to this method, the concentration level can be estimated using multiple pieces of information obtained from the target space 9, thus enabling an accurate estimation of the concentration level of user P in the target space 9. Therefore, notifications based on the concentration level of user P present in the target space 9 can be appropriately delivered.

[0102] Furthermore, the first concentration level estimation unit 416 determines whether or not user P is concentrating on the work based on the concentration level, and the first notification unit 417 notifies user P that they should interrupt their work if user P is not concentrating on the work. According to this, if user P is not focused on the task, user P will be encouraged to interrupt the task. This increases the likelihood that user P will be able to maintain focus on the task after taking a break.

[0103] Furthermore, the notification system 1 includes a sound estimation unit 415 that estimates the user P's behavior from sound, and the first concentration level estimation unit 416 further estimates the concentration level based on the estimation results of the sound estimation unit 415. According to this, the accuracy of the concentration level estimation is improved because the concentration level is estimated based on the user P's behavior, which is inferred from information other than the skeletal data D3.

[0104] Furthermore, skeletal data D3 includes information indicating the position of user P's spine BB. According to this, the information indicating the position of the spine BB improves the accuracy of estimating user P's behavior. Therefore, the accuracy of estimating the level of concentration improves.

[0105] Furthermore, the detection unit 33 includes a transmitter 35 that transmits radio waves to the target space 9 and a receiver 36 that receives radio waves. It also includes a radio wave state acquisition unit 344 that acquires radio wave state information, which is information about the state of the radio waves, based on the received radio waves, and an estimation unit 347 that analyzes the received radio waves based on the radio wave state information and estimates the state of user P present in the target space 9. The detection data D1 includes the state of user P based on the radio waves, and the skeleton data generation unit 413 generates skeleton data D3 based on the state of user P included in the detection data D1. According to this method, detection data D1 can be obtained wirelessly. Furthermore, since the skeletal data D3 obtained wirelessly does not contain any information other than that of user P, personal information can be protected even in the estimation of the skeleton.

[0106] Furthermore, the target space 9 is provided with multiple electrical equipment 2 to which power is supplied, and a detection unit 33 is provided on at least one of the multiple electrical equipment 2. According to this, detection data D1 can be obtained by utilizing the electrical equipment 2 placed in the target space 9. Therefore, the concentration level can be estimated without having to place dedicated components for obtaining detection data D1 in the target space 9.

[0107] (Embodiment 2) [2-1. Structure] Figure 7 shows a schematic configuration of the notification system 1B in Embodiment 2. In Figure 7, elements that are the same as those shown in Figure 1 are indicated by the same reference numerals as those shown in Figure 1, and the explanations of Figure 1 and related Figures 2 to 6 described above are used with reference.

[0108] The notification system 1B in Embodiment 2 includes a computer 1000 used by user P. The notification system 1B in Embodiment 2 differs in that, instead of the server 41 in the notification system 1 in Embodiment 1, the computer 1000 used by user P performs the concentration level estimation.

[0109] In Embodiment 2, User P's work is desk work. User P performs the work using computer 1000 or another computer.

[0110] The reclining chair 97B provided by the notification system 1B is a chair whose backrest can be tilted backward. The reclining chair 97B allows user P to lean their torso backward.

[0111] The notification system 1B includes a vital sensor 7 worn by user P. The vital sensor 7 is a detection device capable of measuring user P's vital signs. The vital sensor 7 is capable of measuring at least heart rate and body temperature as vital signs. The vital sensor 7 is a watch-type detection device and is worn on user P's wrist. The vital sensor 7 is a so-called smartwatch.

[0112] The vital sensor 7 is equipped with a communication device that can communicate with the relay device 40 using a wireless standard. The vital sensor 7 transmits vital data D11, which includes the user P's heart rate, heart rate information linked to the detection time, and the user P's body temperature information linked to the detection time, to the computer 1000 at a predetermined interval (for example, 5 seconds).

[0113] The vital sensor 7 may communicate with the computer 1000 without going through the relay device 40 by using short-range wireless communication such as Bluetooth®. Furthermore, the vital sensor 7 may be a detection device that communicates with the computer 1000 using a wired standard.

[0114] In Embodiment 2, the notification system 1B includes, as an example of electrical equipment 2, a lighting device 10, a switch 20, and outlets 30A and 30B.

[0115] Furthermore, the notification system 1B in Embodiment 2 includes a visible light camera 1004 as an example of a detection device. The visible light camera 1004 includes a communication device capable of communicating with the relay device 40 using a wired or wireless standard. This communication device is a communication interface equipped with a wireless circuit, antenna, and other communication-related configurations in accordance with a predetermined communication standard, and communicates with each part of the notification system 1B via the relay device 40 in accordance with the predetermined communication standard.

[0116] The visible light camera 1004 may communicate with the computer 1000 without going through the relay device 40 by using short-range wireless communication such as Bluetooth.

[0117] A visible light camera 1004A is installed in outlet 30A. The visible light camera 1004A is connected to power line 5C.

[0118] A visible light camera 1004B is installed in outlet 30B. The visible light camera 1004B is connected to power line 5B.

[0119] A visible light camera 1004C is provided in switch 20. The visible light camera 1004C is connected to power line 5A.

[0120] The lighting device 10 is equipped with a visible light camera 1004D. The visible light camera 1004D is connected to power line 5A via a ceiling socket 112. Each visible light camera 1004 may be connected to a separate power line.

[0121] The image data captured by each visible light camera 1004 is the detection data D1 in this embodiment. The detection data D1 is transmitted from each visible light camera 1004 to the computer 1000 at a predetermined interval (for example, every 5 seconds).

[0122] Figure 8 is a block diagram showing the main components of the control system of computer 1000. In Figure 8, elements that are the same as those shown in Figure 5 are indicated using the same reference numerals as those shown in Figure 5, and the explanation of Figure 5 described above is used with reference. The computer 1000 includes a second control unit 1010, a second communication device 1001, a display 1002, and a speaker 1003.

[0123] The second control device 1010 includes a second processor 1110, which is a processor that executes programs such as a CPU or MPU, and a second memory 1120. The second control device 1010 controls various parts of the computer 1000 by having the second processor 1110 read and execute the second program 1121 stored in the second memory 1120. By executing the second program 1121 stored in the second memory 1120, the second processor 1110 functions as a display control unit 1111, a second communication control unit 1112, a sensor acquisition unit 1113, a skeletal data generation unit 413, a posture estimation unit 1114, a second action estimation unit 1115, an environment estimation unit 1116, a second concentration level estimation unit 1117, and a second notification unit 1118. The second concentration estimation unit 1117 corresponds to an example of a "concentration estimation unit," and the second notification unit 1118 corresponds to an example of a "notification unit." The sensor acquisition unit 1113 corresponds to an example of a "first acquisition unit."

[0124] The second memory 1120 has memory for storing programs executed by the second processor 1110 and data processed by the second processor 1110. The second memory 1120 stores the second program 1121 executed by the second processor 1110 and various other data. The second memory 1120 has a non-volatile storage area for storing programs and data non-volatilely. The second memory 1120 may also have a volatile storage area and constitute a work area for temporarily storing programs executed by the second processor 1110 and data to be processed.

[0125] The display control unit 1111 controls the display 1002 of the computer 1000 and displays a predetermined screen.

[0126] The second communication control unit 1112 communicates with the various components of the target space 9, the computer 1000, and the server 41B via the second communication device 1001.

[0127] The sensor acquisition unit 1113 acquires vital data D11 from the vital sensor 7.

[0128] The posture estimation unit 1114 estimates the user P's posture based on the skeletal data D3. Specifically, the posture estimation unit 1114 generates posture estimation data D12 based on the skeletal data D3.

[0129] The posture estimation data D12 shows the estimated probability (e.g., percentage) that user P is in a particular posture type for each type of user P.

[0130] Types of posture include, for example, user P having their back straight, lying face down, sitting cross-legged with legs crossed, sitting on the edge of a chair with legs down, sitting with legs hugged to their chest, leaning forward, torso tilted backward, lying on their back, lying on their stomach, etc. The types of posture are predetermined.

[0131] The posture estimation unit 1114 estimates the type of posture of user P present in the target space 9 using an estimation model that has been machine-learned to determine the relationship between the skeletal data D3 provided by the skeletal data generation unit 413 and the posture estimation data D12 (probability for each type of posture).

[0132] For example, as shown in Figure 8, the posture estimation unit 1114 estimates, based on the skeletal data D3, that there is an 80% probability of the posture being upright and a 5% probability of the posture being slumped over a desk. The posture estimation unit 1114 then uses an estimation model to calculate the probability for each type of posture.

[0133] The second behavior estimation unit 1115 estimates the behavior of user P present in the target space 9 based on the sound collection data D2. Specifically, the second behavior estimation unit 1115 generates second behavior estimation data D5a based on the sound collection data D2.

[0134] The second behavior estimation data, D5a, shows the estimated probability (e.g., percentage) that user P is performing a corresponding behavior for each type of sound-related behavior. The types of actions related to sound are the same as in Embodiment 1.

[0135] The second behavior estimation unit 1115 estimates the type of behavior of user P present in the target space 9 using an estimation model that has been machine-learned to analyze the relationship between the sound collection data D2 provided by the microphone 38 and the second behavior estimation data D5a (probability for each type of behavior related to sound).

[0136] For example, as shown in Figure 8, the second behavior estimation unit 1115 estimates, based on the sound collection data D2, that there is a 98% probability of snoring and an 8% probability of talking as types of sound-related behaviors. The second behavior estimation unit 1115 uses an estimation model to calculate the probability for each type of sound-related behavior.

[0137] The environment estimation unit 1116 estimates the environment of the target space 9 based on the sound collection data D2. Specifically, the environment estimation unit 1116 generates environment estimation data D5b based on the sound collection data D2.

[0138] The environmental estimation data D5b is data that shows the estimated probability (e.g., percentage) of each type of environment being a corresponding environment. The type of environment is the same as in Embodiment 1.

[0139] The environment estimation unit 1116 estimates the environment of the target space 9 using an estimation model that has been machine-learned to analyze the relationship between the sound collection data D2 provided by the microphone 38 and the environment estimation data D5b (probability for each type of environment).

[0140] For example, as shown in Figure 8, the environmental estimation unit 1116 estimates, based on the sound collection data D2, that there is a 9% probability that noise is present and a 40% probability that calming music is playing. The environmental estimation unit 1116 then uses an estimation model to calculate the probability for each type of environment.

[0141] The second concentration level estimation unit 1117 estimates the user P's level of concentration on the task based on vital data D11, posture estimation data D12, first behavior estimation data D4, second behavior estimation data D5a, and environment estimation data D5b.

[0142] The second concentration level estimation unit 1117 estimates the concentration level of user P present in the target space 9 using vital data D11 provided by the sensor acquisition unit 1113, posture estimation data D12 provided by the posture estimation unit 1114, first action estimation data D4 provided by the first action estimation unit 414, second action estimation data D5a provided by the second action estimation unit 1115, environment estimation data D5b provided by the environment estimation unit 1116, and an estimation model that has been machine-learned to analyze the relationship between these factors and the concentration level of user P.

[0143] The second concentration level estimation unit 1117 may estimate the concentration level of user P based on a rule base that associates each type of vital data D11, posture estimation data D12, first behavior estimation data D4, second behavior estimation data D5a, and environment estimation data D5b with the concentration level of user P.

[0144] In this embodiment, the second concentration estimation unit 1117 calculates the concentration level as a percentage. If the percentage is below a predetermined threshold, the second concentration estimation unit 1117 determines that user P is not concentrating on the work.

[0145] For example, suppose posture estimation data D12 indicates that there is an 80% probability that the user's back is straight. Suppose first behavior estimation data D4 indicates that there is a 3% probability that the user is sleeping and a 78% probability that the user is studying. Furthermore, suppose second behavior estimation data D5a indicates that there is a 98% probability that the user is snoring, which is related to sound. Also, suppose environment estimation data D5b indicates that there is a 9% probability that noise is present in the target space 9 and a 40% probability that calming music is playing. In such a case, the level of concentration is likely to be low. In such a case, if the second concentration level estimation unit 1117 calculates the level of concentration as a percentage below a predetermined threshold, it determines that user P is not concentrating on the task. The above example corresponds to a situation where user P is sleeping in a seated position with their torso upright in the target space 9, where there is little noise and calming music is playing.

[0146] Furthermore, if vital data D11 shows characteristic temporal changes in heart rate or body temperature corresponding to a person sleeping, for example, it can be expected that the level of concentration on desk work is low. In this way, estimating the level of concentration based on vital data D11 can improve the accuracy of concentration estimation.

[0147] The detection data D1, sound collection data D2, and vital data D11, which include information about the time of detection, are stored in the second memory 1120 in chronological order from the past to the present. In addition, the generated posture estimation data D12, first action estimation data D4, second action estimation data D5a, and environment estimation data D5b are stored in the second memory 1120 corresponding to the time information contained in the detection data D1 or sound collection data D2.

[0148] The second notification unit 1118 issues a notification based on the estimation results of the second concentration estimation unit 1117. In Embodiment 2, the second notification unit 1118, similar to Embodiment 1, notifies user P that user P should interrupt their work if user P is not concentrating on their work. More specifically, the second notification unit 1118 notifies user P of information including a request to interrupt their work via the display 1002 and speaker 1003.

[0149] [2-2. Operation] Figure 9 is a flowchart FC showing the operation of computer 1000. The flowchart shown in Figure 9 assumes that the work is desk work. During the operation shown in Figure 9, the second communication control unit 1112 receives detection data D1 from each visible light camera 1004 via the relay device 40 at predetermined intervals. The sensor acquisition unit 1113 also acquires vital data D11 at predetermined intervals. The data and information processed during the operation shown in Figure 9 are stored in the second memory 1120 as appropriate.

[0150] As shown in flowchart FC, the second concentration level estimation unit 1117 determines whether a trigger for estimating the concentration level has occurred (step SC1). This trigger is, for example, the elapsed of a predetermined period (for example, 1 minute). In this way, the second concentration level estimation unit 1117 estimates the concentration level of user P at predetermined intervals.

[0151] When the second concentration estimation unit 1117 determines that a trigger for estimating the concentration level has occurred (step SC1: YES), the skeleton data generation unit 413 generates skeleton data D3 based on the detected data D1 (step SC2).

[0152] Next, the posture estimation unit 1114 generates posture estimation data D12 based on the skeletal data D3 generated in step SC2 (step SC3).

[0153] Next, the first behavior estimation unit 414 generates first behavior estimation data D4 based on the skeletal data D3 generated in step SC2 (step SC4).

[0154] Next, the second behavior estimation unit 1115 generates second behavior estimation data D5a based on the sound collection data D2 (step SC5).

[0155] Next, the environmental estimation unit 1116 generates environmental estimation data D5b based on the sound collection data D2 (step SC6).

[0156] Next, the second concentration level estimation unit 1117 estimates the concentration level of user P based on vital data D11, posture estimation data D12, first behavior estimation data D4, second behavior estimation data D5a, and environment estimation data D5b (step SC7).

[0157] Next, the second concentration estimation unit 1117 determines whether or not user P is concentrating on the work (step SC8).

[0158] If the second concentration estimation unit 1117 determines that user P is not concentrating on the work (step SC8: YES), the second notification unit 1118 notifies user P that they should interrupt their work (step SC9).

[0159] After step SC9 is completed, and if the second concentration estimation unit 1117 determines that user P is concentrating on the work (step SC8: NO), the computer 1000 returns to step SC1. In this way, the computer 1000 estimates the degree of concentration based on the data corresponding to the predetermined period each time a predetermined period has elapsed.

[0160] [2-3. Effects] As described above, the notification system 1B includes a visible light camera 1004 installed in the target space 9 to detect information within the target space 9, a skeleton data generation unit 413 that estimates the skeleton of user P present in the target space 9 based on detection data D1 indicating the detection results of the visible light camera 1004 and generates skeleton data D3, a first action estimation unit 414 that estimates user P's actions based on the skeleton data D3, a second concentration level estimation unit 1117 that estimates the degree of concentration of the work performed by user P based on the estimation results of the first action estimation unit 414, and a second notification unit 1118 that provides notifications based on the estimation results of the second concentration level estimation unit 1117. According to this method, the concentration level can be estimated using multiple pieces of information obtained from the target space 9, thus enabling an accurate estimation of the concentration level of user P in the target space 9. Therefore, notifications based on the concentration level of user P present in the target space 9 can be appropriately delivered.

[0161] Furthermore, the notification system 1B includes a posture estimation unit 1114 that estimates the user P's posture based on skeletal data D3, and the second concentration level estimation unit 1117 further estimates the concentration level based on the estimation result of the posture estimation unit 1114. According to this, the accuracy of the concentration level estimation is improved because the concentration level is estimated based on the estimated posture of user P.

[0162] Furthermore, the second concentration level estimation unit 1117 determines whether or not user P is concentrating on the work based on the concentration level, and the second notification unit 1118 notifies user P that they should interrupt their work if user P is not concentrating on the work. According to this, if user P is not concentrating on the task, user P will be prompted to interrupt the task. This increases the likelihood that user P will continue to concentrate on the task by taking a break, etc.

[0163] Furthermore, the notification system 1B includes a second behavior estimation unit 1115 that estimates the user P's behavior from sound, and the second concentration level estimation unit 1117 further estimates the concentration level based on the estimation results of the second behavior estimation unit 1115. According to this, the accuracy of the concentration level estimation is improved because the concentration level is estimated based on the user P's behavior, which is inferred from information other than the skeletal data D3.

[0164] Furthermore, skeletal data D3 includes information indicating the position of user P's spine BB. According to this, information indicating the position of the spine BB improves the accuracy of estimating user P's behavior and user P's posture. Therefore, the accuracy of estimating concentration level improves.

[0165] Furthermore, the notification system 1B includes a sensor acquisition unit 1113 that acquires vital data D11 indicating the vital signs of user P, and the second concentration level estimation unit 1117 further estimates the concentration level based on the vital data D11. According to this, the level of concentration is estimated based on user P's vital signs, thus improving the accuracy of the concentration level estimation.

[0166] Furthermore, the detection device is a visible light camera 1004, and the detection data D1 is the image data captured by the visible light camera 1004. According to this method, skeletal data D3 can be obtained from the video, and the accuracy of skeletal data D3 may be improved. Therefore, the accuracy of estimating user P's concentration level may be improved.

[0167] Furthermore, the computer 1000 includes a skeleton data generation unit 413 that generates skeleton data D3 by estimating the skeleton of user P present in the target space 9 based on detection data D1 showing the detection results of a visible light camera 1004 installed in the target space 9 that detects information within the target space 9; a first action estimation unit 414 that estimates user P's actions based on the skeleton data D3; a first concentration level estimation unit 416 that estimates the degree of concentration of the work performed by user P based on the estimation results of the first action estimation unit 414; and a second notification unit 1118 that provides notifications based on the estimation results of the first concentration level estimation unit 416. According to this, it produces the same effects and functions as described above. Furthermore, since the detection data D1 is used locally and the concentration level is estimated, personal information can be protected.

[0168] (Other embodiments) As described above, Embodiments 1 and 2 have been explained as examples disclosed in this application. However, the technology in this disclosure is not limited thereto and can be applied to embodiments that have been modified, replaced, added, or omitted. Furthermore, it is possible to combine the components described in Embodiments 1 and 2 to create new embodiments. Therefore, other embodiments are described below as examples.

[0169] In Embodiment 1 described above, outlets 30A and 30B were given as examples of electrical equipment 2 to which power is supplied. In Embodiment 2 described above, outlets 30A and 30B, a lighting device 10, and a switch 20 were given as examples of electrical equipment 2 to which power is supplied. However, electrical equipment 2 is not limited to these. Electrical equipment 2 includes switches, outlets, and other power wiring components, as well as ventilation fans, lighting fixtures, intercoms, and other devices that consume power. Electrical equipment 2 also includes distribution boards, building energy management devices, and other power control equipment.

[0170] In Embodiment 1 described above, a detection unit 33 was exemplified as an example of a detection device, but the invention is not limited thereto. In Embodiment 2 described above, a visible light camera 1004 was exemplified as an example of a detection device, but the invention is not limited thereto. In other embodiments, the detection device may be an infrared camera. In this case, the detection data D1 is imaging data from the infrared camera.

[0171] As described above, the notification system 1 of the embodiment described above is configured to acquire detection data D1 of user P without using sensors, but it is not limited to this and may be equipped with sensors such as visible light cameras or infrared cameras. These visible light cameras and infrared cameras may be installed on electrical equipment 2, mounted on surfaces, or installed on walls.

[0172] In another embodiment, the detection device is an infrared camera, and the detection data D1 is the image data captured by the infrared camera. According to this method, skeletal data D3 can be obtained from the video, and the accuracy of skeletal data D3 may be improved. Therefore, the accuracy of estimating user P's concentration level may be improved. In addition, since the skeleton is estimated based on infrared camera imaging data, which contains less personal information than normal images, the personal information of user P can be protected even in the skeletal estimation process.

[0173] Furthermore, the detection unit 33, microphone 38, visible light camera 1004, and infrared camera may be installed on other electrical equipment 2 such as other lighting devices 10, or they may be installed on a wall surface.

[0174] The detection processing unit 345 may store each data point of the radio wave condition information provided by the radio wave condition acquisition unit 344 in a time series. The detection processing unit 345 may extract the change region in the target space 9 from the time series changes in each data point of the radio wave condition information and estimate the state of user P in the change region. This allows detection processing to be narrowed down to the change region within the target space 9, thus enabling faster detection.

[0175] The detection processing unit 345 may also store radio wave state information when user P is not present in the target space 9 as reference channel information in the memory 341 or the like beforehand. The detection processing unit 345 can estimate the state of user P present in the target space 9 using the difference between the current radio wave state information and the reference channel information.

[0176] The detection processing unit 345 may also estimate the state of user P by using as additional information the three-dimensional position of the radio wave transmission source in the target space 9 and the extent of the target space 9, which are estimated by the estimation unit 347 described later. This makes it possible to improve the detection accuracy of the state of user P.

[0177] Alternatively, the arrangement of electrical equipment 2 in the target space 9 may be determined in advance during construction, and the arrangement information of each electrical equipment 2 in the target space 9 may be stored in memory 341 beforehand. The detection processing unit 345 can use the above arrangement information as additional information to estimate the state of user P. This makes it possible to further improve the detection accuracy of the state of user P.

[0178] In the embodiments 1 and 2 described above, the detection processing unit 345, skeletal data generation unit 413, first action estimation unit 414, sound estimation unit 415, first concentration level estimation unit 416, posture estimation unit 1114, second action estimation unit 1115, environment estimation unit 1116, and second concentration level estimation unit 1117 are configured to use estimation models employing machine learning techniques, but are not limited to this. The estimation models of the detection processing unit 345, skeletal data generation unit 413, first action estimation unit 414, sound estimation unit 415, first concentration level estimation unit 416, posture estimation unit 1114, second action estimation unit 1115, environment estimation unit 1116, and second concentration level estimation unit 1117 may each be models employing techniques such as pattern matching, Bayesian estimation, or regression analysis.

[0179] In the embodiments 1 and 2 described above, the first concentration estimation unit 416 and the second concentration estimation unit 1117 use machine learning techniques to estimate the concentration level as a percentage, and determine that user P is not concentrating on the work if the percentage is below a predetermined threshold. However, the invention is not limited to this configuration, and the first concentration estimation unit 416 and the second concentration estimation unit 1117 may estimate the concentration level not as a percentage, but by calculating it using a numerical value defined as appropriate.

[0180] Furthermore, in the above-described embodiment 1, the server 41 is configured to include a first concentration estimation unit 416, but the system is not limited to this. The first concentration estimation unit 416 may be provided in a detection device, a computer used by user P, or a computer installed in the electrical equipment 2.

[0181] Furthermore, the first concentration level estimation unit 416 and the second concentration level estimation unit 1117 may learn the correlation between the feedback from user P to notifications and the concentration level estimation results. The feedback is, for example, the result of a questionnaire administered by user P regarding the validity of their determination that the user P was not concentrating. This questionnaire is answered, for example, by a terminal device 6 used by user P.

[0182] The management DB 422 and the second memory 1120 may also contain prerequisite information about user P. Prerequisite information includes, for example, information about the environment of the target space 9 and information about user P's behavioral habits.

[0183] In the first embodiment described above, the first notification unit 417 is configured to notify via the terminal device 6, but is not limited thereto. In the second embodiment described above, the second notification unit 1118 is configured to notify via the display 1002 and speaker 1003, but is not limited thereto. The first notification unit 417 and the second notification unit 1118 may notify user P to interrupt their work by blowing air from the air conditioner 8 used by user P. Alternatively, the first notification unit 417 may notify user P to interrupt their work by flashing the lighting device 10 used by user P. Furthermore, the method of blowing air from the air conditioner 8 and the method of flashing the lighting device 10 by the first notification unit 417 and the second notification unit 1118 may be appropriately changed depending on the degree of concentration.

[0184] The first notification unit 417 and the second notification unit 1118 may further differentiate the content of the notifications depending on the degree of concentration of user P. For example, if the degree of concentration is high, the notification may state that the user is able to concentrate on the work. Also, for example, the notification content urging the user to interrupt their work may differ depending on whether the degree of concentration is moderate or extremely low. In this case, the degree of concentration is classified by setting appropriate thresholds.

[0185] In the embodiments 1 and 2 described above, the first notification unit 417 and the second notification unit 1118 are shown as having a configuration that notifies the user P to interrupt work, based on the estimation results of the first concentration level estimation unit 416 or the second concentration level estimation unit 1117, but the system is not limited to this. In other embodiments, the first notification unit 417 and the second notification unit 1118 may also notify the user P of the degree of concentration itself. For example, if the user is not concentrating on work, the first notification unit 417 and the second notification unit 1118 may notify the user that the user is not concentrating on work. Even in this case, the user P may be able to recognize that their concentration has broken and may be encouraged to take a break by interrupting work.

[0186] As described above, in the target space 9 in the other embodiment, at least one of an air conditioner 8 as an example of a blowing means and a lighting device 10 as an example of a lighting means is provided, and the first notification unit 417 or the second notification unit 1118 provides notification based on the concentration level via at least one of blowing air from the air conditioner 8 and flashing from the lighting device 10. According to this, notifications based on the degree of concentration can be made using means that can be provided within the target space 9.

[0187] The processor 340, the server processor 410, and the second processor 1110 may consist of a single processor or multiple processors. These processors may also be hardware programmed to implement the corresponding functional units. That is, these processors may consist of, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0188] The configurations of the detection unit 33, server 41, and computer 1000 shown in Figures 4, 5, and 8 are examples, and the specific implementation is not particularly limited. In other words, it is not necessarily required that hardware corresponding to each part be implemented individually, and it is also possible to configure the system so that a single processor executes a program to realize the functions of each part. Furthermore, some of the functions realized by software in the above-described embodiment may be implemented by hardware, or some of the functions realized by hardware may be implemented by software.

[0189] The operational step units shown in Figures 6 and 9 are divided according to the main processing content to facilitate understanding of the operation, and the operation is not limited by the way the processing units are divided or the names of the processing units. Depending on the processing content, it may be further divided into more step units. Alternatively, it may be divided so that one step unit includes even more processing. Furthermore, the order of the steps may be changed as appropriate, as long as it does not impede the intent of this disclosure.

[0190] Since the embodiments described above are for illustrative purposes of the technology described herein, various modifications, substitutions, additions, omissions, etc., can be made within the claims or their equivalents.

[0191] (Note) Based on the above description of embodiments, the following technologies are disclosed.

[0192] (Technology 1) A notification system comprising: a detection device provided in a target space for detecting information within the target space; a microphone provided in the target space for collecting sounds within the target space; a skeleton data generation unit that estimates the skeleton of a user present in the target space and generates skeleton data based on detection data indicating the detection results of the detection device; a first behavior estimation unit that estimates the user's actions based on the skeleton data; an environment estimation unit that estimates the environment of the target space from the sounds; a concentration level estimation unit that estimates the user's level of concentration on a task based on the estimation results of the first behavior estimation unit and the estimation results of the environment estimation unit; and a notification unit that provides notifications based on the estimation results of the concentration level estimation unit. According to this method, the concentration level can be estimated using multiple pieces of information obtained from the target space, thus enabling an accurate estimation of the concentration level of users in the target space. Therefore, notifications can be appropriately delivered based on the concentration level of users present in the target space.

[0193] (Technology 2) The notification system according to Technology 1, comprising a posture estimation unit that estimates the user's posture based on the skeletal data, and the concentration level estimation unit further estimates the concentration level based on the estimation result of the posture estimation unit. According to this, the accuracy of concentration estimation is further improved because concentration is estimated based on the estimated user posture.

[0194] (Technology 3) The notification system according to Technology 1 or 2, wherein the concentration estimation unit determines from the concentration level whether the user is concentrating on the work, and the notification unit notifies the user to interrupt the work if the user is not concentrating on the work. According to this, if a user is not focused on their work, they will be encouraged to take a break. This increases the likelihood that the user will be able to maintain their focus on the work after taking a break.

[0195] (Technical 4) A notification system according to any one of Technical 1 to 3, comprising a second behavior estimation unit that estimates the user's behavior from the sound, and the concentration level estimation unit further estimates the concentration level based on the estimation result of the second behavior estimation unit. According to this, the accuracy of concentration estimation is improved because concentration is estimated based on user behavior inferred from information other than skeletal data.

[0196] (Technical 5) A notification system according to any one of Technical 1 to 4, comprising a second acquisition unit for acquiring vital data indicating the user's vital signs, and the concentration level estimation unit further estimating the concentration level based on the vital data. According to this, the accuracy of the concentration level estimation is further improved because the concentration level is estimated based on the user's vital signs.

[0197] (Technical 6) The notification system according to any one of Technical 1 to 5, wherein the skeletal data includes information indicating the position of the user's spine. According to this, information indicating the position of the spine improves the accuracy of estimating user behavior or user posture. Therefore, the accuracy of estimating concentration levels improves.

[0198] (Technical 7) The detection device comprises a transmitter that transmits radio waves into the target space and a receiver that receives the radio waves, an acquisition unit that acquires radio wave state information which is information about the state of the radio waves based on the received radio waves, and an estimation unit that analyzes the received radio waves based on the radio wave state information to estimate the state of the user present in the target space, the detection data includes the state of the user based on the radio waves, and the skeleton data generation unit generates the skeleton data based on the state of the user included in the detection data, the notification system according to any one of Technical 1 to 6. According to this method, detection data can be obtained wirelessly. Furthermore, since the skeletal data obtained wirelessly does not contain information other than that of user P, personal information can be protected even in the estimation of the skeleton.

[0199] (Technical 8) The notification system according to any one of Technical 1 to 6, wherein the detection device is a visible light camera and the detection data is imaging data from the visible light camera. According to this method, skeletal data can be obtained from video footage, potentially improving the accuracy of the skeletal data. Therefore, the accuracy of estimating user concentration levels may improve.

[0200] (Technical 9) The notification system according to any one of Technical 1 to 6, wherein the detection device is an infrared camera and the detection data is imaging data from the infrared camera. According to this method, skeletal data can be obtained from video footage, potentially improving the accuracy of the skeletal data. Therefore, the accuracy of estimating the user's concentration level may improve. Furthermore, since the skeleton is estimated based on infrared camera imaging data, which contains less personal information than regular images, the protection of the user's personal information can be enhanced even in skeletal estimation.

[0201] (Technical 10) The notification system according to any one of Technical 1 to 9, wherein the target space is provided with a plurality of electrical equipment to which power is supplied, and the detection device is provided on at least one of the plurality of electrical equipment. According to this method, detection data can be obtained by utilizing electrical equipment placed in the target space. Therefore, the concentration level can be estimated without having to place dedicated components for obtaining detection data in the target space.

[0202] (Technical 11) The notification system according to any one of Technical 1 to 10, wherein the target space is provided with at least one of a blower and a lighting means, and the notification unit provides notification based on the concentration level via at least one of blowing air by the blower and flashing by the lighting means. According to this, notifications based on the degree of concentration can be made using means that can be provided within the target space. [Industrial applicability]

[0203] As described above, the notification system according to the present invention can be used for applications that provide notifications based on the user's level of concentration on their work. [Explanation of Symbols]

[0204] 1. 1B Notification System 2. Electrical equipment 6 Terminal devices 7. Vital Sensors 8. Air conditioning system (air blower) 9. Target Space 10 Lighting device (lighting means) 20 switches 30 outlets 33. Detection Unit (Detection Device) 34 Control device 35 Transmitters 36 Receiver 38 Microphones 41, 41B Server 340 processors 341 memory 342 Programs 343 Detection and Communication Control Unit 344 Radio wave condition acquisition unit (second acquisition unit) 345 Detection Processing Unit 346 Radio wave separation section 347 Estimation Department 348 Sound detection processing unit 400 Server Control Units 401 Server Communication Device 410 Server Processors 411 Server Communication Control Unit 412 Management Department 413 Skeleton Data Generation Unit 414 1st Behavior Estimation Department 415 Sound estimation section (second behavior estimation section, environment estimation section) 416 First concentration estimation section (concentration estimation section) 417 Notification Department 420 Server Memory 421 Server Program 1000 computers 1004, 1004A, 1004B, 1004C, 1004D Visible light camera (detection device) 1113 Sensor acquisition unit (first acquisition unit) 1114 Posture estimation section 1115 2nd Behavior Estimation Department 1116 Environmental Estimation Department 1117 Second concentration estimation section (concentration estimation section) 1118 2nd Notification Department (Notification Department) D1 detection data D2 Sound Collection Data D3 Skeleton Data D4 First Action Estimation Data D5 Sound Estimation Data D5a Second Behavior Estimation Data D5b Environmental Estimation Data D11 Vital Data D12 Pose Estimation Data

Claims

1. A detection device installed in the target space for detecting information within the target space, A microphone provided in the aforementioned target space for collecting sound within the target space, A skeleton data generation unit estimates the skeleton of a user present in the target space and generates skeleton data based on detection data showing the detection results of the detection device, A first behavior estimation unit estimates the user's behavior based on the aforementioned skeletal data, An environment estimation unit that estimates the environment of the target space from the sound, A concentration level estimation unit estimates the user's level of concentration on the task based on the estimation results of the first behavior estimation unit and the estimation results of the environment estimation unit, The system includes a notification unit that provides notification based on the estimation results of the concentration estimation unit. Notification system.

2. The system includes a posture estimation unit that estimates the user's posture based on the skeletal data. The concentration estimation unit further estimates the concentration based on the estimation result of the posture estimation unit. The notification system according to claim 1.

3. The concentration level estimation unit determines from the concentration level whether the user is concentrating on the work, The notification unit will notify the user to interrupt their work if the user is not concentrating on their work. The notification system according to claim 2.

4. It includes a second behavior estimation unit that estimates the user's actions from the aforementioned sound, The concentration estimation unit further estimates the concentration level based on the estimation result of the second behavior estimation unit. The notification system according to claim 2.

5. The system includes a first acquisition unit that acquires vital data showing the user's vital signs, The concentration estimation unit further estimates the concentration level based on the vital data. The notification system according to claim 2.

6. The skeletal data includes information indicating the position of the user's spine. The notification system according to any one of claims 1 to 5.

7. The detection device comprises a transmitter that transmits radio waves into the target space and a receiver that receives the radio waves. A second acquisition unit acquires radio wave status information, which is information about the state of the radio wave, based on the received radio wave; The system includes an estimation unit that analyzes the received radio waves based on the radio wave state information to estimate the state of the user present in the target space, The detection data includes the user's state based on the radio waves. The skeletal data generation unit generates the skeletal data based on the user's state included in the detection data. The notification system according to any one of claims 1 to 5.

8. The detection device is a visible light camera, The detection data is the image data from the visible light camera. The notification system according to any one of claims 1 to 5.

9. The detection device is an infrared camera, The detection data is the imaging data from the infrared camera. The notification system according to any one of claims 1 to 5.

10. The aforementioned target space is provided with multiple electrical equipment and materials to which electricity is supplied. The detection device is provided on at least one of the multiple electrical equipment materials. The notification system according to any one of claims 1 to 5.

11. The aforementioned target space is provided with at least one of a blowing means and a lighting means. The notification unit provides notification based on the concentration level via at least one of the following: airflow by the air blowing means and flashing by the lighting means. The notification system according to any one of claims 1 to 5.

Images