Life monitoring system and program

Abstract

To intuitively acquire life behavior of monitoring object persons while suppressing monitoring burden of guardians.SOLUTION: A life monitoring system for providing a guardian with information representing a life situation of a monitoring object person living in a building, the life monitoring system includes an acquisition unit and a control unit. The acquisition unit acquires detection data obtained by detecting movements of the monitoring object person in a monitoring area inside the building. The control unit generates a summary video by reducing a behavior history for at least a partial period from a behavior history of the monitoring object person obtained from the detection data in a prescribed period, and outputs the summary video. The control unit generates the summary video with a reduction amount corresponding to a type of the monitoring area where the monitoring object person is detected.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a life monitoring system and a program therefor that provide a monitor with information about the living conditions of a person being monitored, such as an elderly person residing in a building. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there have been systems that use various sensors to monitor a person being monitored in a building such as a house.

[0003] For example, Patent Document 1 listed below discloses a system that allows a person watching over (a child) to understand the behavior of the person being watched over (an elderly person) by displaying a list of the person being watched over (an elderly person)'s daily living activity patterns in chronological order, color-coding them according to the type of living activity, as detected using multiple motion sensors installed in the home. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-122292 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology of Patent Document 1 uses a time series graph to display a list of daily activity patterns, making it difficult to intuitively grasp the daily activities of the person being watched over. Also, checking the activity history of the person being watched over in detail along a time series is a heavy burden for the person being watched over.

[0006] In view of the above circumstances, the object of the present invention is to provide a daily life monitoring system and program that can intuitively grasp the daily activities of the person being monitored while reducing the monitoring burden on the monitor. [Means for solving the problem]

[0007] To achieve the above object, one embodiment of the present invention provides a lifestyle monitoring system that provides a monitor with information indicating the lifestyle status of a monitored person residing in a building, and includes an acquisition unit and a control unit. The acquisition unit acquires detection data that detects the movements of the monitored person in a monitoring area within the building. The control unit generates a summary video by thinning out at least a portion of the monitored person's behavior history obtained from the detection data over a predetermined period of time, and outputs the summary video. The control unit generates the summary video with a thinning amount that corresponds to the type of monitoring area in which the monitored person is detected.

[0008] As a result, the daily life monitoring system varies the amount of thinning of the behavioral history in the summary video depending on the type of monitoring area, thereby optimizing the behavioral history of the person being watched that the watcher needs to check for each monitoring area, thereby reducing the burden of monitoring on the watcher and allowing the watcher to intuitively understand the daily activities of the person being watched. Here, there is one or more monitoring areas within the building. The summary video is, for example, animation or video. The detection unit includes, for example, radio waves such as millimeter waves, camera photography (visible light, infrared, heat), and distance sensors such as TOF (Time Of Flight) sensors. The difference in the amount of thinning may be, for example, a difference in the predetermined interval when the behavioral history of the person being watched is thinned at a predetermined interval, or a difference in whether a predetermined behavioral period of the person being watched is cut out entirely or partially.

[0009] The behavior history may include information representing a time series of behaviors taken by the watched person. In this case, the control unit may determine the behaviors of the watched person during the predetermined period, and generate the summarized video with a thinning amount according to the type of the watching area, by targeting periods in which a predetermined behavior was detected from the determined behaviors as thinning targets.

[0010] The control unit may set a period in which a behavior type with a small amount of movement is detected among the determined behavior types as the thinning target.

[0011] This allows the daily life monitoring system to create a summary video by thinning out actions of the monitored person with low levels of movement, but still allow the monitor to check even those actions with low levels of movement depending on the purpose of the monitoring area. Here, differences in the amount of thinning include changing whether or not to thin out depending on the monitoring area, or changing the rate of thinning.

[0012] Each of the plurality of buildings may have, as the monitoring area, at least one of a first monitoring area having a sleeping space for the person being watched over and a second monitoring area having no sleeping space. In this case, the control unit may generate the summarized video with a first thinning amount for a period in which the behavior type to be thinned out is detected in the first monitoring area, and may generate the summarized video with a second thinning amount greater than the first thinning amount for a period in which the behavior type to be thinned out is detected in the second monitoring area.

[0013] This allows the lifestyle monitoring system to inform the monitor that sleeping is a normal (important) behavior in the first monitoring area for sleeping, but since sleeping is not a high priority in the second monitoring area for non-sleeping, the burden on the monitor can be reduced by thinning it out.

[0014] The detection data may include information indicating at least the posture of the person being watched over. Furthermore, each of the plurality of buildings may have, as the monitoring area, at least one of a first monitoring area having a sleeping space for the person being watched over and a second monitoring area having no sleeping space. In this case, the control unit may generate an animation of the person being watched over sleeping for a period during which the person being watched over remained lying down in the first monitoring area, and include the animation in the summary video, and omit from the summary video a period during which the person being watched over remained lying down in the second monitoring area.

[0015] This allows the life monitoring system to determine whether or not to include the behavior as an animation in the summary video, even if the monitored person performs the same behavior, depending on the monitoring area in which the behavior was performed.

[0016] The control unit may generate a time series graph that displays the determined behavior types of the monitored person in chronological order on a time axis, and output the generated summary video together with the time series graph that shows the position during playback of the summary video on the time axis to the display unit.

[0017] This allows the life monitoring system to allow the monitor to check the behavior of the person being monitored through a video summary, and to understand at a glance at what time during the specified monitoring period the behavior of the person being monitored occurred.

[0018] A program according to another aspect of the present invention includes: receiving from a server device a behavior history of the person being watched obtained from detection data that detects the movements of the person being watched in a watching area within a building, and area type information that indicates the type of the watching area; generating a summary video by thinning out at least a portion of the behavior history from the behavior history for a predetermined period, with a thinning amount according to the type of the monitoring area; and a step of playing the summary video. [Effects of the Invention]

[0019] As described above, according to the present invention, it is possible to intuitively grasp the daily activities of the person being watched while reducing the burden of watching on the watcher. However, this effect does not limit the present invention. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a diagram showing the configuration of a lifestyle monitoring system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating a hardware configuration of a server device of the above-mentioned life monitoring system. [Figure 3] 10 is a diagram showing an example of a playback screen of the behavior history information of the watched person generated by the server device. FIG. [Figure 4] FIG. 10 is a diagram showing an example of a time series graph displayed on the playback screen of the digest video. [Figure 5] 10 is a table showing differences in the generation method of the digest video generated by the server device depending on the watching area. [Figure 6] 10 is a table showing types of behavior of a watched person that are targets of a summary video generated by the server device. [Figure 7] 10 is a table showing differences in the generation method of the summary video generated by the server device depending on the security mode (presence / absence) during the non-detection period. [Figure 8] 10 is a flowchart showing a flow of a digest video playback process for the living room by the server device. [Figure 9] 10A to 10C are diagrams conceptually showing display transitions of a summary video of a living room generated by the server device. [Figure 10] 10 is a flowchart showing a flow of a summary video playback process of a bedroom by the server device. [Figure 11] 10A to 10C are diagrams conceptually illustrating display transitions of a summary video of a bedroom generated by the server device. [Figure 12] FIG. 10 is a diagram showing types of behavior of a watched person that are the subject of a summary video generated by a server device in a modified example of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0022] [System Configuration] FIG. 1 is a diagram showing the configuration of a lifestyle monitoring system according to this embodiment.

[0023] As shown in the figure, the daily life monitoring system provides a monitoring service that monitors for abnormalities such as intrusions into the building where the person being monitored is located (monitored area), and a monitoring service that monitors the status of the person being monitored, such as an elderly person. In this monitoring service, information showing the behavioral history detected in the monitored person's daily life is provided to a monitor, such as a family member (child) who lives away from the monitored person. This allows the monitor to recognize the status of the monitored person and keep an eye on them.

[0024] The lifestyle monitoring system includes a server device 100, a user terminal 200, a monitoring device 300, a plurality of intrusion monitoring sensors 400, and a monitoring sensor 500. The monitoring device 300, the intrusion monitoring sensor 400, and the monitoring sensor 500 are installed in a building such as the residence of the person being monitored, and are connected via a network such as a LAN (Local Area Network).

[0025] The server device 100 is installed in a monitoring center operated by a company that provides monitoring and watching services. The server device 100 determines the behavior of the person being watched from the time-series changes in the posture and position detection results of the person being watched received from the monitoring device 300, creates a behavior history, and provides the watcher with information about the daily life of the person being watched (behavior history) using animation videos, videos, etc.

[0026] The user terminal 200 is carried by the watcher and is used by the watcher to view various information related to the monitoring service, such as information showing the activity status of the person being watched. Examples of the user terminal 200 include smartphones, mobile phones, personal computers, tablet terminals, and wearable terminals. An application (hereinafter also referred to as a "monitoring app") that supports viewing of various information related to the monitoring service is installed in the user terminal 200.

[0027] The server device 100, the monitoring device 300, and the user terminal 200 are connected via a network 50 including the Internet and a mobile communication network.

[0028] Intrusion monitoring sensors 400 detect intruders into a building and output detection information. If the building is a residence, intrusion monitoring sensors 400 include sensors installed inside the residence (for example, at openings such as the front door and windows, and in traffic lines such as corridors within the residence, and in living rooms) and sensors installed outside the residence (for example, in the garden or gate).

[0029] The intrusion monitoring sensor 400 may be, for example, an open / close sensor or a human presence sensor installed on a window, door, or the like. Examples of the human presence sensor include a sensor that detects the presence of a person by fluctuations in heat rays emitted by the human body, a sensor that detects the presence of a person by sound waves or high frequency waves, a sensor that detects the presence of a person by processing images captured by a surveillance camera, and a sensor that detects the presence of a person by blocking incident light from a light-projecting unit. Other examples of the intrusion monitoring sensor 400 include a TOF (Time Of Flight) sensor and an ultrasonic (millimeter wave, microwave) sensor. The intrusion monitoring sensor 400 may also be used for monitoring purposes.

[0030] The monitoring sensor 500 is installed in a monitoring area (room or space) within a building, detects the movement (amount of movement and posture) of the person being monitored in the monitoring area, and outputs detection information indicating the detection result.

[0031] The monitoring sensor 500 may be, for example, an ultrasonic sensor (such as a millimeter wave or microwave), a TOF sensor, or a camera. When a camera capable of capturing video is used as the monitoring sensor 500, the position and posture of the person being watched are detected for each captured frame. Posture detection methods include, for example, pattern matching between a person image extracted from an image using background subtraction and pre-registered images of each posture, and authentication using a trained model trained on images of various postures for posture detection. Furthermore, when an ultrasonic sensor is used as the monitoring sensor 500, the position and posture of the person being watched are detected, for example, every second. Posture detection methods include, for example, pattern matching between point cloud data obtained from the ultrasonic sensor and pre-registered point cloud data of each posture. The monitoring sensor 500 detects the position and posture of a person within the monitoring area (e.g., standing, sitting, lying down, etc.), and calculates the amount of movement of the person from changes in the person's position and posture. The amount of movement is calculated by scoring the amount of change in position and the amount of change in posture per unit time and adding them up. The monitoring sensor 500 then outputs the calculated amount of movement and posture as detection data. The amount of movement may be calculated only from the amount of change in position. Alternatively, the detection data may include the presence or absence of movement instead of the amount of movement. Alternatively, the monitoring sensor 500 may send the detection results of the position and posture to the server 100, and the server 100 may calculate the amount of movement. The monitoring sensor 500 may also be used for the above-mentioned intrusion monitoring purpose.

[0032] The monitoring device 300 uses the intrusion monitoring sensor 400 to monitor intrusions into the building where the person being watched over is located, and also uses the monitoring sensor 500 to detect the behavior of the person being watched over.

[0033] For example, when an intruder is detected by the intrusion monitoring sensor 400, the monitoring device 300 notifies the server device 100 that an intruder has entered the building. The monitoring device 300 has three security modes: an unmanned security mode, an attended security mode, and a canceled mode.

[0034] The unattended alert mode is a security mode that monitors the entire building. The unattended alert mode is used when the person being watched over is out and the building is unoccupied. In the unattended alert mode, when detection information is output from any of the multiple intrusion monitoring sensors 400, an abnormality signal is sent from the monitoring device 300 to the server device 100. In the unattended alert mode, the detection result of the monitoring sensor 500 is no detection, and the person being watched over is treated as being out.

[0035] The manned alert mode is a security mode that monitors a portion of a building. The manned alert mode is used when there is a person being watched in the building and intruders are to be monitored in areas where the person being watched is not present. For example, in the manned alert mode, of the multiple intrusion monitoring sensors 400, a predetermined perimeter area such as the front door, windows, gates, and the outside of the residence is monitored to detect intrusions into the building from the outside. Alternatively, of the multiple areas included in the building, an area (such as the second floor) designated in response to an operation by the person being watched may be monitored.

[0036] In the manned alert mode, when detection information is output from an intrusion monitoring sensor 400 installed in the outer perimeter area or a designated area, an abnormality signal is sent from the monitoring device 300 to the server device 100. On the other hand, even if detection information is output from an intrusion monitoring sensor 400 installed in a location other than the outer perimeter area or the designated area, this detection information is ignored, and no abnormality signal is sent from the monitoring device 300 to the server device 100. In the manned alert mode, if the person being watched is in a location other than the designated area within the building, the detection results of the monitoring sensor 500 installed in that location are recorded as a behavior history.

[0037] The release mode is a security mode in which the building is not monitored. The release mode is used when there is a person being watched in the building and there is no need to monitor intruders. In the release mode, intruders into the building are not monitored, and even if detection information is output from the intrusion monitoring sensor 400, this detection information is ignored and no abnormality signal is sent from the monitoring device 300 to the server device 100. In the release mode, the detection results of the monitoring sensor 500 are used exclusively to record the behavior history of the person being watched.

[0038] Although not shown, the monitoring device 300 is connected to a setting unit that is installed at the entrance or the like in order to receive an operation to set the security mode of the monitoring device 300.

[0039] When the server device 100 receives an abnormality signal from the monitoring device 300, it outputs information about the building and the person being watched over, prompting the monitoring sensor controller to check on the person being watched over and the security guard rushing to the building to take action. The server device 100 also determines the behavior of the person being watched over based on the detection results of the monitoring sensor 500, and provides the behavior history information to the user terminal 200 of the watcher.

[0040] [Server device configuration] 2 is a diagram showing the configuration of the server device 100. As shown in the figure, the server device 100 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an input / output interface 15, and a bus 14 connecting these components to one another.

[0041] The CPU 11 accesses the RAM 13 and other memory as needed, and performs various arithmetic processing while controlling all of the blocks in the server device 100. The ROM 12 is a non-volatile memory that permanently stores firmware such as the OS, programs, and various parameters to be executed by the CPU 11. The RAM 13 is used as a working area for the CPU 11, and temporarily stores the OS, various applications currently being executed, and various data currently being processed.

[0042] The input / output interface 15 is connected to a display unit 16, an operation reception unit 17, a storage unit 18, a communication unit 19, and the like.

[0043] The display unit 16 is a display device (built-in display) that uses, for example, an LCD (Liquid Crystal Display), an OLED (Organic ElectroLuminescence Display), a CRT (Cathode Ray Tube), or the like.

[0044] The operation reception unit 17 is, for example, a touch panel, a button, a key, or another input device. When the operation reception unit 17 is a touch panel, the touch panel can be integrated with the display unit 16.

[0045] The storage unit 18 is, for example, a flash memory (SSD; Solid State Drive) or other solid-state memory, or a non-volatile memory such as an HDD (Hard Disk Drive). The storage unit 18 stores the OS, various applications, and various data.

[0046] In particular, in this embodiment, the storage unit 18 stores a program and data for generating behavior history information of the person being watched over based on the detection results of the watching sensor 500 and displaying the information on the user terminal 200 .

[0047] The communication unit 19 is, for example, a NIC (Network Interface Card) for Ethernet or various modules for wireless communication such as wireless LAN, and is responsible for communication processing between the user terminal 200 and the monitoring device 300 via the network 50.

[0048] [Behavioral history information] Next, the behavior history information of the person being watched over that is generated by the server device 100 and displayed on the user terminal 200 will be described.

[0049] The server device 100 stores the behavior history of the person being watched in chronological order based on the detection data acquired from the monitoring sensor 500. The behavior history is classified based on the presence or absence of movement and the type of posture, and consecutive periods in which the same behavior type, determined by the presence or absence of movement and the type of posture, are stored separately from periods indicating other behavior types. The behavior types are classified, for example, into standing with movement, standing without movement, sitting, lying down (lying down), and non-detection (a state in which no person is detected). In addition, since changes in posture are also information related to the presence or absence of movement, posture changes may be stored as one of the behavior types. In addition, the amount of movement included in the detection data is stored in chronological order along with each behavior type. Here, standing with movement and posture changes are behavior types with a relatively large amount of movement, while standing without movement, sitting, and lying down are behavior types with a relatively small amount of movement.

[0050] The user terminal 200 receives and displays the behavior history information from the server device 100 via the monitoring app. The behavior history information is generated as, for example, an animated video as described above. Fig. 3 is a diagram showing an example of a playback screen of the behavior history information on the user terminal 200.

[0051] As shown in the figure, the playback screen of the behavior history information has an animation moving image display field 31 and a time series graph display field 32.

[0052] The animation video display area 31 has an icon 33 indicating the position and behavior of the person being watched in the watched area on an image that mimics the space of the watched area, and an animation is played in which the position of the icon 33 changes according to the trajectory of the watched person's movement as behavior history information for a specified period of time, and the display content of the icon 33 changes according to the type of behavior of the watched person.

[0053] Note that while the figure shows an example of the display of behavioral history information when the monitoring area is the living room, a button may be provided, for example, at the right end of the animation video display area 31, to transition to a playback screen of behavioral history information in another monitoring area (for example, a bedroom).

[0054] The time series graph display field 32 displays a time series graph 34 in which the behavior types of the person being watched are associated with the detection times of the detection data of the watching sensor 500 corresponding to the behavior types in time series.

[0055] The time series graph 34 displays a behavior history, starting from a certain point in time (for example, the present) and continuing for, for example, 24 hours prior, in different colors according to the type of behavior. A pointer 36 is displayed on the time series graph 34 at a position indicating the detection time of the behavior of the person being watched played in the animation video display field 31, and the position of the pointer 36 moves in synchronization with the playback time of the animation video. The playback time of the animation video is also displayed, for example, at the top of the time series graph 34.

[0056] As shown in the legend at the bottom of the time series graph display field 32, the types of detected behaviors to be displayed in the animated video and time series graph 34 include, for example, standing with movement, standing without movement, sitting, lying down (sleeping or lying down), going out, and non-detection. Standing with movement refers to a state in which a person is performing an action with a relatively large amount of movement while standing (for example, exercising or cleaning), and standing without movement refers to a state in which a person is performing an action with a relatively small amount of movement while standing (for example, cooking).

[0057] A play button 35 for requesting playback of the animation video is provided, for example, below the time series graph 34. The server device 100 starts generating and displaying the animation video in response to a playback request made by the play button 35. Instead of using the play button 35, a playback request may be sent from the watching app to the server device 100 when the watching app is launched, and the animation video may be automatically played.

[0058] In addition, the above animated video and time series graph 34 may be created using behavioral history information from the morning hours (e.g., after 6:00 a.m.) to the present if the monitoring area is the living room, or from the evening hours (e.g., after 6:00 p.m.) to the present if the monitoring area is the bedroom.

[0059] The animation video playback modes include a normal playback mode and an abridged playback mode. These two modes may be switched by, for example, a mode switching button 37 at the top right corner of the screen. The figure shows the state in which the abridged playback mode is selected.

[0060] The normal playback mode is a mode in which all of the behavior history information, which is the detection results, is played back in chronological order. In the normal playback mode, for example, an animated video of the detection results for each frame or each second of the camera / millimeter wave as the monitoring sensor 500 is displayed together with the time series graph 34.

[0061] On the other hand, the summary playback mode is a mode in which, from the behavior history information of the watched person, actions other than those that should be provided to the watcher are thinned out, and a summary video is generated and displayed. In other words, the summary playback mode is a so-called digest playback mode in which content is played back in a shorter time by focusing on scenes that the user wants to see / show. In the summary playback mode, the server device 100 extracts specific actions from the behavior of the watched person based on the time-series changes in the detection results for each frame or each second, and generates a summary video in which all actions other than the specific actions are omitted (thinned out).

[0062] Alternatively, in normal playback mode, the server device 100 may extract and play back representative actions of the watched person from the detection results for a predetermined period of time. That is, the behavior history information may be divided into predetermined time intervals, such as 1 minute, 5 minutes, or 10 minutes, representative actions may be extracted from each interval, and an animated video and a time series graph 34 created using only those actions may be generated and played back.

[0063] In this case, even if there is a behavior that would be thinned out in the summary playback mode within the divided time range, if that behavior is a representative behavior, it will be played back without thinning out. A representative behavior may be, for example, the behavior with the greatest amount of activity (amount of movement) or the behavior that takes up the largest proportion of time within that time range.

[0064] For example, if the behavior history information for 12:00 to 12:05, 12:05 to 12:10, and 12:10 to 12:15 is 12:00 to 12:05: standing with movement, 5 minutes to 10 minutes: standing and sitting with movement, and 10 minutes to 15 minutes: sitting, the representative behaviors displayed are 12:00 to 12:05: standing with movement, 12:05 to 12:10: standing with movement, and 12:10 to 12:15: sitting.

[0065] The two different display methods in the normal playback mode may be implemented on a single screen. For example, the time-series changes in the detection results for each frame or each second may be displayed as they are as an animated video, and representative behaviors for each time span may be extracted and displayed on the time-series graph 34.

[0066] Furthermore, from the results of extracting the representative actions in the normal playback mode, actions other than those to be thinned out may be extracted based on the thinning process described below to generate a summary video.

[0067] For example, if the representative behaviors for each time span in normal playback mode are 12:00 to 12:05: standing with movement, 12:05 to 12:10: standing with movement, and 12:10 to 12:15: sitting, then in summary playback mode, the standing behaviors with movement between 12:00 and 12:10 are extracted (thinning out the sitting behavior), and a summary video is generated and displayed at double speed. Note that in this case, rather than cutting out the entire sitting behavior from 12:10 to 12:15, a few frames of animation video showing the sitting behavior can be inserted into the summary to show the change in posture from standing to sitting.

[0068] Furthermore, instead of switching between the normal playback mode and the digest playback mode, the video displayed may be changed by a user operation in the animation video display field 31. For example, when an arbitrary time is selected on the time series graph 34 or when the play button 35 is pressed, the animation video display field 31 normally displays the time series changes in the detection results for each frame or each second, and the time series graph 34 extracts and displays representative behavior for each time span (corresponding to the normal playback mode). When the most recent review button in FIG. 3 is pressed or the monitoring app is started, a digest playback request is made, and when the digest playback request is received, the created digest video is played in the animation video display field 31.

[0069] In this embodiment, when generating and playing the summary video, the server device 100 changes the degree to which behavior types with low activity levels (movement levels) are included in the summary video (thinned out from the summary video) depending on the monitoring area (living room / bedroom, etc.).

[0070] In addition, during non-detection periods in which the monitored person cannot be detected by the monitoring sensor 500 in the monitoring area, the server device 100 changes the degree to which the non-detection periods are included in the summary video (thinned out from the summary video) depending on information on whether the monitored person is present in the building (presence / absence information).

[0071] The process of thinning out the summary video according to the monitoring area and the process of thinning out the summary video according to the presence / absence information of the person being monitored will be described in detail below.

[0072] [Video summary reduction according to monitoring area] Fig. 4 shows examples of the time series graph 34 when the monitoring area is the living room and when it is the bedroom. Fig. 5 is a table showing the difference in the thinning process of the summary video depending on the monitoring area. Fig. 6 is a table showing the types of behavior of the person being watched that are the subject of the summary video. The server device 100 stores data such as tables corresponding to Figs. 5 and 6 in the storage unit 18.

[0073] As shown in these figures, the server device 100 includes, without thinning out, the behavior of the person being watched that has a large amount of movement (activity) in the detection data of the monitoring sensor 500, in both the summary video of the living room and the summary video of the bedroom, because this indicates a period when the person being watched was actively active.

[0074] On the other hand, behaviors with low amounts of movement (activity) in the detection data do not indicate periods in which the person being watched was actively moving, so the server device 100 thins out (cuts out) them from the summary video of the living room, but since the bedroom is an area where people sleep and sleeping is also a normal behavior that should be watched over, the server device 100 includes them in the summary video of the bedroom without thinning them out.

[0075] In the living room time series graph 34, among the detected behaviors, standing with movement is included in the summary video, but standing without movement, sitting, and lying down are omitted from the summary video. In addition, if a posture change between different behavior types is detected (for example, from lying down to sitting), it is included in the summary video as a behavior with a large amount of movement.

[0076] In addition, the method of changing the amount of summary video thinning depending on the monitoring area is not limited to the above-mentioned all-display / no-display pattern, but may also thin out activities with low movement in both the living room and bedroom, with the amount of thinning in the living room being greater than the amount of thinning in the bedroom.

[0077] Furthermore, if the monitoring area is an area that serves as both a living room and a bedroom, a living room monitoring mode and a bedroom monitoring mode may be provided, and a summary video corresponding to the mode selected by a user operation may be displayed.

[0078] [Video summary reduction based on presence / absence information] 7 is a table showing the difference in the process of thinning out the summary video depending on the presence or absence of the person being watched during the non-detection period. The server device 100 stores data such as a table corresponding to FIG. 7 in the storage unit 18.

[0079] In this embodiment, the server device 100 determines whether the person being watched over is present or absent in the building based on the security mode of the monitoring device 300. That is, if the security mode is the unmanned security mode during the non-detection period of the monitoring sensor 500, it is determined that the person being watched over is out and therefore active outside.

[0080] Since the summary video informs the watcher of the period during which the watched person was active in their daily life, if the security mode during the non-detection period was unmanned security mode, the server device 100 includes this as an outing action in the summary video.

[0081] Furthermore, if the security mode during the non-detection period is the attended alert mode or the canceled mode, it can be determined that the watched person is at home, but since it is not possible to detect behavior in areas other than the watched area, the server device 100 omits the non-detection period from the summarized video. As shown on the right side of Figure 7, instead of omitting the non-detection period from the summarized video, the server device 100 may include part of it in the summarized video. In other words, it is sufficient that the amount of thinning in the summarized video in the attended alert mode or the canceled mode is greater than the amount of thinning in the unattended alert mode.

[0082] Instead of determining the presence or absence of the person being watched over from the security mode, the person may be detected as having left the house by detecting that the front door is locked and a motion sensor at the front door detects a person, or the presence or absence may be determined from the location information of a smartphone or wearable device carried by the person being watched over.

[0083] Furthermore, in the summary video of the bedroom, the watcher is likely to want to check how the person being watched is sleeping at night, so the non-detection periods during the day are not included in the summary video (as mentioned above, activities with high and low amounts of movement during the day in the bedroom are included in the summary video). Additionally, the non-detection periods at night are included in the summary video regardless of the set security mode (presence or absence).

[0084] [Life Monitoring System Operation] Next, the operation of the lifestyle monitoring system configured as above will be described. The operation is performed by the cooperation of hardware such as the CPU 11 and communication unit 19 of the server device 100 and software (monitoring application) stored in the storage unit 18. For convenience, in the following description, the CPU 11 of the server device 100 is the subject of operation.

[0085] Fig. 8 is a flowchart showing the flow of digest video playback processing for the living room by the server device 100. Fig. 9 is a diagram conceptually showing transitions in display of digest videos for the living room.

[0086] 8, the CPU 11 of the server device 100 first determines whether or not a summary video playback request for the living room has been received from the user terminal 200 via the monitoring app (step 81). As described above, the summary video playback request is sent when the playback button 35 is pressed or the monitoring app is started.

[0087] Next, CPU 11 extracts behavior history information to be included in the summary video of the living room that is the subject of the playback request (step 82). That is, CPU 11 extracts a period in which behaviors to be displayed as a summary were detected from the behavior history information for a predetermined period (one day, half a day, three hours, etc.) (periods that are not subject to summary display are omitted). For example, from the three-hour behavior history information, CPU 11 extracts a period in which behaviors with a large amount of movement (movement, standing position, posture change) were detected.

[0088] Whenever the CPU 11 receives detection data from the monitoring sensor 500, it analyzes the position, posture, amount of movement, etc. in the detection data each time, or periodically, to determine the behavior type, and stores behavior history information including behavior type information in the memory unit 18.

[0089] Next, the CPU 11 determines whether or not there is a period of a behavior type with a small amount of movement (no movement, standing, sitting, lying down) in the behavior history information based on the behavior type (step 83).

[0090] If it is determined that there is a period of an activity type with a small amount of movement (Yes in step 83), the CPU 11 omits (thins out) that period from the digest video (step 84).

[0091] Next, CPU 11 extracts a non-detection period from the behavior history information and determines whether or not there is a non-detection period and a period in which the person is at home (step 85). That is, if there is a non-detection period, CPU 11 obtains information on the security mode that was set during that period from monitoring device 300 and determines whether or not that period is a period in which the person is at home (attended security mode / cancelled mode).

[0092] If it is determined that there is a non-detection period and a period in which the user is at home (Yes in step 85), the CPU 11 omits (thins out) that period from the digest video (step 86).

[0093] The CPU 11 then generates the extracted behavior for the predetermined period so that it is played back at double speed to match the number of seconds of the summary video (animation video), sets it to be displayed in synchronization with the time series graph 34, and transmits it to the user terminal 200 to be played back (step 87).

[0094] For example, when playing back a summary video in 10 seconds, if the total time extracted from three hours of behavior history information is 20 minutes, CPU 11 sets the display of 20 minutes of information (video or behavior detection results) at 115x speed.

[0095] For example, if, in a 20-minute period in the living room, there is a total of 10 minutes of standing with movement (movement), a total of 8 minutes of out-of-home mode and non-detection time, and a total of 2 minutes of posture change behavior, then the standing with movement will normally be displayed for 5 seconds, the non-detection time for 4 seconds, and the posture change for 1 second.

[0096] As shown in Fig. 9, in the summary video (animated video), the display content of the icon 33 indicating the extracted behavior and the position of the person being watched changes according to the type of behavior and the change in position. For example, as shown in the figure, if the person is sitting, an icon 33 showing a person sitting on a sofa is displayed. If the person is lying down in the living room, an icon showing a person lying down (rather than on a futon or bed) is displayed. Furthermore, during the non-detection period and the unmanned alert mode, a message indicating that the person is out is displayed (such as an out mark or character icon).

[0097] Furthermore, for the period when movement and standing is detected, the movement trajectory is displayed at double speed based on the position information of the person being watched. For posture changes, behavior history information for a predetermined period before and after the posture change is extracted and displayed. For example, behavior history information for 10 seconds before and 10 seconds after the posture change is extracted and played back at double speed. Note that even if the behavior before or after the posture change is not included in the summary (for example, from sitting to lying down), the CPU 11 may include the posture change in the summarized video.

[0098] Fig. 10 is a flowchart showing the flow of the bedroom digest video playback process by the server device 100. Fig. 11 is a diagram conceptually showing the transition of the bedroom digest video display.

[0099] As shown in FIG. 10, the CPU 11 of the server device 100 first determines whether or not a request to play back a summary video of the bedroom has been received from the user terminal 200 via the monitoring application (step 91).

[0100] Next, CPU 11 extracts behavior history information to be included in the summary video of the bedroom that is the subject of the playback request (step 92). That is, CPU 11 extracts periods of behavior with high and low amounts of movement in the bedroom. Periods of behavior with low amounts of movement are abbreviated to a specified time (e.g., 1 minute) regardless of the total detected time. Note that instead of a specified time, a period of a specified percentage of the total time (e.g., 12 minutes of lying down, which is 10% of 2 hours) may be detected.

[0101] Next, the CPU 11 determines whether or not there is a period of a behavior type with a small amount of movement (no movement, standing, sitting, lying down) in the behavior history information based on the behavior type (step 93).

[0102] If it is determined that there is a period of an activity type with a small amount of movement (Yes in step 93), the CPU 11 omits (thins out) part of that period from the digest video (step 94).

[0103] Next, the CPU 11 determines whether or not there is a non-detection period during the night from the behavior history information (step 95).

[0104] If it is determined that there is a non-detection period during the night (Yes in step 95), the CPU 11 omits (thins out) part of the non-detection period from the digest video (step 96).

[0105] The CPU 11 then generates the extracted behavior for the predetermined period so that it is played back at double speed to match the number of seconds of the summary video (animation video), sets it to be displayed in synchronization with the time series graph 34, and transmits it to the user terminal 200 to be played back (step 97).

[0106] For example, if there are 18 minutes of standing (moving) with movement, 1 minute of lying down, and 1 minute of posture changes in a 20-minute period, the movement will be played for 8 seconds, the lying down for 1 second, and the posture changes for 1 second. Note that if there is a partial display period within the 20 minutes, the display time may be shortened to allow time to be allocated to displaying other behavioral histories, in which case priority will be given to displaying standing with movement.

[0107] As shown in Fig. 11, in the summarized video (animated video), the display contents of the icon 33 indicating the extracted behavior and the position of the person being watched are changed according to the behavior type and the change in position. For example, as shown in the figure, in the case of a sitting position, an icon 33 showing a person sitting on a sofa is displayed, and in the case of a lying position, an icon showing a person sleeping on a bed is displayed.

[0108] As described above, according to this embodiment, the lifestyle monitoring system varies the amount of thinning of the behavioral history in the summary video depending on the type of monitoring area, thereby optimizing the behavioral history of the watched person that the watcher needs to check for each monitoring area and reducing the burden of monitoring on the watcher. Furthermore, by varying the amount of thinning of non-detection periods in the summary video depending on the presence / absence information of the watched person, the behavioral history of the watched person that the watcher needs to check, including non-detection periods, can be optimized and the burden of monitoring on the watcher can be reduced. Furthermore, by checking the summary display, the watcher can notice behavior that is different from usual behavior, and by contacting the watched person, they can detect illness, etc., early on.

[0109] [Variations] The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit and scope of the present invention.

[0110] In the above-described embodiments, the server device 100 includes the standing position with movement as a behavior of the watched over person with a large amount of movement in the summary video, and thins out the standing position, sitting position, and lying position with no movement as behaviors with a small amount of movement from the summary video. However, the behavior types of the watched over person that are to be thinned out from the summary video are not limited to this.

[0111] For example, as shown in FIG. 12, the server device 100 may include in the summary video, for each posture, standing with movement and standing with movement as actions with a lot of movement, and may also determine the degree to which the actions of each posture without movement should be included in the summary video as actions with a lot of movement if it is detected that the person being watched over is exercising (radio calisthenics, swinging their arms widely, etc.) even when standing, sitting, or lying down, even though the position of the person being watched over does not change.

[0112] In each of the above-described embodiments, the server device 100 thins out the summarized video according to the monitoring area, but the monitoring area may be the living room and the bedroom. However, other rooms may be monitored as the monitoring area, and the specific behaviors included in the summary display may differ depending on the type of monitoring area. For example, if the kitchen is the monitoring area, the motionless standing position, which is the behavior of cooking, which was not included in the summary display in the living room, may be included in the summary display as the specific behavior. Furthermore, if the study is the monitoring area, the behavior of reading a book or sitting down, which was not included in the summary display in the living room, may be included in the summary display as the specific behavior.

[0113] In the above-described embodiment, the server device 100 generates a summary video by omitting (thinning out) actions other than specific actions, or by extracting representative actions of the watched person from detection results for multiple time spans to generate a summary video. Alternatively, the server device 100 may divide the action history information for a predetermined period (e.g., 3 hours) into predetermined time intervals (e.g., every 5 minutes: 12:00 to 12:05, 12:05 to 12:10, etc.) along the timeline, and extract actions to be summarized for each division to generate a summary video.

[0114] In this case, the server device 100 extracts actions with a large amount of movement (moving and standing) without thinning them out, and extracts actions for a predetermined time (e.g., 3 seconds) from, for example, 5 minutes of action history information for actions with a small amount of movement or posture changes, and thins out all other actions for the predetermined time.

[0115] For example, if the monitoring sensor 500 is a camera that takes pictures at 4 fps, the server device 100 extracts 12 frames (6 frames of the posture before the change and 6 frames of the posture after the change) from any position (for example, the start position of the period) within a 5-minute period as a partial thinning period.

[0116] In addition, in periods when there is a change in posture between standing and sitting, for example, six frames in which a sitting posture is detected are extracted, and in the case of a change from standing to sitting, the standing posture with movement is displayed followed by a summary of the six sitting posture frames, and in the case of a change from sitting to standing, the six sitting posture frames are displayed followed by a summary of the standing posture with movement.

[0117] During periods when there is no detection and the unmanned alert mode is set, there is no detection and no images of people are captured, so an icon indicating that the person is out is displayed in summary form for 12 frames.

[0118] If all actions during the 5 minutes were to be cut but there was a change in posture (sitting to lying down, etc.), 6 frames before and 6 frames after the posture change, which is the timing of the change, are extracted and displayed as a summary. Note that, as long as a summary display of the posture change is sufficient, 6 frames each before and after the posture change can be extracted from a time other than before and after the timing of the change.

[0119] In the above-described embodiment, an example has been shown in which an animated video using the icon 33 is generated as the summary video. Alternatively, a summary video may be generated by thinning out a predetermined number of frames from a video of the person being watched over taken by a camera.

[0120] In the above embodiment, the time series graph 34 is displayed together with the summary video (animated video), but it is not necessary to display the time series graph 34. When only the summary video is displayed, the display time for each behavior to be displayed in summary is determined according to the detection period of each behavior.

[0121] In the above-described embodiment, only the detection results of the monitoring sensor 500 are used to generate the summary video. However, the detection results of the intrusion monitoring sensor 400 may also be used to generate the summary video. For example, if the intrusion monitoring sensor detects something during a non-detection period, the result may be displayed on the time series graph 34.

[0122] In the above-described embodiment, the server device 100 generates a summary video, transmits it to the user terminal 200, and causes it to be displayed on the user terminal 200. However, the summary video may be generated and displayed by the monitoring app of the user terminal 200.

[0123] In other words, the monitoring app receives from the server device 100 the behavioral history of the person being watched obtained from detection data indicating the movements of the person being watched detected by the monitoring sensor 500 in the monitoring area, and area type information indicating the type of monitoring area, and based on the detection data and area type information, generates a summary video by thinning out at least a portion of the behavioral history of the person being watched over a specified period of time, with the amount of thinning depending on the type of monitoring area, and plays back the summary video.

[0124] The monitoring app may also receive from the server device 100 the behavioral history of the person being monitored obtained from detection data indicating the movements of the person being monitored by the monitoring sensor 500 in the monitoring area, and presence / absence information indicating whether the person being monitored is present or absent in the building, generate a summary video by thinning out at least a portion of the behavioral history for a specified period of time, with the amount of thinning corresponding to the presence / absence information for the specified period of time, and play the summary video.

[0125] Although only one server device 100 is shown in the above embodiment, the processing executed by the server device 100 may be distributed and executed by a plurality of servers. For example, the processing for acquiring the detection data output from the monitoring sensor 500, determining the behavior of the person being watched over, and generating behavior history information, and the processing for generating a summary video with different amounts of thinning based on the behavior history information in accordance with the monitoring area and presence or absence, may be executed by separate servers.

[0126] The display device and program according to one embodiment of the present invention can contribute to extending healthy lifespan, improving quality of life (QOL), and resolving social issues such as the increase in elderly people living alone due to a super-aging society. Furthermore, the display device and program according to one embodiment of the present invention can contribute to achieving Goal 3 of the Sustainable Development Goals (SDGs) adopted by the United Nations, "Ensure good health and well-being for all." [Explanation of symbols]

[0127] 11...CPU 16…Display section 18...Storage section 19…Communications Department 33...Icon 34...Time series graph 100...Server device 200...User terminal 300...Monitoring device 400...Intrusion monitoring sensor 500...Monitoring sensor

Claims

1. A life monitoring system that provides a watcher with information indicating the living conditions of a person being watched who resides in a building, an acquisition unit that acquires detection data that detects the movement of the person being watched in the watching area within the building; a control unit that generates a summary video by thinning out at least a portion of the behavior history of the person being watched over, from the behavior history of the person being watched over obtained from the detection data for a predetermined period of time, and outputs the summary video; The control unit generates the summarized video with a thinning amount according to the type of the watching area in which the watched-over person is detected. Life monitoring system.

2. The life monitoring system according to claim 1, The behavior history includes information representing the types of behaviors taken by the person being watched over in chronological order, The control unit determines a behavior type of the watched person during the predetermined period, and generates the summarized video with a thinning amount according to the type of the watching area, with a period in which a predetermined behavior type is detected among the determined behavior types as a thinning target. Life monitoring system.

3. The life monitoring system according to claim 2, The control unit sets a period in which a behavior type with a small amount of movement is detected as a target for thinning out. Life monitoring system.

4. The life monitoring system according to claim 3, Each of the plurality of buildings has, as the monitoring area, at least one of a first monitoring area having a sleeping space for the person being watched over and a second monitoring area having no sleeping space, The control unit generating the summarized video with a first thinning amount for a period in which the behavior type to be thinned out is detected in the first monitoring area; generating the summarized video with a second thinning amount that is greater than the first thinning amount for a period in which the behavior type to be thinned out is detected in the second monitoring area; Life monitoring system.

5. The life monitoring system according to claim 1, the detection data includes information indicating at least the posture of the person being watched over; Each of the plurality of buildings has, as the monitoring area, at least one of a first monitoring area having a sleeping space for the person being watched over and a second monitoring area having no sleeping space, The control unit generating an animation of the person being watched over sleeping for a period during which the person being watched over remained in a lying position in the first watching area, and including the animation in the summary video; A period during which the person being watched over remained in a lying position in the second watching area is omitted from the summary video. Life monitoring system.

6. The life monitoring system according to any one of claims 2 to 4, the control unit generates a time series graph that displays the determined behavior types of the watched person on a time axis in chronological order; The time series graph showing the position of the summary video during playback on a time axis is output to a display unit together with the summary video thus generated. Life monitoring system.

7. On the computer, receiving from a server device a behavior history of the person being watched obtained from detection data that detects the movements of the person being watched in a watching area within a building, and area type information that indicates the type of the watching area; generating a summary video by thinning out at least a portion of the behavior history from the behavior history for a predetermined period, with a thinning amount according to the type of the monitoring area; playing the summary video; A program that executes the following.

Images