Display control device and monitoring system

Abstract

We provide a monitoring system that offers a monitoring service that allows caregivers to observe the condition and behavior of the person being monitored. [Solution] A server device comprising: a receiving unit that receives images captured by a first communication device or a second communication device, each equipped with an imaging unit; a transmitting unit that transmits the images to a terminal device used by a user; and a storage unit that stores the user's viewing rights to the images, which are set for the first communication device and the second communication device, respectively, and the terminal device is presented with content based on the viewing rights. The display control device further comprises: a receiving unit that receives event information relating to events occurring at a monitored location; and a presentation unit that displays a history of the event information received by the receiving unit.

Description

Technical Field

[0001] The present invention relates to a display control device and a monitoring system. This application claims priority based on Japanese Patent Application No. 2021-167872 filed on October 13, 2021, and incorporates its content herein by reference.

Background Art

[0002] Conventionally, technologies related to remote monitoring devices have been disclosed (for example, see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

[0004] One embodiment of the present invention is a display control device including: a receiving unit that receives event information related to an event occurring at a location to be monitored; and a presenting unit that presents a history of the event information received by the receiving unit.

Brief Description of the Drawings

[0005] [Figure 1] It is a diagram showing an example of the device configuration of the monitoring system of this embodiment. [Figure 2] It is a diagram showing an example of the functional configuration of the monitoring system of this embodiment. [Figure 3] It is a diagram showing an example of the setting information of the monitoring system of this embodiment. [Figure 4A] It is a diagram showing an example of the initial setting screen of this embodiment. [Figure 4B] It is a diagram showing an example of the initial setting screen of this embodiment. [Figure 4C] It is a diagram showing an example of the operation screen of the emergency notification function of this embodiment. [Figure 5A]This figure shows an example of the display for each monitoring mode in this embodiment. [Figure 5B] This figure shows an example of the display for each monitoring mode in this embodiment. [Figure 6] This figure shows an example of communication in the emergency response function of this embodiment. [Figure 7] This figure shows an example of a pop-up display during an emergency response in this embodiment. [Figure 8] This figure shows an example of the live view display in this embodiment. [Figure 9] This figure shows an example of the corresponding selection screen in this embodiment. [Figure 10] This figure shows an example of the display screen for the correspondence history in this embodiment. [Figure 11] This figure shows another example of communication in the emergency response function of this embodiment. [Figure 12] This figure shows an example of the event history display in this embodiment. [Figure 13] This figure shows an example of the event selection screen for this embodiment. [Figure 14] This figure shows an example of communication using the conversation function of this embodiment. [Figure 15] This figure shows an example of communication for the live view function in the private mode of this embodiment. [Figure 16] This figure shows an example of communication for the live view function in public mode of this embodiment. [Figure 17] This figure shows an example of the operation of the monitoring system in public mode and private mode according to this embodiment. [Figure 18] This figure shows an example of communication in the voltage drop notification function of this embodiment. [Figure 19] This figure shows an example of the voltage drop notification screen of this embodiment. [Figure 20] This figure shows an example of communication in the remote control device search function of this embodiment. [Figure 21] This figure shows an example of communication in the remote control device out-of-range notification function of this embodiment.

Mode for Carrying Out the Invention

[0006] [Configuration of Monitoring System] FIG. 1 is a diagram showing an example of the device configuration of the monitoring system 1 of the present embodiment. The monitoring system 1 includes a terminal device 10, a server device 20, a communication device 30, and a remote control device 40. The monitoring system 1 provides a monitoring service for a monitor to monitor the situation of a monitored person.

[0007] In the following description, the monitored person is also referred to as the person under guardianship or the target of guardianship. The monitoring service provided by the monitoring system 1 is also referred to as a guardianship service. The monitor is also referred to as a guardian, a user of the guardianship service, or a user. That is, the monitoring system 1 provides a guardianship service for a guardian to monitor the state and actions of a person under guardianship. As an example, the person under guardianship is a non-cohabiting family member of the guardian. The monitoring system 1 provides a function that enables a guardian to remotely monitor the situation of the person under guardianship.

[0008] The terminal device 10 is a device used by a guardian, that is, a user. The terminal device 10 is, for example, a device such as a smartphone, a tablet, or a personal computer, and performs information communication with the server device 20 via the network N1. The server device 20 is a computer device operated by a provider of the guardianship service. The server device 20 performs various settings in the monitoring system 1 and holds and provides information acquired by the monitoring system 1. The communication device 30 includes, for example, an imaging unit 350 (imaging device) such as a camera, and images the monitoring target area A. The communication device 30 performs information communication with the server device 20 via the network N2. The remote control device 40 is a device used by the person under guardianship. The remote control device 40 performs information communication with the communication device 30 by short-range wireless communication, infrared communication, or the like.

[0009] Note that the monitoring system 1 may include a plurality of terminal devices 10, communication devices 30, and remote control devices 40 respectively. As an example, the monitoring system 1 can be configured to set a plurality of guardians, and each guardian uses one terminal device 10. For example, when there are siblings among the guardians, the parent who is the object to be monitored can be configured such that a plurality of guardians (siblings) monitor the parent respectively. In addition, the monitoring system 1 can be configured to install communication devices 30 in each room of a certain building to monitor the status and actions of the objects to be monitored in each room. For example, when there are a plurality of rooms in a building, the monitoring system 1 can set the first room (for example, a living room) as the monitoring target area A1 and the second room (for example, a bedroom) as the monitoring target area A2, and be configured to monitor the status and actions of the objects to be monitored in each room. In addition, the monitoring system 1 can be configured to allow a plurality of objects to be monitored living in a certain building to use the remote control device 40 respectively. For example, when the objects to be monitored are the parents of the guardian, the father and the mother can be configured to use the remote control device 40 respectively. In addition, the monitoring system 1 can monitor not only one building but also a plurality of buildings. For example, when there is a spouse of the guardian, the monitoring system 1 can set the first building where the parent of the guardian lives as the monitoring target area A1 and the second building where the parent of the spouse of the guardian lives as the monitoring target area A2, and be configured to monitor the status and actions of the objects to be monitored in each building.

[0010] In the following description, when distinguishing a plurality of terminal devices 10 respectively, the first terminal device 10 is described as the terminal device 10-1, the second terminal device 10 is described as the terminal device 10-2... Also, when distinguishing a plurality of communication devices 30 respectively, the first communication device 30 is described as the communication device 30-1, the second communication device 30 is described as the communication device 30-2... Next, referring to FIG. 2, the functional configurations of the devices of the monitoring system 1 will be described.

[0011] [Functional Configuration of the Monitoring System] Figure 2 shows an example of the functional configuration of the monitoring system 1 of this embodiment.

[0012] [Functional Configuration of Terminal Devices] The terminal device 10 includes a terminal operation detection unit 110, a display unit 120, a terminal control unit 130, a terminal communication unit 140 (receiving unit), a terminal sound pickup unit 150, and a terminal sound generation unit 160. The terminal operation detection unit 110 is equipped with an operation device such as a touch panel or an operation switch, and detects operations performed on the operation device by a user of the terminal device 10. When the terminal operation detection unit 110 detects an operation, it generates operation information indicating the detected operation and outputs it to the terminal control unit 130. The display unit 120, for example, includes a liquid crystal display and displays images based on the control of the terminal control unit 130. The terminal control unit 130, for example, includes a microcomputer and controls each part of the terminal device 10. The terminal communication unit 140 communicates information with the server device 20 based on the control of the terminal control unit 130.

[0013] The terminal sound pickup unit 150, for example, is equipped with a microphone and picks up sounds from the surrounding area of ​​the terminal device 10. The terminal sound pickup unit 150 outputs the information of the picked-up sounds to the terminal control unit 130. The terminal device 10 may operate various functions of the terminal control unit 130 (for example, voice recognition) based on the picked-up sounds. In this case, the terminal sound pickup unit 150 can also be considered an example of a terminal operation detection unit 110 that operates the terminal device 10 by sound. The terminal sound-emitting unit 160, for example, is equipped with a speaker and emits sound based on the control of the terminal control unit 130. The terminal device 10 may also use the emitted sound to alert the user (for example, by emitting an alarm sound). In this case, the terminal sound-emitting unit 160 can be said to be an example of a presentation unit 120 that presents information by sound.

[0014] [Server device functional configuration] The server device 20 includes a server communication unit 210 and a server storage unit 220. The server communication unit 210 communicates information with the terminal device 10 and with the communication device 30. The server storage unit 220 is equipped with a storage device such as semiconductor memory and stores various types of information. An example of the information stored in the server storage unit 220 will be explained with reference to Figure 3.

[0015] Figure 3 shows an example of the configuration information for the monitoring system 1 of this embodiment. As described above, in the monitoring system 1, terminal device 10 can monitor multiple monitoring targets. In this example, among the multiple monitoring targets a to c, terminal devices 10-1 and 10-2 are set to monitor targets a to c. Also, terminal device 10-3 is set to monitor targets a to b, but not target c. Furthermore, when multiple terminal devices 10 monitor a single target, it is possible to configure one terminal device 10 to function as the main monitoring device and another terminal device 10 to function as the sub-monitoring devices. For example, for target a, terminal device 10-1 functions as the main monitoring device, and terminal devices 10-2 and 10-3 function as sub-monitoring devices. In the monitoring system 1, the main monitoring device and the sub-monitoring devices can provide different functions to each other. The server memory unit 220 stores configuration information as shown in Figure 3. This configuration information is accessed as needed by the terminal device 10 or the communication device 30.

[0016] Here, the user of the main monitoring device and the user of the secondary monitoring device may be given different operational permissions. For example, the user of the main monitoring device may be given the right to view real-time video regardless of whether the monitored person has given permission, while the user of the secondary monitoring device may be prevented from viewing real-time video without the monitored person's permission. Furthermore, while we have described the operational permissions as being divided into two levels, main and sub, this is not the only way to do so; operational permissions may be set up in three or more levels. In other words, each user is granted at least one of two levels of operational authority, and the presentation unit 120 presents content based on the operational authority granted to the user.

[0017] This operational authority may be granted for each monitored area A (i.e., each monitored location). In other words, there are multiple locations to be monitored, and the user is granted operational authority for each location to be monitored. The display unit 120 of the terminal device 10 displays content based on the operational authority granted to the user for each location to be monitored.

[0018] [Functional Configuration of Communication Devices] Returning to Figure 2, the communication device 30 includes the following functions: a first communication unit 310, a second communication unit 320, a communication device control unit 330, a measurement unit 340, an imaging unit 350, a communication device sound pickup unit 360, a communication device sound generation unit 370, an operation detection unit 380, and a communication device storage unit 390.

[0019] The first communication unit 310 communicates information with the remote control device 40 based on the control of the communication device control unit 330. The second communication unit 320 communicates information with the server device 20 based on the control of the communication device control unit 330.

[0020] As mentioned above, the second communication unit 320 communicates with the terminal device 10 via the server communication unit 210 of the server device 20. However, in the following description, the explanation that the server device 20 mediates communication between the terminal device 10 and the communication device 30 may be omitted, and it may be stated, for example, that the second communication unit 320 transmits first operation information to the terminal device 10.

[0021] The measuring unit 340 measures various conditions in the monitored area A. For example, the measuring unit 340 may be equipped with a temperature sensor and a humidity sensor to measure the temperature and humidity of the monitored area A. Alternatively, the measuring unit 340 may be equipped with an infrared sensor to measure the movement of people in the monitored area A. The imaging unit 350, for example, is equipped with a camera and, based on the control of the communication device control unit 330, captures images of the area to be monitored A.

[0022] The measuring unit 340 and the imaging unit 350 are collectively referred to as the detection unit. In other words, the communication device 30 detects various conditions in the monitored area A using the detection unit. In addition to the imaging unit 350, the detection unit may also have a determination unit (not shown) that determines the state of the monitored object based on the video captured by the imaging unit 350. The determination unit, for example, analyzes the posture of the monitored object based on the results of machine learning and determines whether or not an abnormality has occurred to the monitored object, such as the object falling. The determination unit outputs the determination result to the communication device control unit 330.

[0023] The communication device control unit 330, for example, is equipped with a microcomputer and controls each part of the communication device 30, including the first communication unit 310, the second communication unit 320, and the imaging unit 350.

[0024] The communication device sound pickup unit 360, for example, is equipped with a microphone and picks up sounds from the vicinity of the communication device 30. The communication device sound pickup unit 360 outputs the sound information it has picked up to the communication device control unit 330. The communication device sound generation unit 370, for example, is equipped with a speaker and emits sound based on the control of the communication device control unit 330. The operation detection unit 380 detects operations on switches or other operating parts (not shown) provided by the communication device 30. For example, the communication device 30 is equipped with an operating mode switching switch (not shown). The operation detection unit 380 detects the setting state of the operating mode switching switch and outputs it to the communication device control unit 330.

[0025] The communication device storage unit 390 is equipped with a memory device such as a semiconductor memory and stores various types of information. For example, the communication device storage unit 390 stores information about images (e.g., moving images) captured by the imaging unit 350 and information about sounds picked up by the communication device sound pickup unit 360.

[0026] [Functional Configuration of Remote Control Device] The remote control device 40 includes an emergency button 410, an OK button 420, an NG button 480, a remote control communication unit 430, a remote control control unit 440, a remote control sound pickup unit 450, a remote control sound generation unit 460, and a voltage detection unit 470. The OK button 420 is an operation button that indicates agreement, approval, or a positive response, and is also called the "Yes button." The NG button 480 is an operation button that indicates disagreement, rejection, or a negative response, and is also called the "No button."

[0027] The emergency button 410 is a button that is operated by the person being monitored if any abnormality occurs to them. The OK button 420 is a button that is operated by the person being monitored when they grant permission when a permission request is made to them by the monitoring system 1.

[0028] The remote control communication unit 430 communicates information with the communication device 30 based on the control of the remote control control unit 440.

[0029] The remote control unit 440, for example, is equipped with a microcomputer and controls various parts of the remote control device 40. For example, when the remote control unit 440 detects an operation on the emergency button 410, it generates first operation information, which is information indicating that the emergency button 410 has been operated. The remote control unit 440 transmits the generated first operation information to the communication device 30 via the remote control communication unit 430. Similarly, when the remote control unit 440 detects an operation on the OK button 420, it generates second operation information, which is information indicating that the OK button 420 has been operated. The remote control unit 440 transmits the generated second operation information to the communication device 30 via the remote control communication unit 430.

[0030] The remote control sound pickup unit 450, for example, is equipped with a microphone and picks up sounds from the surrounding area of ​​the remote control device 40. The remote control sound pickup unit 450 outputs the sound information it has picked up to the remote control control unit 440. The remote control sound-emitting unit 460, for example, is equipped with a speaker and emits sound based on the control of the remote control control unit 440. The voltage detection unit 470 detects the voltage value of the operating power supply of the remote control device 40. The voltage detection unit 470 outputs the detected voltage value of the operating power supply to the remote control control unit 440. If the voltage value detected by the voltage detection unit 470 is below a predetermined value, the remote control control unit 440 transmits voltage drop information to the communication device 30 via the remote control communication unit 430.

[0031] [Description of each function] Next, the functions and operations provided by the monitoring system 1 will be explained with reference to the diagram. In the following explanation, the terminal device 10 has application software (hereinafter also simply referred to as "app") provided by the monitoring system 1 installed. The terminal control unit 130 controls each part of the terminal device 10 based on the installed app. The display unit 120 of the terminal device 10 displays the screen defined by the app. When the app is first launched after it has been installed on the terminal device 10, the initial setup screen is displayed on the display unit 120.

[0032] [Initial settings] Figures 4A to 4B show examples of the initial setup screens of this embodiment. Figure 4A shows an example of the notification settings screen. The monitoring system 1 provides a function to initially set the status of the monitored person that should be notified to the terminal device 10. An example of the status to be notified is shown below.

[0033] • Emergency call: When emergency button 410 is pressed. • Fall: If the person being monitored falls within the monitored area A. • Temperature and humidity anomaly: This occurs when the temperature or humidity in monitored area A is outside the specified range. • Safety confirmation abnormality: When there is no response to calls made to the person being monitored. • Camera malfunction: If a malfunction or other abnormality occurs in the imaging unit 350. • Routine deviation: When the behavior of the person being monitored differs from the usual.

[0034] The user operates the notification settings screen displayed on the display unit 120 of the terminal device 10 and selects a notification target from several types of situations. The terminal device 10 sends the selected notification settings to the server device 20. The server device 20 stores the received notification settings in the server storage unit 220. The monitoring system 1 notifies the terminal device 10 when the configured situation occurs. In this example, notification settings are described as being stored in the server storage unit 220, but this is not the only option. Notification settings only need to be able to determine whether or not a user should be notified when a certain situation occurs, and they can be stored in any of the terminal device 10, server device 20, or communication device 30. The various types of information described below can also be stored in any of the devices, as long as the function of using that information can be fulfilled.

[0035] Figure 4B shows an example of the contact settings screen. The monitoring system 1 provides an emergency notification function that contacts pre-configured contacts with minimal operation (for example, by tapping an icon displayed on the terminal device 10) in the event of an emergency.

[0036] Figure 4C shows an example of the operation screen for the emergency call function. When the above-mentioned notification target situation occurs, the display unit 120 of the terminal device 10 displays the operation screen (emergency response screen) for the emergency call function shown in Figure 4C. For example, if three emergency call destinations are set, the display unit 120 will display emergency call destination icons P11 to P13. The contacts for this emergency call function can be set using the contact settings screen shown in Figure 4B. The user operates the contact setting screen displayed on the display unit 120 of the terminal device 10 to set contacts. The terminal device 10 stores the set contacts in its storage unit (not shown).

[0037] [Monitoring Mode Settings] Figures 5A to 5B show examples of displays for each monitoring mode in this embodiment. The monitoring system 1 provides multiple monitoring modes, including a private mode and a public mode. Figure 5A shows an example of the screen displayed on the terminal device 10 in private mode. In private mode, no image of the monitored person is displayed; instead, an image P21 is displayed that shows the monitored person's status in text. Figure 5B shows an example of the screen displayed on the terminal device 10 in public mode. In public mode, an image P22 of the monitored area A captured by the imaging unit 350 of the communication device 30 is displayed. If there is a monitored person in the monitored area A, the image P22 displays a moving image (real-time image) showing the monitored person's current status. Note that in Figure 5B, for the sake of explanation, the moving image of the monitored person is represented by a line drawing. In this embodiment, the terminal device 10 displays a moving image of the monitored person actually captured by the imaging unit 350, rather than a line drawing. The private mode and the public mode can be switched using the operating mode switching switch (not shown) of the communication device 30 described above. The monitoring system 1 can provide a monitoring function that allows users to select the degree of privacy protection and the level of monitoring function according to the relationship between the person being monitored and the monitor, as well as the situation of the person being monitored, by making it possible to switch between monitoring modes. Next, we will explain each function after the initial setup.

[0038] [Monitoring function (1) Emergency response function] Figure 6 shows an example of communication in the emergency response function of this embodiment. (Step S101) If any abnormality occurs in the person being monitored, the person being monitored operates the emergency button 410 on the remote control device 40. When the remote control device 40 detects that the emergency button 410 has been operated, it transmits first operation information to the communication device 30. The first operation information is information indicating that the emergency button 410 has been operated.

[0039] (Step S102) When the communication device control unit 330 receives information from the remote control device 40 indicating that the emergency button 410 has been operated, the second communication unit 320 transmits first operation information to the terminal device 10. In the following explanation, the information that the communication device control unit 330 transmits to the terminal device 10 when the emergency button 410 is operated will also be referred to as "first operation information." This expression includes cases where the communication device control unit 330 directly transfers the first operation information received from the remote control device 40 to the terminal device 10, and cases where it transmits information generated in response to receiving the first operation information from the remote control device 40 to the terminal device 10.

[0040] In this case, if the communication device 30 includes an imaging unit 350, when the communication device control unit 330 receives first operation information from the remote control device 40, it transmits the image captured by the imaging unit 350 and the first operation information to the terminal device 10 from the second communication unit 320. In addition, the communication device 30 may include a communication device sound pickup unit 360. In this case, when the communication device control unit 330 receives first operation information from the remote control device 40, it transmits the sound information picked up by the communication device sound pickup unit 360 and the first operation information to the terminal device 10. Alternatively, the communication device control unit 330 may transmit sound information picked up by the communication device sound pickup unit 360 without transmitting the image captured by the imaging unit 350. Alternatively, the communication device control unit 330 may transmit both the image captured by the imaging unit 350 and the sound information picked up by the communication device sound pickup unit 360 to the terminal device 10. Furthermore, when the communication device 30 transmits the image captured by the imaging unit 350 and the sound picked up by the communication device's sound pickup unit 360 immediately after the timing of image capture and sound pickup, that is, in real time, this is also referred to as "live view image and sound."

[0041] Hereinafter, the communication device 30 will be described as comprising an imaging unit 350 and a communication device sound pickup unit 360, and transmitting live view images and audio to the terminal device 10.

[0042] Furthermore, the sound from the monitored area A can also be picked up by the remote control sound pickup unit 450, which is installed in the remote control device 40 operated by the person being monitored. In this case, the sound picked up by the remote control sound pickup unit 450 is transmitted from the remote control communication unit 430 to the communication device 30, whereupon it is acquired. However, after the emergency button 410 is operated, communication between the remote control device 40 and the communication device 30 may be interrupted for some reason. Since the communication device 30 is equipped with a communication device sound pickup unit 360, even if communication between the communication device 30 and the remote control device 40 is not possible, the communication device 30 can pick up ambient sounds and provide live view images and audio to the terminal device 10.

[0043] (Step S103) When the terminal device 10 receives the first operation information and the live view image and sound from the communication device 30, it displays an image on the display unit 120 indicating that the emergency button 410 has been operated.

[0044] Figure 7 shows an example of a pop-up display during an emergency response in this embodiment. When the terminal device 10 receives first operation information from the communication device 30, it displays a pop-up image P31 as shown in Figure 7. With the monitoring system 1 configured in this way, even if the user is not operating the terminal device 10 or is operating an application other than the monitoring system 1 app, the user can be notified that the person being monitored has operated the emergency button 410.

[0045] Figure 8 shows an example of the live view display in this embodiment. (Step S104) When the pop-up image P31 displayed in step S103 is operated (for example, tapped), the terminal device 10 displays the live view image P41, the corresponding selection icon P42, and the conversation icon P43. Note that in Figure 8, for the sake of explanation, the video of the person being monitored in the live view image P41 is represented by a line drawing. In this embodiment, however, the terminal device 10 displays the video of the person being monitored that is actually being captured by the imaging unit 350, rather than a line drawing.

[0046] Figure 9 shows an example of the corresponding selection screen in this embodiment. (Step S105) When the corresponding selection icon P42 is operated, the terminal device 10 displays the corresponding selection screen P51 shown in Figure 9.

[0047] Here, the first operation information is an example of event information indicating that the emergency button 410 was operated. The response selection screen P51 contains icons for the corresponding response depending on the type of event. In other words, the event information includes the type of event, and the presentation unit 120 presents content corresponding to the type of event included in the received event information.

[0048] When terminal device 10 receives the first operation information, that is, when the event type is "operation of emergency button 410", terminal device 10 will, for example, • The user rushes to the location where the person being monitored is (Icon P511) • Report to public authorities (fire / ambulance, police, public health center, government office, etc.) (icon P512) • Contact a pre-set contact (icon P513) The corresponding selection screen P51 is displayed, which includes various icons such as the option to launch the phone application (icon P514). In other words, the response includes at least one of the following: the user moving to the monitored location, notification to a public institution, or notification to other users. When any of the icons on the response selection screen P51 is operated, the terminal device 10 performs the action corresponding to the operated icon (for example, making a call to a designated contact). Furthermore, the terminal device 10 transmits the selected response to the server device 20.

[0049] In other words, when the terminal communication unit 140 receives event information, the display unit 120 displays an icon indicating the event information as a pop-up, and then, based on the user's actions after the icon is displayed, it displays content corresponding to the type of event.

[0050] Furthermore, as mentioned above, if the communication device 30 has a judgment unit (not shown), and this judgment unit determines whether or not an abnormality has occurred based on the posture analysis results of the person being monitored, then the judgment result of the judgment unit, "the person being monitored has fallen," can also be said to be an example of event information.

[0051] (Step S106) Returning to Figure 6, when the server device 20 receives the response information from the terminal device 10, it stores the received response information, along with the time information transmitted by the terminal device 10, in the server storage unit 220 as a response history. The server device 20 also transmits information to the communication device 30 indicating that the user has responded to the operation of the emergency button 410. (Step S107) The communication device 30 releases the emergency state and returns to normal operation.

[0052] [Information sharing with other terminal devices] (Step S108) Other terminal devices 10 (for example, terminal devices 10-2 and 10-3) obtain the correspondence history stored in step S106 from the server device 20.

[0053] Here, the monitoring system 1 provides a function that allows other terminal devices 10 to share the selection result of the response content selected in step S105. For example, if the terminal device 10 that selected the response content in step S105 is terminal device 10-1, other terminal devices 10 (for example, terminal devices 10-2 and 10-3) can obtain that selection result (i.e., the user's response content of terminal device 10-1 to the operation of the emergency button 410).

[0054] In other words, event information is transmitted to multiple users, and the display unit 120 displays the status of how one of the multiple users has responded to the event information.

[0055] Figure 10 shows an example of the display screen for the correspondence history in this embodiment. The terminal device 10 displays the correspondence history screen P61 acquired from the server device 20. This correspondence history screen P61 includes the name of the user who was corresponded to, the time of correspondence, and the content of correspondence.

[0056] In other words, the display unit 120 of the terminal device 10 displays a screen for selecting from multiple options (option selection screen P51) and displays the content of the option selected by one of the multiple users.

[0057] As described above, multiple destination terminal devices 10 can be configured for a single communication device 30. In this case, when the communication device 30 receives event information (for example, first operation information) from the remote control device 40, it transmits the event information to each of the configured terminal devices 10. At this time, if a user of any terminal device 10 takes some action in response to event information (for example, rushing to the location of the person being monitored in response to the operation of the emergency button 410), this information is shared among users of multiple terminal devices 10, allowing each user to determine whether further action is necessary. A monitoring system 1 configured in this way can provide users with a sense of security.

[0058] [Video display when the emergency button is pressed] Furthermore, the monitoring system 1 may also provide a function to display images captured by the imaging unit 350 during the time period before and after the operation of the emergency button 410 on the video display screen P62 shown in Figure 10.

[0059] Figure 11 shows another example of communication in the emergency response function of this embodiment. (Step S101) As explained with reference to Figure 6, if any abnormality occurs in the person being monitored, the person being monitored operates the emergency button 410 on the remote control device 40. When the remote control device 40 detects that the emergency button 410 has been operated, it transmits first operation information to the communication device 30.

[0060] (Step S201) In this embodiment, the communication device 30 continuously performs imaging by the imaging unit 350 and sound collection by the communication device sound collection unit 360, regardless of whether the emergency button 410 is operated or not. The communication device control unit 330 stores the captured image and the collected sound in the communication device storage unit 390. Depending on the capacity of the communication device storage unit 390, the communication device control unit 330 may erase information captured and recorded in the past period and store new images and sounds. Here, the predetermined period is, for example, 12 hours, 24 hours, or 7 days. (Step S202) When the operation of the emergency button 410 is detected in step S101, the communication device 30 stores in the communication device storage unit 390, based on a predetermined recording time, the images and sounds from the most recent period among the past images and sounds stored in the communication device storage unit 390, as well as images and sounds captured and recorded after the operation of the emergency button 410 was detected. For example, if the predetermined recording time is 10 minutes, the communication device 30 stores in the communication device storage unit 390 the images and sounds from the 5 minutes before and 5 minutes after the operation of the emergency button 410 was detected.

[0061] (Step S203) The terminal device 10 requests images and audio from the communication device 30 before and after the operation of the emergency button 410 is detected. For example, the user taps the play button image on the video display screen P62 shown in Figure 10. When the terminal device 10 detects the tap on the play button image on the video display screen P62, it requests images and audio from the communication device 30 before and after the operation of the emergency button 410 is detected. (Step S204) When the communication device 30 receives a request for images and sounds from before and after the operation of the emergency button 410 was detected, it transmits the images and sounds from the time period before and after the operation of the emergency button 410 was detected, which were saved in step S202, to the terminal device 10. (Step S205) When the terminal device 10 receives images and sounds from the communication device 30 for the time period before and after the operation of the emergency button 410 is detected, it plays back the received images and sounds.

[0062] Information indicating the operation of the emergency button 410 (first operation information) is an example of event information. The event information includes information indicating a video within a time range that includes the time the event occurred, and the display unit 120 displays the video. Here, the time range that includes the time the event occurred is, for example, 10 minutes, which is 5 minutes before and 5 minutes after the operation of the emergency button 410 as described above. Note that the time width and the ratio of before and after shown here are just examples, and may be longer or shorter than 10 minutes, and the ratio of before and after time may be equal or unequal, with the before (or after) time being longer. Furthermore, the information indicating the video may be the video file itself, or it may be link information indicating the location where the video is saved.

[0063] With the monitoring system 1 configured in this way, the situation before the emergency button 410 was pressed, that is, the situation during the time period in which the cause that led to the pressing of the emergency button 410 occurred, can be displayed to the terminal device 10. As a result, the user can understand the cause of the abnormality that occurred in the person being monitored and take more appropriate action.

[0064] [Display of emergency button operation history] Figure 12 shows an example of the event history display in this embodiment. As described above, the terminal device 10 includes a display unit 120 and a terminal communication unit 140. The terminal communication unit 140 receives event information regarding events that occurred at the monitored location. This event information includes, for example, first operation information indicating that the emergency button 410 was operated. The display unit 120 displays the history of event information received by the terminal communication unit 140. For example, the display unit 120 of the terminal device 10 displays past events among the notifiable events on the event history screen P71, along with the date and time of their occurrence. With the monitoring system 1 configured in this way, not only can notifications be sent when an event occurs (for example, when the emergency button 410 is operated), but past events can also be presented to the user. Furthermore, the terminal device 10 may have a function to exclude (i.e., delete) events selected by the user from the event history. The terminal device 10 may also display the timing of events in a calendar format, or display detailed information about events that occurred on a date selected from the calendar.

[0065] Here, the event information includes the time the event occurred. The display unit 120 displays the history of the event information based on the time the event occurred. Here, the display unit 120 can also be configured to display a history of event information based on the time the event information is received by the terminal device 10. However, since the event information is transmitted from the remote control device 40 to the terminal device 10 via the communication device 30 and the server device 20, there may be a discrepancy between the time the event information is received and the time the event occurs, depending on the communication conditions. Also, for example, if multiple events occur, the order in which the events occur and the order in which the event information is received may be reversed. According to the monitoring system 1 configured to present a history of event information based on the time of event occurrence, as in this embodiment, the event occurrence history can be presented more accurately regardless of the communication status.

[0066] Furthermore, as described above, a communication device 30 equipped with an imaging unit 350 is installed in the monitored area A. The event information transmitted to the terminal device 10 (in the example above, information indicating that the emergency button 410 was operated; in other words, first operation information) includes information about the image captured by the imaging unit 350 (for example, a thumbnail image or video). The presentation unit 120 presents this image information.

[0067] Furthermore, the terminal device 10 may mark events shown on the event history screen P71 as "unread" (P72) for events whose content has not been viewed by the user, i.e., unread events. In other words, the display unit 120 may indicate that the event information has not been viewed by the user.

[0068] Furthermore, with respect to an unopened event, if a predetermined time has elapsed since the event occurred, the terminal device 10 may notify the user that the predetermined time has elapsed by changing the color of the unopened indicator P72, displaying a pop-up image, making a sound, vibrating, etc. In other words, the display unit 120 indicates that a predetermined amount of time has elapsed since the event information was viewed by the user.

[0069] Furthermore, the terminal device 10 may change the color and shape of the unopened indicator P72 depending on the type of event. In other words, the display unit 120 may display different forms depending on the type of event indicated by the event information.

[0070] Figure 13 shows an example of the event selection screen for this embodiment. The terminal device 10 may narrow down the notifications when an event occurs based on the user's settings on the event selection screen P81. The presentation unit 120 selects the event information to be presented based on at least one of the event occurrence time and event type.

[0071] With the monitoring system 1 configured as described above, users can easily understand the situation when an event occurs, and they can take more appropriate action in response to the event.

[0072] [Monitoring function (2): Conversation function using a remote control device] The monitoring system 1 provides a conversation function using the remote control device 40. For example, by tapping the conversation icon P43 shown in Figure 8 above, the user can have a conversation with the person being monitored.

[0073] As described above, the remote control device 40 is equipped with a remote control sound pickup unit 450. When a conversation request is received from the terminal device 10, the communication device control unit 330 transmits information about the sound picked up by the remote control sound pickup unit 450 to the terminal device 10 based on the second operation information. Here, the second operation information is information indicating that the OK button 420 has been pressed. That is, when the remote control device 40 detects an operation on the OK button 420, it transmits the second operation information. Let's explain the more specific steps for using the conversation function.

[0074] Figure 14 shows an example of communication using the conversation function of this embodiment. (Step S301) When the terminal device 10 detects a tap on the conversation icon P43, it sends a conversation request to the communication device 30. (Step S302) When the communication device 30 receives a conversation request from the terminal device 10, the communication device sound-emitting unit 370 emits a conversation request sound. The person being monitored hears the conversation request sound, realizes that a conversation has been requested, and decides whether or not to allow the conversation. If the person being monitored allows the conversation, they operate the OK button 420 on the remote control device 40. (Step S303) When the remote control device 40 detects an operation on the OK button 420, it notifies the terminal device 10 via the communication device 30 and the server device 20 that the OK button 420 has been operated. Here, the information indicating that the OK button 420 has been operated is also called the second operation information. That is, when the remote control device 40 detects an operation on the OK button 420, it transmits the second operation information.

[0075] (Step S304) When the terminal device 10, server device 20, communication device 30, and remote control device 40 detect that permission to the conversation request has been granted in step S303, they start a two-way conversation. Here, the two-way conversation function is a function that enables two-way conversation between the user and the person being monitored by having the sound picked up by the terminal sound pickup unit 150 of the terminal device 10 be emitted from the communication device sound output unit 370 of the communication device 30, and the sound picked up by the remote control sound pickup unit 450 of the remote control device 40 be emitted from the terminal sound output unit 160 of the terminal device 10. In the two-way conversation function, the monitoring system 1 may display the image of the person being monitored during the conversation, captured by the imaging unit 350, on the terminal device 10. (Step S305) The communication device 30 captures an image of the monitored area A using the imaging unit 350. (Step S306) The remote control device 40 receives the voice of the person being monitored using the remote control sound receiving unit 450. (Step S307) The remote control device 40 transmits the collected audio as audio information to the communication device 30. (Step S308) The communication device 30 transmits the image captured in step S305 and the audio information received from the remote control device 40 in step S307 to the terminal device 10. (Step S309) The terminal device 10 presents the image and sound received from the communication device 30 to the user. (Step S310) In parallel with steps S305 to S309, the terminal device 10 transmits the sound picked up by the terminal sound pickup unit 150 as voice information to the communication device 30. (Step S311) The communication device 30 receives voice information from the terminal device 10 and emits it from the communication device sound-producing unit 370.

[0076] As described above, the monitoring system 1 uses the remote control sound pickup unit 450 of the remote control device 40 carried by the person being monitored to pick up the voice of the person being monitored. This allows for clearer voice pickup compared to, for example, using the communication device sound pickup unit 360 of the communication device 30. Furthermore, the monitoring system 1 uses the communication device sound output unit 370 of the communication device 30 to produce sound. This allows for louder or better-sounding sound compared to, for example, using the remote control sound output unit 460 of the remote control device 40. Therefore, the monitoring system 1 can improve the quality of conversations between the user and the person being monitored.

[0077] As described above, the remote control device 40 is equipped with an OK button 420. This OK button 420 is a different operation button from the emergency button 410. The remote control device 40 transmits second operation information indicating the operation of the OK button 420. The communication device control unit 330 of the communication device 30 controls the type of information to transmit to the terminal device 10 based on the receipt of the second operation information from the remote control device 40. In the example described above, when the OK button 420 is operated, the communication device control unit 330 controls the type of information to transmit to the terminal device 10 by transmitting an image and sound to the terminal device 10.

[0078] [Monitoring function (3) Live view function] The monitoring system 1 provides a live view function that displays the current image and sound of the monitored area A to the terminal device 10 in real time. In the live view function, the terminal device 10 displays, for example, the live view image P41 shown in Figure 8 above. In the aforementioned conversation function, the remote control sound pickup unit 450 of the remote control device 40 picked up the sound, but in the live view function, the communication device sound pickup unit 360 of the communication device 30 picked up the sound, which is a difference from the conversation function described above. Furthermore, the live view function operates differently in private mode and public mode. In private mode, permission from the monitored person is required. An example of this procedure is described below.

[0079] <Live View Function: Private Mode> Figure 15 shows an example of communication for the live view function in the private mode of this embodiment. (Step S401) When the terminal device 10 detects an operation in which the user requests the live view function (for example, selecting the "Live" tab or the "Current Status" tab), it sends a live view request to the communication device 30. (Step S402) When the communication device 30 receives a live view request from the terminal device 10, the communication device sound unit 370 emits a live view request sound. The person being monitored hears the live view request sound, realizes that a live view request has been made, and decides whether or not to grant permission for live view. If the person being monitored grants permission for live view, they operate the OK button 420 on the remote control device 40. (Step S403) When the remote control device 40 detects an operation on the OK button 420, it notifies the terminal device 10 via the communication device 30 and the server device 20 that the OK button 420 has been operated. Here, the information indicating that the OK button 420 has been operated is also called the second operation information. That is, when the remote control device 40 detects an operation on the OK button 420, it transmits the second operation information. Furthermore, in the conversation function described above, the remote control device 40 also transmits second operation information. Depending on the requested function, the meaning of the operation of the OK button 420, that is, the meaning of the second operation information, changes.

[0080] (Step S404) The communication device 30 captures an image of the monitored area A using the imaging unit 350. The communication device 30 also captures sound from the monitored area A using the sound-collecting unit 360. (Step S405) The communication device 30 transmits the image and audio information to the terminal device 10. (Step S406) The terminal device 10 presents the image and audio received from the communication device 30 to the user.

[0081] <Live View Function: Public Mode> Figure 16 shows an example of communication for the live view function in public mode of this embodiment. In public mode, the communication device 30 continuously captures images of the monitored area A using the imaging unit 350.

[0082] (Step S501) When the terminal device 10 detects an operation in which the user requests the live view function (for example, selecting the "Live" tab or the "Current Status" tab), it sends a live view request to the communication device 30. (Step S502) When the communication device 30 receives a live view request from the terminal device 10, the communication device sound unit 370 emits a live view request sound. The person being monitored hears the live view request sound, realizes that a live view request has been made, and decides whether or not to grant permission for live view. If the person being monitored grants permission for live view, they operate the OK button 420 on the remote control device 40. (Step S503) When the remote control device 40 detects an operation on the OK button 420, it notifies the terminal device 10 via the communication device 30 and the server device 20 that the OK button 420 has been operated.

[0083] (Step S504) The communication device 30 collects sound from the monitored area A using the communication device sound collection unit 360. (Step S505) The communication device 30 transmits the image and audio information to the terminal device 10. (Step S506) The terminal device 10 presents the image and audio received from the communication device 30 to the user.

[0084] A live view request is also called an image transmission request. That is, when an image transmission request is received from the terminal device 10, the communication device control unit 330 of the communication device 30 transmits the image captured by the imaging unit 350 to the terminal device 10 based on the second operation information.

[0085] Here, the differences in the operation of monitoring system 1 between public mode and private mode will be explained with reference to the diagram.

[0086] Figure 17 shows an example of the operation of the monitoring system 1 in public mode and private mode according to this embodiment. In Figure 17, "OFF" indicates that the button is not operated, and "ON" indicates that the button is operated. Regarding images, in public mode, the image captured by the imaging unit 350 is displayed to the terminal device 10 regardless of whether the emergency button 410 or the OK button 420 is operated. On the other hand, in private mode, the image captured by the imaging unit 350 is displayed to the terminal device 10 when the emergency button 410 is operated, or when the OK button 420 is operated in response to a conversation request or a live view request (i.e., when the person being monitored gives permission). In other words, in private mode, if neither the emergency button 410 nor the OK button 420 is operated (OFF), no image is displayed. Regarding audio, in both public and private modes, the recorded audio is presented to the terminal device 10 when the emergency button 410 is operated, or when the OK button 420 is operated in response to a conversation request or live view request (i.e., when the person being monitored gives permission). In other words, regarding audio, in both public and private modes, it is not presented when neither the emergency button 410 nor the OK button 420 is operated (when they are OFF).

[0087] Regarding audio, in both public and private modes, when the emergency button 410 is operated, the sound picked up by the communication device sound pickup unit 360 of the communication device 30 is presented. Also, in both public and private modes, when the OK button 420 is operated (i.e., when the person being monitored gives permission), if a conversation is requested, the sound picked up by the remote control sound pickup unit 450 is presented, and if a live view is requested, the sound picked up by the communication device sound pickup unit 360 is presented.

[0088] As mentioned above, the information indicating the operation of the OK button 420, that is, the second operation information, changes in meaning depending on the content of the user's request and the operating mode. The second operation information is information indicating an operation that authorizes providing at least one of the sound information and the captured image to the terminal device 10 in a non-public mode in which the image captured by the imaging unit 350 is not made public.

[0089] Furthermore, the second operation information is information indicating an operation that authorizes providing the information of the sound picked up by the imaging unit 350 to the terminal device 10 in the public mode in which the image captured by the imaging unit 350 is made public.

[0090] [Monitoring function (4) Voltage drop notification function for remote control device] Figure 18 shows an example of communication in the voltage drop notification function of this embodiment. (Step S601) The voltage detection unit 470 of the remote control device 40 detects the power supply voltage of the remote control device 40. If the detected power supply voltage is below a predetermined value, the remote control control unit 440 sends a voltage drop notification to the terminal device 10. That is, when the communication device control unit 330 receives voltage drop information from the remote control device 40, it sends information to the terminal device 10 indicating that the power supply voltage of the remote control device 40 has dropped. (Step S602) The terminal device 10 receives a voltage drop notification. (Step S603) The terminal device 10 notifies the user of the voltage drop.

[0091] Figure 19 shows an example of the voltage drop notification screen in this embodiment. In step S603, the terminal device 10 displays the voltage drop notification screen P91 shown in Figure 19. Furthermore, if the power supply voltage of the remote control device 40 is restored by replacing the batteries or charging the device, the remote control device 40 may send a voltage restoration notification to the communication device 30 or terminal device 10. The communication device 30 or terminal device 10 may manage the period from when it receives the voltage drop notification until when it receives the voltage restoration notification as the power supply voltage drop period of the remote control device 40.

[0092] [Monitoring function (5) Remote control device search function] The monitoring system 1 provides a remote control device search function that, if the remote control device 40 is lost, sounds the remote control device 40 to assist in the search for the remote control device 40.

[0093] Figure 20 shows an example of communication in the remote control device search function of this embodiment. (Step S701) The communication device 30 is equipped with a remote control device search switch (not shown). The communication device 30 detects the operation of the remote control device search switch. This remote control device search switch is an example of an operation receiving unit that accepts an operation to search for the remote control device 40. (Step S702) When the operation of the remote control device search switch is detected, the communication device 30 transmits a remote control device search instruction to the remote control device 40. (Step S703) When the remote control device 40 receives a remote control device search command from the communication device 30, it emits a sound from the remote control sound-emitting unit 460 to notify the person being monitored or the user of its location.

[0094] The remote control device 40 may also be equipped with a vibrating unit (not shown) and a light-emitting unit (not shown), and may notify its location by vibration or light when it receives a remote control device search instruction from the communication device 30.

[0095] [Monitoring function (6) Remote control device out of range notification function] The monitoring system 1 provides a remote control device out-of-range notification function that notifies the terminal device 10 when the remote control device 40 is outside the communication range of the communication device 30.

[0096] Figure 21 shows an example of communication in the remote control device out-of-range notification function of this embodiment. (Step S801) The communication device 30 determines whether or not it can communicate with the remote control device 40. The communication device 30 may perform this determination periodically or in response to a request from the terminal device 10. If the communication device 30 cannot communicate with the remote control device 40, it determines that the remote control device 40 is outside the communication range and transmits remote control range out-of-range information to the terminal device 10. Furthermore, the communication device 30 may, in combination with the voltage drop notification function of the remote control device described above, determine that the remote control device 40 is outside the communication range if the power supply voltage of the remote control device 40 has not dropped and communication with the remote control device 40 is not possible.

[0097] (Step S802) The terminal device 10 receives remote control out-of-range information from the communication device 30. (Step S803) The terminal device 10 informs the user that the remote control device 40 is outside the communication range of the communication device 30.

[0098] As explained above, monitoring system 1 can provide a monitoring system that is convenient for both the monitor and the monitored.

[0099] Although embodiments of the present invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments, and modifications can be made as appropriate without departing from the spirit of the present invention. The configurations described in each of the above embodiments may also be combined.

[0100] Furthermore, each component of the device in the above embodiment may be implemented by dedicated hardware, or it may be implemented by memory and a microprocessor.

[0101] Furthermore, each component of each device may consist of memory and a CPU (Central Processing Unit), and its functions may be realized by loading a program into memory and executing it to realize the functions of each component of each device.

[0102] Alternatively, programs for realizing the functions of each part of each device may be recorded on a computer-readable recording medium, and the programs recorded on this recording medium may be loaded into a computer system and executed to perform processing by the parts of the control unit. The term "computer system" here includes hardware such as the operating system and peripheral devices.

[0103] Furthermore, "computer system" shall also include the homepage provisioning environment (or display environment) if a WWW system is being used. Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. In addition, "computer-readable recording media" also includes those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs over networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period of time, such as volatile memory inside computer systems that act as servers or clients in such cases. Moreover, the above-mentioned programs may be for the purpose of realizing some of the functions described above, and may also be able to realize the above-mentioned functions in combination with programs already recorded in the computer system. [Explanation of Symbols]

[0104] 1…Monitoring system, 10…Terminal device, 110…Terminal operation detection unit, 120…Display unit, 130…Terminal control unit, 140…Terminal communication unit, 150…Terminal sound pickup unit, 160…Terminal sound generation unit, 30…Communication device, 310…First communication unit, 320…Second communication unit, 330…Communication device control unit, 340…Measurement unit, 350…Imaging unit, 360…Communication device sound pickup unit, 370…Communication device sound generation unit, 380…Operation detection unit, 390…Communication device memory unit, 40…Remote control device, 410…Emergency button, 420…OK button, 430…Remote control communication unit, 440…Remote control unit, 450…Remote control sound pickup unit, 460…Remote control sound generation unit, 470…Voltage detection unit, 480…NG button

Claims

[Claim 1] A receiving unit that receives an image captured by at least one of the first communication device or the second communication device, each of which is equipped with an imaging unit, A transmission unit that transmits the aforementioned image to a terminal device used by the user, A storage unit that stores the user's viewing rights to the image, which are set for each of the first and second communication devices, Equipped with, The terminal device is a server device on which content based on the aforementioned viewing permissions is presented.

Images