System, information processor, and program

Abstract

To enable more appropriate watching by presenting information about a person P to be watched to a user U while respecting the privacy of the person P.SOLUTION: A watching system (100) includes: an acquisition device (10) that acquires information including at least one of an image and audio of the person (P); a user terminal (30) where the information is output; and a server device (20) that is an information processor having a state detection unit (211) that detects a state based on the information, a setting unit (212) that sets a privacy protection level according to the information output to the user terminal (30), an abstraction processing unit (213) that performs abstraction processing on the information, and an output unit (214) that outputs the information to the user terminal (30).SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a system, an information processing device, and a program. [Background technology]

[0002] In recent years, with the aging of society, there has been a demand for improved care and monitoring services for the elderly, especially for elderly people living alone in their homes (elderly people living alone). To this end, technologies have been developed that allow family members or caregivers to remotely monitor elderly people and other subjects using surveillance cameras capable of capturing images and audio (see, for example, Patent Documents 1 and 2).

[0003] Patent Document 1 discloses a technology related to an at-home care support system that connects a support center device and a care recipient device via a line and bidirectionally communicates audio data and image data. The technology disclosed in Patent Document 1 enables the transmission of image data to the support center device to be stopped, and the transmission of audio data only or the transmission of audio data and image data to be stopped depending on the number of times the care recipient presses a privacy button provided on the care recipient device. This prevents the audio and images from the care recipient device from being transferred regardless of the care recipient's wishes, thereby preventing the privacy of the care recipient from being violated.

[0004] Patent Document 2 discloses a bathroom surveillance system that blurs images of bathers captured by a camera installed in the bathroom, using techniques such as mosaic or blurring, before displaying them on a display device outside the bathroom. Audio collected by a microphone installed in the bathroom is output from a speaker outside the bathroom. The technology disclosed in Patent Document 2 can adjust the blurring level based on the privacy protection level set according to the bather's age (e.g., infant, elementary school student, elderly person) and the need for monitoring within the bathroom. Furthermore, the audio output from the speaker can be switched on and off depending on the blurring level. This makes it possible to detect accidents in the bathroom without compromising the bather's privacy.

[0005] Incidentally, when monitoring a subject, especially an elderly person, on a daily basis, it is sufficient to know that the subject is responding and that they are doing well, and images and audio that involve privacy are not necessary. However, when there is no response from the subject or when there is an abnormality such as poor health, it is desirable to check the subject's image and audio. In this case, too, it is desirable to be able to check the image and audio by switching the blurring level etc. according to the elderly person's condition so as not to violate their privacy. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 10-234796 [Patent Document 2] Patent No. 3798271

[0007] However, with the technology disclosed in the above-mentioned Patent Document 1, the person requiring care simply switches between sending and stopping image data and audio data, and if the sending is permitted, the image data and audio data are sent without any processing. Also, with the technology disclosed in Patent Document 2, the level of privacy protection is not changed based on the condition of the person, so there is room for improvement. Summary of the Invention

[0008] According to a first aspect, the system includes an acquisition unit that acquires information about the subject, including at least one of an image or voice of the subject, a status detection unit that detects the status of the subject based on the information acquired by the acquisition unit, and a setting unit that sets a level to protect the privacy of the subject, and the setting unit gradually changes the level based on the status of the subject. According to a second aspect, an information processing device includes a receiving unit that receives information about a subject, including at least one of an image and audio of the subject, a status detection unit that detects the status of the subject based on the information, and a setting unit that sets a level for protecting the privacy of the subject, and the setting unit gradually changes the level based on the status of the subject. According to a third aspect, the program is a program executed by a computer, causing the computer to function as a receiving unit that receives information about the subject, including at least one of an image and audio of the subject, a status detection unit that detects the status of the subject based on the information, and a setting unit that sets a level to protect the privacy of the subject, and causes the setting unit to function to gradually change the level based on the status of the subject. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a conceptual diagram showing an example of the overall configuration of a monitoring system according to a first embodiment. [Figure 2] 1 is a functional block diagram of a monitoring system according to a first embodiment. [Figure 3A] FIG. 10 is a schematic diagram illustrating an example of the data structure of a privacy protection level table. [Figure 3B] FIG. 4 is a schematic diagram illustrating an example of a data structure of an information setting table. [Figure 3C] FIG. 10 is a schematic diagram illustrating an example of a data structure of a periodic check information table. [Figure 4A] 10 is an example of an image output to a user terminal, showing an image in which a person's image is replaced with an avatar. [Figure 4B] 10 is an example of an image output to a user terminal, showing an image in which a pixelated image of a person has been applied. [Figure 4C] 10 is an example of an image output to a user terminal, showing an image in which a person's image has not been subjected to abstraction processing. [Figure 5A] FIG. 10 is a diagram illustrating an example of a call screen output to a user terminal. [Figure 5B] FIG. 10 is a diagram for explaining an example of a first information display screen output to a user terminal. [Figure 5C] FIG. 10 is a diagram for explaining an example of a second information display screen output to a user terminal. [Figure 5D] FIG. 10 is a diagram illustrating an example of a third information display screen output to a user terminal. [Figure 5E] FIG. 10 is a diagram for explaining an example of a fourth information display screen output to a user terminal. [Figure 6] 10 is a flowchart for explaining an example of the flow of a monitoring operation in a "special mode" in the monitoring system according to the first embodiment. [Figure 7] 10 is a flowchart for explaining an example of the flow of a monitoring operation in a "special mode" in a monitoring system according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] (First embodiment) A monitoring system 100 as one embodiment of the system of the present disclosure will be described below with reference to the drawings. Fig. 1 is a conceptual diagram showing an example of the overall configuration of the monitoring system 100 according to the first embodiment, and Fig. 2 is a functional block diagram of the monitoring system 100.

[0011] 1 and 2, the monitoring system 100 according to this embodiment includes an acquisition device 10 as an acquisition unit, a server device 20 as an information processing device, and a user terminal 30 as a terminal. The acquisition device 10, the server device 20, and the user terminal 30 are connected to each other via a communication network N such as the Internet or a mobile phone network, either wired by a connection cable or wirelessly by Wi-Fi (registered trademark) or the like, so that data can be transmitted and received between them.

[0012] In this embodiment and the following embodiments, the monitoring system 100 is described as a system for monitoring a target P, such as an elderly person living alone at home or a person requiring care, by a remote person (i.e., a user U of the monitoring system 100) who is the target P's child or the like and who acts as the monitor. The monitoring system 100 of this embodiment constantly monitors the target P, notifying the user U that there is no abnormality if there is no abnormality, and notifying the user U of the abnormality if there is an abnormality. This normal monitoring is hereinafter referred to as the "normal mode." The monitoring system 100 of this embodiment also performs special monitoring according to a level of privacy protection for the target P (hereinafter referred to as the "privacy protection level"), based on an instruction from the user U after notifying the user U of the abnormality, or based on an instruction of the user U's own will, regardless of whether there is an abnormality or not. Hereinafter, this special monitoring is referred to as the "special mode." Monitoring in the "special mode" is a special function within the monitoring in the "normal mode." There are preferably multiple privacy protection levels (levels), and one of the privacy protection levels is set.

[0013] Note that the target person P is not limited to elderly people living alone, but can be one or both of an elderly couple living together, a person requiring care, or an elderly person or person requiring care in a nursing home or care facility. Even if the target person P lives with their family, they can also be elderly people who spend time alone during the day. Furthermore, the target person P is not limited to elderly people or people requiring care, but can also be an elementary school student in after-school care.

[0014] Furthermore, the user U is not limited to the child of the subject P, but can also be a parent, grandchild, sibling, other relative, or even a guardian. The user U is not limited to one person, but multiple users U can monitor the subject P. The user U is not limited to a relative of the subject P, but can also be a third party such as a caregiver such as a home helper, a facility staff member, a hospital doctor, or a nurse. Both a relative and a third party can also be the user U.

[0015] The acquisition device 10 acquires information including at least one of an image and a voice of the subject P (hereinafter, sometimes referred to as "subject information"). In this embodiment, the acquisition device 10 acquires both an image and a voice as the subject information. In this embodiment, the acquisition device 10 repeatedly acquires the subject information based on an input from the user terminal 30 via the server device 20, and outputs the acquired information to the server device 20. In addition, the acquisition device 10 outputs the image and voice of the user U input from the user terminal 30 via the server device 20 to the subject P.

[0016] The acquisition device 10 is installed in the home of the subject P. The acquisition device 10 is preferably installed in a place where the subject P spends a lot of time or a place that the subject P uses daily. Suitable installation locations for the acquisition device 10 include, for example, a living room, kitchen, bedroom, and washroom. In the case of a subject P who spends a lot of time in bed, such as a physically disabled person or a bedridden person, a bedroom is particularly suitable. Furthermore, in consideration of the possibility of a person falling in a bathroom or toilet, or losing consciousness due to a sudden change in temperature, suitable installation locations for the acquisition device 10 include a bathroom, a dressing room, a toilet, an entrance, a hallway, near stairs, and the like.

[0017] The acquisition device 10 includes an imaging unit 11, a sound collection unit 12, a sound output unit 13, an operation unit 14, a display unit 15, a control unit 16, a communication unit 17, and a storage unit 18. The acquisition device 10 may be, for example, a camera device equipped with these components. The acquisition device 10 may also be a dedicated controller, a personal computer (PC), or a tablet terminal. The acquisition device 10 is not limited to an all-in-one device, and may also be configured with a camera equipped with imaging, sound collection, and sound output functions, and a monitor equipped with a touch panel function, and each unit may be installed in a physically separate location.

[0018] The imaging unit 11 captures an image of a predetermined target area ("imaging area", for example, the living room) including the subject P, converts it into a digital image signal, and outputs it to the control unit 16. The imaging unit 11 includes an imaging element such as a CCD element or a C-MOS element, an imaging optical system that forms an image of the subject on the imaging element, an A / D converter that amplifies the electrical signal output from the imaging element and performs analog-to-digital (A / D) conversion, and the like. The imaging unit 11 can use a known camera such as a digital still camera, a digital video camera, or a CCD camera, but can also use an infrared camera that can capture images in a dark room. The image acquired by the imaging unit 11 may be a still image or a moving image.

[0019] The sound collection unit 12 collects sounds in the target area including the voice of the subject P, converts them into digital audio signals, and outputs them to the control unit 16. The sound collection unit 12 includes a microphone that converts air vibrations into electrical signals, an A / D converter that amplifies the electrical signals output from the microphone and performs analog / digital (A / D) conversion, and the like.

[0020] The sound output unit 13 outputs sounds such as the voice of the user U, music, and sound effects such as chimes and buzzers under the control of the control unit 16. The sound output unit 13 can be, for example, a speaker, but may also have a chime, buzzer, etc.

[0021] The operation unit 14 accepts operation inputs from the subject P and outputs an operation signal corresponding to the accepted operation to the control unit 16. The operation unit 14 can be, for example, a touch panel, and may also have mechanical push buttons, operation levers, etc. Furthermore, the operation unit 14 can also include a mouse, keyboard, etc. The touch panel is provided with at least a "normal" and an "abnormal" key. The "normal" key is a key that the subject P presses when there are no problems with his or her physical condition, etc., and is normal. The "abnormal" key is a key that the subject P presses when he or she complains of poor health, etc., or when he or she requests help. The operation unit 14 also has an "away button" that the subject P presses to notify the control unit 16 that the subject P is not at home, for example, when he or she goes out. When the "away button" is pressed, the operation unit 14 outputs the operation signal to the control unit 16, and the control unit 16 outputs to the server device 20 that the away button has been set.

[0022] The display unit 15 (monitor) displays various types of information such as images and text on its display surface under the control of the control unit 16. The display unit 15 is made up of, for example, a liquid crystal display, an organic EL display, etc. The display unit 15 of this embodiment is made up of a touch panel type liquid crystal display, and the touch panel provided on the display surface functions as the above-mentioned operation unit 14.

[0023] The control unit 16 is configured with a processor such as a CPU or MPU, semiconductor memory such as a ROM or RAM, etc. The control unit 16 may also include a programmable logic device such as a GPU or FPGA, or an arithmetic element such as an ASIC. The processor of the control unit 16 loads a computer program pre-stored in the storage unit 18 onto, for example, the RAM of the storage unit 38, thereby allowing the control unit 16 to comprehensively control the operation of the acquisition device 10.

[0024] Furthermore, the control unit 16 outputs the image signal input from the imaging unit 11, the audio signal input from the sound collection unit 12, and the operation signal input from the operation unit 14 as subject information to the server device 20 via the communication unit 17 and the communication network N. Furthermore, the control unit 16 causes the sound output unit 13 to output audio and the display unit 15 to display an image based on the instruction signal and data input from the server device 20.

[0025] The communication unit 17 is composed of, for example, a communication interface circuit conforming to TCP / IP or the like, and is connected to the communication network N. The communication unit 17 outputs data received from the communication network N to the control unit 16, and transmits data input from the control unit 16 to the communication network N.

[0026] The storage unit 18 includes semiconductor memories such as ROM and RAM, large-capacity storage media such as hard disk drives, and other storage media. The storage unit 18 temporarily or non-temporarily stores various programs and data used for various operations in the acquisition device 10.

[0027] In this embodiment, only one acquisition device 10 is provided, but this is not limited thereto, and multiple acquisition devices 10 may be provided and installed in each room. Furthermore, when multiple subjects P are monitored in a facility or the like, one or more acquisition devices may be installed for each subject P. Furthermore, the acquisition device 10 may be configured to include multiple sets of an imaging unit 11, a sound collection unit 12, a sound output unit 13, an operation unit 14, and a display unit 15, and these sets may be installed in each room, with one or more control units 16 controlling the operations of these units.

[0028] Next, the server device 20 will be described. The server device 20 sets a privacy protection level for the subject P in accordance with the subject information input from the acquisition device 10, and outputs the subject information to the user terminal 30 in accordance with the set privacy protection level. There are preferably multiple privacy protection levels, and at least one of the privacy protection levels is set in accordance with the subject information.

[0029] The server device 20 can be, for example, a cloud server that uses a cloud computing service, or can be an in-house server owned by a monitoring service provider. The server device 20 is, for example, composed of one or more server PCs, general-purpose computers (mainframes), etc., but is not limited to these.

[0030] The server device 20 includes a control unit 21, a communication unit 22, a storage unit 23, etc., and may also include an input unit such as a keyboard, a display unit such as a liquid crystal display, a printer, etc.

[0031] The control unit 21 is configured with a processor such as a CPU or MPU, semiconductor memories such as a ROM or RAM, etc. The control unit 16 may also include a programmable logic device such as a GPU or FPGA, or an arithmetic element such as an ASIC. The processor of the control unit 21 loads a computer program stored in advance in the storage unit 18 onto, for example, the RAM of the storage unit 18, thereby allowing the control unit 21 to comprehensively control the operation of the server device 20.

[0032] Furthermore, the control unit 21 functions as a state detection unit 211, a setting unit 212, an abstraction processing unit 213, and an output unit 214 by executing a program on the processor.

[0033] In the "normal mode" and "special mode," the status detection unit 211 detects whether the status of the subject P is normal or abnormal by image analysis or audio analysis based on the subject information input from the acquisition device 10 to the server device 20, and outputs the detection result to the user terminal 30. In addition, the status detection unit 211 detects whether the subject P is at home or away based on the presence or absence of an operation signal from the away button or based on the results of the image analysis or audio analysis, and outputs the detection result to the user terminal 30.

[0034] In addition, in the "special mode", the status detection unit 211 detects whether or not a response action has been made based on the subject information input from the acquisition device 10 to the server device 20, and based on this detection result, detects whether the subject P's status is normal, abnormal, or indeterminable, and outputs the detection result to the user terminal 30.

[0035] When detecting the state in this "special mode," in this embodiment, the control unit 21 (state detection unit 211) receives input from the user terminal 30 and instructs the acquisition device 10 to output a predetermined "call sound" from the sound output unit 13 to call out to the target person P. The "call sound" can be, for example, a "ringing sound" such as a chime or music, or a "calling voice" such as "How are you feeling today?" In the case of a call sound, it may be a pre-registered voice or the voice of the user U, or it may be a voice called out by the user U in real time.

[0036] Then, during monitoring in the "special mode," the status detection unit 211 detects a predetermined response action based on the images and sounds acquired by the acquisition device 10, and detects the status of the subject P based on this response action. When the status detection unit 211 detects a "normal" action, it outputs a detection result of "normal," and when it detects an "abnormal" action, it outputs a detection result of "abnormal." Furthermore, the status detection unit 211 acquires information regarding the response action of the subject P in response to the call sound from the acquisition device 10. For example, when an operation of the "normal" or "abnormal" key is input from the operation unit 14, the operation signal is input from the acquisition device 10 to the status detection unit 211 as information about the response action.

[0037] When the state detection unit 211 receives an operation signal of the "normal" key, it outputs a detection result of "normal." When it receives an operation signal of the "abnormal" key, it outputs a detection result of "abnormal." On the other hand, if no operation signal is input even after a predetermined time (for example, 1 minute, 3 minutes, 5 minutes, 10 minutes, etc.) has passed since transitioning to the current privacy protection level, it outputs a detection result of "unable to determine." This detection result is also included in the subject information output to the user terminal 30.

[0038] The setting unit 212 sets the privacy protection level according to the subject information to be output to the user terminal 30. In this embodiment, the setting unit 212 changes the privacy protection level every time a predetermined time according to the current privacy protection level elapses, or every time at least one of the acquisition of subject information by the acquisition device 10 and the output of subject information to the user terminal 30 is repeated. Alternatively, as a modified example, the setting unit 212 may change the privacy protection level every time a call is made from the user terminal 30 to the subject P. In this embodiment, the privacy protection level for each subject P ranges from level 1 to 5, and the initial value is level 5, which is the highest level.

[0039] In this embodiment, the setting unit 212 sets the initial value of the privacy protection level to the maximum level of level 5. Then, every time a predetermined time corresponding to the current privacy protection level elapses, the privacy protection level of the subject P is lowered by one level (level 5 → level 4 → level 3 → level 2 → level 1).

[0040] The abstraction processing unit 213 performs a predetermined abstraction process on the subject information to be output to the user terminal 30, depending on the privacy protection level set by the setting unit 212. Examples of the abstraction process include mosaic processing, blurring processing, and de-identification processing for the image of the subject P acquired by the imaging unit 11, and sound modification processing for the sound acquired by the sound collection unit 12.

[0041] Mosaic processing is an image process in which a predetermined square area of ​​an image is filled with the same color to create a mosaic effect, making it difficult to see. Blur processing is an image process in which an image is blurred to make it difficult to see. Hereinafter, images in which mosaic processing or blur processing has been performed on a person's image may be referred to as a "mosaic image" or a "blurred image," respectively. Mosaic processing and blur processing modify the image of a person captured by the imaging unit 11, while de-identification processing is an image process that prevents the identification of subject P by, for example, replacing the image of a person with another image. Examples of de-identification processing include, but are not limited to, image processing in which a person's image is replaced with an "avatar." Hereinafter, replacing a person's image with an avatar is referred to as "avatarization," and the avatarized image may be referred to as an "avatar image." An "avatar" is a representation of a person using line drawings such as straight lines and circles, or a portrait or character that resembles a person. The voice change process is a process of converting the voice into a voice different from the voice of the person using a voice changer or the like.

[0042] In this embodiment, the abstraction processing unit 213 avatarizes a person's image when the privacy protection level is level 3, performs mosaic processing when the privacy protection level is level 2, and does not perform abstraction processing when the privacy protection level is level 1, the lowest level. FIG. 4A shows an image of an avatarized person's image. FIG. 4B shows an image of an image where the person's image has been subjected to mosaic processing. FIG. 4C shows an image where the person's image has not been subjected to abstraction processing. In this embodiment, the abstraction processing unit 213 does not perform voice modification processing. The correspondence between privacy protection levels and abstraction processing will be described in detail below in the description of the privacy protection level table 231.

[0043] The output unit 214 selects one of the image, the sound, and the state detection result according to the privacy protection level and outputs it as subject information to the user terminal 30. The correspondence between the privacy protection level and the subject information output by the output unit 214 will be described in detail below in the description of the privacy protection level table 231.

[0044] The communication unit 22 is composed of, for example, a communication interface circuit conforming to TCP / IP or the like, and is connected to the communication network N. The communication unit 22 outputs data received from the communication network N to the control unit 21, and transmits data input from the control unit 21 to the communication network N.

[0045] The storage unit 23 includes semiconductor memories such as ROM and RAM, large-capacity storage media such as hard disk drives, and other storage media. The storage unit 23 temporarily or non-temporarily stores various programs and data used for various operations in the server device 20.

[0046] The memory unit 23 stores, as data, a privacy protection level table 231, an information setting table 232, etc. Furthermore, for each user U, the memory unit 23 stores, as data, information such as the user ID, password, terminal information of the user terminal 30, name, name and physical condition of the person P being watched over, etc., linked to one another. Furthermore, for each person P, the memory unit 23 stores, as data, the current privacy protection level of each person P.

[0047] The privacy protection level table 231 is a table in which conditions for outputting subject information to the user terminal 30 are stored for each privacy protection level. Fig. 3A is a schematic diagram showing an example of the data structure of the privacy protection level table 231. As shown in Fig. 3A, the privacy protection level table 231 has a privacy protection level 231a, a state detection result output condition 231b, a sound output condition 231c, and an image output condition 231d.

[0048] As described above, the privacy protection level 231a is set to five levels, from a maximum of 5 to a minimum of 1. The status detection result output condition 231b sets whether or not the status detection result is to be output by the output unit 214 to the user terminal 30. The sound output condition 231c sets whether or not the output unit 214 is to output sound to the user terminal 30. The image output condition 231d sets whether or not the output unit 214 is to output an image to the user terminal 30 and the type of image processing (abstraction processing).

[0049] In the state detection result output condition 231b, the sound output condition 231c, and the image output condition 231d, "o" is set if output is to be performed, and "-" is set if output is not to be performed. Note that the parameters to be set are not limited to "o" and "-", but may be characters such as "possible" and "not possible", or numbers such as "1 (possible)" and "0 (not possible)". In addition to whether output is possible, the image output condition 231d is set with characters or numbers corresponding to "avatar", "processed", and "unprocessed" as image processing conditions.

[0050] More specifically, in this embodiment, the state detection result output condition 231b is set to "◯" so that the state detection result is output at all levels. Furthermore, the sound output condition 231c is set to "-" at level 5 so that the voice of the subject P collected by the sound collection unit 12 is not output, and is set to "◯" at levels 4 to 1 so that the voice is output. Note that in this embodiment, the sound output condition 231c is set only to whether or not to output sound, but is not limited to this. For example, it is also possible to set whether or not to perform abstraction processing (processing) such as voice processing that converts the voice using a voice changer or the like.

[0051] Furthermore, in the image output condition 231d, at levels 5 and 4, "-" is set so that the image of the subject P captured by the imaging unit 11 is not output. At level 3, "o" is set so that the image is output, and "avatar" is set as the processing condition so that the person's image is displayed as an avatar. "Avatar" means that the image of the subject P is output as an "avatar" rather than as a real video (an image actually captured).

[0052] At level 2, "Yes" is set to output an image, and "Real video (processed)" is set as the processing condition. "Real video (processed)" means that an image of subject P will be output after mosaic processing has been applied to the actual video of subject P. At level 1, "Yes" is set to output an image, and "Real video (unprocessed)" is set as the processing condition. "Real video (processed)" means that the image of subject P will be output as real video without any processing.

[0053] The information setting table 232 is a table in which subject information including images, sounds, and state detection results acquired by the acquisition device 10 is stored for each subject P. Fig. 3B is a schematic diagram showing an example of the data structure of the information setting table 232. As shown in Fig. 3B, the information setting table 232 has a subject ID 232a and a recording area 232b.

[0054] Subject ID 232a stores the subject ID of subject P. Recording area 232b has a plurality of recording areas 1 to N for each date and time when subject information was acquired, each of which stores information such as images, sounds, and state detection results by state detection unit 211 input from acquisition device 10. Each recording area 232b has date and time 232c, state determination results 232d, sound signals 232e, and image signals 232f.

[0055] The date and time 232c stores the date and time when the state detection unit 211 detected the state of the subject P based on the information acquired by the acquisition device 10. The state determination result 232d stores the state detection result (characters, numbers, etc. indicating normal, abnormal, or impossible to determine). The audio signal 232e stores the audio signal collected by the sound collection unit 12 of the acquisition device 10. The image signal 232f stores the image signal acquired by the imaging unit 11 of the acquisition device 10. Note that, when the acquisition device 10 acquires a moving image including audio, the audio signal and the image signal can be stored without being separated. Furthermore, images and audio acquired by the acquisition device 10 may be stored in the information setting table 232 even when there is no instruction for state detection. By allowing the administrator of the monitoring system 100 or the user U to refer to the information setting table 232, it is possible to check the past state, images, and audio of the subject P.

[0056] The user terminal 30 is a terminal owned by the user U, and one or more terminals are provided for each user U. In this embodiment, the user terminal 30 is a smartphone, which allows the user to easily monitor the user even when the user is out. Note that the user terminal 30 is not limited to a smartphone, and can also be a mobile phone, tablet terminal, notebook PC, desktop PC, etc., or a dedicated controller.

[0057] Next, we will explain the user terminal 30. The user terminal 30 includes an imaging unit 31, a sound collection unit 32, a sound output unit 33, an operation unit 34, a display unit 35, a control unit 36, a communication unit 37, and a storage unit 38. The hardware configurations of the imaging unit 31, the sound collection unit 32, the sound output unit 33, the operation unit 34, the display unit 35, the control unit 36, the communication unit 37, and the storage unit 38 can be similar to those of the acquisition device 10, and therefore detailed explanations will be omitted.

[0058] The imaging unit 31, sound collection unit 32, sound output unit 33, operation unit 34, display unit 35, control unit 36, communication unit 37, and memory unit 38 may be, for example, a smartphone camera, microphone, speaker, touch panel and buttons, LCD display, processor and memory, communication interface circuit, and storage medium, respectively.

[0059] The user terminal 30 can connect to the communication network N via the communication unit 37, access the server device 20, and use the monitoring service by using a general-purpose browser or by launching an application program (app) that has been pre-installed on the user terminal 30.

[0060] The imaging unit 31 takes an image of the user U, converts it into a digital image signal, and outputs it to the control unit 36. The sound collection unit 32 collects the voice of the user U, converts it into a digital audio signal, and outputs it to the control unit 36. The sound output unit 33 outputs audio under the control of the control unit 36. The operation unit 34 accepts operation input by the user U, and outputs an operation signal corresponding to the accepted operation to the control unit 36. The display unit 35 displays various information such as images and text on its display surface under the control of the control unit 36.

[0061] The control unit 36 ​​comprehensively controls the operation of the user terminal 30 by causing a processor to load a computer program pre-stored in the storage unit 38, for example, onto the RAM of the storage unit 38. The control unit 36 ​​outputs image signals input from the imaging unit 31, audio signals input from the sound collection unit 32, and operation signals input from the operation unit 34 to the server device 20 via the communication unit 37 and the communication network N. Furthermore, the control unit 36 ​​causes the sound output unit 33 to output sound such as the voice of the subject P, and causes the display unit 35 to display images such as an image of the subject P, based on instruction signals and data such as subject information input from the server device 20.

[0062] The communication unit 37 outputs data received from the communication network N to the control unit 36, and transmits data input from the control unit 36 ​​to the communication network N. The memory unit 38 temporarily or non-temporarily stores various programs and data used for various operations in the user terminal 30.

[0063] Here, an example of a screen displayed on the display unit 35 of the user terminal 30 will be described. FIG. 5A is an example of a call screen 50, which has a call button 50a and an emergency button 50b. When the user U presses a "special mode button" (not shown) displayed on the display unit 35 for transitioning to special mode while monitoring in "normal mode," the mode switches from "normal mode" to "special mode," and this call screen 50 is displayed on the display unit 35. In other words, this "special mode button" functions as a trigger for starting the special mode.

[0064] The call screen 50 displays text and images (see "i" in FIG. 5A) indicating whether the target person P is at home or away (away). When the detection result in the "normal mode" is normal and the "away button" on the operation unit 14 of the acquisition device 10 is not pressed (no operation signal for the away button is input), the control unit 21 displays "At home" on the call screen 50. When the "away button" on the operation unit 14 is pressed (an operation signal for the away button is input), the control unit 21 displays "Away" or "Away" on the call screen 50. By visually checking this display, the user U can recognize whether the target person P is at home or away, enabling more appropriate monitoring. Furthermore, since only "at home" or "away" is displayed, the user U can recognize that there is nothing particularly abnormal with the target person P (normal). In contrast, when the control unit 21 receives an input of a detection result indicating an abnormality from the server device 20, the control unit 21 displays a message such as "An abnormality has occurred" on the call screen 50, or on a separate screen or pop-up screen (pop-up display). It is also possible to have a configuration in which whether the person is at home or not is not displayed, and a message or the like is displayed only when an abnormality occurs.

[0065] The call button 50a is a button for instructing a call to the target person P. When the user U presses the call button 50a, the control unit 21 starts counting the elapsed time at level 5 (initial value). Furthermore, a call sound is output from the sound output unit 13 of the acquisition device 10, and the acquisition device 10 starts acquiring information including the response action.

[0066] The emergency button 50b is a button for making an emergency call. When the user U presses the emergency button 50b, a call is automatically or manually made to a predetermined emergency contact, allowing the user U to call for rescue. The screens shown in Figures 5B, 5C, 5D, and 5E also have emergency buttons 51b, 52b, 53b, and 54b, and when these buttons are pressed, a call is made to the emergency contact, just as when the emergency button 50b is pressed.

[0067] Emergency contact points include, for example, telephone numbers of 119, hospitals, care service providers, neighbors of the subject P, etc. One of these contact points may be set, or multiple contact points may be set and the user U may choose one depending on the degree of urgency, etc.

[0068] FIG. 5B shows an example of the first information display screen 51 on which the state detection result is displayed, and includes an information display area 51a, an emergency button 51b, and an end button 51c.

[0069] The information display area 51a displays a message according to the status detection result. For example, the message may be "Normality detected," "Normal button pressed," or "Normal response," if the status detection result is normal; "Abnormality detected," "Abnormal button pressed," or "Abnormal response," if the status detection result is abnormal; or "No response" if the status cannot be determined, but is not limited to these.

[0070] The emergency button 51b is a button for making an emergency call, similar to the emergency button 50b. The end button 51c is a button for ending monitoring in the "special mode" and switching (returning) to monitoring in the "normal mode." When the emergency button 51b or the end button 51c is pressed, or an operation to end the application is performed by swiping the screen, etc., a command signal to end monitoring in the "special mode" is output from the user terminal 30 to the server device 20.

[0071] 5C to 5E are examples of second to fourth information display screens 52 to 54 on which information (images and / or audio) corresponding to the privacy protection level is displayed. These second to fourth information display screens 52 to 54 also display messages instructing the user U to call out, and therefore serve as call instruction screens. When displaying the second to fourth information display screens 52 to 54 of FIGS. 5C to 5E, the user terminal 30 collects the user U's call voice using the sound collection unit 32 and outputs the audio signal to the server device 20. Furthermore, when displaying the fourth information display screen 54 of FIG. 5E, the user terminal 30 captures an image of the user U using the imaging unit 31 and outputs the image signal to the server device 20.

[0072] FIG. 5C shows an example of the second information display screen 52 displayed when the privacy protection level is level 5. The second information display screen 52 includes an information display area 52a, an emergency button 52b, and an end button 52c. Since level 5 is set to not display images, the second information display screen 52 does not include an image display area. A message instructing the user U to call the target person P, such as "Please call," is displayed in the information display area 52a. A call from the user U is collected by the sound collection unit 32, converted into an audio signal, output to the server device 20, output from the server device 20 to the acquisition device 10, and then output from the sound output unit 13 to the target person P. The emergency button 52b is a button for making an emergency call. The end button 52c is a button for ending the "special mode" monitoring and returning to the "normal mode" monitoring.

[0073] In addition, in the case of a mode in which the voice of the target person P is detected and the state is detected based on that voice, the message "Voices in the room are being detected. Please call out." is displayed in the information display area 52a in Fig. 5C. This prompts the user U to call out, and the voice of the target person P is detected, and the state detection unit 211 can detect the state based on that voice.

[0074] 5D is an example of the third information display screen 53 that is displayed when the privacy protection level is level 4, and has an information display area 53a, an emergency button 53b, and an end button 53c. Even at level 4, the setting is such that images are not displayed, so the third information display screen 53 does not have an image display area.

[0075] The information display area 53a displays a message instructing the user to make a call. For example, when the privacy protection level is level 4, the setting is to output the audio of the target area including the voice of the target person P, so a message such as "Please make a call while listening to the sounds in the room" is displayed. The emergency button 53b is a button for making an emergency call. The end button 53c is a button for ending monitoring in the "special mode" and returning to monitoring in the "normal mode."

[0076] FIG. 5E is an example of a fourth information display screen 54 that is displayed when the privacy protection level is level 3, level 2, or level 1, and has an information display area 54a, an emergency button 54b, and an exit button 54c.

[0077] The information display area 54a displays an image based on an image signal acquired by the acquisition device 10, subjected to abstraction processing by the server device 20, and output, as well as a message instructing a call, such as "Please call out." At level 3, the information display area 54a displays an avatarized image of a person as shown in FIG. 4A; at level 2, a pixelated image of a person as shown in FIG. 4B; and at level 1, an unprocessed image without any abstraction processing as shown in FIG. 4C. The emergency button 54b is a button for making an emergency call. The end button 54c is a button for ending "special mode" monitoring and returning to "normal mode" monitoring.

[0078] 5B to 5E also display characters or images (see "i" in each figure) indicating whether the target person P is at home or away (away), similar to FIG. 5A. By visually checking this display, the user U can recognize whether the target person P is at home or away, and can more easily determine whether there is no response despite the target person P being at home, or whether there is no response because the target person P is away, enabling more appropriate monitoring. Note that when each of the screens shown in FIGS. 5B to 5E is displayed, the user U can recognize whether the target person P is at home or away, or whether it is not possible to determine whether the target person P is at home or away, based on the detection results, the presence or absence of a response operation or sound from the target person P, or the image of the target area, so it is also possible to configure these screens so that the "at home or away" display is not displayed.

[0079] An example of the flow of the operation of the monitoring system 100 according to this embodiment will be described below with reference to the flowchart in Fig. 6. The flowchart shown in Fig. 6 is a flowchart for explaining an example of the flow of the monitoring operation in the "special mode".

[0080] A user U registered in the monitoring system 100 according to this embodiment can use the monitoring service by starting a browser or an app on the user terminal 30, connecting to the communication network N, accessing the server device 20, and having the server device 20 authenticate the user U using a user ID and password. This allows the monitoring system 100 to constantly monitor in "normal mode."

[0081] The "special mode" monitoring operation shown in the flowchart of FIG. 6 is initiated when user U presses the "special mode button" on the operation screen of an app or the like on user terminal 30. User U may press the "special mode button" when a message indicating an abnormality is displayed during monitoring in "normal mode," or when, even though there is no abnormality notification from monitoring system 100, user U wants to check the status of subject P in more detail, for example, when subject P does not reply to an email or answer a phone call or remains "away" despite being "at home" during monitoring in "normal mode." This operation may also be performed periodically, such as every morning or once a week. When the "special mode button" is pressed, a call screen 50 shown in FIG. 5A is displayed on display unit 35, and "At home" or "Away" is displayed depending on whether the "Away button" is pressed.

[0082] In this first embodiment, the monitoring system 100 performs a "special mode" monitoring process in which, upon receiving a call instruction from the user U via the user terminal 30, the user U acquires information about the target person P and outputs this information to the user terminal 30 according to the privacy protection level. The monitoring system 100 also changes the privacy protection level every time a predetermined time elapses, and repeats the "special mode" monitoring process while making a call from the user U. In another different embodiment, a button may be provided on the first or fourth information display screen 51, 54 to instruct the user to lower the privacy protection level and check the status. Instead of waiting for time to pass, when the status cannot be determined or when the user U wants to know the status in more detail, the privacy protection level may be lowered in response to the user U's operation of this button, and the status may be determined or displayed at the new privacy protection level.

[0083] First, in step S1 of the flowchart shown in FIG. 6, the server device 20 accepts a call input from the user U through the user terminal 30. For example, when the privacy protection level is level 5, if the user U presses the call button 50a on the call screen 50, the operation signal is input from the user terminal 30 to the server device 20. Furthermore, when the privacy protection level is level 4 or lower, the user terminal 30 displays a message on the display unit 35 urging the user U to make a call. As a result, when the user U makes a call toward the sound collection unit 32, an audio signal of the user U's call collected by the sound collection unit 32 is input from the user terminal 30 to the server device 20. In addition, when the privacy protection level is level 3 to level 1, an image signal of the user U captured by the imaging unit 31 is input from the user terminal 30 to the server device 20.

[0084] Upon receiving this call, the server device 20 instructs the acquisition device 10 to output a call sound from the sound output unit 13 of the acquisition device 10. For calls at level 5, this call sound may be a chime, music, or other sounds, a pre-registered voice, or the voice of the user U. Furthermore, while the level is changed (lowered), specifically for calls at level 4 or lower, the real-time call voice of the user U collected by the sound collection unit 32 of the user terminal 30 is output. At this time, the second to fourth information display screens 52 to 54 shown in FIGS. 5C to 5E are displayed on the user terminal 30 according to the privacy protection level, allowing the user U to make a call.

[0085] During execution of this special mode, the imaging unit 11 of the acquisition device 10 captures images continuously or at regular intervals, the sound collection unit 12 collects audio, and the operation unit 14 accepts input of a response action. The acquisition device 10 outputs target person information including image signals, audio signals, and operation signals to the server device 20 and temporarily or non-temporarily stores this information in the storage unit 18 as necessary. The acquisition device 10 may also restrict the output of images and audio to the server device 20 according to the privacy protection level. For example, at level 5, the acquisition device 10 may not output images and audio to the server device 20, and at level 4, the acquisition device 10 may output only audio to the server device 20 without outputting images, or the acquisition device 10 may output images and audio to the server device 20, but the server device 20 may restrict the output to the user terminal 30 according to the privacy protection level.

[0086] In step S2, upon receiving the subject information from the acquisition device 10, the output unit 214 of the server device 20 outputs the subject information, including images and audio, according to the privacy protection level to the user terminal 30. At this time, the abstraction processing unit 213 of the server device 20 performs abstraction processing on the image signal according to the privacy protection level. Specifically, the abstraction processing unit 213 turns the person's image into an avatar when the privacy protection level is level 3, and performs mosaic processing on the person's image when the privacy protection level is level 2. On the other hand, at levels 5 and 4, no image is output, and at level 1, an actual video is output, so the abstraction processing unit 213 does not perform abstraction processing.

[0087] Furthermore, the control unit 21 stores images (unprocessed real images or images after abstraction processing), sounds, and state detection results in the information setting table 232 of the storage unit 23.

[0088] Upon receiving the subject information from the server device 20, the user terminal 30 presents the subject information to the user U according to the privacy protection level. For example, when the privacy protection level is level 5, the user terminal 30 displays the call screen 50 shown in Fig. 5A or the second information display screen 52 shown in Fig. 5C. At level 5, neither images nor sounds are output to the user terminal 30, so the privacy of the subject P is protected to the maximum extent.

[0089] On the other hand, when the privacy protection level is level 4, the user terminal 30 displays the third information display screen 53 shown in Fig. 5D on the display unit 35, and because audio output is permitted, outputs the audio of the target area acquired by the sound collection unit 12 of the acquisition device 10 from the sound output unit 33. The user U can determine whether the subject P is normal or abnormal by calling out in his or her own voice in accordance with the call instructions displayed on the third information display screen 53 and listening to the audio of the target area output from the sound output unit 33.

[0090] Furthermore, when the privacy protection level is level 3 or lower, the user terminal 30 embeds an avatar image of a person, a mosaicked image, or an unprocessed image in the information display area 54a according to the level, displays the fourth information display screen 54 shown in Fig. 5E, and outputs the voice of the subject P from the sound output unit 33. By checking the subject information according to the privacy protection level displayed in the information display area 54a of this fourth information display screen 54, the user U can determine whether the subject P is normal or abnormal, and can also call out in his or her own voice according to the call instructions displayed in the information display area 54a.

[0091] Furthermore, at level 3 or lower, the acquisition device 10 outputs a voice call from the sound output unit 13 and displays an image of the user U on the display unit 15. This allows both the user U and the target person P to hear the voice and have a conversation, allowing the user U to check the condition of the target person P in more detail. The target person P feels a greater sense of security by checking the voice and image of the user U.

[0092] In the next step S3, the state detection unit 211 of the server device 20 automatically detects the state of the subject P based on the subject information and determines whether normal or abnormal has been detected. If the state detection unit 211 detects normal or abnormal (determination of step S3 is YES), the program proceeds to step S10, and if it is not possible to determine whether the state is normal or abnormal (the state is unknown) (determination of step S3 is NO), the program proceeds to step S4.

[0093] Here, a specific description will be given of automatic detection of the state by the state detection unit 211. The state detection unit 211, for example, performs image analysis on the image signal from the imaging unit 11, detects a response action based on a gesture indicating a response from the subject P (for example, a gesture indicating normality such as a nod or an OK sign, or a gesture indicating abnormality such as shaking the head or hand), and detects the state based on this response action. Alternatively, even if a gesture indicating a response cannot be detected, the state detection unit 211 detects the facial expression, complexion, body position (posture), etc. of the subject P through image analysis, compares these with those when the subject P is normal (and even when abnormal), and detects the state based on the comparison result.

[0094] Furthermore, the state detection unit 211, for example, performs sound analysis on the sound collected by the sound collection unit 12 to detect waveform data and words (text data), detects a response action based on the sound indicating a response (for example, sound indicating normality such as "I'm fine," sound indicating abnormality such as "I'm not feeling well," "Please help me," groaning, screaming, or sounds indicating abnormality such as hitting the floor or wall, etc.), and detects the state based on this response action. Alternatively, the state detection unit 211 compares the waveform data and words obtained by the sound analysis with waveform data of the subject P's normal voice, or with the above-mentioned words indicating normality or abnormality, and detects the state based on the comparison result.

[0095] The state detection by the state detection unit 211 using image analysis or audio analysis may be performed by a program, or may use artificial intelligence (AI) technology such as machine learning, reinforcement learning, or deep learning using a neural network including multiple intermediate layers.

[0096] Then, in the next step S4, the state detection unit 211 determines whether or not a response operation has been made from the subject P, specifically, whether or not a response operation of pressing the "normal" or "abnormal" button on the operation unit 14 has been made based on the subject information. If a response operation has been made (determination of YES in step S4), the program proceeds to step S10, and if no response operation has been made (determination of NO in step S4), the program proceeds to step S5 as it is not possible to make a determination.

[0097] In the next step S5, the control unit 21 determines whether or not an operation to terminate the special mode from the user U has been detected, specifically, whether or not the control unit 21 has detected an operation of the emergency button 51b or the end button 51c by the user U on the user terminal 30, or an operation to terminate the application. If an end operation has been detected (determination of YES in step S5), the program proceeds to step S10, and if an end operation has not been detected (determination of NO in step S5), the program proceeds to step S6.

[0098] Here, the operation for ending the special mode will be explained. After checking the image displayed on the display unit 35 of the user terminal 30 or the sound output from the sound output unit 33, the user U can end the monitoring in the "special mode" by pressing the end button 51c or by swiping the screen. Furthermore, if the user U notices an abnormality, he or she can press the emergency button 51b to make an emergency call. By these operations, an operation signal for ending the monitoring in the "special mode" is output from the user terminal 30 to the server device 20.

[0099] In the next step S6, it is determined whether a predetermined time corresponding to the privacy protection level has elapsed. This "predetermined time" is a threshold value of the time elapsed since transitioning to the current level. The "elapsed time" at level 5 is the time elapsed since user U pressed the "special mode button" to transition to the special mode, and the "elapsed time" at levels 4 to 1 is the time elapsed since transitioning to that level. More specifically, for example, the "predetermined time" can be set to 3 minutes for all levels 5 to 1, but is not limited to this. As another example, the predetermined time can be changed for each level. For example, since the lower the level, the greater the urgency, and therefore the predetermined time can be set shorter for each lower level.

[0100] On the other hand, at level 1, monitoring is not performed at a lower level, so as a variant, the specified time may be set to unlimited (infinite) so that the presentation of subject information at level 1 continues until an end operation is performed.

[0101] Then, while the predetermined time has not elapsed (the determination in step S6 is NO), the program returns to step S1, and while no termination operation is performed, the processing in steps S1 to S6 is repeated, and once the predetermined time has elapsed (the determination in step S6 is YES), the program proceeds to step S7.

[0102] In step S7, the monitoring system 100 outputs a status determination result to the user terminal 30 according to the privacy protection level. This step S7 processing is executed when the status is not automatically detected in step S3 and a response operation from the subject P is not detected in step S4 even after a predetermined time has elapsed. Therefore, the output unit 214 of the server device 20 outputs a message indicating that determination is impossible to the user terminal 30. When the user terminal 30 receives this input indicating that determination is impossible, if the privacy protection level is level 5 or level 4, the user terminal 30 displays the first information display screen 51 on the display unit 35 by presenting a message saying "No response" in the information display area 51a of the first information display screen 51 shown in FIG. 5B. On the other hand, if the privacy protection level is level 3 or lower, the user terminal 30 displays the message "No response" on the display unit 35 alongside the image of the subject P (avatar image, mosaic image, or unprocessed image) presented in the information display area 54a of the fourth information display screen 54 shown in FIG. 5E, or by displaying it as a separate screen or pop-up.

[0103] Thereafter, the program proceeds to step S8, where the setting unit 212 of the server device 20 changes the privacy protection level (lowering the level by one level). Next, the program proceeds to step S9, where it is determined whether the privacy protection level is less than 1. If the privacy protection level is level 1 or higher (determination in step S9 is NO), the program returns to step S1, whereby monitoring in the "special mode" at the privacy protection level after the change is repeated. On the other hand, if the privacy protection level is less than level 1 (determination in step S9 is YES), the program proceeds to step S10 to end the special mode.

[0104] Finally, the process of step S10 will be described. The process of step S10 is executed in any of the following cases. (1) When a normal or abnormal state of subject P is detected in step S3. (2) When a normal or abnormal response operation from the subject P is detected in step S4. (3) When an operation to end the special mode from the user terminal 30 is detected in step S5. (4) When it is determined in step S9 that the privacy protection level is lower than level 1.

[0105] In step S10, the monitoring system 100 outputs the reason for terminating the special mode to the user terminal 30. The program then terminates the special mode monitoring and returns to the normal mode monitoring. Specifically, (1) when a normal or abnormal state of the subject P is detected, the output unit 214 of the server device 20 outputs the normal or abnormal state detection result to the user terminal 30 as the reason for termination. The user terminal 30 then displays a message such as "Normal has been detected" or "Abnormality has been detected" on the display unit 35 depending on the state detection result and the privacy protection level. For example, at privacy protection levels 5 and 4, these messages are displayed in the information display area 51a, and the first information display screen 51 shown in FIG. 5B is displayed. At levels 3 and below, these messages are displayed alongside an image of the subject P (avatar image, mosaic image, no abstraction) displayed in the information display area 54a, and the fourth information display screen 54 shown in FIG. 5E is displayed.

[0106] Furthermore, when (2) a normal or abnormal response operation from the subject P is detected, the output unit 214 outputs the normal or abnormal state detection result and a signal indicating the response operation to the user terminal 30 as the reason for termination. The user terminal 30 then displays a message such as "The normal button has been pressed" or "The abnormal button has been pressed" on the display unit 35 depending on the state detection result. Meanwhile, when (3) an operation to terminate the special mode is performed, the output unit 214 sets the privacy protection level to the initial value and outputs a signal to terminate the special mode to the user terminal 30 as the reason for termination. The user terminal 30 then displays a message such as "The special mode will be terminated and the normal mode will be returned to." on the display unit 35. Furthermore, when (4) the privacy protection level is lower than level 1, the output unit 214 outputs a signal indicating an indeterminable state detection result to the user terminal 30 as the reason for termination. The user terminal 30 then displays a message such as "The subject's state could not be confirmed. The special mode will be terminated." on the display unit 35. Once each message is displayed on the display unit 35 in this manner, the monitoring system 100 then ends the special mode monitoring and returns to the normal mode monitoring.

[0107] Next, a more specific example of the operation of the monitoring system 100 of this embodiment in response to the state of the target person P during monitoring in the "special mode" will be described. First, the user U presses the "special mode button" (not shown) displayed on the display unit 35 upon receiving an abnormality notification from the monitoring system 100 or of their own volition. Upon receiving this operation, the monitoring mode switches to the "special mode," and a call screen 50 is displayed on the display unit 35. When the user U presses the call button 50a on the call screen 50, the program proceeds from step S1 to step S2, causing the acquisition device 10 to output a call tone from the sound output unit 13. However, because the current privacy protection level is level 5, no image or audio of the target person P is output to the user terminal 30 in step S2. The user terminal 30 then displays on the display unit 35 a second information display screen 52 shown in FIG. 5C , which displays "Call me" or "Voices in the room are being detected. Please call me." in the information display area 52a.

[0108] Next, when the state detection unit 211 detects whether the subject P is normal or abnormal based on the subject information in step S3, or when a response operation by the subject P is detected in step S4 (the "normal" or "abnormal" button on the operation unit 14 is pressed), the program proceeds to step S10. In this step S10, the user terminal 30 displays on the display unit 35 a first information display screen 51 presenting a message indicating that normal or abnormal conditions have been detected, depending on the reason for termination. By visually checking this first information display screen 51, the user U can confirm whether the subject P is normal or abnormal. Thereafter, special mode monitoring ends, and monitoring returns to normal mode.

[0109] On the other hand, while neither normality nor abnormality is detected at level 5, the processes of steps S1 to S6 are repeated until a predetermined time has elapsed unless an operation to end the special mode is performed from the user terminal 30. When the predetermined time corresponding to level 5 has elapsed, the program proceeds to step S7, and the first information display screen 51 shown in FIG. 5B, in which "No response" is displayed in the information display area 51a, is displayed on the display unit 35. Then, the program proceeds to step S8, in which the setting unit 212 changes the privacy protection level to level 4, and the program proceeds to step S9.

[0110] The program then returns from step S9 to step S1, and monitoring in the special mode at level 4 continues. In step S1, the server device 20 receives a call voice from the user terminal 30 and instructs the acquisition device 10 to output the call voice from the sound output unit 13 to the target person P. Next, in step S2, the user terminal 30 receives the user information from the server device 20 and displays on the display unit 35 the third information display screen 53 shown in FIG. 5D , which displays "Voices in the room are being detected. Please call out." corresponding to level 4. At this time, the sound output unit 33 of the user terminal 30 outputs the sound collected in the target area. That is, at level 4, the user U can hear the sound in the target area. While listening to the sound in the target area, the user U can call out to the sound collection unit 32 in accordance with the message displayed on the second information display screen 52. This call voice from the user U is output from the sound output unit 13 of the acquisition device 10. Therefore, while calling out, the user U can recognize whether the target person P is normal or abnormal based on whether or not there is a response by voice from the sound output unit 33 and the content of the response.

[0111] In step S3, when a normal or abnormal state of subject P is detected based on subject information such as images and audio, or when a response operation is detected in step S4, the program proceeds to step S10, and displays a message on the display unit 35 indicating that normal or abnormal status has been detected depending on the reason for termination, after which special mode monitoring ends and monitoring returns to normal mode.

[0112] On the other hand, while neither normality nor abnormality is detected at level 4, the processes of steps S1 to S6 are repeated until a predetermined time has elapsed unless an operation to end the special mode is performed from the user terminal 30. When the predetermined time corresponding to level 4 has elapsed, the program proceeds to step S7, and the first information display screen 51 shown in FIG. 5B, in which "No response" is displayed in the information display area 51a, is displayed on the display unit 35. Then, the program proceeds to step S8, in which the setting unit 212 changes the privacy protection level to level 3, and the program proceeds to step S9.

[0113] The program then returns from step S9 to step S1, and monitoring in the special mode at level 3 continues. In step S1, server device 20 receives a call voice from user terminal 30 and instructs acquisition device 10 to output the call voice to target person P from sound output unit 13. Next, in step S2, abstraction processing unit 213 performs abstraction processing corresponding to level 3 on the image acquired by acquisition device 10, i.e., avatarization of a human figure, and output unit 214 outputs target person information including the avatar image and the voice of target person P to user terminal 30. In response to the input of this target person information, user terminal 30 embeds the avatar image in information display area 54a, displays fourth information display screen 54 shown in FIG. 5E on display unit 35, and outputs the voice of target person P from sound output unit 33. User U can call out to target person P while listening to the voice and checking their condition.

[0114] The acquisition device 10 continues to output the call voice from the sound output unit 13 and display the image of the user U on the display unit 15. This allows both the user U and the target person P to hear the voice and have a conversation, allowing the user U to check the condition of the target person P in more detail. The target person P feels a greater sense of security by checking the voice and image of the user U.

[0115] Furthermore, since the image of the person displayed on the fourth information display screen 54 is an avatar, the privacy of the subject P can be protected, while the user U can confirm the presence and posture of the subject P (e.g., standing, sleeping, fallen, etc.), and can check the condition of the subject P in more detail and more easily.

[0116] Next, when the status detection unit 211 detects that the subject P is normal or abnormal based on the subject information in step S3, or when a response operation by the subject P is detected in step S4 (the "normal" or "abnormal" button on the operation unit 14 is pressed), the program proceeds to step S10, and after displaying a message indicating that normal or abnormal has been detected on the display unit 35 depending on the reason for termination, special mode monitoring ends and monitoring returns to normal mode.

[0117] On the other hand, while neither normal nor abnormal is detected at level 3, the processes of steps S1 to S6 are repeated until a predetermined time has elapsed unless an operation to end the special mode is performed from user terminal 30. When the predetermined time corresponding to level 3 has elapsed, the program proceeds to step S7, where a message "There was no response" is displayed on display unit 35 alongside the avatar image in information display area 54a. The program then proceeds to step S8, where setting unit 212 changes the privacy protection level to level 2, and proceeds to step S9.

[0118] The program then returns from step S9 to step S1, and monitoring in the special mode at level 2 continues. In step S1, the server device 20 receives a call voice from the user terminal 30 and instructs the acquisition device 10 to output the call voice to the target person P from the sound output unit 13. Next, in step S2, the abstraction processing unit 213 performs abstraction processing corresponding to level 2 on the image acquired by the acquisition device 10, i.e., mosaic processing of the person's image, and the output unit 214 outputs target person information including the mosaic image and the voice of the target person P to the user terminal 30. The user terminal 30, which has received the input of this target person information, embeds the mosaic image in the information display area 54a, displays the fourth information display screen 54 shown in FIG. 5E on the display unit 35, and outputs the voice of the target person P from the sound output unit 33.

[0119] The acquisition device 10 continues to output the call voice from the sound output unit 13 and display the image of the user U on the display unit 15. This allows both the user U and the target person P to hear the voice and have a conversation, allowing the user U to check the condition of the target person P in more detail. The target person P feels a greater sense of security by checking the voice and image of the user U.

[0120] In addition, since the image of the person displayed on the fourth information display screen 54 is pixelated, the privacy of the subject P is protected, while the user U can check the subject P's complexion, movements, etc., and can check the subject P's condition in more detail and more easily.

[0121] Next, when the status detection unit 211 detects that the subject P is normal or abnormal based on the subject information in step S3, or when a response operation by the subject P is detected in step S4 (the "normal" or "abnormal" button on the operation unit 14 is pressed), the program proceeds to step S10, and after displaying a message indicating that normal or abnormal has been detected on the display unit 35 depending on the reason for termination, special mode monitoring ends and monitoring returns to normal mode.

[0122] On the other hand, while no normal or abnormal state is detected at level 2, the processes of steps S1 to S6 are repeated until a predetermined time has elapsed unless an operation to end the special mode is performed from the user terminal 30. When the predetermined time corresponding to level 2 has elapsed, the program proceeds to step S7, where the message "No response" is displayed on the display unit 35 alongside the mosaic image in the information display area 54a. The program then proceeds to step S8, where the setting unit 212 changes the privacy protection level to level 1, and the program proceeds to step S9.

[0123] The program then returns from step S9 to step S1, and monitoring in the special mode at level 1 continues. In step S1, in response to an instruction from server device 20, user terminal 30 collects a call from user U, and this call is output from sound output unit 13 of acquisition device 10. The program then proceeds to step S2, but because it is at level 1, abstraction processing is not performed on the image acquired by acquisition device 10. Output unit 214 outputs subject information including an unprocessed image that has not been abstracted and the voice of subject P to user terminal 30. In response to the input of this subject information, user terminal 30 fits the unprocessed image into information display area 54a, displays fourth information display screen 54 shown in FIG. 5E on display unit 35, and outputs the voice of subject P from sound output unit 33.

[0124] The acquisition device 10 continues to output the call voice from the sound output unit 13 and display the image of the user U on the display unit 15. This allows both the user U and the target person P to hear the voice and have a conversation, allowing the user U to check the condition of the target person P in more detail. The target person P feels a greater sense of security by checking the voice and image of the user U.

[0125] Furthermore, since the fourth information display screen 54 displays an actual image of the subject P, the user U can confirm the subject P's condition in more detail and more realistically through this actual image and sound.

[0126] Next, when the status detection unit 211 detects that the subject P is normal or abnormal based on the subject information in step S3, or when a response operation by the subject P is detected in step S4 (the "normal" or "abnormal" button on the operation unit 14 is pressed), the program proceeds to step S10, and after displaying a message indicating that normal or abnormal has been detected on the display unit 35 depending on the reason for termination, special mode monitoring ends and monitoring returns to normal mode.

[0127] On the other hand, while no normal or abnormal state is detected at level 1, the processes of steps S1 to S6 are repeated until a predetermined time has elapsed unless an operation to end the special mode is performed from the user terminal 30. When the predetermined time corresponding to level 1 has elapsed, the program proceeds to step S7, where the message "There was no response" is displayed on the display unit 35 alongside the actual image in the information display area 54a. The program then proceeds to step S8, where the setting unit 212 changes the privacy protection level to level 0. Next, the program proceeds to step S9. However, because the privacy protection level is lower than level 1 (level 0), the determination in step S9 is YES, and the program proceeds to step S10. In step S10, the user terminal 30 displays a message such as "The subject's condition could not be confirmed. The special mode will be terminated," depending on the reason for termination. The program then terminates the special mode monitoring and returns to normal mode monitoring.

[0128] If the predetermined time at level 1 is set to unlimited (infinite), the processing of steps S1 to S6 is repeated as long as no normal or abnormal condition is detected. That is, the fourth information display screen 54 at level 1 continues to be displayed on the display unit 35 of the user terminal 30. Therefore, the user U can check the condition of the subject P at level 1 by using the fourth information display screen 54 displayed on the display unit 35 and the sound output from the sound output unit 33. Then, if the user U checks the condition of the subject P and performs an end operation while the processing at level 1 is being repeated, the program proceeds from step S5 to S10, and the user terminal 30 displays a message such as "Special mode will end and return to normal mode" on the display unit 35 depending on the reason for ending, and then special mode monitoring ends and returns to normal mode monitoring.

[0129] (Second embodiment) Next, a monitoring system 100 according to a second embodiment will be described. The monitoring system 100 according to the second embodiment has the same basic configuration as the monitoring system 100 according to the first embodiment shown in Fig. 2 etc., and therefore a detailed description of the configuration will be omitted. Below, functions different from the first embodiment will be mainly described.

[0130] The monitoring system 100 of the first embodiment starts the "special mode" monitoring process when the user U operates the "special mode button" on the user terminal 30 as a trigger. In contrast, the monitoring system 100 of the second embodiment is configured so that the monitoring system 100 automatically starts the "special mode" monitoring process at a predetermined timing (specifically, when a predetermined regular check time is reached as a trigger). Therefore, the user U does not need to operate any buttons or speak to the system. Furthermore, in this second embodiment, the user U may check the status of the special mode processing by viewing the information displayed on the user terminal 30 in real time, or may check the status of the special mode processing recorded in the monitoring system 100 later by selecting it from a list displayed on the user terminal 30.

[0131] In the second embodiment, the storage unit 23 includes a periodic confirmation information table 233, as indicated by the dashed lines in Fig. 2, in addition to the privacy protection level table 231 and the information setting table 232. The privacy protection level table 231 and the information setting table 232 have the same data structures as those in the first embodiment shown in Figs. 3A and 3B.

[0132] The periodic check information table 233 is a table in which an initial value of a privacy protection level and a periodic check time are stored for each subject P. Fig. 3C is a schematic diagram showing an example of the data structure of the periodic check information table 233. As shown in Fig. 3C, the periodic check information table 233 has a subject ID 233a, an initial level value 233b, and a periodic check time 233c.

[0133] The subject ID of subject P is stored in subject ID 233a. The initial level value 233b stores the initial value of the privacy protection level. This initial level value 233b is normally set to the maximum level 5, but can also be set to a value below level 4 if more immediate monitoring is required. The regular check time 233c stores the time to start calling out to subject P (monitoring process in "special mode"). Note that if the calls are not made every day but at intervals, such as once or multiple times a week or once or multiple times a month, the day of the week, date, and time are stored.

[0134] An example of the flow of operation of the monitoring system 100 according to the second embodiment will be described below with reference to the flowchart in FIG. 7. The flowchart shown in FIG. 7 is a flowchart for explaining an example of the flow of monitoring operation in "special mode." The operation shown in FIG. 7 is started when the current time reaches the periodic check time 233c obtained from the periodic check information table 233 during monitoring operation in normal mode. Note that, in the second embodiment as well, while the special mode is being executed, the imaging unit 11 of the acquisition device 10 captures images, the sound collection unit 12 collects sound, and the operation unit 14 accepts input of a response action, either continuously or at regular intervals.

[0135] First, in step S21, in response to an instruction from the server device 20, the acquisition device 10 outputs a predetermined call sound from the sound output unit 13. When the privacy protection level is level 5, this call sound may be a chime, music, or other sound, a pre-registered voice, or the voice of the user U. Furthermore, when the call count is the second time or later (level 4 or lower), instead of a chime, the real-time call sound of the user U collected by the sound collection unit 32 of the user terminal 30 may be output.

[0136] In the next step S22, upon receiving input of subject information including images, audio, operation signals, etc. from the acquisition device 10, the abstraction processing unit 213 in the server device 20 performs abstraction processing on the image signal in accordance with the privacy protection level, and the output unit 214 outputs the subject information to the user terminal 30 in accordance with the privacy protection level. In addition, the control unit 21 stores the images (unprocessed actual video or images after abstraction processing), audio, and state detection results in the information setting table 232 of the storage unit 23.

[0137] Upon receiving the input of the target person information from the server device 20, the user terminal 30 displays on the display unit 35 the first or fourth information display screen 51, 54 shown in FIG. 5B or FIG. 5E.

[0138] In the next step S23, the state detection unit 211 of the server device 20 automatically detects the state of the subject P based on the subject information. If the state detection unit 211 detects normality or abnormality (determination of step S23 is YES), the program proceeds to step S30, and if it is impossible to determine whether the state is normal or abnormal (the state is unknown) (determination of step S23 is NO), the program proceeds to step S24.

[0139] Then, in the next step S24, the state detection unit 211 determines whether or not the subject P has made a response operation, specifically, whether or not the "normal" or "abnormal" button on the operation unit 14 has been pressed, based on the subject information. If a response operation has been made (determination of YES in step S24), the program proceeds to step S30, and if no response operation has been made (determination of NO in step S24), the program proceeds to step S25 as it is not possible to make a determination.

[0140] In the next step S25, the control unit 21 determines whether or not an operation to terminate the special mode from the user U has been detected, specifically, whether or not the control unit 21 has detected an operation to terminate the application by the user U of the emergency button 51b or the end button 51c on the user terminal 30. If an end operation has been detected (determination of YES in step S25), the program proceeds to step S30, and if an end operation has not been detected (determination of NO in step S25), the program proceeds to step S26.

[0141] In the next step S26, it is determined whether a predetermined time corresponding to the privacy protection level has elapsed. As in the first embodiment, this "predetermined time" is set to 3 minutes for levels 5 to 1 in the second embodiment, but the predetermined time may be different for each level, or may be unlimited (infinite) for level 1, and is not limited to these.

[0142] Then, while the predetermined time has not elapsed (determination in step S26 is NO), the program returns to step S21 and repeats the processing of steps S21 to S26. Once the predetermined time has elapsed (determination in step S26 is YES), the program proceeds to step S27.

[0143] In step S27, the monitoring system 100 outputs a status determination result (undeterminable) to the user terminal 30 according to the privacy protection level. When the privacy protection level is level 5 or 4, the user terminal 30 receives this "undeterminable" input and displays a first information display screen 51 on the display unit 35, in which only the message "No response" is presented in the information display area 51a as the status detection result, as shown in FIG. 5B. At level 4, the sound of the target area collected by the sound collection unit 12 is also output from the sound output unit 33. At level 3 or lower, the user terminal 30 displays a fourth information display screen 54 on the display unit 35, in which an abstracted image or an unprocessed image is embedded in the information display area 54a, as shown in FIG. 5E. Even at level 3 or lower, the sound of the target area collected by the sound collection unit 12 of the acquisition device 10 is output from the sound output unit 33 of the user terminal 30, and the user U can make a call from the user terminal 30.

[0144] The user terminal 30 may notify the user U by voice, vibration, or the like that the status detection result of the periodic check has been received. Upon receiving this notification, the user U can check the status of the subject P on the first or fourth information display screen 51, 54.

[0145] Thereafter, the program proceeds to step S28, where the setting unit 212 of the server device 20 changes the privacy protection level (lowering the level by one level). Next, the program proceeds to step S29, where it is determined whether the privacy protection level is less than 1. If the privacy protection level is level 1 or higher (determination in step S29 is NO), the program returns to step S21, whereby monitoring in the "special mode" at the privacy protection level after the change is repeated. On the other hand, if the privacy protection level is less than level 1 (determination in step S29 is YES), the program proceeds to step S30.

[0146] The final step S30 is executed in one of the following cases: (5) When a normal or abnormal state of subject P is detected in step S23. (6) When a normal or abnormal response action from the subject P is detected in step S24. (7) When an operation to end the special mode from the user terminal 30 is detected in step S25. (8) When it is determined in step S29 that the privacy protection level is lower than level 1.

[0147] In step S30, the monitoring system 100 outputs the reason for terminating the special mode to the user terminal 30, terminates the special mode monitoring, and returns to normal mode monitoring. The specific processing is similar to the processing of step S10 in the first embodiment, and therefore a description thereof will be omitted. Furthermore, upon receiving the input of the reason for termination, the user terminal 30 displays the first and fourth information display screens 51 and 54, which present a message according to the reason for termination, on the display unit 35. At this time, the user terminal 30 may notify the user U by voice, vibration, or the like that the reason for terminating the periodic check has been received.

[0148] Next, a more specific example of operation according to the state of the subject P in the monitoring system 100 of the second embodiment will be described. The initial value of the privacy protection level of the subject P is set to level 5. First, when the regular check time arrives, the program proceeds to step S21, and under the instruction of the server device 20, a call sound such as a chime is output from the sound output unit 13 of the acquisition device 10. Next, in step S22, subject information according to the privacy protection level is output to the user terminal 30, but since the current privacy protection level is level 5, no image or sound is output to the user terminal 30.

[0149] When the state detection unit 211 detects whether the subject P is normal or abnormal based on the subject information in the next step S23, or when a response operation by the subject P is detected in step S24, the program proceeds to step S30. In this step S30, the user terminal 30 displays on the display unit 35 a first information display screen 51 presenting a message indicating that normal or abnormal has been detected, depending on the reason for termination. By visually checking this first information display screen 51, the user U can confirm whether the subject P is normal or abnormal. Thereafter, special mode monitoring ends, and monitoring returns to normal mode.

[0150] On the other hand, while neither normality nor abnormality is detected at level 5, the processes of steps S21 to S26 are repeated until a predetermined time has elapsed unless an operation to end the special mode is performed from the user terminal 30. When the predetermined time corresponding to level 5 has elapsed, the program proceeds to step S27, and the first information display screen 51 shown in FIG. 5B is displayed on the display unit 35, with "No response" displayed in the information display area 51a. Then, the program proceeds to step S28, where the setting unit 212 changes the privacy protection level to level 4, and the program proceeds to step S29.

[0151] The program then returns from step S29 to step S21, and monitoring continues in the special mode at level 4. First, in step S21, the sound output unit 13 of the acquisition device 10 outputs a pre-registered sound to call out to the subject P. Next, the program proceeds to step S22, but because the privacy protection level is level 4, no image or sound of the subject P is output to the user terminal 30.

[0152] When the state detection unit 211 detects whether the subject P is normal or abnormal based on the subject information in the next step S23, or when a response operation by the subject P is detected in step S24, the program proceeds to step S30. In this step S30, the user terminal 30 displays on the display unit 35 a first information display screen 51 presenting a message indicating that normal or abnormal status has been detected, depending on the reason for termination. By visually checking this first information display screen 51, the user U can confirm whether the subject P is normal or abnormal. Thereafter, special mode monitoring ends, and monitoring returns to normal mode.

[0153] On the other hand, while no normal or abnormal state is detected at level 4, the processing of steps S21 to S26 is repeated until a predetermined time has elapsed unless an operation to end the special mode is performed from the user terminal 30. When the predetermined time corresponding to level 4 has elapsed, the program proceeds to step S27, and the user terminal 30 displays on the display unit 35 the first information display screen 51 shown in FIG. 5B, in which "No response" is displayed in the information display area 51a. The program then proceeds to step S28, in which the setting unit 212 changes the privacy protection level to level 3, and the program proceeds to step S29. Note that the information on the first information display screen 51 displayed at level 4 and the information on the fourth information display screen 54 displayed at levels 3 to 1 (described later) are recorded in the monitoring system 100 (e.g., the storage unit 38 of the user terminal 30) together with the information on the first information display screen 51 output at level 5. The recorded information is displayed on the user terminal 30 as a list of image names and thumbnails, and the user U can select a desired image from this list and view it later.

[0154] The program then returns from step S29 to step S21, and monitoring in the special mode at level 3 continues. First, in step S21, sound output unit 13 of acquisition device 10 outputs a pre-registered sound to call out to subject P. Next, the program proceeds to step S22, where abstraction processing unit 213 performs abstraction processing corresponding to level 3 on the image acquired by acquisition device 10, i.e., avatarization of a human figure, and output unit 214 outputs subject information including the avatar image and the voice of subject P to user terminal 30. Upon receiving this input of subject information, user terminal 30 fits the avatar image into information display area 54a, displays fourth information display screen 54 shown in FIG. 5E on display unit 35, and outputs the voice of subject P from sound output unit 33.

[0155] When the state detection unit 211 detects whether the subject P is normal or abnormal based on the subject information in the next step S23, or when a response operation by the subject P is detected in step S24, the program proceeds to step S30. In this step S30, the user terminal 30 displays on the display unit 35 a fourth information display screen 54 presenting a message indicating that normal or abnormal has been detected, depending on the reason for termination. By visually checking this fourth information display screen 54, the user U can confirm whether the subject P is normal or abnormal.

[0156] At this time, an image of the user U is displayed on the display unit 15 of the acquisition device 10, and the user U's real-time voice call is output from the sound output unit 13. This allows both the user U and the target person P to see the voice and image and to converse, allowing the user U to check the target person P's condition in more detail. Furthermore, the target person P feels a sense of security by seeing the voice and image of the user U. After that, the special mode monitoring ends and the monitoring returns to the normal mode.

[0157] On the other hand, while neither normal nor abnormal is detected at level 3, the processes of steps S21 to S26 are repeated until a predetermined time has elapsed unless an operation to end the special mode is performed from user terminal 30. When the predetermined time corresponding to level 3 has elapsed, the program proceeds to step S27, and user terminal 30 displays the message "No response" on display unit 35 alongside the avatar image in information display area 54a. The program then proceeds to step S28, where setting unit 212 changes the privacy protection level to level 2, and proceeds to step S29.

[0158] The program then returns from step S29 to step S21, and monitoring in the special mode at level 2 continues. First, in step S21, the sound output unit 13 of the acquisition device 10 outputs a pre-registered sound to call out to the subject P. Next, the program proceeds to step S22, where the abstraction processing unit 213 performs abstraction processing corresponding to level 3 on the image acquired by the acquisition device 10, i.e., mosaic processing of the human image, and the output unit 214 outputs subject information including the mosaic image and the voice of the subject P to the user terminal 30. Upon receiving this input of the subject information, the user terminal 30 embeds the mosaic image in the information display area 54a, displays the fourth information display screen 54 shown in FIG. 5E on the display unit 35, and outputs the voice of the subject P from the sound output unit 33.

[0159] When the state detection unit 211 detects whether the subject P is normal or abnormal based on the subject information in the next step S23, or when a response action of the subject P is detected in step S24, the program proceeds to step S30. In this step S30, the user terminal 30 displays a fourth information display screen 54 on the display unit 35, which displays a message indicating that normal or abnormal status has been detected, depending on the reason for termination. By visually checking this fourth information display screen 51, the user U can confirm whether the subject P is normal or abnormal.

[0160] Even at level 2, an image of user U is displayed on the display unit 15 of the acquisition device 10, and user U's real-time voice call is output from the sound output unit 13. This allows both user U and target person P to see the voice and image and to converse, allowing user U to check the target person P's condition in more detail. Also, target person P feels a sense of security by seeing user U's voice and image. After that, special mode monitoring ends and monitoring returns to normal mode.

[0161] On the other hand, while neither normal nor abnormal conditions are detected at level 2, the processes of steps S1 to S6 are repeated until a predetermined time has elapsed unless an operation to end the special mode is performed from the user terminal 30. When the predetermined time corresponding to level 2 has elapsed, the program proceeds to step S27, where the user terminal 30 displays the message "No response" on the display unit 35 alongside the mosaic image in the information display area 54a. The program then proceeds to step S28, where the setting unit 212 changes the privacy protection level to level 1, and the program proceeds to step S29.

[0162] The program then returns from step S29 to step S21, and monitoring in the special mode at level 1 continues. First, in step S21, the sound output unit 13 of the acquisition device 10 outputs a pre-registered sound to call out to the subject P. Next, the program proceeds to step S22, but because it is at level 1, the output unit 214 outputs subject information including an unprocessed image and the voice of the subject P to the user terminal 30. Upon receiving this input of the subject information, the user terminal 30 fits the unprocessed image into the information display area 54a, displays the fourth information display screen 54 shown in FIG. 5E on the display unit 35, and outputs the voice of the subject P from the sound output unit 33.

[0163] When the state detection unit 211 detects whether the subject P is normal or abnormal based on the subject information in the next step S23, or when a response operation by the subject P is detected in step S24, the program proceeds to step S30. In this step S30, the user terminal 30 displays on the display unit 35 a fourth information display screen 54 presenting a message indicating that normal or abnormal has been detected, depending on the reason for termination. By visually checking this fourth information display screen 54, the user U can confirm whether the subject P is normal or abnormal.

[0164] Even at level 1, an image of user U is displayed on the display unit 15 of the acquisition device 10, and user U's real-time voice call is output from the sound output unit 13. This allows both user U and target person P to see the voice and image and to converse, allowing user U to check the target person P's condition in more detail. Also, target person P feels a sense of security by seeing user U's voice and image. After that, special mode monitoring ends and monitoring returns to normal mode.

[0165] On the other hand, while no normal or abnormal state is detected at level 1, the process of steps S21 to S26 is repeated until a predetermined time has elapsed unless an operation to terminate the special mode is performed from the user terminal 30. After the predetermined time corresponding to level 1 has elapsed, the program proceeds to step S27, where the message "No response" is displayed on the display unit 35 alongside the actual image in the information display area 54a. The program then proceeds to step S28, where the setting unit 212 changes the privacy protection level to level 0. Next, the program proceeds to step S29. However, because the privacy protection level is lower than level 1 (level 0), the determination in step S29 is YES, and the program proceeds to step S30. In step S30, the user terminal 30 displays a message such as "The subject's condition could not be confirmed. The special mode will be terminated." depending on the reason for termination. The program then terminates the special mode monitoring and returns to normal mode monitoring. Therefore, as described above, the user U can check the status of the special mode processing by viewing the information displayed on the user terminal 30 in real time, or select it from the list displayed on the user terminal 30 and check it later.

[0166] In the second embodiment, when the predetermined time at level 1 is set to unlimited, the processes of steps S21 to S26 are repeated as long as no normal or abnormal state is detected. That is, the fourth information display screen 54 at level 1 continues to be displayed on the display unit 35 of the user terminal 30. Therefore, the user U can check the condition of the subject P at level 1 by using the fourth information display screen 54 displayed on the display unit 35 and the sound output from the sound output unit 33. Then, if the user U checks the condition of the subject P and performs an end operation while the process at level 1 is being repeated, the program proceeds from step S25 to step S30, and the user terminal 30 displays a message such as "Special mode will end and return to normal mode" on the display unit 35 depending on the reason for ending, and then special mode monitoring ends and returns to normal mode monitoring.

[0167] (Third embodiment) Next, a monitoring system 100 of a third embodiment will be described. As shown by the dashed line in FIG. 2 , the monitoring system 100 of the third embodiment has the same basic configuration as the monitoring system 100 of the first embodiment, except that the acquisition device 10 has an input / output interface (I / F) 19, and a biometric information acquisition unit 41 and an environmental information acquisition unit 42 are connected to the acquisition device 10 via this input / output interface 19. Therefore, a detailed description of the same configuration will be omitted. Below, the configuration and functions that differ from the first embodiment will be mainly described. Note that such a configuration of the third embodiment can also be applied to the monitoring system 100 of the second embodiment.

[0168] The input / output interface 19 is composed of an interface circuit conforming to a serial bus standard such as USB, or an interface circuit conforming to a short-range wireless communication standard such as Wi-Fi (registered trademark), Bluetooth (registered trademark), or NFC, and transmits and receives data to and from the biometric information acquisition unit 41 and the environmental information acquisition unit 42 via wired or wireless communication.

[0169] The biological information acquiring unit 41 acquires biological information of the subject P. Examples of the biological information acquiring unit 41 include a sensor that acquires the subject P's heart rate, a sensor that acquires respiration, a sensor that acquires pulse, a sensor that acquires blood pressure, a sensor that acquires body temperature, a sensor that acquires oxygen saturation, a sensor that acquires brain waves, a sensor that acquires weight, a sensor that acquires body movement, etc., and one or more of these sensors can be selected from these sensors, but the biological information acquiring unit 41 is not limited to these sensors.

[0170] The biological information acquisition unit 41 is basically a device that the subject P uses to measure the biological information themselves, but it is not necessarily the subject P who measures it. For this reason, it is preferable to use a sensor that can automatically measure biological information without wearing it, using a laser, infrared rays, radio waves, etc. Specific examples include a body temperature sensor that uses infrared rays to detect body temperature from a remote location, and a vital sensor that uses radio waves with a microwave Doppler element to detect pulse waves, breathing, body movements, etc. from a remote location.

[0171] Additionally, beds and the like equipped with body temperature sensors, blood pressure sensors, weight sensors, etc., that can measure body temperature, blood pressure, and weight while the subject P is lying down are also available on the market. Because beds and the like are always used by the subject P, using these sensors as the biometric information acquisition unit 41 allows for more appropriate and reliable acquisition of biometric information.

[0172] The environmental information acquisition unit 42 acquires environmental information related to the target area. Examples of the environmental information acquisition unit 42 include a human presence sensor that detects a person, a CO sensor that detects carbon monoxide concentration, a smoke sensor that detects smoke, a heat sensor that detects heat, a vibration sensor that detects vibration, a gas sensor that detects gas leaks, a noise sensor that detects noise, and a sound collection sensor that collects environmental sounds. One or more of these sensors can be selected from these sensors, but the environmental information acquisition unit 42 is not limited to these sensors.

[0173] The biometric information from the biometric information acquisition unit 41 and the environmental information from the environmental information acquisition unit 42 are used as one of the criteria for determining the state by the state detection unit 211 and one of the criteria for the user U to confirm the state. For example, the state detection unit 211 can further increase the reliability of the state detection result by detecting the state based on the biometric information in addition to the response action, image, and voice. Furthermore, if the biometric information from the biometric information acquisition unit 41 is output to the user terminal 30, the user U can check the state of the subject P in more detail. Furthermore, even when the subject P is not aware of poor health and is performing normal actions, for example, if the state detection unit 211 detects that the oxygen saturation level is lower than the standard, the state detection unit 211 can determine that there is an "abnormality" and notify the user U via the user terminal 30, allowing appropriate action to be taken before a sudden deterioration in health occurs.

[0174] Furthermore, even when the condition cannot be determined by the response action, image, or sound, it is possible to determine whether the subject P is normal or abnormal based on the information detected by the biological information acquisition unit 41. For example, if no response action is confirmed and the subject P's biological information is an abnormal value, such as an abnormally high body temperature or heavy breathing, it is determined that there is some abnormality in the subject P. On the other hand, even if there is no response action, if the biological information is a normal value, it is determined that the subject P is normal. More specifically, if the respiratory rate, etc. is abnormal, it is determined that there is an abnormality that prevented the subject P from responding, and if the respiratory rate, etc. is normal, it is determined that the subject P was unable to respond because they were asleep, etc.

[0175] As a variant example, when a button is provided on each information presentation screen to instruct the user to lower the privacy protection level and check the status, user U can make a more appropriate decision by referring to biometric information and environmental information when deciding whether or not to operate this button.

[0176] Furthermore, when a human presence sensor is used as the environmental information acquisition unit 42, it is possible to detect whether the subject P is at home or out. Furthermore, when a plurality of acquisition devices 10 or imaging units 11, etc. are installed, more efficient monitoring can be achieved by acquiring information such as images, sounds, and operation signals only with the acquisition device 10 or imaging unit 11, etc. that detects the subject P with the human presence sensor.

[0177] Furthermore, by using a CO sensor, smoke sensor, heat sensor, vibration sensor, gas sensor, etc. as the environmental information acquisition unit 42, it is possible to prevent these accidents and take prompt action by detecting or predicting abnormalities such as carbon monoxide poisoning, fire, earthquake, gas leak, etc. and outputting the detected abnormalities to the user terminal 30. Furthermore, by using a noise sensor or an environmental sound collection sensor, it is possible to check whether the subject P is unwell (abnormal) due to noise, etc., and to check for the presence of a third party other than the subject P.

[0178] As described above, the monitoring system 100 according to the first to third embodiments is a system comprising an acquisition unit (acquisition device 10, imaging unit 11, sound collection unit 12, operation unit 14) that acquires information (subject information) including at least one of an image and audio of the subject P, a terminal (user terminal 30) to which the information is input, and a setting unit 212 that sets a level of privacy protection based on at least one of the information input to the terminal and the state of the subject.

[0179] In addition, the systems according to the first to third embodiments are systems that include an acquisition unit (acquisition device 10, imaging unit 11, sound collection unit 12, operation unit 14) that acquires information (subject information) related to the condition of the subject P, a terminal (user terminal 30) into which information is input and which can make a call to the subject P, and a setting unit 212 that sets a level for protecting the privacy of the subject P each time a call is made from the terminal to the subject P.

[0180] In addition, the information processing device (server device 20) according to the first to third embodiments is an information processing device that includes a setting unit 212 that sets a level for protecting the privacy of the subject P based on information (subject information) including at least one of an image and audio of the subject P and at least one of the state of the subject P, and outputs information to a terminal (user terminal 30) according to the level.

[0181] In addition, the information processing device (server device 20) according to the first to third embodiments is an information processing device that includes a setting unit 212 that sets a level for protecting the privacy of the subject P each time a call is repeatedly made to the subject P from the terminal (user terminal 30), and outputs information regarding the subject's condition (subject information) to the terminal according to the level.

[0182] In addition, the information processing methods according to the first to third embodiments are information processing methods having an acquisition process for acquiring information (subject information) including at least one of an image and audio of the subject P, an output process for outputting the information to a terminal (user terminal 30), and a setting process for setting a level for protecting the privacy of the subject P based on at least one of the information input to the terminal and the state of the subject P.

[0183] In addition, the information processing methods according to the first to third embodiments are information processing methods that include an acquisition process for acquiring information (subject information) relating to the status of the subject P, and a setting process for setting a level for protecting the privacy of the subject P each time a call is repeatedly made to the subject P by the terminal (user terminal 30) into which the information is input.

[0184] Furthermore, the programs according to the first to third embodiments are programs executed by a computer, and cause the computer to function as an acquisition unit (acquisition device 10, imaging unit 11, sound collection unit 12, operation unit 14) that acquires information (subject information) including at least one of an image and audio of the subject P, an output unit 214 that outputs the information to a terminal (subject information), and a setting unit 212 that sets a level of privacy protection for the subject P based on at least one of the information output to the terminal and the state of the subject P.

[0185] Furthermore, the programs according to the first to third embodiments are programs executed by a computer, and cause the computer to function as an acquisition unit (acquisition device 10, imaging unit 11, sound collection unit 12, operation unit 14) that acquires information relating to the subject's condition (subject information), and a setting unit 212 that sets a level of privacy protection for subject P each time a call is made to subject P by the terminal (user terminal 30) into which information is input.

[0186] Therefore, according to the first to third embodiments, the user U can appropriately check the condition (normal or abnormal) of the subject P by checking the information output to the user terminal 30 according to the privacy protection level. Furthermore, the minimum necessary information according to the privacy protection level can be provided to the user U, enabling appropriate monitoring of the subject P while respecting his / her privacy.

[0187] Furthermore, as a modification of the first and third embodiments, the acquisition unit (acquisition device 10, etc.) may repeatedly acquire information based on input from a terminal (user terminal 30), and the setting unit 212 may change the privacy protection level each time at least one of the acquisition of information by the acquisition unit and output to the terminal is repeated. Furthermore, as a modification of the second embodiment, the acquisition unit (acquisition device 10, etc.) may repeatedly acquire information at a predetermined timing (for example, when a regular check time is reached), and the setting unit 212 may change the privacy protection level each time at least one of the acquisition of information by the acquisition unit and output to the terminal is repeated.

[0188] With the above configuration, the monitoring system 100 can provide information to the user U while gradually lifting the privacy protection level until it can determine whether the person P being monitored is normal or abnormal. As a result, the reliability of the normal or abnormal determination is increased while respecting the privacy of the person P as much as possible, and the user U can more appropriately confirm whether the person is normal or abnormal. Furthermore, in a configuration in which processing is repeated based on input from a terminal, as in the variations of the first and third embodiments, the user U can monitor at their own timing, and the privacy protection level can be gradually lowered at the user U's will, allowing monitoring that respects the user U's will.

[0189] In contrast, in a configuration in which the monitoring system 100 repeats processing at a predetermined timing, as in the second embodiment or its variant, the user U does not need to perform input operations himself, and only needs to check the information when it is output from the monitoring system 100 to the terminal, making monitoring easier.

[0190] Furthermore, the first to third embodiments include a condition detection unit 211 that detects the condition of the subject P based on information acquired by an acquisition unit (such as the acquisition device 10). With this configuration, the user U can be presented with not only images and sounds but also detection results of normal, abnormal, and indeterminable, and the user U can refer to these to more appropriately check the condition of the subject P.

[0191] Furthermore, in the first to third embodiments, the setting unit 212 sets one of a plurality of preset levels for the subject P based on the information (subject information) and the state detected by the state detection unit 211. At this time, the setting unit 212 changes the privacy protection level when the state of the subject P cannot be determined. With this configuration, it is possible to output information according to the changed privacy protection level to the user U, and the user U can more clearly confirm the state of the subject P.

[0192] In addition, in the modified examples of the first and third embodiments, the setting unit 212 changes the privacy protection level based on an input from a terminal (user terminal 30). With this configuration, the user U can obtain information according to the changed privacy protection level by issuing an instruction from the terminal, and can more clearly confirm the state of the subject P.

[0193] Furthermore, in the first to third embodiments, when the number of calls to the target person from the terminal is the initial value, the setting unit 212 sets the privacy protection level to the maximum level, and lowers the privacy protection level each time the number of calls increases. Alternatively, in a modified example, when a button is provided to instruct lowering the privacy protection level and checking the status, the setting unit 212 lowers the privacy protection level each time this button is pressed. With these configurations, it becomes possible to monitor the target person P while maximally respecting their privacy at the beginning of monitoring, and since the privacy protection is gradually released as the number of calls and button operations increases, more appropriate monitoring according to the target person P becomes possible.

[0194] Furthermore, in the first to third embodiments, an abstraction processing unit 213 is provided that performs a predetermined abstraction process on information in accordance with the privacy protection level set by the setting unit 212. This configuration makes it possible to appropriately protect the privacy of the subject P. At this time, when the privacy protection level is at the minimum level, the abstraction processing unit 213 does not perform the abstraction process on the information, thereby allowing the user U to confirm actual video and live audio of the subject P, enabling more appropriate monitoring, and enabling abnormalities to be quickly and appropriately confirmed and responded to promptly.

[0195] If the abstraction processing by the abstraction processing unit 213 is at least one of mosaic processing, blurring processing, de-identification processing for a person's image, and voice change processing for a voice, it is possible to provide the user U with information that makes it easier to confirm the status of the subject P while appropriately protecting the privacy of the subject P. Note that in the first to third embodiments, the subject P's image is mosaic processed or de-identified into an avatar according to the privacy protection level, so it is possible to protect the privacy of the subject P according to the privacy protection level and provide the user U with appropriate information.

[0196] Furthermore, in the first to third embodiments, by providing an output unit 214 that outputs information to a terminal (user terminal 30), information can be appropriately supplied to the user terminal 30 and processing can be distributed, thereby improving the information processing speed.

[0197] Furthermore, in the first to third embodiments, when the privacy protection level is at the maximum level, the output unit 214 does not output information (images, audio) to the terminal, so that the privacy of the subject P can be protected to the maximum extent during normal monitoring. Furthermore, when such information is not output, it is clear that the subject P is normal, which gives the user U a sense of security.

[0198] Furthermore, in the first to third embodiments, storage units 18, 23, 38 are provided that store information acquired by an acquisition unit (acquisition device 10, etc.). These storage units 18, 23, 38 may be in any of the acquisition device 10, the server device 20, and the user terminal 30. Storing information in the storage units 18, 23, 38 in this manner allows the user U or a system administrator, etc., to check the past state of the subject P, enabling more appropriate monitoring.

[0199] Furthermore, in the first to third embodiments, the acquisition unit (acquisition device 10, etc.) includes at least one of an imaging unit 11 that captures an image of a predetermined target area including the subject P, a sound collection unit 12 that collects sound from the target area, and an operation unit 14 that is operated by the subject P, and acquires at least one of an image, sound, and operation as information. With this configuration, the acquisition unit can acquire more detailed information, enabling more appropriate monitoring, and the subject P can operate the operation unit 14 to check the state of the subject P's movements and more appropriately detect whether they are normal or abnormal.

[0200] Furthermore, in the first to third embodiments, the acquisition unit (acquisition device 10, etc.) includes a sound output unit 13 that outputs sound, and upon receiving input from a terminal (user terminal 30) or at a predetermined timing, notifies the subject P of a predetermined call sound from the sound output unit 13, and acquires the subject P's response action to the call sound as information. With this configuration, the monitoring system 100 detects whether the subject P is normal or abnormal based on the response action, and if there is no response action, detects that the determination is impossible, thereby more appropriately detecting the state.

[0201] The third embodiment also includes a biological information acquisition unit 41 that acquires biological information of the subject P. The biological information acquisition unit 41 has a sensor that acquires at least one selected from the heart rate, respiration, pulse wave, blood pressure, body temperature, oxygen saturation, brain wave, weight, and body movement of the subject P as the biological information.

[0202] Furthermore, the third embodiment includes an environmental information acquisition unit 42 that acquires environmental information related to a predetermined target area. The environmental information acquisition unit 42 has at least one selected from a human sensor that detects a person, a CO sensor that detects a carbon monoxide concentration, a smoke sensor that detects smoke, a heat sensor that detects heat, a vibration sensor that detects vibrations, a gas sensor that detects gas leaks, a noise sensor that detects noise, and a sound collection sensor that collects environmental sounds.

[0203] Therefore, in the third embodiment, by outputting the biological information and environmental information to the user terminal 30 in addition to the information on the subject P, the user U can check the condition of the subject P more appropriately.

[0204] Moreover, in the first to third embodiments, an information processing device (server device 20) having a setting unit 212 is provided, and this information processing device is connected to an acquisition unit (acquisition device 10) and a terminal (user terminal 30) via a communication network N. Furthermore, this information processing device (server device 20) is provided with the setting unit 212 that sets a privacy protection level according to information including at least one of an image and a voice of the subject P, and is configured to output information to the terminal according to the privacy protection level.

[0205] With this configuration, the information processing device can comprehensively control and manage the monitoring system 100. Furthermore, even if the monitoring system 100 has multiple acquisition units and terminals, the information processing device can manage and control them, thereby enabling appropriate monitoring.

[0206] The above describes in detail the embodiments and modifications of the present disclosure with reference to the drawings, but the specific configurations are not limited to these embodiments and modifications, and design changes that do not deviate from the gist of the present disclosure are included in the present disclosure.

[0207] For example, in each of the above embodiments, the setting unit 212 lowers the privacy protection level one level at a time from the maximum level to the minimum level, but this is not limited to this. For example, as a modified example, the setting unit 212 can be configured to set one of a plurality of pre-set privacy protection levels for the subject P based on the information (image, audio) and the state detected by the state detection unit 211. For example, if an indeterminable or normal state is detected, but the subject P has a painful expression or movement or an abnormality is detected based on the results of image analysis or audio analysis, and an emergency is required, setting the privacy protection level lower allows for a more rapid response. Furthermore, if no particular abnormality is found in the analysis results or state, setting the privacy protection level higher suppresses unnecessary cancellation of privacy protection, enabling more efficient and appropriate monitoring.

[0208] Furthermore, in each of the above embodiments, the acquisition unit (acquisition device 10) monitors a subject P who lives alone, but this is not limited to this. Alternatively, as a modified example, the acquisition unit may be configured to detect a predetermined subject P based on information, and when this subject P is detected, the information may be output to a terminal (user terminal 30). With this configuration, for example, even when there is a home helper or visitor, when there is a roommate, or when a caregiver or the like enters and leaves a facility, the subject P can be identified from among multiple people, enabling more efficient and appropriate monitoring. Furthermore, output and storage of images, etc., when the subject P is absent can be suppressed, thereby reducing the communication load and the amount of storage in the storage unit 23, etc.

[0209] Furthermore, in the first to third embodiments, one acquisition unit (acquisition device 10) is installed, but the number is not limited to one, and multiple acquisition units may be installed in the residence of the subject P. With this configuration, it is possible to acquire information about the subject P not only from places where the subject P spends his or her daily time, such as the living room or bedroom, but also from places where falls or the like are likely to occur, such as the bathroom, hallway, and toilet, thereby enabling more appropriate monitoring and enabling earlier detection of abnormalities and more rapid response.

[0210] In this case, the multiple acquisition units (acquisition devices 10) are configured to detect a predetermined subject P based on the information, and if the information acquired by one of the multiple acquisition units that detected the subject P is output to the terminal (user terminal 30), the information of the subject P can be output more reliably, and the amount of information can be reduced, reducing communication load and improving information processing efficiency. Furthermore, the acquisition unit that acquires information from the multiple acquisition units may be selected depending on the time of day. For example, if an acquisition unit installed in the living room or hallway acquires information during the day, and an acquisition unit installed in the bedroom or bathroom acquires information at night, the information of the subject P can be acquired more reliably and efficiently.

[0211] Furthermore, in the first to third embodiments, the setting unit 212 is provided in the server device 20 as an information processing device, but is not limited to this configuration. The setting unit 212 may be provided in either the acquisition unit (acquisition device 10) or the terminal (user terminal 30), and the monitoring system 100 can have a simpler configuration.

[0212] More specifically, without providing the server device 20, the control unit 16 of the acquisition device 10 or the control unit 36 ​​of the user terminal 30 can function as the state detection unit 211, the setting unit 212, the abstraction processing unit 213, and the output unit 214, and the acquisition device 10 and the user terminal 30 can be connected via the communication network N to enable data transmission and reception.

[0213] Furthermore, in the first to third embodiments, the state detection unit 211 and the abstraction processing unit 213 are provided in the server device 20 serving as an information processing device, but the present invention is not limited to this configuration. The state detection unit 211 and / or the abstraction processing unit 213 may be provided in the acquisition unit (acquisition device 10). This configuration not only simplifies the configuration and functions of the server device 20, but also increases the degree of freedom in installing each unit. Furthermore, state detection processing and abstraction processing based on images and sounds can be performed on the acquisition device 10 side, and the amount of image data and sound data to be transmitted to the server device 20 via the communication network N can be reduced, thereby reducing the communication load.

[0214] Furthermore, in each of the above embodiments, the privacy protection level is lowered by one level from level 5 regardless of who the user U is, but this is not limited to this and the number of levels to be lowered may be increased or decreased. For example, the initial value of the privacy protection level and the level to be lowered may be different depending on the user U. Specifically, if the user U is a child of the target person P, the privacy cancellation may be relaxed and the initial value may be set to level 4 or lower, or the initial value may be set to level 5, but when changing from level 4, the level may be lowered from level 4 to level 1 without going through levels 3 and 2. Furthermore, even if there are multiple users U, even if the user U is a child, the initial value may be set low for the representative user U and high for the other users U.

[0215] On the other hand, if the user U is a third party such as a staff member of a nursing facility, the initial value may be set to level 5, and the changeable range may be set from level 5 to level 2, with no output of information of level 1 (unprocessed image). It is preferable to store such conditions for changing the privacy protection level for each user U in the storage unit 23 or the like.

[0216] In addition, in the above embodiments, the processing level of the image and audio of the subject P is described as being changed in stages according to the transition of each privacy protection level. However, this is not limited thereto. As a modified example, the processing level may be changed continuously within the same level. Specifically, for example, when mosaic processing is performed on an image at level 2, the coarseness of the mosaic may be changed as a change in processing level. In addition, in the above embodiments, the privacy protection level is described as being decreased in stages or automatically decreased over time. However, this is not limited thereto. As a modified example, the user U may manually change (raise) the level, for example, from the current level 4 to level 5. In addition, in the above embodiments, the number of privacy protection levels is set to five. However, this is not limited thereto. The number of levels may be ten, three, or any other number.

[0217] Furthermore, even if the privacy protection level is the same, the information to be output (for example, audio only, image only, audio and image, etc.) may be changed depending on the user U. With the above configuration, it is possible to protect the privacy of the subject P while monitoring more appropriately depending on the user U.

[0218] In addition, in each of the above embodiments, the server device 20 is used as the information processing device, but the present invention is not limited to this. For example, a notebook PC, a desktop PC, a tablet terminal, a smartphone, or the like can also be used as the information processing device, and such an information processing device can be installed in a management company or the like, or can be installed on the side of the target person P or the user U. [CROSS-REFERENCE TO RELATED APPLICATIONS]

[0219] This application claims priority based on Patent Application No. 2021-142503 filed with the Japan Patent Office on September 1, 2021, the entire disclosure of which is incorporated herein by reference in its entirety.

Claims

1. an acquisition unit that acquires information about the subject, including at least one of an image and a voice of the subject; a state detection unit that detects a state of the subject based on the information acquired by the acquisition unit; a setting unit for setting a level for protecting the privacy of the subject; Equipped with The setting unit changes the level in stages based on the condition of the subject.

2. A transmitting unit that transmits at least one of an image and a voice of the subject to a terminal, The system according to claim 1 , wherein the terminal comprises an output unit that outputs at least one of an image and a sound of the subject.

3. The system of claim 2 , wherein output of at least one of an image or sound of the subject by the output unit is limited based on the level.

4. The system of claim 3 , wherein the levels include at least three levels.

5. The system according to claim 4 , wherein the at least three levels include a first level in which output of at least one of an image and a sound of the subject by the output unit is not restricted.

6. The system according to claim 4 or 5, wherein the at least three levels include a second level in which the output unit does not output an image or sound of the subject.

7. The system according to claim 1 , wherein the acquisition unit includes at least one of an imaging unit that captures an image of a predetermined target area and a sound collection unit that collects sound from the predetermined target area.

8. the acquisition unit includes a biometric information acquisition unit that acquires biometric information of the subject, The system according to claim 1 , wherein the condition detection unit detects the condition of the subject based on the biological information.

9. the acquisition unit includes an environmental information acquisition unit that acquires environmental information of a predetermined target area; The system according to claim 1 , wherein the state detection unit detects the state of the subject based on the environmental information.

10. The system according to claim 1 , wherein the setting unit changes the level when the condition detection unit fails to detect the condition of the subject.

11. an abstraction processing unit that performs a predetermined abstraction process on at least one of the image and the sound acquired by the acquisition unit according to the level; The system according to claim 2 , wherein the transmitting unit transmits at least one of the image and the audio that have been subjected to the abstraction processing to the terminal.

12. The system according to claim 11 , wherein the abstraction processing by the abstraction processing unit is at least one of mosaic processing, blurring processing, and non-identification processing for a person's image, and voice modification processing for a voice.

13. a receiving unit that receives information about the subject, including at least one of an image and a voice of the subject; a state detection unit that detects a state of the subject based on the information; a setting unit for setting a level for protecting the privacy of the subject; Equipped with The information processing device wherein the setting unit changes the level in stages based on the condition of the subject.

14. A program executed by a computer, the program causing the computer to: a receiving unit that receives information about the subject, including at least one of an image and a voice of the subject; a state detection unit that detects a state of the subject based on the information; a setting unit for setting a level for protecting the privacy of the subject person; It functions as A program that causes the setting unit to function to gradually change the level based on the condition of the subject.

Images