Monitoring system, monitoring device, and monitoring program

Abstract

To provide a monitoring system, a monitoring device and a monitoring program for easily performing proper monitoring regardless of whether or not a monitoring object is positioned on a bed.SOLUTION: A monitoring system includes an imaging device that images a monitoring target area, and a skeleton information generation unit that generates skeleton information 10 indicating a skeleton of a monitoring target in the monitoring target area based on an imaging result by the imaging device, in which the imaging result includes a first image that is an image indicating a temperature distribution in the monitoring target area, and an image generation unit that generates a second image 12 that is an image in which the skeleton information 10 and the first image are superimposed.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a monitoring system, a monitoring device, and a monitoring program. [Background technology]

[0002] Patent Document 1 describes a monitoring system for monitoring a target. This monitoring system (referred to as a "monitoring system" in Patent Document 1) detects signs that the target (referred to as a "person being monitored" in Patent Document 1) is making a dangerous move in bed, based on two-dimensional thermal image information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-113243 Summary of the Invention [Problem to be solved by the invention]

[0004] The monitoring system described in Patent Document 1 can only be used when the monitoring target is asleep or otherwise positioned on the bed.

[0005] An object of the present invention is to provide a monitoring system, a monitoring device, and a monitoring program that can easily perform appropriate monitoring regardless of whether the monitored subject is positioned on a bed or not. [Means for solving the problem]

[0006] The monitoring system according to the present invention is characterized in that it comprises an imaging device that images a monitored area, and a skeletal information generation unit that generates skeletal information indicating the skeleton of the monitored object in the monitored area based on the imaging results obtained by the imaging device, wherein the imaging results include a first image that is an image showing the temperature distribution in the monitored area, and the monitoring system comprises an image generation unit that generates a second image that is an image in which the skeletal information and the first image are superimposed.

[0007] The monitoring device according to the present invention is characterized in that it comprises an imaging device that images a monitored area, and a skeletal information generation unit that generates skeletal information indicating the skeleton of the monitored object in the monitored area based on the imaging results obtained by the imaging device, wherein the imaging results include a first image that is an image showing the temperature distribution in the monitored area, and the monitoring device comprises an image generation unit that generates a second image that is an image in which the skeletal information and the first image are superimposed.

[0008] The monitoring program of the present invention is characterized in that it is a monitoring program that causes a computer to realize an imaging function for photographing a monitored area and a skeletal information generation function for generating skeletal information indicating the skeleton of the monitored object in the monitored area based on the imaging results obtained by the imaging function, wherein the imaging results include a first image that is an image showing the temperature distribution in the monitored area, and the computer realizes an image generation function for generating a second image that is an image in which the skeletal information and the first image are superimposed.

[0009] In a configuration in which the target is monitored based only on the skeletal information, out of the skeletal information and the first image, it is conceivable that the patterns on the floor or wall may be mistakenly recognized as the skeleton of the target, making it impossible to monitor properly.

[0010] Furthermore, in a configuration in which the target is monitored based only on the first image out of the skeletal information and the first image, although it is possible to identify the position of the target, the state (behavior) of the target tends to become unclear.

[0011] According to this configuration, a second image is generated. The second image is an image in which skeletal information and the first image, which is an image showing temperature distribution, are superimposed. As a result, for example, if a pattern on a floor or wall is mistakenly recognized as the skeleton of a monitoring target as described above, it is easy to determine that the skeleton is not the target of monitoring but the result of a mistaken recognition based on the temperature distribution around the recognized skeleton. This makes it possible to eliminate mistaken recognition and facilitate appropriate monitoring.

[0012] Furthermore, the inventors of the present application have found that, with this configuration, it is possible to determine the state (behavior) of the monitoring target, regardless of whether the monitoring target is positioned on the bed or not.

[0013] Therefore, with this configuration, it is easy to perform appropriate monitoring regardless of whether the monitoring target is positioned on the bed or not.

[0014] Furthermore, in the present invention, it is preferable to include a recognition unit that recognizes the monitoring target based on the second image.

[0015] With this configuration, for example, if a pattern on a floor or wall is mistakenly recognized as the skeleton of a monitoring target as described above, it is easy to determine based on the temperature distribution around the recognized skeleton that the skeleton is not the target of monitoring but the result of a mistaken recognition, making it easier for the recognition unit to accurately recognize the monitoring target.

[0016] Furthermore, in the present invention, it is preferable to further include an abnormality detection unit that detects an abnormality in the monitoring target based on the second image.

[0017] This configuration makes it easier to accurately detect abnormalities in the monitored object compared to a configuration in which abnormalities in the monitored object are detected based on only the skeletal information or only the first image, making it easier for the monitor to take appropriate action.

[0018] Furthermore, in the present invention, it is preferable that the image generation unit generates the second image over time, and the abnormality detection unit detects an abnormality in the monitored object based on changes in the monitored object over time in the second image.

[0019] According to this configuration, the abnormality detection unit can detect an abnormality in the monitored object based on a change in the position of the monitored object's skeleton over time and a change in the temperature distribution of the monitored object over time, which makes it easier to detect an abnormality in the monitored object with high accuracy.

[0020] Furthermore, in the present invention, it is preferable to provide an alarm unit that issues an alarm in response to the abnormality detection unit detecting an abnormality in the monitored object, and a display unit that displays the second image in conjunction with the issuance of the alarm.

[0021] According to this configuration, if an abnormality (such as a fall) occurs in the monitored object, an alarm is issued and the second image is displayed on the display unit. By checking the displayed second image, the monitor can easily understand the status (such as the position and posture) of the monitored object.

[0022] Another monitoring system according to the present invention is characterized in that it comprises an imaging device that images a monitored area, a skeletal information generation unit that generates skeletal information indicating the skeleton of the monitored object in the monitored area based on the imaging results obtained by the imaging device, and a processing execution unit that executes a corresponding process that is a process corresponding to the behavior of the monitored object based on the skeletal information, wherein the imaging results include a first image that is an image showing the temperature distribution in the monitored area, and the processing execution unit executes the corresponding process based on the skeletal information and the first image.

[0023] When the behavior of the monitored subject (e.g., a fall) is detected based only on the skeletal information or only on the first image, the detection result is likely to be erroneous. Therefore, in a configuration in which a response process is performed based only on the skeletal information or only on the first image, it is relatively easy for a response process (e.g., issuing an alert) to be erroneously performed in an inappropriate situation (e.g., a situation in which issuing an alert is unnecessary).

[0024] According to this configuration, the correspondence process is executed based on the skeleton information and the first image, which makes it easier to avoid the correspondence process being executed in an inappropriate situation compared to when the correspondence process is executed based only on the skeleton information or only on the first image. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 1 is a diagram illustrating an overview of a monitoring system. [Figure 2]FIG. 1 is a block diagram of a monitoring system. [Figure 3] FIG. 4 is a diagram showing an example of a captured image acquired by a capturing unit. [Figure 4] FIG. 10 is a diagram illustrating an example of skeletal information. [Figure 5] FIG. 2 is a diagram showing an example of a first image. [Figure 6] FIG. 10 is a diagram showing an example of a second image. [Figure 7] FIG. 4 is a diagram showing an example of a captured image acquired by a capturing unit. [Figure 8] FIG. 10 is a diagram illustrating an example of skeletal information. [Figure 9] FIG. 2 is a diagram showing an example of a first image. [Figure 10] FIG. 10 is a diagram showing an example of a second image. [Figure 11] FIG. 4 is a diagram showing an example of a captured image acquired by a capturing unit. [Figure 12] FIG. 10 is a diagram illustrating an example of skeletal information. [Figure 13] FIG. 2 is a diagram showing an example of a first image. [Figure 14] FIG. 10 is a diagram showing an example of a second image. [Figure 15] 10 is a flowchart of an alarm execution flow. DETAILED DESCRIPTION OF THE INVENTION

[0026] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described with reference to the drawings.

[0027] [Overall configuration of the monitoring system] 1, a monitoring system A in this embodiment includes a monitoring device 1 and a management terminal 2. The monitoring device 1 and the management terminal 2 are connected to each other via a predetermined network.

[0028] The monitoring device 1 is installed in a monitoring target area B. The monitoring device 1 monitors the monitoring target area B. In the example shown in FIG.

[0029] The monitoring device 1 is equipped with an imaging device 3 (see Figure 2). The monitoring device 1 captures an image of the monitored area B using the imaging device 3. An image showing the situation within the monitored area B is sent from the monitoring device 1 to the management terminal 2. By viewing the image displayed on the management terminal 2, a user E (monitoring person) of the monitoring system A can grasp the situation within the monitored area B while being outside the monitored area B.

[0030] In this embodiment, the monitored area B is a room in a welfare facility for the elderly, the user E is a staff member of the welfare facility for the elderly, and the management terminal 2 is an information terminal (smartphone, tablet, etc.) carried by the staff member. In the example shown in Fig. 1, the monitored area D is a resident of the welfare facility for the elderly, and the staff member, user E, is located in a room other than the room (for example, a waiting room).

[0031] The monitored object D is not limited to a resident (human), but may be a non-human (animal such as a dog or cat).

[0032] The monitoring system A may include one or more monitoring devices 1. The monitoring system A (or the monitoring device 1) may monitor one monitoring target D, or may monitor multiple monitoring targets D.

[0033] A welfare facility for the elderly may have multiple monitoring devices 1. For example, one monitoring device 1 may be installed in each room of the welfare facility for the elderly, or may be installed in an area other than the rooms (for example, a hallway).

[0034] There may be one or more users E. The monitoring system A may include one or more management terminals 2. Each of the multiple users E may carry one management terminal 2. The management terminal 2 may be a personal computer installed in a room (e.g., a waiting room) for user E.

[0035] [Generation of the second image] As shown in FIG. 2, the photographing device 3 has a photographing unit 4. The photographing unit 4 photographs the monitored area B (see FIG. 1). In this embodiment, the photographing unit 4 is configured by an infrared camera. However, the present invention is not limited to this, and the photographing unit 4 may be any device other than an infrared camera as long as it is capable of photographing the monitored area B. It is preferable that the photographing unit 4 is a ToF (Time of flight) camera.

[0036] 3 shows an example of a captured image acquired by the imaging unit 4. The captured image shows the monitoring target D, a bed, a wall, etc. It is preferable that the captured image acquired by the imaging unit 4 is a depth image, which is an image including depth information.

[0037] As shown in Fig. 2, the monitoring device 1 includes a skeletal information generation unit 5. The captured image acquired by the imaging unit 4 is sent from the imaging unit 4 to the skeletal information generation unit 5. The skeletal information generation unit 5 generates skeletal information 10 as shown in Fig. 4 based on the captured image acquired from the imaging unit 4. The skeletal information 10 is information indicating the skeleton of the monitoring target D in the monitoring target area B.

[0038] That is, the monitoring system A (or the monitoring device 1) includes an imaging device 3 that captures an image of the monitored area B, and a skeleton information generation unit 5 that generates skeleton information 10 indicating the skeleton of the monitored object D in the monitored area B based on the image capture results by the imaging device 3.

[0039] The skeletal information 10 in this embodiment is specifically a skeletal model. The skeletal model is generated by analyzing the captured image acquired by the image capturing unit 4, recognizing the monitoring target D using a known person recognition technology, and recognizing the positions of the joints of the monitoring target D three-dimensionally based on the depth information, and connecting the joints with straight lines.

[0040] As shown in Figure 2, the photographing device 3 has a thermographic camera 6. The thermographic camera 6 photographs the monitored area B (see Figure 1). In this embodiment, the photographing direction and photographing range of the thermographic camera 6 match the photographing direction and photographing range of the photographing unit 4. However, the present invention is not limited to this, and the photographing direction and photographing range of the thermographic camera 6 do not have to completely match the photographing direction and photographing range of the photographing unit 4.

[0041] Based on infrared rays emitted from an object, the thermographic camera 6 outputs a first image 11, which is an image showing the temperature distribution in the monitored area B. Note that the technology for outputting an image showing the temperature distribution based on infrared rays emitted from an object is well known, and therefore a description thereof will be omitted.

[0042] Fig. 5 shows an example of the first image 11 output by the thermographic camera 6. In the first image 11, the temperature of each part is indicated by color. For example, relatively high temperature areas are indicated in red, and relatively low temperature areas are indicated in blue. In the example shown in Fig. 5, the area where the monitored object D is present and the area where the bed is present are indicated in red or a color close to red, and other areas (areas corresponding to the floor and walls) are indicated in blue or a color close to blue.

[0043] More specifically, the first image 11 in Fig. 5 shows a first high temperature section 21, a second high temperature section 22, and a medium temperature section 23. The first high temperature section 21 is the hottest area in the first image 11 and is shown in red. The second high temperature section 22 is a relatively hot area, although its temperature is lower than that of the first high temperature section 21, and is shown in orange or yellow. The medium temperature section 23 is a lower temperature area than the second high temperature section 22 but is hotter than the floor and walls, and is shown in green.

[0044] Although not particularly limited, in this embodiment, the area in the first image 11 where the monitoring target D exists is the first high temperature area 21 or the second high temperature area 22. Also, the area where the bed exists is the medium temperature area 23.

[0045] With the configuration described above, the photographing results obtained by the photographing device 3 include the photographed image (see FIG. 3) acquired by the photographing unit 4 and the first image 11 (see FIG. 5) output by the thermographic camera 6. That is, the photographing results obtained by the photographing device 3 include the first image 11, which is an image showing the temperature distribution in the monitored area B.

[0046] As shown in FIG. 2, the monitoring device 1 includes an image generation unit 7. The image generation unit 7 acquires skeletal information 10 from the skeletal information generation unit 5. The image generation unit 7 also acquires a first image 11 from the thermographic camera 6. The image generation unit 7 generates a second image 12 based on the skeletal information 10 and the first image 11. The second image 12 is an image in which the skeletal information 10 and the first image 11 are superimposed. FIG. 6 shows an example of the second image 12 generated by the image generation unit 7.

[0047] In this way, the monitoring system A (or the monitoring device 1) includes an image generating unit 7 that generates the second image 12, which is an image in which the skeleton information 10 and the first image 11 are superimposed.

[0048] The second image 12 may be formed by overlapping a layer of an image representing the skeletal information 10 with a layer of the first image 11. In this case, the pixel values ​​of each layer may not change before and after the overlapping, and each layer may be separable as an independent image even after the second image 12 is generated.

[0049] The second image 12 may also be a single image obtained by combining (an example of "overlapping") the image representing the skeleton information 10 and the first image 11. In this case, for example, the pixel values ​​of the portion of the first image 11 that overlaps with the skeleton represented by the skeleton information 10 may be overwritten with the pixel values ​​of the image representing the skeleton information 10 to generate the second image 12. In this case, after the second image 12 is generated, it may not be possible to separate the second image 12 into the two original images. The second image 12 may also be generated by calculating pixel values ​​on a pixel-by-pixel basis based on the image representing the skeleton information 10 and the first image 11.

[0050] [Recognition of the target of surveillance] As shown in FIG. 2, the monitoring device 1 includes a recognition unit 8. The recognition unit 8 acquires a second image 12 from the image generation unit 7. The recognition unit 8 is configured to recognize the monitoring target D based on the second image 12. That is, the monitoring system A (or the monitoring device 1) includes the recognition unit 8 that recognizes the monitoring target D based on the second image 12. Recognition of the monitoring target D by the recognition unit 8 will be described in detail below.

[0051] Although not particularly limited, in the present embodiment, when the first high temperature part 21 or the second high temperature part 22 exists around (near) the skeleton indicated by the skeleton information 10 in the second image 12, the recognition unit 8 determines that the skeleton belongs to the monitoring target D. In this way, the recognition unit 8 recognizes the monitoring target D (or the existence of the monitoring target D).

[0052] Furthermore, if neither the first high temperature part 21 nor the second high temperature part 22 exists around (near) the skeleton indicated by the skeleton information 10 in the second image 12, the recognition unit 8 determines that the skeleton does not belong to the monitoring target D. In this case, the recognition unit 8 does not recognize the monitoring target D (or the existence of the monitoring target D).

[0053] 6, a first high temperature portion 21 and a second high temperature portion 22 exist around (near) the skeleton indicated by the skeleton information 10. Therefore, the recognition unit 8 determines that the skeleton belongs to the monitoring target D, and recognizes the monitoring target D (or the existence of the monitoring target D).

[0054] FIG. 7 shows an example of a captured image acquired by the imaging unit 4. In the example shown in FIG. 7, there is no monitoring target D in the monitoring target area B (see FIG. 1). This captured image does not show monitoring target D, but shows a fake target G. A fake target G is an object that is not a monitoring target D (human), but has a similar external shape (appearance) to that of monitoring target D, and is therefore recognized (or may be recognized) as monitoring target D by the skeleton information generation unit 5. Specific examples of fake target G include dirt on the floor or wall, an empty wheelchair, a mannequin, etc. In the example shown in FIG. 7, the fake target G is dirt on the floor.

[0055] When the captured image shown in Figure 7 is acquired by the photographing unit 4, the skeletal information generating unit 5 recognizes that the fake target G is the monitored target D based on the captured image, and generates skeletal information 10 corresponding to the fake target G as shown in Figure 8.

[0056] In this case, the thermographic camera 6 outputs a first image 11 shown in Fig. 9. In this example, the monitoring target D does not exist in the monitoring target area B (see Fig. 1). Therefore, the first high temperature part 21 and the second high temperature part 22 do not exist in this first image 11.

[0057] Then, in this example, the image generation unit 7 generates the second image 12 shown in Fig. 10. In this second image 12, neither the first high temperature part 21 nor the second high temperature part 22 exists around (near) the skeleton indicated by the skeleton information 10. Therefore, the recognition unit 8 determines that the skeleton does not belong to the monitoring target D. Therefore, in this case, the recognition unit 8 does not recognize the monitoring target D (or the existence of the monitoring target D).

[0058] 11 shows an example of a captured image acquired by the imaging unit 4. In the example shown in FIG. 11, a monitoring target D exists in the monitoring target area B (see FIG. 1). Therefore, the monitoring target D appears in this captured image. In this example, the monitoring target D is sleeping in bed.

[0059] When the photographed image shown in Fig. 11 is acquired by the photographing unit 4, the skeletal information generating unit 5 recognizes the monitoring target D based on the photographed image. However, in this example, most of the monitoring target D except for the head is covered by a comforter. Therefore, the skeletal information generating unit 5 generates skeletal information 10 of only a part of the body of the monitoring target D (specifically, only the head and arms), as shown in Fig. 12.

[0060] In this case, the thermographic camera 6 outputs a first image 11 shown in Fig. 13. In this example, the monitoring target D exists in the monitoring target area B (see Fig. 1). Therefore, a first high temperature part 21 and a second high temperature part 22 exist in this first image 11.

[0061] In this example, the image generation unit 7 generates the second image 12 shown in Fig. 14. In this second image 12, a first high temperature portion 21 and a second high temperature portion 22 exist around (near) the skeleton indicated by the skeleton information 10. Therefore, the recognition unit 8 determines that the skeleton is that of the monitoring target D, and recognizes the monitoring target D (or the existence of the monitoring target D).

[0062] [Detection of abnormalities] As shown in Fig. 2, the monitoring device 1 includes a processing execution unit 30. The processing execution unit 30 includes an abnormality detection unit 31. The abnormality detection unit 31 detects an abnormality in the monitoring object D based on the second image 12. That is, the monitoring system A (or the monitoring device 1) includes the abnormality detection unit 31 that detects an abnormality in the monitoring object D based on the second image 12. The detection of an abnormality by the abnormality detection unit 31 will be described in detail below.

[0063] 2, the processing execution unit 30 acquires a second image 12 from the image generation unit 7. The processing execution unit 30 also acquires a recognition result from the recognition unit 8. The abnormality detection unit 31 identifies a skeleton (skeleton information 10) determined to be that of the monitoring target D based on the second image 12 and the recognition result from the recognition unit 8, and detects an abnormality in the monitoring target D (for example, a fall or a state of being lying on the floor) based on the posture and movement of the skeleton. Note that the technology for detecting an abnormal state such as a fall based on the posture and movement of the skeleton is publicly known, and therefore will not be described here.

[0064] Furthermore, the abnormality detection unit 31 in this embodiment is configured to detect abnormalities not only based on the posture and movement of the skeleton as described above, but also based on changes in the monitoring target D over time.

[0065] More specifically, in this embodiment, the image generation unit 7 generates the second image 12 over time. The abnormality detection unit 31 detects an abnormality in the monitoring object D based on a change in the monitoring object D in the second image 12 over time.

[0066] Although not particularly limited, for example, the abnormality detection unit 31 may identify the skeleton (skeleton information 10) that has been determined to belong to the monitored object D as described above, and may detect that an abnormality has occurred in the monitored object D if the skeleton remains in a certain posture and does not move for a predetermined period of time in the second image 12.

[0067] Furthermore, for example, the abnormality detection unit 31 may identify the skeleton (skeleton information 10) determined to belong to the monitored object D as described above, and may also detect that an abnormality has occurred in the monitored object D if the temperature around (near) the skeleton indicated by the skeleton information 10 in the second image 12 changes beyond a predetermined temperature range (for example, if there is a temperature change of 5 degrees or more).

[0068] [Response process] 2, the processing execution unit 30 has an imaging direction control unit 32 and an alarm issuing unit 33. The processing execution unit 30 is configured to execute response processing using the imaging direction control unit 32 and the alarm issuing unit 33. The response processing is processing that corresponds to the behavior of the monitoring target D. The response processing will be described in detail below.

[0069] The imaging direction control unit 32 is configured to be able to control the imaging direction of the imaging device 3. For example, in the examples shown in Figs. 3 and 7, the imaging direction of the imaging device 3 is controlled in a direction such that only a part of the bed falls within the imaging range of the imaging unit 4. In contrast to this, in the example shown in Fig. 11, the imaging direction of the imaging device 3 is directed downward more than in the examples shown in Figs. 3 and 7, and the imaging direction of the imaging device 3 is controlled in a direction such that the entire bed falls within the imaging range of the imaging unit 4.

[0070] The imaging direction control unit 32 identifies the skeleton (skeleton information 10) determined to be that of the monitoring target D based on the second image 12 and the recognition result by the recognition unit 8, and controls the imaging direction of the imaging device 3 so as to follow the movement of the monitoring target D (a specific example of "behavior") based on the position and movement of the skeleton. In other words, the imaging direction control unit 32 controls the imaging direction of the imaging device 3 based on the second image 12. Controlling the imaging direction of the imaging device 3 is a specific example of the above-mentioned "response processing."

[0071] As described above, the second image 12 is generated based on the skeletal information 10 and the first image 11. Therefore, the imaging direction control unit 32 controls the imaging direction of the imaging device 3 based on the skeletal information 10 and the first image 11.

[0072] When the abnormality detection unit 31 detects that an abnormality has occurred in the monitoring target D, it sends a predetermined signal to the alarm issuing unit 33. When the alarm issuing unit 33 receives the signal, it issues an alert. That is, the monitoring system A (or the monitoring device 1) is equipped with the alarm issuing unit 33 that issues an alert in response to the abnormality detection unit 31 detecting an abnormality in the monitoring target D. As a result, for example, when the monitoring target D falls (a specific example of "behavior"), the abnormality is detected by the abnormality detection unit 31, and the alarm issuing unit 33 issues an alert. That is, the alarm issuing unit 33 issues an alert in response to the monitoring target D falling. Announcement is a specific example of the above-mentioned "response process."

[0073] Specific examples of "behavior" are not limited to moving and falling. Specific examples of "behavior" include the body position of the monitored subject D (for example, standing, lying down, sitting), lying down, slipping down, boundary position, getting up, getting out of bed, etc. An "boundary position" is, for example, sitting beside the bed.

[0074] As described above, the abnormality detection unit 31 detects an abnormality in the monitoring target D based on the second image 12. The second image 12 is generated based on the skeleton information 10 and the first image 11. Therefore, the alarm issuing unit 33 issues an alarm based on the skeleton information 10 and the first image 11.

[0075] The alarm unit 33 in this embodiment issues an alert via the management terminal 2. More specifically, in issuing an alert, the alarm unit 33 transmits a predetermined alert signal to the management terminal 2. The management terminal 2 performs an alert operation using light or sound in accordance with this alert signal. The light alert operation may be, for example, the display of an icon or message on a display unit 35 (e.g., a display) provided in the management terminal 2. The display may indicate that an abnormality has occurred in the monitoring target D. Furthermore, the sound alert operation may be an audio output from a speaker (not shown) provided in the management terminal 2. The audio output may indicate that an abnormality has occurred in the monitoring target D.

[0076] As described above, the monitoring system A (or monitoring device 1) includes an imaging device 3 that captures an image of a monitored area B, a skeleton information generation unit 5 that generates skeleton information 10 indicating the skeleton of a monitored object D in the monitored area B based on the image capture results from the imaging device 3, and a processing execution unit 30 that executes a corresponding process that is a process according to the behavior of the monitored object D based on the skeleton information 10. The processing execution unit 30 also executes the corresponding process based on the skeleton information 10 and the first image 11.

[0077] Furthermore, when issuing an alert, the alert issuing unit 33 sends the second image 12 acquired by the processing execution unit 30 to the management terminal 2. Then, the display unit 35 displays the second image 12 in response to the issuance of an alert (more specifically, the above-mentioned alert issuing operation). That is, the monitoring system A (or the monitoring device 1) includes a display unit 35 that displays the second image 12 in response to the issuance of an alert.

[0078] [Alarm execution flow] The monitoring device 1 is configured to issue the above-described alert in accordance with the alert execution flow shown in FIG.

[0079] When the alarm execution flow is started, first, the process of step S01 is executed. In step S01, the recognition unit 8 recognizes the monitored object D based on the second image 12 as described above. Thereafter, the process proceeds to step S02.

[0080] In step S02, the process execution unit 30 determines whether or not the monitoring target D exists in the monitoring target area B. This determination is made based on the recognition result by the recognition unit 8 in step S01.

[0081] For example, if the second image 12 contains a skeleton indicated by the skeleton information 10 and the recognition unit 8 determines that the skeleton belongs to the monitored object D, the processing execution unit 30 determines that the monitored object D exists in the monitored area B.

[0082] Also, for example, if the recognition unit 8 determines that a skeleton indicated by the skeleton information 10 exists in the second image 12 and that the skeleton does not belong to the monitored object D, the processing execution unit 30 determines that the monitored object D does not exist in the monitored area B.

[0083] Furthermore, for example, if the skeleton indicated by the skeleton information 10 does not exist in the second image 12, the recognition unit 8 does not recognize the monitoring target D (or the existence of the monitoring target D). In this case, the processing execution unit 30 determines that the monitoring target D does not exist in the monitoring target area B.

[0084] If it is determined that the monitoring target D does not exist in the monitoring target area B ("No" in step S02), this alarm execution flow ends. If it is determined that the monitoring target D exists in the monitoring target area B ("Yes" in step S02), the process proceeds to step S03.

[0085] In step S03, the abnormality detection unit 31 detects an abnormality in the monitoring object D as described above. If an abnormality in the monitoring object D is not detected (there is no abnormality) ("No" in step S03), this alert execution flow ends for the time being. If an abnormality in the monitoring object D is detected (there is an abnormality) ("Yes" in step S03), the process proceeds to step S04.

[0086] In step S04, the alarm is issued as described above by the alarm issuing unit 33. After that, this alarm issuing execution flow ends for the time being.

[0087] [Air conditioning monitoring] As shown in FIG. 1, an air conditioner 40 for conditioning the monitored area B is provided in the monitored area B.

[0088] In this embodiment, the monitoring device 1 is configured to be able to output the first image 11 or the second image 12 to the outside, regardless of whether or not an abnormality has been detected by the abnormality detection unit 31. Both the first image 11 and the second image 12 show the temperature distribution in the monitored area B. Therefore, for example, if the display unit 35 is configured to be able to display the first image 11 or the second image 12 output from the monitoring device 1, the user E can check whether or not the state of air conditioning by the air conditioner 40 is appropriate by looking at the displayed first image 11 or second image 12.

[0089] The abnormality detection unit 31 is also configured to determine whether the state of air conditioning by the air conditioner 40 is appropriate based on the second image 12. This determination may be made, for example, by comparing the temperature distribution of the monitoring target D with the temperature distribution around the monitoring target D.

[0090] Each functional unit included in the monitoring device 1 and the management terminal 2 may be a physical device such as a microcomputer, or may be a software functional unit. For example, a configuration may be adopted in which a program corresponding to each functional unit is stored in a ROM or non-volatile memory (not shown), and the program is loaded into a CPU and executed, thereby executing a process corresponding to each functional unit.

[0091] According to the configuration described above, the second image 12 is generated. The second image 12 is an image in which the skeleton information 10 and the first image 11, which is an image showing the temperature distribution, are superimposed. As a result, if, for example, a pattern on a floor or wall (fake object G) is mistakenly recognized as the skeleton of the monitoring target D, it is easy to determine that the skeleton is not that of the monitoring target D but is the result of a mistaken recognition, based on the temperature distribution around the recognized skeleton. This makes it possible to eliminate mistaken recognition and facilitate appropriate monitoring.

[0092] Furthermore, the inventors of the present application have found that, with the configuration described above, it is possible to determine the state (behavior) of the monitoring target D, regardless of whether the monitoring target D is positioned on the bed or not.

[0093] Therefore, according to the configuration described above, it is easy to perform appropriate monitoring regardless of whether the monitoring target D is positioned on the bed or not.

[0094] Other Embodiments (1) The functions of the components in the above embodiment may be configured as a monitoring program that causes a computer to realize them. For example, the functions of the components in the above embodiment may be configured as a monitoring program that causes a computer to realize a photographing function corresponding to the photographing device 3, a skeletal information generating function corresponding to the skeletal information generating unit 5, and an image generating function corresponding to the image generating unit 7.

[0095] (2) The captured image acquired by the photographing unit 4, the first image 11 output by the thermographic camera 6, and the second image 12 generated by the image generating unit 7 may be either a moving image or a still image.

[0096] (3) Some or all of the skeletal information generation unit 5, image generation unit 7, recognition unit 8, processing execution unit 30, abnormality detection unit 31, shooting direction control unit 32, and alarm issuing unit 33 may be provided outside the monitoring device 1 (for example, on the management terminal 2 or a management server not shown).

[0097] (4) The recognition unit 8 does not have to be provided. For example, the monitoring system A may detect an abnormality in the monitoring target D by the abnormality detection unit 31 without recognizing the monitoring target D, and may execute a corresponding process by the process execution unit 30.

[0098] (5) The abnormality detection unit 31 does not have to be provided. For example, the monitoring system A may execute a response process using the process execution unit 30 without detecting an abnormality in the monitoring target D.

[0099] (6) The alarm unit 33 does not have to be provided.

[0100] (7) Instead of the photographing unit 4 and the thermographic camera 6, a single device (camera) having both the functions of the photographing unit 4 and the thermographic camera 6 may be provided.

[0101] (8) In the alarm execution flow shown in Fig. 15, some steps may not be present, new steps may be added, or the order of the steps may be changed. For example, step S03 may be configured to be executed before step S01. In this case, even if an abnormality is detected by the abnormality detection unit 31, if the recognition unit 8 does not recognize the presence of the monitoring target D, the alarm issuing unit 33 will be prevented from issuing an alarm (in other words, the alarm will not be issued).

[0102] The configurations disclosed in the above-described embodiments (including other embodiments, the same applies hereinafter) can be applied in combination with the configurations disclosed in other embodiments, unless a contradiction arises. Furthermore, the embodiments disclosed in this specification are merely examples, and the present invention is not limited to these, and can be modified as appropriate within the scope of the purpose of the present invention. [Industrial Applicability]

[0103] The present invention can be used not only in elderly care facilities but also in various other facilities such as medical facilities and public facilities, and can also be used outdoors. [Explanation of symbols]

[0104] 1: Monitoring device 3: Imaging device 5: Skeleton information generation unit 7: Image generation unit 8: Recognition part 10: Skeleton information 11: First image 12: Second image 30: Processing execution unit 31: Anomaly detection unit 33: Reporting Department 35: Display section A: Surveillance system B: Monitoring area D: Monitoring target

Claims

1. 1. A monitoring system comprising: an imaging device that images a monitored area; and a skeleton information generation unit that generates skeleton information indicating a skeleton of a monitored object in the monitored area based on an imaging result obtained by the imaging device, the photographing result includes a first image that is an image showing a temperature distribution in the monitored area, A monitoring system comprising an image generation unit that generates a second image in which the skeletal information and the first image are superimposed.

2. The monitoring system according to claim 1 , further comprising a recognition unit that recognizes the monitoring target based on the second image.

3. The monitoring system according to claim 1 or 2, further comprising an abnormality detection unit that detects an abnormality in the monitoring target based on the second image.

4. the image generation unit generates the second image over time, The monitoring system according to claim 3 , wherein the abnormality detection unit detects an abnormality in the monitoring target based on a change over time of the monitoring target in the second image.

5. an alarm issuing unit that issues an alarm in response to an abnormality in the monitoring target being detected by the abnormality detection unit; The monitoring system according to claim 3 , further comprising: a display unit that displays the second image in response to the issuance of the alarm.

6. A monitoring system comprising: an imaging device that images a monitored area; a skeleton information generation unit that generates skeleton information indicating a skeleton of a monitored object in the monitored area based on an imaging result obtained by the imaging device; and a processing execution unit that executes a corresponding process that is a process corresponding to a behavior of the monitored object based on the skeleton information, the photographing result includes a first image that is an image showing a temperature distribution in the monitored area, The processing execution unit executes the response processing based on the skeletal information and the first image.

7. A monitoring device comprising: an imaging device that images a monitoring target area; and a skeleton information generation unit that generates skeleton information indicating a skeleton of a monitoring target in the monitoring target area based on an imaging result obtained by the imaging device, the photographing result includes a first image that is an image showing a temperature distribution in the monitored area, A monitoring device comprising an image generation unit that generates a second image in which the skeletal information and the first image are superimposed.

8. A monitoring program that causes a computer to realize an imaging function for imaging a monitored area, and a skeleton information generation function for generating skeleton information indicating a skeleton of a monitored object in the monitored area based on an imaging result obtained by the imaging function, the photographing result includes a first image that is an image showing a temperature distribution in the monitored area, A monitoring program that causes a computer to realize an image generation function that generates a second image that is an image in which the skeletal information and the first image are superimposed.

Images