Monitoring device, control method, and control program
The monitoring device enhances tracking accuracy by dynamically adjusting the camera direction based on motion and characteristics, addressing the issue of fixed detection in existing systems and ensuring continuous monitoring of care recipients.
Patent Information
- Application Number
- JP2023075981
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-02
- Publication Date
- 2024-11-15
AI Technical Summary
Existing monitoring systems may fail to accurately track individuals in need of care due to the detection direction of sensor units being fixed at predetermined positions, potentially missing the care recipient for extended periods.
A monitoring device with a photographing section, motion sensor, drive section, recognition unit, and determination unit that adjusts the photographing direction based on detected motion and characteristics to follow the tracking target, using methods such as face authentication and joint point estimation to enhance accuracy.
Improves the accuracy of capturing the monitoring target by dynamically adjusting the camera direction to ensure continuous tracking of individuals in need of care, reducing the likelihood of missed observations.
Smart Images

Figure 2024160736000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a monitoring device, a control method, and a control program. [Background technology]
[0002] In Japan, the standard of living has improved, the sanitary environment has improved, and the level of medical care has improved, all of which have come with the rapid economic growth since the war. This, combined with the declining birth rate, has resulted in an aging society with a high aging rate. In such an aging society, it is expected that the number of people who require nursing care due to illness, injury, aging, etc. will increase.
[0003] In facilities such as hospitals and elderly care facilities, people who need care may fall while walking or fall off their beds and get injured. For this reason, systems are being developed that enable caregivers, nurses, and other staff to rush to the scene immediately in response to the actions of people who need care.
[0004] The following prior art is disclosed in Patent Document 1 below. A sensor device is provided that includes an infrared array sensor unit that detects the presence of a person, and a drive mechanism that changes the detection direction of the infrared array sensor unit up, down, left, and right. The drive mechanism changes the detection direction of the infrared array sensor unit at regular intervals to detect people requiring care at multiple predetermined positions. Then, the detection direction of the infrared array sensor unit is fixed at the predetermined position that has the longest presence capture time among the times the person requiring care is captured at the multiple predetermined positions, and monitoring is performed. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] JP 2022-110634 A Summary of the Invention [Problem to be solved by the invention]
[0006] However, in the above-mentioned prior art, the detection direction of the sensor unit may be fixed to a predetermined position where a person other than the person requiring care is captured, which may result in the person requiring care not being watched over for a long period of time.
[0007] The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to provide a monitoring device, a control method, and a control program that can improve the accuracy of capturing a target being monitored by an imaging unit. [Means for solving the problem]
[0008] The above-mentioned object of the present invention is achieved by the following means.
[0009] (1) A monitoring device having an imaging unit that captures images of people, a human presence sensor that detects the position of people, a drive unit that changes the imaging direction of the imaging unit based on detection information from the human presence sensor, a recognition unit that recognizes people from the images captured by the imaging unit, a determination unit that determines whether the person recognized by the recognition unit is a target to be followed based on characteristics of the person recognized by the recognition unit, and a control unit that causes the imaging direction of the imaging unit to be changed by the drive unit so as to follow the person determined to be the target to be followed by the determination unit.
[0010] (2) The monitoring device according to (1) above, wherein the determination unit determines whether or not a person is the target to be tracked based on the moving speed of the person recognized by the recognition unit.
[0011] (3) The monitoring device described in (1) above, wherein the determination unit determines whether a person recognized by the recognition unit is the target to be tracked by authenticating the face or an item worn by the person.
[0012] (4) A monitoring device as described in (1) above, further comprising an estimation unit that estimates joint points of a person recognized by the recognition unit, and the judgment unit determines whether or not the person is to be tracked based on the joint points estimated by the estimation unit.
[0013] (5) The monitoring device described in (1) above, wherein the recognition unit further recognizes objects other than humans from the image captured by the photographing unit, and the judgment unit determines whether the person recognized by the recognition unit is the tracking target based on the relative position between the person recognized by the recognition unit and the object recognized by the recognition unit.
[0014] (6) A monitoring device described in any of (1) to (5) above, wherein when a predetermined condition is satisfied, the control unit again causes the drive unit to change the shooting direction of the shooting unit so as to track the person determined by the determination unit to be the tracking target.
[0015] (7) The monitoring device described in (6) above, wherein the specified condition is that a person determined by the judgment unit to be the target to be followed is no longer recognized by the recognition unit in the image, a specified time has elapsed since the person determined by the judgment unit to be the target to be followed was no longer recognized by the recognition unit in the image, or a person is detected by the human presence sensor after the person determined by the judgment unit to be the target to be followed is no longer recognized by the recognition unit in the image.
[0016] (8) A control method for a monitoring device having an imaging unit that captures images of people, a human presence sensor that detects the position of the person, and a drive unit that changes the imaging direction of the imaging unit, comprising the steps of: (a) changing the imaging direction of the imaging unit based on detection information from the human presence sensor; (b) recognizing a person from the image captured by the imaging unit; (c) determining whether the person recognized in (b) is a target to be followed based on characteristics of the person recognized in (b); and (d) changing the imaging direction of the imaging unit to the drive unit so as to follow the person determined to be the target to be followed in (c).
[0017] (9) The control method according to (8) above, wherein in the step (c), it is determined whether the person is the target to be followed based on the moving speed of the person recognized in the step (b).
[0018] (10) The control method according to (8) above, wherein in the step (c), it is determined whether the person recognized in the step (b) is the target to be followed by authenticating the face or an article attached to the person.
[0019] (11) The control method according to (8) above, further comprising a step (e) of estimating joint points of the person recognized in the step (b), and in the step (c), determining whether or not the person is the target to be tracked based on the joint points estimated in the step (e).
[0020] (12) The control method described in (8) above, wherein in the step (b), an object other than a human is further recognized from the image captured by the photographing unit, and in the step (c), it is determined whether the human recognized in the step (b) is the tracking target based on the relative position between the human and the object recognized in the step (b).
[0021] (13) The control method described in (8) above, further comprising a step (f) of causing the driving unit to change the shooting direction of the shooting unit again to follow the person determined to be the target to be tracked when a predetermined condition is satisfied.
[0022] (14) The control method described in (13) above, wherein the specified condition is that the person determined to be the target to be followed in step (c) is no longer recognized in the image, that a specified time has elapsed since the person determined to be the target to be followed in step (c) was no longer recognized in the image, or that a person is detected by the human presence sensor after the person determined to be the target to be followed in step (c) is no longer recognized in the image.
[0023] (15) A control program for causing a computer to execute the control method according to any one of (8) to (14) above. Effect of the Invention
[0024] The driving unit changes the shooting direction of the shooting unit based on the detection information of a person by the human sensor.Then, it is determined whether the person is a tracking target based on the characteristics of the person recognized from the image of the shooting unit, and the shooting direction of the shooting unit is changed by the driving unit so that the person determined to be a tracking target is tracked.This improves the accuracy of capturing the target by the shooting unit. [Brief description of the drawings]
[0025] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a monitoring system. [Diagram 2] FIG. 2 is a block diagram showing a hardware configuration of the information processing device. [Diagram 3] FIG. 1 is an explanatory diagram showing a schematic configuration of an information processing device. [Figure 4] 2 is a block diagram showing functions of a control unit of the information processing device. FIG. [Diagram 5] FIG. [Figure 6] FIG. [Figure 7] FIG. 2 is a schematic diagram showing joint points. [Figure 8] FIG. 13 is a schematic diagram showing joint points detected based on a human rectangle; [Figure 9] 4 is a flowchart showing the operation of the monitoring system. [Figure 10] FIG. 2 is a schematic diagram showing a face area. [Figure 11] 4 is a flowchart showing the operation of the monitoring system. [Figure 12] 4 is a flowchart showing the operation of the monitoring system. [Figure 13] 11 is an explanatory diagram showing a situation in which a person determined to be a tracking target is no longer recognized in a captured image. FIG. [Figure 14] 11 is an explanatory diagram showing a situation in which, after the person being followed is no longer recognized, another person detected by the human presence sensor is determined to be the tracking target. FIG. [Figure 15] 11 is an explanatory diagram showing a situation in which a person is not detected by the human presence sensor after the person being followed is no longer recognized; FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0026] Hereinafter, a monitoring device, a control method, and a control program according to an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same elements are given the same reference numerals, and duplicated explanations will be omitted. In addition, the dimensional ratios in the drawings are exaggerated for the convenience of explanation, and may differ from the actual ratios.
[0027] (First embodiment) FIG. 1 is a diagram showing a schematic configuration of a monitoring system 10. As shown in FIG.
[0028] The watching system 10 includes an information processing device 100, a communication network 200, and a mobile terminal 300. The information processing device 100 constitutes a watching device. A part of the information processing device 100, or a part or all of the watching system 10 may constitute the watching device.
[0029] The information processing device 100 is connected to the mobile terminal 300 via the access point 210 by the communication network 200 so that they can communicate with each other. The information processing device 100 may be one integrated device. The information processing device 100 may be a plurality of devices that are separately arranged. A server (not shown) that can communicate with the information processing device 100 and the mobile terminal 300 via the communication network 200 may be provided. Then, some of the functions of the information processing device 100 may be implemented by the server. Also, a fixed terminal (not shown) that can communicate with the information processing device 100 and the mobile terminal 300 via the communication network 200 may be provided. Then, some or all of the functions of the mobile terminal 300 may be implemented by the fixed terminal.
[0030] The information processing device 100 may be disposed at a high position on the ceiling or wall of each room of the subject 500. The subject 500 is, for example, a person who needs care or nursing. The mobile terminal 300 is carried by each staff member who provides care or nursing for the subject 500.
[0031] (Information processing device 100) 2 is a block diagram showing a hardware configuration of the information processing device 100. The information processing device 100 includes a control unit 110, a communication unit 120, a storage unit 130, and a camera unit 140, which are interconnected via a bus.
[0032] The control unit 110 is configured with a CPU (Central Processing Unit) and memories such as a RAM (Random Access Memory) and a ROM (Read Only Memory), and controls each unit of the information processing device 100 and performs arithmetic processing according to a program. The functions of the control unit 110 will be described in detail later.
[0033] The communication unit 120 is an interface circuit (eg, a LAN card) for communicating with the mobile terminal 300 and the like via the communication network 200.
[0034] The storage unit 130 stores various programs and various data, and is configured, for example, by an SDD or HDD.
[0035] Camera unit 140 includes a camera 141, a driving mechanism 142, and a human sensor 143. Camera unit 140 is integrated with information processing device 100 and is disposed, for example, on the ceiling or a high position on a wall of a room in which subject 500 lives.
[0036] The camera 141 captures a two-dimensional captured image 700 (see FIG. 6; hereinafter, also simply referred to as "captured image 700"). The captured image 700 includes moving images and still images. The captured image 700 may be a moving image made up of a plurality of frames (still images). The camera 141 is an infrared camera, but a visible light camera may be used instead, or both may be used in combination. For ease of explanation, the following description will be given taking as an example a case where the captured image 700 is a moving image made up of a plurality of frames captured by an infrared camera.
[0037] 3 is an explanatory diagram showing a schematic configuration of the information processing device 100. As described above, the camera unit 140 may be integrated with the information processing device 100. The driving mechanism 142 changes the shooting direction of the camera 141. The driving mechanism 142 may be configured to include a motor, a gear, and a joint member with the camera 141. The shooting direction of the camera 141 may be changed by rotating about a predetermined axis in the direction indicated by the arrow in FIG. 3, for example.
[0038] The human sensor 143 is a sensor for detecting the position of a person 50 (see FIG. 5, etc.) including a target person 500. The human sensor 143 is configured, for example, by a stereo camera or a TOF (Time Of Flight) sensor. The human sensor 143 generates a distance image 800 (see FIG. 5; hereinafter, also simply referred to as "distance image 800") by detecting depth information for each pixel within a predetermined detection range. The depth information is information on the distance from the camera unit 140 to an object (for example, a person, a floor, a wall, a bed, and furniture). The depth information can be information on the distance in a direction perpendicular to the captured image 700 (normal direction). The distance image 800 is depth information for each pixel. Therefore, the distance image 800 can also be said to be a three-dimensional image. The distance image 800 constitutes the detection information of the human sensor 143.
[0039] The detection direction and detection range of the depth information by the motion sensor 143 may be fixed. The detection range of the depth information by the motion sensor 143 may be wider than the shooting range of each frame of the captured image 700 by the camera 141. Note that the shooting range by the camera 141 is expanded by changing the shooting direction of the camera 141. The fixed detection range of the motion sensor 143 may include the shooting ranges by the camera 141 (before and after changing the shooting direction). Note that the distance image 800 does not include texture information including patterns, colors, and the like.
[0040] It should be noted that the motion sensor 143 may be replaced by another device that detects the position of the person 50. The other device may include, for example, a beacon.
[0041] The function of the control unit 110 will now be described.
[0042] 4 is a block diagram showing functions of the control unit 110. The control unit 110 functions as a position detection unit 111, a drive unit 112, a recognition unit 113, an estimation unit 114, a determination unit 115, a tracking control unit 116, and a behavior recognition unit 117.
[0043] The position detection unit 111 acquires a distance image 800 from the human presence sensor 143 and detects the position of a person from the distance image 800.
[0044] FIG. 5 is a diagram showing a distance image 800. In the distance image 800 of FIG. 5, depth information (the distance from the camera unit 140 to an object) is shown by color tones. Note that the distance image 800 shown in FIG. 5 is an image obtained by converting the distance image 800 in which depth information is shown by differences in color into gray. For this reason, the tone density of the distance image 800 shown in FIG. 5 does not necessarily accurately reflect the depth information. In the example of FIG. 5, two people 50 (images of people 50) are detected in the distance image 800.
[0045] The position detection unit 111 may detect the position of the person 50 by known techniques such as a time subtraction method (frame subtraction method) or a background subtraction method. The position detection unit 111 may detect the position of the person 50 by using a trained model of a neural network that has been trained to detect the person 50 from the distance image 800.
[0046] The driving unit 112 rotates the camera 141 using the driving mechanism 142 so that the person 50 detected by the position detection unit 111 is within the shooting range of the camera 141. This changes the shooting direction of the camera 141 so that the person 50 detected by the position detection unit 111 is included in the captured image 700. When multiple people 50 are detected in the distance image 800, the shooting direction of the camera 141 can be changed so that the multiple people 50 are included in the captured image 700. When multiple people 50 are detected in the distance image 800, the shooting direction of the camera 141 may be changed so that any one of the people is included in the captured image 700. The distance image 800 and the captured image 700 are calibrated to correspond to each other's pixels.
[0047] The recognition unit 113 recognizes (detects) the person 50 from the photographed image 700. When the photographed image 700 includes images of a plurality of people 50, the recognition unit 113 can recognize all of the plurality of people 50.
[0048] FIG. 6 is a diagram showing a human rectangle 710 detected in a photographed image 700. The recognition unit 113 may detect a person 50 included in the photographed image 700 as a human rectangle 710 that is an area including the person 50. Specifically, the recognition unit 113 detects an area in which an object exists on the photographed image 700. Then, the recognition unit 113 detects the person 50 by estimating the category of the object included in the detected area. The area in which the object exists is detected as a candidate rectangle including the object on the photographed image. Among the detected candidate rectangles, a candidate rectangle estimated to have an object category of a person is detected as the human rectangle 710 to detect the person 50. The recognition unit 113 may use a trained model of a neural network reflecting a dictionary (parameters) for detecting the human rectangle 710 from the photographed image 700. Then, the recognition unit 113 may detect the human rectangle 710 from the photographed image 700 using the trained model. As the trained model, for example, a known model, Region Proposal Network (RPN), may be used. The recognition unit 113 may output a human rectangle 710 as a combination of the captured image 700 and the coordinates of two opposing vertices e1 and e2 of the human rectangle 710.
[0049] The recognition unit 113 may detect the person 50 as a silhouette from the captured image 700 using a known technique such as a time subtraction method (frame subtraction method) or a background subtraction method.
[0050] The estimation unit 114 estimates the joint points 720 of the person 50 recognized by the recognition unit 113. Specifically, the estimation unit 114 estimates the joint points 720 from the person rectangle 710.
[0051] FIG. 7 is a schematic diagram showing an articulation point 720.
[0052] 7, the joint points 720 include a head joint point 720a, a shoulder joint point 720b, a waist joint point 720c, and a joint point 720d constituting the trunk. The distance between the two head joint points 720a reflects the size of the head of the person 50.
[0053] FIG. 8 is a schematic diagram showing articulation points 720 detected based on a human rectangle 710.
[0054] In the example of FIG. 8, joint points 720 of a person 50 in a sitting position on a chair are shown. The estimation unit 114 may estimate the joint points 720 using a trained model of a neural network that has been trained to detect the joint points 720 from the human rectangle 710. As the trained model, a known model such as Deep Pose may be used. Details of DeepPose are described in a known document (Alexander Toshev, et al. "DeepPose: Human Pose Estimation via Deep Neural Networks", in CVPR, 2014).
[0055] The determination unit 115 determines whether the person 50 recognized by the recognition unit 113 is a target to be tracked by the camera 141, based on the features of the person 50 recognized by the recognition unit 113. The features of the person 50 used to determine whether the person 50 is a target to be tracked include a wide range of features of the person 50 that can be determined whether the person 50 is a target to be tracked. For example, the features of the person 50 include the moving speed of the person 50, the face of the person 50, the clothing worn by the person 50, and the range and pattern of movement of the person 50.
[0056] In this embodiment, the determination unit 115 determines whether the person 50, whose joint points 720 have been estimated, is the target to be tracked, based on the features of the joint points 720. The target to be tracked is the target person 500. Therefore, the determination unit 115 determines whether the person 50 detected as the human rectangle 710 is the target person 500, based on the features of the joint points 720.
[0057] The determination unit 115 can determine whether the recognized person 50 is a target to be tracked, based on the moving speed of the person 50 recognized by the recognition unit 113. Specifically, the determination unit 115 determines one of the people detected as human rectangles 710 as a target to be tracked, based on the moving speed of the human rectangle 710 or the joint point 720. Note that, when only one human rectangle 710 is detected, the determination unit 115 determines the person 50 included in the detected human rectangle 710 as a target to be tracked.
[0058] The determination unit 115 determines the person 50 with the slowest moving speed among the people 50 detected as the human rectangle 710 as the tracking target. This is because the target 500 to be tracked is considered to be moving slower than other people 50 such as staff members included in the photographed image 700 because the target 500 requires care or the like. The moving speed can be calculated by dividing the amount of movement of the center of the human rectangle 710 detected from each of the frames of the photographed image adjacent in time series by the frame interval (time). The moving speed may be calculated by dividing the amount of movement of the center of gravity of the joint point 720 detected from each of the frames of the photographed image adjacent in time series by the frame interval (time). In calculating the moving speed, the determination unit 115 may perform tracking to determine the identity of the person 50 between frames adjacent in time series. In the tracking, for example, the human rectangle 710 with the highest degree of overlap (or proximity) of the human rectangles 710 detected from each of the frames adjacent in time series is determined. Then, the determined human rectangles 710 are determined as human rectangles 710 of the same person 50. In tracking, the joint point 720 that is most close to the joint points 720 detected from adjacent frames in time series is determined. Then, the determined joint points 720 are determined as joint points 720 of the same person 50. Tracking may be performed using DeepSORT, which is a known tracking algorithm included in the tracking genre MTSCT (Multi Target Single Camera Tracking).
[0059] The determination unit 115 may determine whether the recognized person 50 is a target to be followed based on the features of the joint points 720 of the person 50 recognized by the recognition unit 113. Specifically, the determination unit 115 may calculate the posture of the person 50 from the joint points 720, and determine whether the person 50 corresponding to the joint point 720 is a target to be followed based on the calculated posture. The posture of the person 50 may be calculated (estimated) using a known machine learning technique using a neural network. The determination unit 115 may determine, for example, that the person 50 in a slouching posture is a target to be followed.
[0060] The determination unit 115 may determine whether the person 50 recognized by the recognition unit 113 is a target to be followed, based on both the moving speed and the posture. For example, the person 50 who moves at a relatively slow moving speed while maintaining a hunched posture may be determined to be a target to be followed.
[0061] The tracking control unit 116 changes the shooting direction of the camera 141 to the driving mechanism 142 so as to track the person 50 determined by the determination unit 115 as the tracking target. At this time, the tracking control unit 116 may perform tracking to determine the identity of the person 50 in the frames of the captured image 700 in order to track the person 50 determined as the tracking target. By performing tracking, it is possible to eliminate the need for the determination unit 115 to determine the tracking target for each frame of the captured image 700. In tracking, for example, the person rectangle 710 with the highest degree of overlap (or proximity) between the person rectangles 710 detected from adjacent frames in a time series is determined. Then, the determined person rectangles 710 are determined as person rectangles 710 of the same person 50. The above-mentioned DeepSORT may be used for tracking.
[0062] The behavior recognition unit 117 recognizes the behavior of the subject 500 determined to be a follow-up target based on the joint points 720 estimated by the estimation unit 114. For example, when the subject 500 is detected in a lying position within a short time from a state in which the subject 500 is detected in an upright position from the joint points 720, the behavior recognition unit 117 may recognize the behavior of "falling down". The behavior of the subject 500 may be recognized using a trained model of a neural network. The trained model is a model trained to estimate behavior based on a plurality of joint points 720, each of which is estimated from a plurality of photographed images adjacent in time series. The behavior of the subject 500 may be recognized using a relative positional relationship between the joint points 720 and an object (e.g., a bed 70). For example, when a lying position is detected from the joint points 720 outside the bed 70 and within a relatively close range from the bed 70 in the photographed image 700, the behavior of "falling down" may be recognized.
[0063] The control unit 110 may detect an event related to the subject 500 from the behavior of the subject 500 recognized by the behavior recognition unit 117. An event is a change in the state or situation of the subject 500, such as getting up, getting out of bed, falling, or tripping, which should be reported (notified) to a caregiver, etc. When the control unit 110 detects an event, it may transmit an event notification notifying the contents of the event to an external device such as the mobile terminal 300 or a fixed terminal (not shown) of the administrator.
[0064] The operation of the monitoring system 10 will now be described.
[0065] 9 is a flowchart showing the operation of the monitoring system 10. This flowchart can be executed by the control unit 110 of the information processing device 100 in accordance with a program.
[0066] The control unit 110 acquires a distance image 800 by the human presence sensor 143 (S101).
[0067] The control unit 110 detects the person 50 from the distance image 800 (S102).
[0068] The control unit 110 changes the shooting direction of the camera 141 so that the detected person 50 is within the shooting range of the camera 141 (S103).
[0069] The control unit 110 detects the person 50 from the image 700 captured by the camera 141 (S104).
[0070] The control unit 110 determines whether the detected person 50 is one person (S105). When the control unit 110 determines that the detected person 50 is one person (S105: YES), the control unit 110 determines that the detected person 50 is a tracking target (S107).
[0071] When the control unit 110 determines that the detected person 50 is not one person but multiple people (S105: NO), it calculates the moving speed of each of the detected people 50. Then, it determines the person 50 with the slowest moving speed (S106: YES) as the person to be followed (S107). The control unit 110 determines the person 50 other than the person 50 with the slowest moving speed (S106: NO) as not the person to be followed (S107).
[0072] The control unit 110 controls the shooting direction of the camera 141 by the driving mechanism 142 so as to follow the person 50 determined to be the follow-up target (S109).
[0073] The control unit 110 recognizes the behavior of the person 50 determined to be the follow-up target based on the joint points 720 and the like of the person 50.
[0074] Second embodiment A second embodiment of the present invention will be described. This embodiment differs from the first embodiment in the following respects. In the first embodiment, whether or not a person 50 is a target to be followed is determined based on the moving speed of the person 50, etc. On the other hand, in this embodiment, whether or not a person 50 is a target to be followed is determined by authenticating the face of the person 50 or an item worn by the person 50. In other respects, this embodiment is similar to the first embodiment, so duplicated explanations will be omitted.
[0075] The function of the control unit 110 will be described with reference to FIG.
[0076] The recognition unit 113 recognizes (detects) the person 50 and the face of the person 50 from the photographed image 700. When the photographed image 700 includes images of a plurality of people 50, the recognition unit 113 may recognize the plurality of people 50 and their faces. Specifically, the recognition unit 113 detects a person rectangle 710 from the photographed image 700. The recognition unit 113 detects the face of the person 50 included in the person rectangle 710 as a face region 711 (see FIG. 10 ) that is a region including the face. Specifically, for example, the recognition unit 113 uses a trained model of a neural network reflecting a dictionary (parameters) for detecting the face region 711 from the person rectangle 710. The recognition unit 113 may detect the face region 711 from the person rectangle 710 by the trained model. For example, a Region Proposal Network (RPN), which is a well-known model, may be used as the trained model. The recognition unit 113 can output the face region 711 as a combination of the photographed image 700 and the coordinates of two opposing vertices of the face region 711 .
[0077] FIG. 10 is a schematic diagram showing the face area 711. As shown in FIG.
[0078] In the example of Fig. 10, two people 50 are included in the photographed image 700, and their respective face regions 711 are detected. In Fig. 7, the person rectangles 710 are shown as rectangles drawn with dashed lines.
[0079] The determination unit 115 determines whether the person 50 is a target to be followed by authenticating the face of the person 50 recognized by the recognition unit 113. Specifically, the determination unit 115 authenticates the face of the person 50 as follows, for example. The image of the face of the target person 500 is registered by storing it in advance in the storage unit 130. The determination unit 115 may use a known face authentication technology. The determination unit 115 compares the registered face image with the face image included in the face area 711. Then, the determination unit 115 determines whether a face image whose features match the face image included in the face area 711 is registered. When the determination unit 115 determines that a face image whose features match the face image included in the face area 711 is registered, the determination unit 115 determines that the person 50 corresponding to the face area 711 is a target to be followed. Specifically, the determination unit 115 determines that the person 50 corresponding to the person rectangle 710 including the face area 711 is a target to be followed.
[0080] The determination unit 115 may determine whether the person 50 is a target to be followed by authenticating the wear of the person 50 included in the person rectangle 710. The wear includes a marker attached to clothing, etc. Specifically, the determination unit 115 authenticates the wear as follows, for example. An image of a specific wear worn by the subject 500 is registered by storing it in the storage unit 130 in advance. The determination unit 115 may use a known machine learning technique using a neural network. The determination unit 115 compares the registered image of the wear with the image of the wear included in the person rectangle 710. Then, the determination unit 115 determines whether an image of the wear that matches the image of the wear included in the person rectangle 710 is registered. When the determination unit 115 determines that an image of the wear that matches is registered, the determination unit 115 determines that the person 50 corresponding to the person rectangle 710 including the wear is a target to be followed.
[0081] The operation of the monitoring system 10 will now be described.
[0082] 11 is a flowchart showing the operation of the monitoring system 10. This flowchart can be executed by the control unit 110 of the information processing device 100 in accordance with a program.
[0083] Steps S201 to S204 in the flowchart of Fig. 11 are the same as steps S101 to S104 in the flowchart of Fig. 9 described in the first embodiment. Therefore, a description of steps S201 to S204 will be omitted. Also, steps S209 to S210 are the same as steps S109 to S110 in the flowchart of Fig. 9 described in the first embodiment. Therefore, a description of steps S209 to S210 will be omitted.
[0084] The control unit 110 recognizes the face of the person 50 detected in the captured image 700 (S205).
[0085] The control unit 110 authenticates the recognized face by determining whether the recognized face matches a registered face image (S206). When the control unit 110 determines that the recognized face matches a registered face image, it determines that the person 50 corresponding to the recognized face is a target to be followed (S207). When the control unit 110 determines that the recognized face does not match any of the registered face images, it determines that the person 50 corresponding to the recognized face is not a target to be followed (S208).
[0086] Third embodiment A third embodiment of the present invention will be described. This embodiment differs from the first embodiment in the following respects. In the first embodiment, it is determined whether or not a person 50 is a target to be tracked based on the moving speed of the person 50, etc. On the other hand, in this embodiment, it is determined whether or not the person 50 is a target to be tracked based on the relative positions of the person 50 and an object recognized from the captured image 700. In other respects, this embodiment is similar to the first embodiment, so duplicated explanations will be omitted.
[0087] The function of the control unit 110 will be described with reference to FIG.
[0088] The recognition unit 113 recognizes (detects) the person 50 and the object from the captured image 700. When the captured image 700 includes images of a plurality of people 50 and a plurality of objects, the recognition unit 113 can recognize all of the plurality of people 50 and objects. The objects include objects used by the subject 500. Specifically, the objects include, for example, a bed 70 and a wheelchair. In the following, for the sake of simplicity, a case where the object is a bed 70 will be described as an example.
[0089] The recognition unit 113 may detect the person 50 and the bed 70 included in the photographed image 700 as a person rectangle 710 and an object rectangle, respectively. Specifically, the recognition unit 113 detects an area in which an object (object) exists on the photographed image 700. The recognition unit 113 detects the person 50 and the bed 70 by estimating the category of the object included in the detected area. The area in which the object exists is detected as a candidate rectangle including the object on the photographed image 700. The person 50 is detected by detecting a candidate rectangle estimated to have an object category of the person 50 as the person rectangle 710 from among the detected candidate rectangles. The bed 70 is detected by detecting a candidate rectangle estimated to have an object category of the bed 70 from among the detected candidate rectangles as the bed area. Note that the bed area may be registered in advance as a predetermined area in the photographed image 700.
[0090] The determination unit 115 determines whether the person 50 is a target to be tracked, based on the relative positions of the person 50 and the bed 70 recognized by the recognition unit 113. Specifically, when the person rectangle 710 and the bed area overlap, the determination unit 115 determines that the person 50 corresponding to the person rectangle 710 is a target to be tracked. This is because the person 50 present on the bed 70 is highly likely to be the target person 500. The overlap of the person rectangle 710 and the bed area indicates that the person 50 is present on the bed 70.
[0091] The operation of the monitoring system 10 will now be described.
[0092] 12 is a flowchart showing the operation of the monitoring system 10. This flowchart can be executed by the control unit 110 of the information processing device 100 in accordance with a program.
[0093] Steps S301 to S304 in the flowchart of Fig. 12 are the same as steps S101 to S104 in the flowchart of Fig. 9 described in the first embodiment. Therefore, a description of steps S301 to S304 will be omitted. Furthermore, steps S309 to S310 are the same as steps S109 to S110 in the flowchart of Fig. 9 described in the first embodiment. Therefore, a description of steps S309 to S310 will be omitted.
[0094] The control unit 110 detects the person 50 and the bed 70 in the captured image 700 (S304).
[0095] The control unit 110 determines whether or not the detected person 50 is one person (S305). When the control unit 110 determines that the detected person 50 is one person (S305: YES), the control unit 110 determines that the detected person 50 is a tracking target (S307).
[0096] When the control unit 110 determines that the detected person 50 is not one person but multiple people (S305: NO), it determines whether the person 50 has been detected on the bed 70 (S306). When the control unit 110 determines that the detected person 50 has been detected on the bed 70 (S306: YES), it determines that the person 50 is a target to be followed (S307). When the control unit 110 determines that the detected person 50 is not detected on the bed 70 (S306: NO), it determines that the person 50 is not a target to be followed (S308).
[0097] (Fourth embodiment) A fourth embodiment of the present invention will now be described.
[0098] In this embodiment, after determining the target to be followed, when a predetermined condition is satisfied, the shooting direction of the camera 141 is changed again to determine the target to be followed. The predetermined condition includes, for example, the following. (1) The person 50 determined to be the target to be followed by the determination unit 115 is no longer recognized by the recognition unit 113 in the captured image 700. (2) A predetermined time has elapsed since the person 50 determined to be the target to be followed by the determination unit 115 is no longer recognized by the recognition unit 113 in the captured image 700. (3) The person 50 determined to be the target to be followed by the determination unit 115 is no longer recognized by the recognition unit 113 in the captured image 700. Thereafter, the person 50 is detected by the human presence sensor 143.
[0099] Fig. 13 is an explanatory diagram showing a situation in which person 50, who has been determined to be a tracking target by determination unit 115, is no longer recognized by recognition unit 113 in photographed image 700. In Fig. 13, arrows indicate the passage of time. Also, the shooting range of camera 141 is indicated by a dashed line.
[0100] 13, a subject 500 is lying on a bed 70. Meanwhile, a staff member 50s, who is a person 50 other than the subject 500, is standing and starting to walk towards the exit.
[0101] The upper diagram of FIG. 13 shows a situation where a staff member 50s, who is a person 50 other than the subject 500, is mistakenly determined to be a tracking target. The middle diagram shows a situation where the shooting direction of the camera 141 is changed erroneously to track the staff member 50s. Since image distortion, occlusion, etc. may occur, it cannot be said that there is no possibility of a misjudgment of a tracking target. The lower diagram shows a situation where the staff member 50s leaves the room of the subject 500. As a result, the staff member 50s that the camera 141 has mistakenly been tracking disappears from the shooting range of the camera 141. When the situation shown in the lower diagram occurs, the person 50 is not recognized from the captured image 700. That is, the person 50 determined to be a tracking target by the determination unit 115 is no longer recognized by the recognition unit 113 in the captured image 700. If the shooting direction of the camera 141 is not changed in this state, the subject 500 to be tracked is not monitored.
[0102] Therefore, in this embodiment, the following control is performed. Person 50 determined as the target to be followed by determination unit 115 is no longer recognized by recognition unit 113 in captured image 700. At this time, control unit 110 changes the shooting direction of camera 141 again to determine the target to be followed. Then, control unit 110 changes the shooting direction of camera 141 to cause person 50 determined as the target to be followed to be followed.
[0103] The control unit 110 may perform the following control. Person 50 determined as the target to be followed by the determination unit 115 is no longer recognized by the recognition unit 113 in the captured image 700. After a predetermined time has elapsed, the control unit 110 may again change the shooting direction of the camera 141 to determine the person to be followed. Then, the control unit 110 may follow person 50 determined as the target to be followed by changing the shooting direction of the camera 141. The predetermined time may be appropriately set by experiment or the like in terms of the proportion of time during which the target to be followed is correctly determined, etc.
[0104] The control unit 110 may perform the following control. The person 50 determined as the target to be followed by the determination unit 115 is no longer recognized by the recognition unit 113 in the captured image 700. Thereafter, when the person 50 is detected by the human sensor 143, the control unit 110 may again determine that the person is the target to be followed by changing the shooting direction of the camera 141. Then, the control unit 110 may track the person 50 determined as the target to be followed by changing the shooting direction of the camera 141.
[0105] Fig. 14 is an explanatory diagram showing a situation in which, after the person 50 being followed is no longer recognized, another person 50 detected by the human sensor 143 is determined to be the tracking target. In Fig. 14, the person 50 being followed is no longer recognized by the recognition unit 113. Thereafter, the shooting direction of the camera 141 is changed again, and the other person 50 detected by the human sensor 143 is determined to be the tracking target.
[0106] In the example of FIG. 14, the control unit 110 changes the shooting direction of the camera 141 again and correctly determines the target person 500 as the target to be tracked.
[0107] 15 is an explanatory diagram showing a situation in which the person 50 that was being followed is no longer recognized, and then the person 50 is no longer detected by the human presence sensor 143. In FIG. 15, the person 50 that was being followed is no longer recognized by the recognition unit 113. Thereafter, the person 50 is no longer detected by the human presence sensor 143.
[0108] 15, since person 50 has not been detected by motion sensor 143, control unit 110 fixes the shooting direction of camera 141 toward the doorway of the room. There is a high possibility that the next person 50 entering the room is target person 500 to be tracked. Therefore, person 50 to be tracked can be quickly tracked.
[0109] The embodiment provides the following advantages.
[0110] The driving unit changes the shooting direction of the shooting unit based on the detection information of a person by the human sensor.Then, it is determined whether the person is a tracking target based on the characteristics of the person recognized from the image of the shooting unit, and the shooting direction of the shooting unit is changed by the driving unit so that the person determined to be a tracking target is tracked.This improves the accuracy of capturing the target by the shooting unit.
[0111] In addition, the system determines whether or not a person is a target to be tracked based on the recognized moving speed of the person. This makes it possible to simply and effectively improve the accuracy with which the image capture unit captures the target to be watched over.
[0112] In addition, the system determines whether a person is a target to be tracked by authenticating their face or the person they are wearing. This improves the accuracy of the camera unit's capture of the person being watched over.
[0113] In addition, the system estimates the joint points of the person recognized from the image captured by the camera and judges whether or not the person is a target to be tracked based on the joint points. This further improves the accuracy of capturing the target by the camera.
[0114] In addition, an object is further recognized from an image captured by the image capture unit. Then, based on the relative positions of a person and the recognized object from the image, it is determined whether the person is the tracking target. This makes it possible to more flexibly improve the accuracy of capturing the target by the image capture unit.
[0115] Furthermore, when a predetermined condition is satisfied, the shooting direction of the shooting unit is changed again to follow the person determined to be the tracking target, thereby making it possible to prevent the occurrence of a state in which the tracking target is not tracked.
[0116] Furthermore, the above-mentioned predetermined condition is (1) that the person determined to be the target to be followed is no longer recognized in the image, or (2) that a predetermined time has elapsed since the person determined to be the target to be followed was no longer recognized in the image, or (3) that a person is detected by the human presence sensor after the person determined to be the target to be followed is no longer recognized in the image. This makes it possible to simply and effectively prevent the occurrence of a state in which the target to be followed is not followed.
[0117] The above-described configurations of the monitoring device, the control method, and the control program are the main configurations for explaining the features of the above-described embodiment. Therefore, the monitoring device, the control method, and the control program are not limited to the above-described configurations, and can be modified in various ways within the scope of the claims. In addition, the configurations of general monitoring devices, control methods, and control programs are not excluded.
[0118] For example, some steps may be omitted from the above-described flowchart, other steps may be added, some of the steps may be executed simultaneously, and one step may be divided into multiple steps and executed.
[0119] The means and methods for performing various processes in the above-mentioned system can also be realized by a dedicated hardware circuit. The means and methods for performing various processes in the above-mentioned system can also be realized by a programmed computer. The above-mentioned program may be provided by a computer-readable recording medium such as a USB memory or a DVD (Digital Versatile Disc)-ROM, or may be provided online via a network such as the Internet. In this case, the program recorded in the computer-readable recording medium is usually transferred to and stored in a storage unit such as a hard disk. The above-mentioned program may also be provided as a standalone application software, or may be incorporated as a function into the software of a device such as an information processing device. [Explanation of symbols]
[0120] 10 Monitoring system, 50 people, 50s caregiver, 70 beds, 100 Information processing device, 110 control section, 111 Position detection unit, 112 Drive unit, 113 Recognition Department, 114 Estimation Department, 115 Judgment Department, 116 Tracking control section, 117 Behavioral Recognition Department, 120 Communications Department, 130 storage section, 140 camera unit, 141 cameras, 142 Drive mechanism, 143 Human sensor, 200 communication networks, 210 access points, 300 mobile devices, 500 subjects, 700 captured images, 710 people rectangle, 711 face area, 720 articulation points.
Claims
1. An imaging unit that captures an image of a person; A human presence sensor that detects the location of people, A drive unit that changes the shooting direction of the shooting unit based on detection information of the human presence sensor; a recognition unit that recognizes a person from the image captured by the imaging unit; a determination unit that determines whether or not the person recognized by the recognition unit is a target to be followed based on features of the person recognized by the recognition unit; a control unit that changes the shooting direction of the shooting unit to the driving unit so as to follow the person determined to be the follow-up target by the determination unit; A monitoring device having the above configuration.
2. The monitoring device according to claim 1 , wherein the determination unit determines whether the person is the target to be followed based on a moving speed of the person recognized by the recognition unit.
3. The monitoring device according to claim 1 , wherein the determination unit determines whether the person recognized by the recognition unit is the target to be followed by authenticating a face or an article worn by the person.
4. an estimation unit that estimates joint points of a person recognized by the recognition unit; The monitoring device according to claim 1 , wherein the determination unit determines whether the object is the target to be tracked based on the joint point estimated by the estimation unit.
5. The recognition unit further recognizes an object other than a human from the image captured by the imaging unit, The monitoring device according to claim 1 , wherein the determination unit determines whether a person recognized by the recognition unit is the tracking target based on a relative position between the person and the object recognized by the recognition unit.
6. The monitoring device according to any one of claims 1 to 5, wherein, when a predetermined condition is satisfied, the control unit again causes the drive unit to change the shooting direction of the shooting unit so as to track the person determined by the determination unit to be the tracking target.
7. The monitoring device of claim 6, wherein the specified condition is that a person determined to be the target to be followed by the judgment unit is no longer recognized by the recognition unit in the image, a specified time has elapsed since the person determined to be the target to be followed by the judgment unit was no longer recognized by the recognition unit in the image, or a person is detected by the human presence sensor after the person determined to be the target to be followed by the judgment unit is no longer recognized by the recognition unit in the image.
8. An imaging unit that captures an image of a person; A human presence sensor that detects the location of people, A control method for a monitoring device having a driving unit that changes the shooting direction of the shooting unit, (a) changing the photographing direction of the photographing unit based on detection information of the human presence sensor; (b) a step of recognizing a person from the image captured by the imaging unit; A step (c) of determining whether the recognized person is a target to be followed based on characteristics of the person recognized in the step (b); A step (d) of changing the shooting direction of the shooting unit to the driving unit so as to follow the person determined to be the follow-up target in the step (c); The control method includes:
9. 9. The control method according to claim 8, wherein in said step (c), it is determined whether or not the person recognized in said step (b) is the target to be followed based on a moving speed of the person.
10. 9. The control method according to claim 8, wherein in the step (c), it is determined whether the person recognized in the step (b) is the target to be followed by authenticating a face or an article worn by the person.
11. a step (e) of estimating articulation points of the person recognized in step (b); 9. The control method according to claim 8, wherein in said step (c), it is determined whether or not the object is the target to be tracked based on the joint points estimated in said step (e).
12. In the step (b), an object other than a human is further recognized from the image captured by the imaging unit; The control method according to claim 8 , wherein in the step (c), it is determined whether the person recognized in the step (b) is the tracking target based on a relative position between the person recognized in the step (b) and the object recognized in the step (b).
13. The control method according to claim 8, further comprising a step (f) of causing the driving unit to change the shooting direction of the shooting unit again to follow the person determined to be the target to be followed, when a predetermined condition is satisfied.
14. The control method of claim 13, wherein the specified condition is that the person determined to be the target to be followed in step (c) is no longer recognized in the image, a specified time has elapsed since the person determined to be the target to be followed in step (c) was no longer recognized in the image, or a person is detected by the human presence sensor after the person determined to be the target to be followed in step (c) is no longer recognized in the image.
15. A control program for causing a computer to execute the control method according to any one of claims 8 to 14.
Citation Information
Patent Citations
Care recipient watching system
JP2022110634A
