Notification device, notification method, and program

Abstract

To provide a notification device capable of notifying a user of an abnormality or sickness of a target person, a notification method, and a program.SOLUTION: A notification device 10 comprises: an image capturing section 11 which captures a picked-up image; a portion discrimination section 12 which discriminates portions of a target person on the basis of the picked-up image; an action detection section which detects a repeated action based on movements of the portions of the target person; a notification determination section 14 which determines whether to perform a notification to a user corresponding to the target person based on a detection result of the repeated action; and a communication section 15 which performs the notification based on a determination result of the notification determination section 14.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a notification device, a notification method, and a program. [Background technology]

[0002] Patent Document 1 discloses a system that can ensure the safety of a monitored subject. In this system, an image of a non-monitored subject is captured. The system detects the coordinates of a predetermined body part of the monitored subject and the coordinates of predetermined body parts of another person from the image. The system outputs a warning signal when the right or left wrist of the non-monitored subject moves toward the chest, waist, buttocks, hand, or shoulder of the other person.

[0003] There is also a nursing care system that watches over elderly people in remote locations (Patent Document 2). The system in Patent Document 2 analyzes camera footage to determine whether the elderly person has fallen. If the elderly person has fallen, a notification is sent. Furthermore, if the duration of the fall exceeds a reference time, a notification is sent to the monitor. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-55250 [Patent Document 2] Japanese Patent Application Laid-Open No. 2010-40017 Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, there is a monitoring service that watches over elderly people and the like in remote locations (Patent Document 2). Such a service notifies the monitor when a serious abnormality occurs in the monitored person, but does not notify of malfunctions that the monitored person, the monitor, or people around them are not aware of, or changes in the monitored person that occur before the monitored person becomes seriously ill.

[0006] The present disclosure has been made in consideration of the above points, and aims to provide a notification device, a notification method, and a program that can appropriately notify a user of abnormalities or malfunctions in a target person. [Means for solving the problem]

[0007] The notification device of this embodiment includes an image acquisition unit that acquires an image of a subject, a body part discrimination unit that discriminates body parts of the subject based on the image, a movement detection unit that detects repetitive movements based on the movements of the discriminated body parts of the subject, a notification determination unit that determines whether or not to notify a user associated with the subject based on the detection result of the repetitive movement, and a communication unit that issues a notification based on the determination result of the notification determination unit.

[0008] The notification method of this embodiment is a notification method in which a notification device executes the following steps: acquiring an image of a subject; determining a body part of the subject based on the image; detecting a repetitive motion based on the movement of the determined body part of the subject; determining whether or not to notify a user associated with the subject based on the detection result of the repetitive motion; and notifying based on the result of the determination.

[0009] The program of this embodiment is a program for causing a computer to execute a notification method, and the notification method includes the steps of acquiring an image of a subject, determining a body part of the subject based on the image, detecting a repetitive motion based on the movement of the determined body part of the subject, determining whether or not to notify a user associated with the subject based on the detection result of the repetitive motion, and notifying based on the determination result. [Effects of the Invention]

[0010] According to the present disclosure, it is possible to provide a notification device, a notification method, and a program that can appropriately notify a user of an abnormality or malfunction in a target person. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram schematically illustrating the overall configuration of a notification system. [Figure 2] FIG. 1 is a control block diagram showing the configuration of a system using a notification device. [Figure 3] FIG. 1 is a diagram schematically illustrating the skeleton of a subject estimated by image analysis. [Figure 4] 4 is a flowchart showing a notification method according to the first embodiment. [Figure 5] 10 is a flowchart showing a notification method according to the second embodiment. [Figure 6] 10 is a flowchart showing a notification method according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. However, the present disclosure is not limited to the following embodiments. In addition, the following description and drawings have been simplified as appropriate for clarity of explanation.

[0013] Embodiment 1 FIG. 1 is a schematic diagram showing the system configuration of a notification system 1 using a notification device according to this embodiment. The notification system 1 is used for a monitoring service for elderly people and the like. The notification system 1 notifies a user 202, who is a monitor, of any abnormalities or malfunctions of a target person 102. Here, the target person 102 is a person to be monitored, such as an elderly person or a sick person. The monitoring site 101 is the residence of the target person 102, for example. The monitoring site 101 is provided with a camera 103 that captures an image of the target person 102, and a notification device (not shown in FIG. 1).

[0014] The user 202 is a monitor who monitors the subject 102. Specifically, the user 202 is a relative such as a child or sibling of the subject 102, or a caregiver. The user 202 registers information about the notification destination (email address, etc.) of the subject 102 in advance in a notification device. The user site 201 is the residence of the user 202, for example. The user site 201 is located, for example, in a remote location from the monitoring site 101. The monitoring site 101 where the subject 102 is located and the user site 201 where the subject 102 is located are connected via a network, etc. The address of the notification destination may be a caregiver who is near the subject 102, for example. Furthermore, two or more notification destination addresses may be specified.

[0015] When the notification system 1 detects an abnormality, malfunction, or the like in the target person 102, it notifies the user 202. Specifically, the notification is sent as an email, a message, or the like to the user terminal 203. The user terminal 203 is a smartphone, a personal computer, or the like owned by the user 202. Therefore, the user 202 can be aware of the abnormality in the target person 102 even when he or she is not near the target person 102.

[0016] Assume that the subject 102 is in a situation where there is no hindrance to daily life, but is experiencing discomfort in the lower back and is frequently rubbing the lower back. The notification system 1 analyzes the image captured by the camera 103 to detect a predetermined notification action of the subject 102. The notification system 1 transmits a notification to the user 202 based on the detection result of the notification action. Here, the notification action is a rubbing action. Of course, the part of the body that is the target of the rubbing action is not limited to the lower back, and may be other parts of the subject 102, such as the knees, legs, feet, abdomen, buttocks, head, arms, or shoulders. The notification system 1 may identify the target part that is being rubbed.

[0017] When the notification system 1 detects a repetitive motion of repeatedly moving a hand to a target area, it determines that the subject 102 is rubbing the target area. Then, the notification system 1 determines that there is something wrong with the subject 102 or a malfunction, and notifies the user 202. In this way, the notification system 1 can notify the subject 102 of an abnormality, malfunction, etc. at an early stage before the condition becomes serious.

[0018] An example of the configuration and processing of the notification system 1 will be described using Fig. 2. Fig. 2 is a block diagram that schematically shows the configuration of the notification system 1. The notification system 1 includes a notification device 10, a camera 103, and a user terminal 203. The notification device 10, the camera 103, and the user terminal 203 are connected via a wireless or wired network.

[0019] The notification device 10 includes an image acquisition unit 11, a body part discrimination unit 12, a motion detection unit 13, a notification determination unit 14, and a communication unit 15. The notification device 10 may further include a recording unit 16.

[0020] The notification device 10 is an information processing device such as a personal computer, a smartphone, or a tablet PC. For example, the notification device 10 includes a processor and a memory, and the processor executes a program stored in the memory to realize the functions of each unit. The notification device 10 stores a user 202 and a target person 102 in advance in association with each other. The notification device 10 stores the email address of the notification destination. The notification device 10 functions as an image analysis device that analyzes images from the camera 103. For example, when the notification device 10 detects through image analysis that the target person 102 has performed a rubbing action as a notification action, it notifies the user 202 based on the detection result.

[0021] The notification device 10 is not limited to a personal computer, a smartphone, or the like, as long as it can perform image analysis processing. For example, at least some of the functions of the notification device 10 may be implemented in the camera 103. Alternatively, the notification device 10 may be mounted on the camera 103. In other words, the notification device 10 may be a device that mounts the camera 103. Furthermore, the notification device 10 may be a single physical device or may be different devices. For example, some of the processing or functions may be implemented by a server located in a remote location.

[0022] The notification device 10 includes a communication unit 15 and is capable of communicating with the user terminal 203 wirelessly or via a wired connection. For example, the notification device 10 is connected to a network via a wireless LAN (Local Area Network) device such as Wi-Fi (registered trademark). The notification device 10 and the user terminal 203 are capable of receiving data from the network and transmitting data to the network. A known network communication protocol can be used, and therefore a description thereof will be omitted.

[0023] The image acquisition unit 11 acquires captured images captured by the camera 103. The captured images are moving images or continuous still images of the subject 102. The captured images may be RGB images or grayscale images. The captured images may also be infrared images.

[0024] The part discrimination unit 12 discriminates the parts of the subject 102 contained in the captured image by performing image analysis on the captured image. For example, the part discrimination unit 12 identifies the position coordinates of the shoulders, waist, arms, hands, feet, legs, and each joint in the captured image. The part discrimination unit 12 identifies the left and right hands of the subject 102 as predetermined parts. Furthermore, the part discrimination unit 12 estimates the distance of each part from the predetermined parts. The part discrimination unit 12 extracts parts close to the hands as target parts. In other words, the part discrimination unit 12 focuses on the hands of the subject 102 and determines which part of the body the hands are in.

[0025] The processing in the part discrimination unit 12 can utilize an AI model (machine learning model) that has learned the shape of a hand. When the part discrimination unit 12 inputs the captured image into the machine learning model, the machine learning model performs image analysis. This allows the part discrimination unit 12 to identify the hand of the subject 102. In other words, the part discrimination unit 12 outputs the position coordinates of the hand in each frame of the captured image. Furthermore, the part discrimination unit 12 identifies the coordinates of each part in each frame of the captured image. The coordinates here may be coordinates in the captured image, or may be coordinates based on a predetermined part of the subject 102. The part discrimination unit 12 estimates the distance of each part from the predetermined part. The part discrimination unit 12 identifies the target part based on the distance of each part, etc.

[0026] The part determination unit 12 determines the part of the subject 102 using known skeletal structure estimation technology and posture estimation technology. OpenPose is a known example of such skeletal structure estimation technology. FIG. 3 is a diagram schematically showing a skeleton F estimated by the part determination unit 12. The skeleton F is a model in which shoulder joints, elbow joints, wrist joints, hip joints, knee joints, ankle joints, tips of the feet, tips of the hands, neck joints, the center of the head, etc. are used as nodes, and lines connecting the nodes are used as links. In FIG. 3, the right hand is located near the abdomen in the captured image. If the right hand is the predetermined part P1, then the abdomen becomes the target part P2. Of course, the predetermined part P1 may be the left hand, not just the right hand.

[0027] The motion detection unit 13 detects that the subject 102 has performed a notifying motion. In this example, the notifying motion is a motion of rubbing the target part P2. For example, the motion detection unit 13 detects the motion of rubbing the predetermined part P1 against the target part P2 based on a change in the position of the predetermined part P1 relative to the target part P2 of the subject 102. The rubbing motion is a repetitive motion in which the predetermined part P1 repeatedly moves relative to the target part P2. The motion detection unit 13 obtains the relative position of the predetermined part P1 with respect to the target part P2. The motion detection unit 13 calculates the change in position of the predetermined part P1. Based on the change in position, the motion detection unit 13 detects that the subject 102 is performing a motion of rubbing the target part P2 with his / her hand. As shown in FIG. 3, the right hand is repeatedly moving left and right around the abdomen, so the motion detection unit 13 determines that the subject 102 is performing a motion of rubbing the abdomen.

[0028] The motion detection unit 13 determines that a notification motion has occurred when the predetermined part P1 has moved to the same part multiple times. When the position of the predetermined part P1 repeatedly coincides with the position of the target part P2 and then moves away from it, the motion detection unit 13 detects that a rubbing motion has occurred. The motion detection unit 13 determines whether the predetermined part P1 has repeatedly moved over the target part P2.

[0029] When the movement detection unit 13 detects movement of the predetermined part P1 relative to the target part P2, it stores the coordinates of the movement and starts counting the number of movements. When the movement detection unit 13 extracts movement that includes part of the stored coordinates, it counts up the count. The movement detection unit 13 detects a repeated movement, for example, by counting two or more times. Furthermore, a movement may be determined by counting the cessation of movement at the same part (coordinates) for a predetermined period of time, and a repeated movement may be determined, for example, by counting two or more times. In other words, when the predetermined part P1 presses the target part P2 for a certain period of time or more, the movement detection unit 13 may detect that movement.

[0030] Alternatively, the motion detection unit 13 obtains a vector and a distance by comparing the position of the predetermined part P1 with the position of the target part P2. The motion detection unit 13 determines whether the motion of the predetermined part P1 with respect to the target part P2 is a rubbing motion based on the change in the vector and the distance over time. When the subject 102 rubs the target part P2 with his / her hand, the predetermined part P1 moves so as to repeatedly approach and move away from the target part P2, and the direction of this repeated movement is constant. Therefore, the motion detection unit 13 can detect the notification motion based on the change in the vector and the distance between the target part P2 and the predetermined part P1.

[0031] For example, when a hand moves back and forth within a given range of the skeleton, the motion detection unit 13 determines that the motion is a hand rubbing motion within the given range. Alternatively, when a hand repeatedly contacts or approaches a specific position on the skeleton, the motion detection unit 13 determines that the motion is a hand rubbing motion. In this way, the motion detection unit 13 can detect a rubbing motion on the target part P2.

[0032] The notification determination unit 14 determines whether to notify the user 202 based on the detection result of the repetitive motion by the motion detection unit 13. For example, when the motion detection unit 13 detects a rubbing motion, the notification determination unit 14 determines to notify the user 202.

[0033] The communication unit 15 transmits a notification to the user 202 based on the determination result of the notification determination unit 14. The communication unit 15 transmits status information based on the rubbing action to the user terminal 203. The status information includes the target part P2 on which the notification action was performed, the time for which the rubbing action was performed, etc. In other words, the communication unit 15 notifies the user 202 of the part that was rubbed and the time for which it was rubbed. The communication unit 15 may transmit the notification by email, text message, voice message, etc. For example, the email address of the notification destination is stored in a memory or the like.

[0034] In this way, the motion detection unit 13 detects the repetitive motion of the predetermined part P1 repeatedly moving over the target part P2 as the notification motion. Therefore, the notification device 10 can notify the user 202 of any abnormality or problem in the subject 102 at an appropriate time. When the subject 102 rubs a part of concern or a painful part, the notification is sent to the user 202. The user 202 can recognize any abnormality or problem in the subject 102 at an early stage. Therefore, the user 202 can recognize any disease or abnormality that is not a major hindrance but is chronically occurring in the subject 102. Therefore, the subject 102 can be cared for at an early stage before the user 202 becomes seriously ill.

[0035] Furthermore, the notification device 10 may have a recording unit 16. The recording unit 16 has a memory that records the history of the rubbing action in chronological order. The recording unit 16 may also be a cloud server. The notification determination unit 14 may make a notification determination based on the history of the notification action.

[0036] When the action detection unit 13 detects a rubbing action, the recording unit 16 records the start time, end time, etc. of the rubbing action. The end time can be the timing when the predetermined part P1 moves away from the target part P2 for a certain period of time or more. Furthermore, the action detection unit 13 may record the duration of the rubbing action, the target part P2, the number of times the rubbing is performed, etc. The duration is the difference between the end time and the start time. The recording unit 16 stores action information related to the above-mentioned notification action in memory. The action information is information in which the start time, end time, target part P2, etc. of the notification action are associated with each other.

[0037] For example, the notification determination unit 14 may determine whether to issue a notification based on the duration of the notification action. If the duration is equal to or greater than a threshold, the notification determination unit 14 determines that a notification should be issued. In other words, if the duration is less than the threshold, the notification determination unit 14 determines that a notification should not be issued. Specifically, if the duration is 10 seconds or more, the notification determination unit 14 determines that a notification should be issued. Alternatively, the notification determination unit 14 may make a notification determination based on the number of times rubbing occurs in one duration. The notification determination unit 14 may also make a notification determination based on the total time of repeated actions on the same part in one day. In this way, the notification device 10 can more appropriately issue notifications.

[0038] Furthermore, the notification determination unit 14 may determine whether or not to issue a notification based on the frequency of the notification action. For example, if the frequency is equal to or greater than a threshold, the notification determination unit 14 determines that a notification should be issued. In other words, if the frequency is less than the threshold, the notification determination unit 14 determines that a notification should not be issued. Specifically, if the action is repeated three or more times per hour, the notification determination unit 14 determines that a notification should be issued. Alternatively, if there are three or more consecutive days in which the user rubs the screen five or more times per day, the notification determination unit 14 issues a notification.

[0039] Furthermore, the notification determination unit 14 may make a notification determination based on an increase or decrease in the frequency of the notification action. That is, if the increase in frequency is equal to or greater than a threshold, the notification determination unit 14 determines that a notification should be made. In other words, if the increase in frequency is less than the threshold, the notification determination unit 14 determines that a notification should not be made. For example, suppose that the average frequency on the previous day was three times per hour, and the average frequency on the following day was five times per hour. In this case, since the frequency of the rubbing action has increased by more than a threshold (for example, two times), the notification determination unit 14 determines that a notification should be made. In this way, the notification device 10 can more appropriately make a notification.

[0040] In this way, the notification device 10 can more appropriately notify the subject 102. For example, the subject 102 may have a habit of rubbing a specific part of the body. The notification determination unit 14 compares the current frequency with the previous frequency and makes a determination based on an increase in frequency. In this way, the habitual movement can be detected as a notifiable movement and a notification can be made at an appropriate time. Therefore, the notification device 10 can appropriately notify the subject 102 of a change in the state of the subject 102.

[0041] The notification determination unit 14 may make a determination based on at least one of the following criteria: detection of a rubbing motion, duration of the rubbing motion, frequency of the rubbing motion, and increase or decrease in frequency. Of course, the notification determination unit 14 may make a determination by combining two or more of these criteria. If two or more criteria are met, the notification determination unit 14 determines that a notification should be issued. The criteria for notification determination may also be set by the user 202.

[0042] Furthermore, the notification determination unit 14 may use the posture of the subject 102 as a determination condition. For example, suppose the subject 102 frequently rubs or holds his / her knees when standing up. The action of rubbing or holding his / her knees when standing up can be used as a notification action. Alternatively, the action of rubbing or holding his / her shoulders when raising his / her arms can be used as a notification action. By using posture and movement as determination conditions in this way, it is possible to accurately detect any abnormality or malfunction in the subject 102.

[0043] The notification method according to this embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the notification method.

[0044] First, the image acquisition unit 11 acquires an image captured by the camera 103 (S11). The part discrimination unit 12 discriminates parts based on the image (S12). Here, the part discrimination unit 12 obtains the position coordinates of each part using a skeleton estimation technique or the like. The part discrimination unit 12 determines whether or not the hand has been identified (S13). Specifically, the part discrimination unit 12 determines whether or not the hand is included in the discriminated parts, and if the hand is included, identifies the position of the hand. If the hand cannot be identified (No in S13), the process returns to step S12. That is, the part discrimination process is repeated until the part discrimination unit 12 identifies the hand. Furthermore, if the hand is in the blind spot of the camera 103, the notification device 10 may perform the process in the next or subsequent frames.

[0045] If the hand can be identified (Yes in S13), the motion detection unit 13 performs motion detection (S14). The motion detection unit 13 determines whether the hand motion is a rubbing motion (S15). If the hand motion is not a rubbing motion (No in S15), the process returns to S14. That is, the process is repeated until the motion detection unit 13 detects a rubbing motion.

[0046] If the hand motion is a rubbing motion (Yes in S15), the notification determination unit 14 determines whether or not to notify (S16). As described above, the recording unit 16 may record a history of the detection results. The notification determination unit 14 can make a notification determination based on the detection of a rubbing motion, the duration of the rubbing motion, the frequency of the rubbing motion, and an increase in the frequency. If a notification is to be made (Yes in S16), the communication unit 15 transmits a notification to the user terminal 203 (S17). If a notification is not to be made (No in S16), the process returns to S14.

[0047] In this way, the notification device 10 can send a notification to the user 202 at an appropriate time.

[0048] In the above description, the notification action was a rubbing action, but actions other than rubbing action may also be the notification action. For example, the notification action may be repeated stops of action for a predetermined time at the same part (coordinate), i.e., pressing the same part with the hand for a certain period of time or more. The notification determination unit 14 can make a notification determination based on the detection of the pressing action, the duration of the pressing action, the frequency of the pressing action, and an increase in the frequency. Furthermore, the notification action may be detected by setting a part other than the hand as the predetermined part.

[0049] In the above description, the part identification process and the motion detection process were performed based on the captured image, but the detection results of other sensors may also be used. For example, if the subject 102 wears a wearable device such as a smart watch on his / her arm, the sensor of the wearable device may be used. For example, if the smart watch has an acceleration sensor, it is possible to accurately detect hand motion.

[0050] Even if the hand is in the blind spot of the camera 103, the motion detection unit 13 can detect the rubbing motion. For example, suppose that the camera 103 is capturing an image of the subject 102 from the front, and the subject 102 performs a motion of rubbing his / her back. In this case, the hand is hidden by the body in the image captured by the camera 103, so the motion of the hand cannot be detected. By using the sensor of the wearable device, the motion detection unit 13 can detect that the hand is moving repeatedly. This allows the motion detection unit 13 to detect the notification motion with higher accuracy.

[0051] Furthermore, the notification device 10 may detect abnormalities or malfunctions of the target person 102 using thermal images acquired by a far-infrared camera or the like. For example, if the temperature of the target part P2 is high, the notification determination unit 14 may determine to issue a notification. Alternatively, if the body temperature of the target person 102 is high, a notification may be issued. Furthermore, the action detection unit 13 may detect a repetitive action that serves as a notification action based on the movement of the identified part of the target person 102.

[0052] Embodiment 2 A notification device according to the second embodiment will be described. The basic configuration of the notification device 10 is the same as that of the first embodiment, and therefore illustrations and detailed descriptions will be omitted. For example, the notification device 10 includes an image acquisition unit 11, a body part discrimination unit 12, a motion detection unit 13, a notification determination unit 14, a communication unit 15, and a recording unit 16, as shown in FIG.

[0053] In this embodiment, the repetitive motion detected by the motion detection unit 13 is different from that in the first embodiment. For example, the repetitive motion is different from a continuous motion such as rubbing with the hands. Specifically, the repetitive motion is a motion in which the subject 102 stands up from a chair or the like while repeatedly making sudden stops midway through due to pain or the like, or a motion in which the subject 102 repeatedly tries to stand up after falling because he or she is unable to lose balance. The motion detection unit 13 detects a motion with a sudden stop as a repetitive motion. In other words, the motion detected by the motion detection unit 13 is a discontinuous motion. Furthermore, the body part determination unit 12 and the motion detection unit 13 may be AI models constructed by machine learning.

[0054] 5 is a flowchart showing a notification method according to this embodiment. When image acquisition unit 11 acquires an image (S21), body part determination unit 12 determines the body parts of subject 102 (S22). For example, body part determination unit 12 acquires the positions of subject 102's head, figure, arms, waist, legs, feet, etc., and performs skeletal estimation. Movement detection unit 13 performs movement analysis of each body part (S23).

[0055] For example, the motion detection unit 13 acquires a motion vector of each body part based on a plurality of frame images. The motion detection unit 13 analyzes the motion of the subject by evaluating the amount of change in the body part per unit time. The motion detection unit 13 determines whether the detected motion is a notification motion (S24).

[0056] As described above, the notifiable motion to be detected is a motion of suddenly stopping when standing up. The motion detection unit 13 detects that the subject is standing up from a chair from a change in posture. Furthermore, the motion detection unit 13 extracts deceleration of a predetermined part such as the head. For example, the motion detection unit 13 extracts a rapid deceleration in the amount of change in the head or shoulders in the vertical direction, and determines this as a notifiable motion. If the motion obtained by the motion analysis is a notifiable motion (YES in S24), the recording unit 16 records the status and number of times of the notifiable motion (S25). That is, the motion detection unit 13 counts the number of times the notifiable motion is detected, and writes this to the recording unit 16.

[0057] If the motion obtained by the motion analysis is not a notification motion (NO in S24), the process returns to S22. That is, the notification device 10 performs the same process on the next acquired image. The notification determination unit 14 determines whether the counted number of extractions is equal to or greater than a predetermined number. If the number of extractions is less than the predetermined number (NO in S26), the process returns to step S22. That is, the notification device 10 performs the same process on the images that are acquired sequentially.

[0058] If the number of detections is equal to or greater than a predetermined number (YES in S26), the communication unit 15 transmits a notification to the user terminal 203 (S27). In this way, the motion detection unit 13 can detect motions other than continuous motions as notification motions. Then, the notification device 10 issues a notification according to the number of detections of notification motions. If the subject 102 is unable to stand up properly and performs a repetitive motion such as trying to stand up multiple times, a notification is received.

[0059] In this way, a repetitive motion is detected based on the movement of the identified body part. The notification determination unit 14 determines whether or not to notify the user associated with the target person based on the detection result of the repetitive motion. Here, the motion detection unit 13 extracts a motion that stops when standing up. The motion detection unit 13 determines that a motion that stops when the head or the like slows down when standing up.

[0060] For example, suppose that the subject 102 stops moving once while standing up, and then stops moving again when standing up. In this case, the subject 102 repeats the deceleration action twice. Therefore, if the action detection unit 13 detects deceleration twice or more, the notification determination unit 14 determines to issue a notification. In other words, the action detection unit 13 starts counting when it detects deceleration of a predetermined part while the subject is moving in a predetermined direction. The action detection unit 13 detects repeated actions when the count is equal to or greater than a predetermined number of times. The action detection unit 13 may use an AI model to extract deceleration of a predetermined part. In this case, supervised machine learning can be performed by using a set of image data when deceleration occurs and image data when no deceleration occurs as training data.

[0061] The notification action detected by the action detection unit 13 is not limited to the stopping action when standing up. For example, the action detection unit 13 can detect a stumble or fall while walking as a notification action. The action detection unit 13 detects a fall when the position of the head in the vertical direction moves rapidly toward the feet. Alternatively, the action of reflexively pulling back the hand after touching a hot object may be detected as a notification action. If these actions are repeated a predetermined number of times or more, the notification device 10 will issue a notification. Furthermore, a notification may be issued if no movement is detected for a certain period of time or more after a fall is detected.

[0062] Embodiment 3 A notification device 10 and a notification system 1 according to the third embodiment will be described. The basic configuration of the notification device 10 is the same as that of the first embodiment, and therefore illustrations and detailed descriptions will be omitted. For example, the notification device 10 includes an image acquisition unit 11, a body part discrimination unit 12, a motion detection unit 13, a notification determination unit 14, a communication unit 15, and a recording unit 16, as shown in FIG.

[0063] The part discrimination unit 12 discriminates parts of the face. For example, the part discrimination unit 12 discriminates the eyes, nose, mouth, ears, eyebrows, and the space between the eyebrows by performing image processing. The part discrimination unit 12 calculates the coordinates of the extraction point of each part for each frame. The movement detection unit 13 detects facial movements based on the movements of each part of the face. The movement detection unit 13 can also extract facial expressions based on the movements of the parts. For example, the movement detection unit 13 can detect facial expressions based on the movements of each part. If the facial expression becomes a repetitive movement, a notification is made.

[0064] Specifically, the motion detection unit 13 analyzes the shape of the eyes, the eyebrows, and the state of the space between the eyebrows. The motion detection unit 13 extracts an expression of agony. Examples of expressions of agony include frowning or tightly closing the eyes. For example, if a target expression is extracted multiple times within a few minutes, the notification determination unit 14 determines that the motion is a repeated motion and issues a notification. Furthermore, the body part determination unit 12 and the motion detection unit 13 may be AI models constructed by machine learning.

[0065] 6 is a flowchart showing the notification method. When image acquisition unit 11 acquires an image (S31), body part determination unit 12 determines body parts of subject 102 (S32). For example, body part determination unit 12 acquires position coordinates of subject 102's eyes, nose, mouth, eyebrows, between the eyebrows, ears, etc. The movement detection unit 13 performs facial expression analysis based on the movement of each facial part (S33).

[0066] For example, the motion detection unit 13 acquires a motion vector of each body part based on a plurality of frame images. The motion detection unit 13 analyzes the facial expression of the subject by evaluating the amount of change in the body part per unit time. The motion detection unit 13 determines whether the detected facial expression is a notification motion (S34).

[0067] As described above, the notifying action (notifying facial expression) to be detected is a facial expression of anguish, such as frowning. If the facial expression detected by the action detection unit 13 is a facial expression of anguish, it is determined to be a notifying action. If the facial expression obtained by the facial expression analysis is a notifying action (YES in S34), the recording unit 16 records the status and number of times of the notifying action (S35). In other words, the action detection unit 13 counts the number of times the notifying action is detected and writes the count to the recording unit 16.

[0068] If the facial expression obtained by the facial expression analysis is not a notification movement (NO in S34), the process returns to S32. That is, the notification device 10 performs the same process on the next acquired image. The notification determination unit 14 determines whether the number of times the notification movement has been detected is equal to or greater than a predetermined number. If the number of times the notification movement has been detected is less than the predetermined number (NO in S36), the process returns to step S32. That is, the notification device 10 performs the same process on the images that are acquired sequentially.

[0069] If the number of detections is equal to or greater than a predetermined number (YES in S36), the communication unit 15 transmits a notification to the user terminal 203 (S37). In this way, the movement detection unit 13 can detect movements other than continuous movements as notification movements. Then, the notification device 10 issues a notification according to the number of detections of notification movements. If the subject 102 makes a pained expression multiple times, a notification is received. The movement detection unit 13 extracts a predetermined expression based on the movement of parts of the subject's face. The movement detection unit 13 detects repeated movements by counting equal to or greater than a predetermined number of times. The movement detection unit 13 may extract the predetermined expression using an AI model. In this case, supervised machine learning can be performed using a set of image data of the predetermined expression and image data of other expressions as training data.

[0070] Two or more of the first to third embodiments can be combined as appropriate. For example, when it is determined in the processing of any one of the first to third embodiments that a notification should be made, the communication unit 15 may be configured to make the notification. For example, the second embodiment and the third embodiment can be combined. In the detection of a stumble or fall while walking or a movement such as reflexively withdrawing a hand in the second embodiment, even if no repeated movement is detected or no movement is detected for a certain period of time after a fall is detected, if it is determined in the processing of the third embodiment that a notification should be made, the communication unit 15 may be configured to make the notification.

[0071] As described in the first to third embodiments above, the notification device 10 may include an image acquisition unit, a body part determination unit, a motion detection unit, a notification determination unit, and a communication unit. The image acquisition unit acquires a captured image of a target person. The body part determination unit determines a body part of the target person based on the captured image. The body part determination unit may determine a body part such as a hand or a waist, or a facial part such as an eye or an eyebrow. The motion detection unit detects a repetitive motion based on the movement of the determined body part of the target person. The notification determination unit determines whether or not to notify the user associated with the target person based on the detection result of the repetitive motion. The communication unit performs notification based on the determination result of the notification determination unit. This allows the notification device 10 to perform notification at an appropriate timing.

[0072] Some or all of the above processes may be executed by a computer program. The above-described program can be stored and provided to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be provided to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can provide the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0073] The present invention has been specifically described above based on the embodiments, but it goes without saying that the present invention is not limited to the above embodiments and can be modified in various ways without departing from the spirit of the invention. Two or more of the above embodiments can also be combined as appropriate. [Explanation of symbols]

[0074] 1. Notification System 10 Notification device 11 Image acquisition unit 12 Part identification section 13 Motion detection unit 14 Notification determination section 15 Communications Department 16 Recording section 101 Monitoring Site 102 Target 103 Camera 201 User Site 202 users 203 User terminal

Claims

1. an image acquisition unit that acquires a captured image of a subject; a body part discrimination unit that discriminates a body part of the subject based on the captured image; a movement detection unit that detects a repetitive movement based on the movement of the identified part of the subject; a notification determination unit that determines whether to notify a user associated with the target person based on a detection result of the repetitive motion; a communication unit that issues a notification based on a determination result of the notification determination unit.

2. The notification device according to claim 1 , wherein the notification determination unit determines whether or not to make the notification based on a duration of the repeated action.

3. The notification device according to claim 1 , wherein the notification determination unit determines whether or not to issue the notification based on a frequency of the repeated action.

4. The notification device according to claim 1 , wherein the movement detection unit detects a repetitive movement of a predetermined part relative to the target part of the subject based on a change in the position of the predetermined part relative to the target part.

5. The notification device according to claim 1 , wherein the movement detection unit detects deceleration of a predetermined part while the subject is moving in a predetermined direction, and counts the number of times the detection occurs.

6. The notification device according to claim 1 , wherein the movement detection unit detects a predetermined facial expression based on the movement of a part of the face of the subject and counts the number of times the predetermined facial expression is detected.

7. acquiring a captured image of a subject; determining a body part of the subject based on the captured image; detecting a repetitive motion based on the movement of the identified part of the subject; determining whether to notify a user associated with the target person based on a result of the detection of the repetitive motion; and a step of issuing a notification based on the result of the determination.

8. A program for causing a computer to execute a notification method, The notification method includes: acquiring a captured image of a subject; determining a body part of the subject based on the captured image; detecting a repetitive motion based on the movement of the identified part of the subject; determining whether to notify a user associated with the target person based on a result of the detection of the repetitive motion; and issuing a notification based on the determination result. program.

Images