Behavior estimation device, behavior estimation method, and behavior estimation program

Abstract

The present invention provides a behavior estimation system, a behavior estimation method, and a behavior estimation program capable of detecting falling, regardless of the way of falling or the orientation of the body. A behavior estimation device 300 includes an image acquisition unit 310, a skeleton detection unit 320, a distance information calculation unit 330, and behavior estimation units 350 and 360. The image acquisition unit 310 acquires an image obtained by shooting an object. The skeleton detection unit 320 detects skeleton information, including information regarding a plurality of skeleton joint points of the object, on the basis of the image. The distance information calculation unit 330 calculates, on the basis of the skeleton information, first distance information relating to a distance between two different skeleton joint points in the object and second distance information relating to a distance between two skeleton joint points of which at least one skeleton joint point is different from the above two skeleton joint points. The behavior estimation units 350 and 360 estimate behavior of the object on the basis of the first distance information and the second distance information.

Description

Behavior estimation device, behavior estimation method, and behavior estimation program

[0001] The present invention relates to a behavior estimation device, a behavior estimation method, and a behavior estimation program.

[0002] In recent years, surveillance cameras have been installed everywhere to ensure the safety of pedestrians on the streets and users of facilities. For example, in nursing care facilities such as hospitals and nursing homes for the elderly, patients who are hospitalized or residing in the facility may fall while walking within the facility. To enable nurses, caregivers, etc. to quickly respond when a patient falls, technology is being developed that detects falls based on video footage of the patient (see, for example, Patent Document 1 below).

[0003] Patent Literature 1 proposes a system for monitoring a person who has fallen in order to provide early follow-up care to the person. This system estimates the person's posture based on skeletal information detected from video footage from a camera installed directly above the person, and performs two-stage fall detection using the position of each body part detected based on the skeletal information of the person, who can be assumed to be standing, and the person's aspect ratio. More specifically, the system determines whether the upper body is positioned above the lower body based on the skeletal information, and makes a primary determination of a fall if the upper body is not positioned above the lower body. Furthermore, even if the upper body is positioned above the lower body, a secondary determination of a fall is made based on the aspect ratio, thereby improving detection accuracy through two-stage determination.

[0004] Japanese Patent Application Laid-Open No. 2021-33343

[0005] However, fall detection using the positioning of each body part and the aspect ratio of a person can lead to false positives and over-detections, as it is difficult to distinguish between a person who has actually fallen and a non-fall depending on the manner of fall and the orientation of the body.

[0006] The present invention has been made to solve the above-mentioned problems. That is, a main object of the present invention is to provide a behavior estimation device, a behavior estimation method, and a behavior estimation program that can detect a fall regardless of the manner of fall or the orientation of the body.

[0007] The above-mentioned problems of the present invention are solved by the following means.

[0008] (1) A behavior estimation device having: an image acquisition unit that acquires an image of an object; a skeleton detection unit that detects skeleton information including information about a plurality of skeleton joint points of the object based on the image; a distance information calculation unit that calculates, based on the skeleton information, first distance information regarding a distance between two different skeleton joint points of the object and second distance information regarding a distance between two skeletal joint points that differ from the two skeletal joint points by at least one skeleton joint point; and a behavior estimation unit that estimates behavior of the object based on the first distance information and the second distance information.

[0009] (2) The behavior estimation device described in (1) above, wherein the image acquisition unit acquires an image of a person as the object, the distance information calculation unit calculates first distance information for an upper body of the person included in the image and calculates the second distance information for a lower body of the person, and the behavior estimation unit determines that the behavior of the person is a specific behavior when, for the upper body and the lower body of the person, the inter-joint point distances between two skeletal joint points of the person are reduced relative to respective predetermined reference distances.

[0010] (3) The behavior estimation device described in (2) above, wherein the behavior estimation unit includes a primary behavior determination unit that provisionally determines that the behavior of the person is the specific behavior when the inter-joint point distance has shrunk to a predetermined ratio or less of the predetermined reference distance, and a secondary behavior determination unit that formally determines that the behavior of the person is the specific behavior when the provisional determination that the behavior of the person is the specific behavior continues for a predetermined first time period.

[0011] (4) The behavior estimation device described in (3) above, wherein the primary behavior determination unit provisionally determines that the behavior of the person is the specific behavior if the time it takes for the inter-joint point distance to reduce to the specified ratio with respect to the specified reference distance is less than or equal to a specified second time.

[0012] (5) The behavior estimation device described in (3) above, wherein the primary behavior determination unit determines that the behavior of the person is not the specific behavior when the inter-joint point distance has not reduced to the specified ratio with respect to the specified reference distance, and the secondary behavior determination unit, after formally determining that the behavior of the person is the specific behavior, cancels the formal determination that the behavior of the person is the specific behavior when the state in which the primary behavior determination unit has determined that the behavior of the person is not the specific behavior continues for a specified third time period.

[0013] (6) The behavior estimation device according to (2), wherein the behavior estimation unit determines that the behavior of the person is the specific behavior based on a ratio, an amount of change, or a speed of change of the inter-joint point distance relative to the predetermined reference distance.

[0014] (7) The behavior inference device according to any one of (1) to (6) above, further comprising a tracking processing unit that continuously holds the same ID information for the same object based on skeletal information of the object.

[0015] (8) The behavior estimation device described in any one of (1) to (6) above, wherein the skeleton detection unit estimates a degree of certainty for the skeleton information of the object, and the distance information calculation unit calculates the first distance information and the second distance information when the skeleton information has the degree of certainty equal to or greater than a predetermined value.

[0016] (9) A behavior estimation device according to any one of (1) to (6), further comprising: a display device that displays a video captured by an imaging device that captures the object and outputs a video consisting of a plurality of images; and a display control unit that controls the display device, wherein the skeleton detection unit analyzes the video acquired by the image acquisition unit to calculate the skeleton information of the object, and the display control unit causes the display device to display the skeleton information of the object as an image together with the video.

[0017] (10) The behavior estimation device described in (9) above, further comprising a marking processing unit that displays, on the video, figures and / or characters for distinguishing the object whose behavior has been estimated by the behavior estimation unit from other objects other than the object on the video.

[0018] (11) The behavior estimation device described in (9) above, wherein the image acquisition unit acquires an image of a person as the object, and the display control unit performs mosaic or blurring processing at the position of the person in the video based on position information included in skeletal information of the person so that the person is not identified as an individual.

[0019] (12) The behavior estimation device described in (11) above, wherein the display control unit superimposes and displays a figure representing skeletal information of the person on the image of the person on which the mosaic processing or the blurring processing has been performed.

[0020] (13) The behavior estimation device according to any one of (1) to (6), further comprising a behavior notification unit, which, when the behavior estimation unit determines that the behavior of the object is a specific behavior, notifies an alarm that outputs an alarm by at least one of sound and light of the specific behavior of the object.

[0021] (14) The behavior estimation device according to any one of (1) to (6), further comprising a behavior notification unit, wherein when the behavior estimation unit determines that the behavior of the object is a specific behavior, the behavior notification unit notifies the specific behavior of the object by transmitting information via a network.

[0022] (15) The behavior estimation device according to any one of (1) to (6), further comprising an effective area input unit for inputting an effective area in which an estimation result of the person's behavior is valid, wherein the image acquisition unit acquires an image of the person as the object, and the behavior estimation unit validates the estimation result of the person's behavior when the position coordinates of at least one skeletal joint point of the person's legs, head, and torso are within the effective area.

[0023] (16) The behavior estimation device described in (15) above, further comprising: a display device that displays a video captured by an imaging device that captures the person and outputs a video consisting of a plurality of images; and a display control unit that displays on the display device a figure indicating the effective area together with the video captured by the imaging device.

[0024] (17) The behavior estimation device according to any one of (1) to (6), further comprising an invalid area input unit that inputs an invalid area in which the estimation result of the person's behavior is invalid, wherein the image acquisition unit acquires an image of the person as the object, and the behavior estimation unit invalidates the estimation result of the person's behavior when the position coordinates of at least one skeletal joint point of the person's legs, head, and torso are within the invalid area.

[0025] (18) The behavior estimation device described in (17) above, further comprising: a display device that displays a video captured by an imaging device that captures the person and outputs a video consisting of a plurality of images; and a display control unit that displays on the display device a figure indicating the invalid area together with the video captured by the imaging device.

[0026] (19) A behavior estimation method including the steps of: (a) acquiring an image of an object; (b) detecting skeletal information based on the image, the skeletal information including information on a plurality of skeletal joint points of the object; (c) calculating, based on the skeletal information, first distance information relating to a distance between two different skeletal joint points of the object and second distance information relating to a distance between two skeletal joint points that are different from the two skeletal joint points by at least one skeletal joint point; and (d) estimating behavior of the object based on the first distance information and the second distance information.

[0027] (20) A behavior estimation program for causing a computer to execute a process including the steps of: (a) acquiring an image of an object; (b) detecting skeletal information based on the image, the skeletal information including information on a plurality of skeletal joint points of the object; (c) calculating, based on the skeletal information, first distance information relating to the distance between two different skeletal joint points of the object and second distance information relating to the distance between two skeletal joint points that differ from the two skeletal joint points by at least one skeletal joint point; and (d) estimating the behavior of the object based on the first distance information and the second distance information.

[0028] According to the present invention, the behavior of an object is estimated based on the distance between two different skeletal joint points of the object and the distance between two skeletal joint points that are different from the two skeletal joint points by at least one skeletal joint point, thereby making it possible to detect a fall regardless of the way the object falls or the orientation of the body.

[0029] 1 is a block diagram illustrating a schematic configuration of a behavior inference system according to an embodiment of the present invention. FIG. 1 is a block diagram illustrating a schematic configuration of an imaging device shown in FIG. 1. FIG. 2 is a schematic diagram illustrating a method for acquiring an image (video) of a person. FIG. 3 is a schematic block diagram illustrating a hardware configuration of the behavior inference device shown in FIG. 1. FIG. 4 is a schematic diagram illustrating the distance between joint points of the upper body and the lower body when walking normally and when falling. FIG. 5 is a schematic diagram illustrating the distance between joint points of the upper body and the lower body when walking normally and when sitting down. FIG. 6 is a schematic diagram illustrating the distance between joint points of the upper body and the lower body when walking normally and when leaning forward. FIG. 7 is a flowchart illustrating a behavior determination flow in a primary determination process. FIG. 8 is a schematic diagram illustrating a case where a person falls from normal walking. FIG. 9 is a graph illustrating the change over time in the distance between joint points of the torso when a person falls from normal walking. FIG. 10 is a graph illustrating the change over time in the distance between joint points of the legs when a person falls from normal walking. FIG. 11 is a schematic diagram illustrating a case where a person crouches down from normal walking. 1 is a graph illustrating a change over time in the distance between joint points of the torso when a person crouches down from normal walking. FIG. 2 is a graph illustrating a change over time in the distance between joint points of the legs when a person crouches down from normal walking. FIG. 3 is a flowchart illustrating a processing procedure of a behavior estimation method by a behavior estimation system according to an embodiment. FIG. 4 is a schematic diagram illustrating detection of skeletal information. FIG. 5 is a schematic diagram illustrating differentiation between a person in a fallen state and a person in a non-falling state in a video. FIG. 6 is a schematic diagram illustrating protection of a person's privacy. FIG. 7 is a schematic diagram illustrating a method of setting an effective area. FIG. 8 is a schematic diagram illustrating behavior estimation in an effective area. FIG. 9 is a schematic diagram illustrating a method of setting an effective area. FIG. 10 is a schematic diagram illustrating behavior estimation in an invalid area.

[0030] Hereinafter, a behavior estimation device, a behavior estimation method, and a behavior estimation program according to embodiments of the present invention will be described with reference to the drawings. In the drawings, identical elements are denoted by the same reference numerals, and duplicated explanations will be omitted. Also, the dimensional proportions in the drawings are exaggerated for the sake of explanation, and may differ from the actual proportions.

[0031] (Embodiment) A behavior estimation system according to this embodiment estimates the behavior of an object based on skeletal information detected from video images of the object. The behavior estimation system can particularly effectively detect falls of a person as an object. The skeletal information may include information about a plurality of skeletal joint points (hereinafter simply referred to as "joint points") of the object and information about connections between the joint points. The information about the joint points may be two-dimensional or three-dimensional position coordinates of the joint points. Furthermore, the information about connections between the joint points may be represented, for example, by line segments connecting the joint points. The object is a living organism or machine whose skeleton can be detected, and may be, for example, a person, an animal, an industrial robot, a humanoid robot, a structure or machine with a skeleton, etc. The following description will mainly focus on the case where the object is a person, but the same applies to cases where the object is an animal, a machine, etc.

[0032] In this specification, "falling" means that a person falls on the floor or the like. "Falling off" means that a person falls from a place such as a bed to a lower place. In the following, this specification will mainly describe the case of detecting a person falling off, but the same can be applied to the case of detecting a person falling off.

[0033] 1 is a block diagram illustrating a schematic configuration of a behavior estimation system 100 according to an embodiment. The behavior estimation system 100 includes, for example, an image capture device 200 and a behavior estimation device 300. The image capture device 200 is connected to the behavior estimation device 300 via a communication network so that they can communicate with each other.

[0034] The behavior inference device 300 may be, for example, a personal computer, a server, a smartphone, a tablet terminal, or the like.

[0035] <Image capture device 200> Fig. 2 is a block diagram illustrating a schematic configuration of the image capture device 200 shown in Fig. 1. The image capture device 200 has a control unit 210, a communication unit 220, and a camera 230, which are interconnected by a bus 201.

[0036] 3 is a schematic diagram illustrating a method for acquiring an image (video) of a person. As shown in the figure, the imaging device 200 is installed in a location where at least a portion of the movement path PA of the person 400 on the floor surface FL falls within the field of view of the camera 230. In the figure, the depth direction of the video image captured by the camera 230 of the imaging device 200 (the direction of the optical axis of the lens of the camera 230) is defined as the Y direction, and the horizontal direction perpendicular to the depth direction on the same plane as the Y direction is defined as the X direction.

[0037] For example, when estimating the behavior of a worker engaged in work in a company factory, imaging device 200 is installed in a location where at least a part of the worker's movement route PA falls within the field of view of camera 230. Furthermore, when estimating the behavior of a care recipient in a nursing care facility, imaging device 200 is installed in a location where the inside of the care recipient's room, a hallway within the facility, etc. fall within the field of view of camera 230.

[0038] The control unit 210 is configured with, for example, a CPU (Central Processing Unit) and memories such as a RAM (Random Access Memory) and a ROM (Read Only Memory). The CPU controls each unit of the imaging device 200 and performs calculation processing in accordance with a behavior estimation program stored in the memory.

[0039] The control unit 210 transmits a plurality of captured images (frames) obtained by the camera 230 capturing an image of a predetermined area to the behavior inference device 300 via the communication unit 220. The predetermined area is a three-dimensional area that includes the movement path of an object.

[0040] The communication unit 220 includes an interface circuit for communicating with the behavior inference device 300 and the like via a communication network, for example.

[0041] Camera 230 may be, for example, a standard camera (bullet camera) used for security cameras and surveillance cameras. Camera 230 may also be a wide-angle camera (for example, a ceiling-mounted 360-degree camera) with a wider angle of view than a standard camera. For simplicity of explanation, the following description will be given assuming that camera 230 is a standard camera.

[0042] It is desirable that imaging device 200 adjusts the position and lens distortion of camera 230 in advance so that a person is captured in a generally vertical orientation when standing. Specifically, if camera 230 is a bullet-type camera, camera 230 can be positioned to capture the person from the side or diagonally from above. Furthermore, if a ceiling-mounted 360-degree camera is used as camera 230, the camera can be configured to adjust the lens distortion and viewpoint so that the angle of view is the same as when capturing images using the bullet-type camera.

[0043] The camera 230 may be, for example, a visible light camera, which may capture an image of a predetermined area by receiving visible light reflected by an object within the predetermined area using a complementary metal oxide semiconductor (CMOS) sensor.

[0044] Furthermore, camera 230 may be configured using a near-infrared camera instead of a visible light camera. The infrared camera may capture an image of a predetermined area by irradiating the predetermined area with near-infrared light using an LED (Light Emitting Device) and receiving the near-infrared light reflected by objects within the predetermined area using a CMOS sensor. In this case, the image captured by camera 230 may be a monochrome image in which each pixel has a reflectance of near-infrared light.

[0045] The imaging device 200 can capture a predetermined area as a video consisting of a plurality of temporally consecutive captured images (frames) at a frame rate of, for example, 15 fps to 30 fps. The camera 230 may also be configured to use both a near-infrared camera and a visible light camera.

[0046] Note that some or all of the functions of the imaging device 200 may be executed by the behavior inference device 300 or another information processing device (not shown).

[0047] <Behavior Inference Device 300> Fig. 4 is a schematic block diagram illustrating an example of the hardware configuration of the behavior inference device 300 shown in Fig. 1. The behavior inference device 300 includes, for example, a CPU 301, a RAM 302, a ROM 303, an auxiliary storage unit 304, a communication I / F 305, an operation display unit 306, and an alarm 307. The components of the behavior inference device 300 are connected to each other via an internal bus.

[0048] The auxiliary storage unit 304 may be configured with, for example, a solid state drive (SSD), a hard disk drive (HDD), etc. The auxiliary storage unit 304 stores, for example, a program such as a behavior estimation program, video images of a predetermined area including a person, the date and time when the video images were captured, and various parameters used in behavior determination (described later). Examples of the various parameters include the distance ratio, determination time, duration, and release time when falling, sitting, and leaning forward.

[0049] The communication I / F 305 is an interface circuit for communicating with other devices, such as the imaging device 200, via a network. The interface circuit complies with standards such as LAN (Local Area Network), USB (Universal Serial Bus), and MIPI (Mobile Industry Processor Interface).

[0050] The operation display unit 306 has an input unit and an output unit. The input unit includes, for example, a touch panel, a keyboard, a mouse, etc. The input unit is used by the user to perform touch operations using the touch panel, input characters using the keyboard, mouse, etc., and perform various settings. The output unit includes a display (display device) and displays moving images, etc., including an object captured by the imaging device 200.

[0051] The alarm device 307 outputs an alarm in accordance with a person's behavior notification from the notification unit 370 (see FIG. 1 ). The alarm device 307 has, for example, a speaker and can output a warning sound or voice from the speaker as an alarm. The alarm device 307 also has a lamp or LED (Light Emitting Diode) and can light or flash the lamp or LED as an alarm.

[0052] <Functions of Behavior Inference Device 300> Next, functions of the behavior inference device 300 (behavior inference application) will be described with reference to Fig. 1 again. The behavior inference device 300 functions as, for example, an image acquisition unit 310, a skeleton detection unit 320, a distance information calculation unit 330, a tracking processing unit 340, a primary behavior determination unit 350, a secondary behavior determination unit 360, a notification unit 370, and a UI control unit 380. These functions are realized by the CPU 301 executing a behavior inference program. The primary behavior determination unit 350 and the secondary behavior determination unit 360 function as behavior inference units.

[0053] [Image Acquisition Process] The image acquisition unit 310 acquires a moving image captured of a predetermined area including a person. The moving image may be, for example, a plurality of frames of images captured by the imaging device 200 of the predetermined area including a person. The image acquisition unit 310 acquires the moving image, for example, by receiving the moving image from the imaging device 200 via the communication I / F 305. The acquired moving image is decoded to obtain a two-dimensional image (snapshot), which is a still image.

[0054] The image acquisition unit 310 also performs predetermined preprocessing on the acquired two-dimensional image and inputs the preprocessed image data to the skeleton detection unit 320. The preprocessing may be, for example, convolution (smoothing, sharpening) or tone correction using a one-dimensional lookup table. By performing preprocessing on the two-dimensional image, people are highlighted even if there are parts of the video where people are difficult to see, thereby improving the accuracy of detection of skeleton information by the skeleton detection unit 320.

[0055] Furthermore, if the moving images captured by the imaging device 200 are stored in advance in the auxiliary storage unit 304 or the like, the image acquisition unit 310 can also acquire the moving images by reading them from the auxiliary storage unit 304 or the like. Note that the moving images captured by the imaging device 200 may be stored in an external storage device or the like, and the image acquisition unit 310 may be configured to read the moving images from the storage device. Alternatively, the moving images captured by the imaging device 200 may be recorded on a recording medium such as a USB memory, and the image acquisition unit 310 may acquire the moving images offline by reading them from the recording medium.

[0056] [Person Skeleton Detection Processing] The skeleton detection unit 320 analyzes the images acquired by the image acquisition unit 310 to detect (estimate) skeletal information of a person included in a moving image. The skeletal information may include position information of a plurality of joint points, such as the eyes, nose, neck, shoulders, elbows, wrists, waist, knees, and ankles, which are skeletal feature points (key joint points) of the person, as well as line segments connecting the joint points (see, for example, FIG. 10 ). The position information of the joint points may be coordinates (X coordinate, Y coordinate) of the joint points on the captured image. Note that in this specification, for convenience of illustration, some joint points may be omitted in the drawings. The person may be one person or multiple people.

[0057] The skeleton detection unit 320 estimates skeleton information using, for example, a trained model for detecting a person's joint points from a rectangle containing a person (hereinafter also referred to as a "person rectangle"). The skeleton information includes the person's multiple joint points as well as a confidence level (a score indicating the likelihood of estimation) for each joint point. In estimating behavior in this embodiment, it is desirable to use joint points equal to or greater than a predetermined threshold. This is because, due to the nature of machine learning, a low confidence level increases the likelihood of an inaccurate detection result. The skeleton detection unit 320 outputs the skeleton information to the distance information calculation unit 330 and the tracking processing unit 340.

[0058] A human rectangle is a region that contains the joint points and nodes of a person in an image. Known examples of such trained models include OpenPose (https: / / arxiv.org / abs / 1812.08008), DeepPose (https: / / arxiv.org / abs / 1312.4659), and ResNet (https: / / arxiv.org / abs / 1512.03385).

[0059] [Distance Information Calculation Process] Based on the skeleton information, the distance information calculation unit 330 calculates first distance information regarding the distance between two different joint points of a person (hereinafter also referred to as "inter-joint point distance"). Furthermore, the distance information calculation unit 330 calculates second distance information regarding the distance between two joint points, each of which has at least one joint point different from the first two joint points. In this specification, the skeleton corresponds to, for example, a line connecting a specific joint point with another joint point, or a line connecting a joint point with a center point of two joint points. The specific joint point may be included, for example, in the person's upper body (torso) and lower body (legs). The first distance information and the second distance information may include, for example, the Euclidean distance between two joint points. In this embodiment, the first distance information may include, for example, the distance between the joint points of the neck (or the center of both shoulders) and the center of the waist. Furthermore, the second distance information may include, for example, the distance between the joint points of the foot (or ankle). That is, the first distance information and the second distance information include a distance between the joint points calculated using the joint point at the center of the waist as one of the two joint points.

[0060] In this embodiment, in order to ensure the reliability of the calculated first distance information and second distance information, the distance information calculation unit 330 calculates the first distance information and the second distance information when the skeletal information has a certainty equal to or greater than a predetermined threshold (predetermined value). For example, the distance information calculation unit 330 calculates the first distance information for the upper body of a person included in the image, and calculates the second distance information for the lower body of the person.

[0061] [Person Tracking Processing] The tracking processing unit 340 identifies people over time by continuously storing the same ID (Identification) information for the same person based on the person's skeletal information. The tracking processing unit 340 assigns unique ID information to each of multiple people included in a video and stores the same ID information for the same person. More specifically, the tracking processing unit 340 determines the degree of proximity of the position information of joint points from the person's skeletal information for the most recent captured image and the image captured immediately before, and associates the degree of proximity with the ID information of the person being tracked.

[0062] Furthermore, although the example has been given in which coordinate information detected by skeleton detection is used, position information of a human rectangle may also be used. Furthermore, if a separate means for detecting a person by object detection or the like can be used, the ID information of the person to be tracked may be associated with the degree of similarity between image data of the area where the person was detected for the latest captured image and the image captured immediately before. These tracking processing methods are well-known technologies, and therefore detailed explanations thereof will be omitted.

[0063] [Primary behavior determination process] Fig. 5A is a schematic diagram illustrating the distance between joint points of the upper and lower body when walking normally and when falling. Fig. 5B is a schematic diagram illustrating the distance between joint points of the upper and lower body when walking normally and when sitting down. Fig. 5C is a schematic diagram illustrating the distance between joint points of the upper and lower body when walking normally and when leaning forward.

[0064] The primary behavior determination unit 350 estimates the behavior of the person based on the first distance information and the second distance information. For example, the primary behavior determination unit 350 determines that the behavior of the person is a specific behavior when the inter-joint distances Lu, Lb for the upper body and / or lower body of the person are reduced relative to the respective predetermined reference distances. The provisional determination result by the primary behavior determination unit 350 is transmitted to the secondary behavior determination unit 360. The specific behavior may be, for example, falling, sitting down, or leaning forward. The following description will mainly use as an example a case where the specific behavior is falling.

[0065] 5A to 5C , for example, the inter-joint distances Lsu and Lsb of the upper body (torso) and lower body (legs) when a person is not moving (e.g., walking normally) are set as reference distances. Similarly, when the reference distances Lsu and Lsb are set, the inter-joint distances Lu1, Lb1, Lu2, Lb2, Lu3, and Lb3 of the upper body and lower body when a person falls, sits down, or leans forward are set experimentally or empirically. For example, when the inter-joint distance Lu is equal to or less than Lu1 and the inter-joint distance Lb is equal to or less than Lb1 relative to the reference distances Lsu and Lsb, the person's behavior is considered to be a fall.

[0066] The primary behavior determination unit 350 provisionally determines that the behavior of the person is a specific behavior when the inter-joint distances Lu, Lb of the person's torso and / or legs have shrunk to or below a predetermined ratio (threshold) of the reference distances Lsu, Lsb. For example, the primary behavior determination unit 350 provisionally determines that the behavior of the person is a fall when the ratios (distance ratios) of the inter-joint distances Lu, Lb to the reference distances Lsu, Lsb of the person's torso and legs have shrunk to or below Lu1 / Lsu and Lb1 / Lsb, respectively. Furthermore, the primary behavior determination unit 350 provisionally determines that the behavior of the person is sitting down when the distance ratios of the person's torso and legs have shrunk to or below Lu2 / Lsu and Lb2 / Lsb, respectively. Furthermore, for example, if the distance ratios of the person's torso and legs are reduced to Lu3 / Lsu or less and Lb3 / Lsb or less, respectively, the primary behavior determination unit 350 provisionally determines that the person's behavior is leaning forward.

[0067] For example, the thresholds for the distance ratio between the torso and legs when falling may be 0.6 and 0.65, the thresholds for the distance ratio between the torso and legs when sitting down may be 1.0 and 0.65, and the thresholds for the distance ratio between the torso and legs when leaning forward may be 0.6 and 1.0. The thresholds for the distance ratio between the torso and legs when falling, sitting down, and leaning forward are stored in the auxiliary storage unit 304.

[0068] 6 is a flowchart illustrating an example of a flow of determining behavior in the primary determination process. The priorities of the determinations of falling, sitting down, and leaning forward are predetermined, for example, in this order. The priorities can be set, for example, taking into account the urgency of assistance required for the person who performed the behavior.

[0069] The primary behavior determination unit 350 determines whether the torso distance ratio is 0.6 or less and the leg distance ratio is 0.65 or less (step S101). If the torso distance ratio is 0.6 or less and the leg distance ratio is 0.65 or less (step S101: YES), the primary behavior determination unit 350 tentatively determines that the person's behavior is a fall because it is highly likely that the person has fallen (step S102).

[0070] Next, if the torso distance ratio is not equal to or less than 0.6 and the leg distance ratio is not equal to or less than 0.65 (step S101: NO), the primary behavior determination unit 350 determines whether the torso distance ratio is equal to or less than 1.0 and the leg distance ratio is equal to or less than 0.65 (step S103).If the torso distance ratio is equal to or less than 1.0 and the leg distance ratio is equal to or less than 0.65 (step S103: YES), the primary behavior determination unit 350 tentatively determines that the person's behavior is sitting down because it is highly likely that the person is sitting down (step S104).

[0071] Next, if the torso distance ratio is 1.0 or less and the leg distance ratio is not 0.65 or less (step S103: NO), the primary behavior determination unit 350 determines whether the torso distance ratio is 0.6 or less and the leg distance ratio is 1.0 or less (step S105). If the torso distance ratio is 0.6 or less and the leg distance ratio is 1.0 or less (step S105: YES), the primary behavior determination unit 350 determines that the person's behavior is leaning forward because it is highly likely that the person is leaning forward (step S106). On the other hand, if the torso distance ratio is 0.6 or less and the leg distance ratio is not 1.0 or less (step S105: NO), the primary behavior determination unit 350 determines that the person's behavior does not correspond to falling, sitting down, or leaning forward, and therefore determines that the person's behavior is other.

[0072] The distance ratios for each of the above-mentioned actions are merely examples, and can be changed as appropriate depending on the age, sex, attributes, etc. of the person.

[0073] Although the case where the primary behavior determination unit 350 determines that the behavior of the person is a specific behavior based on the distance ratio has been exemplified, the present invention is not limited to this case. The primary behavior determination unit 350 may be configured to determine that the behavior of the person is a specific behavior based on the amount or speed of change from the reference distances Lsu, Lsb to the inter-joint distances Lu, Lb for the specific behavior.

[0074] More specifically, the amount of change is the difference (distance) between the inter-joint distances Lu, Lb in the specific behavior and the reference distances Lsu, Lsb. If the amount of change is equal to or greater than a predetermined distance, the primary behavior determination unit 350 may provisionally determine that the behavior of the person is the specific behavior, and if the amount of change is less than the predetermined distance, may determine that the behavior of the person is not the specific behavior.

[0075] The rate of change is the difference divided by the time required for the change. If the rate of change is equal to or greater than a predetermined rate, the primary behavior determination unit 350 may provisionally determine that the person's behavior is a specific behavior, and if the amount of change is less than the predetermined rate, the primary behavior determination unit 350 may determine that the person's behavior is not a specific behavior.

[0076] - Exclusion of other actions having a distance ratio similar to that of a specific action. Fig. 7A is a schematic diagram illustrating an example of a case where a person falls from normal walking. Fig. 7B is a graph illustrating an example of a change over time in the distance Lu between joint points of the torso when the person falls from normal walking. Fig. 7C is a graph illustrating an example of a change over time in the distance Lb between joint points of the legs when the person falls from normal walking. Fig. 8A is a schematic diagram illustrating an example of a case where a person crouches from normal walking. Fig. 8B is a graph illustrating an example of a change over time in the distance Lu between joint points of the torso when the person crouches from normal walking. Fig. 8C is a graph illustrating an example of a change over time in the distance Lb between joint points of the legs when the person crouches from normal walking.

[0077] The primary behavior determination unit 350 can be configured to determine that the behavior of the person is a specific behavior (e.g., a fall) only when the inter-joint distances Lu and Lb satisfy a predetermined determination condition. For example, as shown in FIG. 7A , assume that a person who was walking normally at time t1 falls at time t2.

[0078] The primary behavior determination unit 350 determines, as a determination condition, whether the time (t2-t1=Δt) until the inter-joint distance is reduced to a predetermined distance ratio with respect to the reference distance is within a predetermined determination time Δt ｔｈ 7B and 7C, the primary behavior determining unit 350 may determine that the person's behavior is a fall when Δt is equal to or shorter than the determination time Δt ｔｈ If the determination time Δt is less than or equal to the predetermined time, it can be provisionally determined that the person's behavior is a fall. ｔｈ Although there are no particular limitations on the determination time, it is preferable that the determination time be, for example, 300 ms or less. The determination time is stored in the auxiliary storage unit 304 in advance.

[0079] On the other hand, as shown in FIG. 8A , assume that a person who was walking normally at time t1 crouches down at time t3 (>t2). When a person crouches down, the inter-joint distances Lu and Lb decrease to a predetermined distance ratio or less, similar to when a person falls. However, a person crouches down more slowly than when a person falls. For example, a person may crouch down when picking up something that has fallen on the floor, cleaning, carrying luggage, etc.

[0080] As shown in FIGS. 8B and 8C, when the person is crouching, the time (t2-t1=Δt') required for the inter-joint distances Lu and Lb to decrease to a predetermined distance ratio with respect to the reference distances Lsu and Lsb is the determination time Δt ｔｈ Therefore, the primary behavior determining unit 350 does not determine the person's behavior as a specific behavior (falling).

[0081] In this way, in the primary determination process, by imposing determination conditions on each of the upper and lower body and excluding other actions that have similar distance ratios to a specific action, it is possible to prevent the person from mistakenly determining that the action is a specific action even when the person is in a position similar to another action.

[0082] In the above example, the primary behavior determination unit 350 estimates a person's behavior and provisionally determines that the behavior is a specific behavior (e.g., a fall) only when the inter-joint distances Lu and Lb are a predetermined distance ratio and satisfy predetermined conditions. However, the present invention is not limited to this case, and the primary behavior determination unit 350 may be configured to estimate a person's behavior only when the inter-joint distances Lu and Lb satisfy predetermined conditions. In other words, the primary behavior determination unit 350 may be configured to impose predetermined conditions on its behavior estimation.

[0083] As described above, the primary behavior determination unit 350 provisionally determines that a person's behavior is a specific behavior when the inter-joint distances Lu, Lb of the person's torso and / or legs are reduced to a predetermined distance ratio or less with respect to the reference distances Lsu, Lsb. Therefore, in the provisional determination by the primary behavior determination unit 350, in addition to a specific behavior, a specific behavior may also be provisionally determined when the person temporarily assumes a posture similar to a specific behavior. Therefore, the provisional determination by the primary behavior determination unit 350 is not, so to speak, a final determination of the person's behavior, but indicates the possibility that the person has performed a specific behavior.

[0084] [Secondary Behavior Determination Processing] The secondary behavior determination unit 360 associates the provisional determination result by the primary behavior determination unit 350 with ID information and the number of the corresponding captured image, and stores the association in RAM 302. By processing to associate the provisional determination result with the ID information and the number of the captured image, the provisional determination result of the person's behavior is accumulated as history in RAM 302. Note that this processing may be configured to be performed by the primary behavior determination unit 350.

[0085] The secondary behavior determination unit 360 determines that the person's behavior is a specific behavior (e.g., a fall) when the provisional determination that the person's behavior is a specific behavior continues for a predetermined duration (first time). Specifically, the secondary behavior determination unit 360 officially determines that the person has fallen if the provisional determination result of a fall for a person with the same ID information continues even after the duration has elapsed since the person fell. The duration can be set to a time sufficiently longer than the duration of a temporary action that a person takes in their daily life, such as temporarily crouching down to pick up something that has fallen on the floor. In other words, the secondary behavior determination unit 360 officially determines that the provisionally determined fall is a fall when it continues for a time longer than a temporary action similar to a fall that a person takes in their daily life.

[0086] The duration of the provisional fall determination result can be calculated, for example, by dividing the number of consecutive frames during which the primary behavior determination unit 350 provisionally determines that a fall has occurred by the frame rate. The duration value can be input by the user via the operation and display unit 306. Alternatively, the duration value can be stored in advance in the auxiliary storage unit 304.

[0087] Note that the secondary behavior determination unit 360 may be configured to have a timing unit that measures the duration of the provisional determination result of a fall, instead of using the number of consecutive frames to calculate the duration of the provisional determination result of a fall.

[0088] Alternatively, the secondary behavior determination unit 360 may be configured to officially determine that the person's behavior is a fall when the number of frames during which the provisional fall determination continues exceeds a predetermined number of frames, instead of calculating the duration of the provisional fall determination result. The predetermined number of frames may be input by the user via the operation display unit 306. Alternatively, the predetermined number of frames may be stored in the auxiliary storage unit 304.

[0089] Cancellation of Determination by the Secondary Behavior Determination Unit 360 After officially determining that the person's behavior is a specific behavior, the secondary behavior determination unit 360 determines whether the state in which the primary behavior determination unit 350 determined that the person's behavior is not a specific behavior continues for a predetermined cancellation time (third time). If the state in which the primary behavior determination unit 350 determined that the person's behavior is not a specific behavior continues for the cancellation time, the secondary behavior determination unit 360 cancels the official determination that the person's behavior is a specific behavior. This makes it possible to accurately capture the continuation of the person's fall. The value of the cancellation time can be input by the user via the operation and display unit 306, for example. The value of the cancellation time can also be stored in advance in the auxiliary storage unit 304.

[0090] In this way, the secondary behavior determination unit 360 determines to cancel the fall determination of the behavior when it detects that the user has not fallen continuously over the cancellation time. This makes it possible to prevent chattering in detection and enables stable operation.

[0091] [Behavior Notification Processing] The notification unit 370 functions as a behavior notification unit and notifies the alarm device 307 of the person's behavior when the secondary behavior determination unit 360 determines that the person's behavior is a specific behavior. The notification unit 370 can also notify another terminal device (not shown) of the person's behavior via communication over a network. For example, the notification unit 370 can send an email containing a video (image) of the person who has performed the behavior (e.g., fallen) to the other terminal device. The image to be sent may be either a photographed image or an image obtained by processing the original photographed image, such as an image in which skeletal information is superimposed on the photographed image. Adding an image to the notification allows the person receiving the notification to know the situation at the site where the person 400 is located in more detail, which is more effective in providing support to the person 400.

[0092] The notification unit 370 can also send videos / messages to other terminal devices via SNS (Social Networking Service) or by using push notifications. By requesting assistance from available members via email or an alarm 307, the person who has fallen can receive appropriate assistance according to the situation. Furthermore, if a person has a terminal capable of receiving notifications from the notification unit 370, they can know that action is required even if they are remotely located.

[0093] Furthermore, the notification unit 370 can be configured to notify and record the fall of the person 400 to a system other than the behavior estimation system 100, such as a core system. This makes it possible to keep a record of the fall of the person 400 for personal use by the person 400, for example. Alternatively, the notification unit 370 can transmit the skeletal information of the person whose behavior has been estimated to a database system intended for commercial, medical, academic, or other purposes, and store the information as data.

[0094] [Display Processing] The UI control unit 380 functions as a display control unit and causes the display to display the image acquired by the image acquisition unit 310. The UI control unit 380 can also cause the image to display the joint points of a person and the line segments connecting the joint points superimposed on the person.

[0095] Note that some or all of the functions of the behavior estimation device 300 may be configured to be executed by the imaging device 200 or another information processing device (not shown).

[0096] <Processing Procedure of Behavior Inference Method> Fig. 9 is a flowchart illustrating the processing procedure of the behavior inference method performed by the behavior inference system 100 according to the embodiment. The processing of the flowchart shown in the figure is realized by the CPU 301 executing the behavior inference program. Fig. 10 is a schematic diagram illustrating detection of skeletal information.

[0097] In the following description, it is assumed that at least one person is photographed by the camera 230 (see FIG. 3). For convenience of illustration, the drawings illustrate a case in which one person is photographed, but multiple people may also be photographed.

[0098] 3 , the imaging device 200 is installed so as to be able to capture an image of a floor surface FL within a predetermined area and a person 400 standing on the floor surface FL. More specifically, the installation position of the imaging device 200, the orientation of the camera 230 (optical axis direction of the lens), lens distortion, etc. are adjusted so that the floor surface FL and the standing person 400 fit within the field of view of the camera 230. For example, the imaging device 200 is installed on the floor surface FL at a predetermined distance from the position where the person 400 is located. Furthermore, the imaging device 200 is adjusted so that the orientation of the camera 230 is slightly downward relative to the horizontal direction so that the entire floor surface FL within the predetermined area fits within the field of view of the camera 230.

[0099] First, the image acquisition unit 310 acquires an image of a person (step S201). The imaging device 200, for example, captures an image of a person 400 moving in a predetermined area and transmits the captured video of the person 400 to the image acquisition unit 310. The image acquisition unit 310 receives and acquires a moving image of the person 400 from the imaging device 200, and decodes the acquired moving image to generate a plurality of frames of images. The image acquisition unit 310 transmits these plurality of frames of images to the skeleton detection unit 320.

[0100] Next, the skeleton detection unit 320 detects skeleton information of the person included in the image (step S202). The skeleton detection unit 320 detects skeleton information of the person 400 from multiple frames of images acquired by the image acquisition unit 310. If multiple people are included in the image, skeleton information is detected for the multiple people. The detected skeleton information is output to the distance information calculation unit 330 and the tracking processing unit 340.

[0101] As shown in FIG. 10 , the UI control unit 380 displays skeletal information of a person 400 on the display DP along with the moving image acquired by the image acquisition unit 310. In FIG. 10 , the person 400 in FIG. 3 is represented by a person HM on the video IM. The UI control unit 380 also displays a human rectangle BX of the person HM, whose skeletal information has been detected, on the display DP. The UI control unit 380 superimposes the joint points of the person HM and the line segments (skeleton) connecting the joint points from the skeletal information onto the person HM on the video IM and displays them on the display DP. In FIG. 10 , the black dots (●) displayed at the positions of the nose, neck, shoulders, elbows, wrists, waist, knees, ankles, etc. of the person HM correspond to the joint points, and the black lines connecting these dots correspond to the line segments.

[0102] Furthermore, the tracking processing unit 340 retains the same ID information for the same person over time. For example, the tracking processing unit 340 assigns ID information (ID: 01234) to the person HM and retains the assigned ID information until monitoring ends. This allows the user to easily visually identify who has fallen, even when multiple people are included in the video IM. Furthermore, even when the person HM stands up after falling, the user can determine from the video that the person HM has fallen. In other words, this prevents or suppresses the situation in which the fallen person HM becomes indistinguishable from other people who have not fallen because they have stood up after falling. As a result, the user can take necessary measures, such as calling out to the fallen person HM or providing assistance.

[0103] Next, the distance information calculation unit 330 calculates first distance information and second distance information (step S203). Based on the skeletal information, the distance information calculation unit 330 calculates first distance information for the upper body of a person included in the image and calculates second distance information for the lower body of the person. The first distance information includes, for example, the distance between the joint points at the center of the neck and the waist. The second distance information includes, for example, the distance between the center of the waist and the joint points at the feet. Note that if the reliability of the skeletal information does not meet a predetermined threshold, the distance information calculation unit 330 skips calculating the first distance information and the second distance information for that person.

[0104] Next, the primary behavior determination unit 350 estimates the behavior of the person (step S204). Based on the first distance information and the second distance information, the primary behavior determination unit 350 provisionally determines that the behavior of the person is a specific behavior when the inter-joint distances Lu, Lb of the person's torso and / or legs have shrunk to a predetermined distance ratio or less with respect to the reference distances Lsu, Lsb. For example, the primary behavior determination unit 350 provisionally determines that the behavior of the person is a fall when the distance ratios of the person's torso and legs have shrunk to 0.6 or less and 0.65 or less, respectively.

[0105] Next, the primary behavior determination unit 350 determines whether the first distance information and the second distance information each satisfy a determination condition (step S205). The determination condition may be, for example, whether the time Δt required for the inter-joint distances Lu and Lb to be reduced to a predetermined distance ratio with respect to the reference distances Lsu and Lsb is equal to or exceeds a predetermined determination time t ｔｈ If the first distance information and the second distance information do not satisfy the determination condition (step S205: NO), the process proceeds to step S202, and the skeleton detection unit 320 detects skeleton information of the person in the next captured image.

[0106] On the other hand, if the first distance information and the second distance information each satisfy the determination conditions (step S205: YES), the secondary behavior determination unit 360 determines whether the tentative determination of the specific behavior by the primary behavior determination unit 350 continues for the duration (step S206). If the tentative determination of the specific behavior does not continue for the duration (step S206: NO), the process proceeds to step S202, and the skeleton detection unit 320 detects skeleton information of the person in the next captured image.

[0107] On the other hand, if the provisional determination that the behavior is a specific behavior continues for a duration (step S206: YES), the secondary behavior determination unit 360 officially determines that the behavior of the person HM is a fall (step S207). This makes it possible to prevent the behavior of the person HM from being erroneously determined as a fall, even if the person HM temporarily assumes a posture close to a fall, for example, when crouching down to pick up a dropped object.

[0108] Furthermore, the secondary behavior determination unit 360 can delete or invalidate data on the determination results by the primary behavior determination unit 350 that is associated with the ID information of the person HM from the RAM 302. When the data on the determination results by the primary behavior determination unit 350 is deleted, no history of the determination results by the primary behavior determination unit 350 remains. On the other hand, when the data on the determination results by the primary behavior determination unit 350 is invalidated, the history of the determination results by the primary behavior determination unit 350 remains, but is not used for official determination by the secondary behavior determination unit 360. Furthermore, the secondary behavior determination unit 360 can also delete or invalidate data on the determination results that is associated with the ID information of a person for whom the provisional determination by the primary behavior determination unit 350 is no longer continued from the RAM 302.

[0109] Next, when the secondary behavior determination unit 360 officially determines that the behavior of the person HM is a specific behavior, the notification unit 370 notifies the specific behavior of the person HM (step S208). The notification unit 370 notifies another terminal device of the specific behavior of the person HM via the alarm 307 or by transmitting information via a network. For example, the alarm 307 outputs an alarm by outputting audio (sound) from a speaker and lighting a lamp (light) in accordance with the notification from the notification unit 370.

[0110] This makes it possible to prompt people around the fallen person 400 and members who can provide support (such as medical professionals) to take appropriate measures for the person 400 as soon as possible.

[0111] <Differentiation Between Persons Who Have Fallen and Persons Who Have Not Fallen in Video> Figure 11 is a schematic diagram illustrating an example of differentiation between a person HM1 who has fallen and a person HM who has not fallen in video. The UI control unit 380 differentiates the color of the person HM1 who has been officially determined to have fallen by the secondary behavior determination unit 360 from the color of the person HM who has not been determined to have fallen (persons who have not fallen) in the video. For example, the UI control unit 380 colors the person HM1 whose behavior has been determined to be a fall red and the person HM whose behavior has not been determined to be a fall green. This makes it easier for the user to visually distinguish between the person HM1 whose behavior has been determined to be a fall and the person HM whose behavior has not been determined to be a fall.

[0112] Alternatively, the UI control unit 380 assigns a mark MK to a person HM1 whose behavior has been determined to be a fall by the secondary behavior determination unit 360 in the video, while not assigning a mark MK to a person HM whose behavior has not been determined to be a fall. This makes it possible to distinguish the person HM1 who has been determined to have fallen from other people other than the person HM1 in the video. It is also possible to differentiate between people who have fallen and people who have not fallen in the video. The mark MK may be a figure and / or a character. The UI control unit 380 functions as a marking processing unit.

[0113] Furthermore, the UI control unit 380 can keep the color of the person HM1 different even if the determination that the person HM1's behavior is a fall is subsequently canceled, for the person HM1 whose behavior has been determined to be a fall by the secondary behavior determination unit 360. Alternatively, the UI control unit 380 can keep the mark attached to the person HM1.

[0114] In this way, by differentiating the person HM1 whose behavior is falling from the person HM whose behavior is not falling in the video, the user can clearly identify the person HM1 who has fallen (the person who needs treatment) from the video.

[0115] <Person Privacy Protection> FIG. 12 is a schematic diagram illustrating an example of protecting a person's privacy. For example, when a video is to be made public or when the person being photographed is hesitant to be photographed, consideration must be given to the person's privacy. As shown in the figure, the UI control unit 380 can perform a predetermined process at the position of the person HM in the video based on position information included in the skeletal information of the person HM to prevent the individual in the video from being identified. The predetermined process can be, for example, mosaic processing, blurring processing, or masking processing of a rectangular area including the face, upper body, or entire body of the person HM. This allows the person 400 to be monitored / watched while protecting the person's privacy.

[0116] The UI control unit 380 can also superimpose a graphic representing the skeletal information of the person HM on the video of the person HM for which a predetermined process has been performed. This allows the user to visually recognize the skeletal information of the person HM even while a predetermined process is being performed, and therefore the user can easily see the posture of the person HM in the video. As a result, the user can easily determine the posture of the person 400 when he or she fell.

[0117] [Detection in an effective area] Fig. 13 is a schematic diagram illustrating a method for setting an effective area. In the figure, the shaded area is the effective area. Fig. 14 is a schematic diagram for explaining behavior estimation in the effective area.

[0118] The UI control unit 380 allows the user to input an effective area (monitoring area) VA on the image via, for example, the operation and display unit 306. The operation and display unit 306 and the UI control unit 380 function as an effective area input unit. The effective area VA may be input, for example, by touching a touch panel with a finger FI to specify the range of the effective area VA, or by using an input device such as a keyboard or a mouse. Alternatively, if the position of the imaging device 200 is fixed, the value of the effective area VA may be stored in advance in an auxiliary storage device.

[0119] The valid area VA is an area in which the determination result of the primary behavior determination unit 350 (estimation result of the behavior estimation unit) is valid when the position coordinates of at least one joint point of the legs, head, or torso of the person HM are within the area. The valid area VA can be at least one polygon (figure).

[0120] 14, for person HM1 who is within the effective area VA, the result of the determination of the behavior (e.g., falling) by the primary behavior determination unit 350 is valid. On the other hand, for person HM2 who is not within the effective area VA (i.e., outside the effective area VA), the result of the determination of the behavior by the primary behavior determination unit 350 is not valid.

[0121] This makes it possible to perform only necessary behavior estimation by monitoring the effective area including the movement path of a person where a specific behavior of the person may occur. Furthermore, in skeleton detection, it is possible to prevent or suppress false detection of equipment with a complex structure that has parts structurally similar to a person (for example, industrial robots, humanoid robots, etc.). As a result, it is possible to prevent or suppress false alarms in the estimation of a person's behavior.

[0122] Furthermore, the UI control unit 380 displays a graphic indicating the effective area VA on the display DP together with the moving image captured by the imaging device 200. This allows the user to intuitively understand which area is set as the effective area.

[0123] [Detection in Invalid Area] Fig. 15 is a schematic diagram illustrating a method for setting an invalid area. In the figure, the gray area is the invalid area. Fig. 16 is a schematic diagram illustrating behavior estimation in an invalid area. The UI control unit 380, for example, allows the user to input an invalid area (non-monitored area) IVA on the video via the operation display unit 306. The operation display unit 306 and the UI control unit 380 function as an invalid area input unit. Input of the invalid area IVA is similar to input of the valid area VA, so detailed description will be omitted.

[0124] The invalid area IVA is an area in which the determination result of the primary behavior determination unit 350 (estimation result of the behavior estimation unit) becomes invalid when the position coordinates of at least one joint point of the legs, head, or torso of the person HM are within the area. The invalid area IVA can be at least one polygon (figure).

[0125] 16, for person HM1 who is within the invalid area IVA, the determination result of the behavior (for example, falling) by the primary behavior determination unit 350 is invalid. On the other hand, for person HM2 who is not within the invalid area IVA (i.e., outside the invalid area IVA), the determination result of the specific behavior by the primary behavior determination unit 350 is not invalid.

[0126] This makes it possible to perform only necessary behavior estimation by monitoring areas outside the person's movement path, where specific behaviors of a person are unlikely to occur, including the invalid area. Furthermore, in skeleton detection, it is possible to prevent or suppress false detection of equipment with complex structures (e.g., industrial robots, humanoid robots) that have structurally similar parts to a person as a person. As a result, it is possible to prevent or suppress false alarms in the estimation of a person's behavior.

[0127] Furthermore, the UI control unit 380 displays a graphic indicating the invalid area IVA on the display together with the moving image captured by the imaging device 200. This allows the user to intuitively understand which area is set as the invalid area (non-monitored area).

[0128] According to the present embodiment described above, the behavior inference device 300 provisionally determines that a person's behavior is a specific behavior when the inter-joint distances Lu and Lb for the person's upper body and / or lower body are each equal to or less than a predetermined distance ratio. Then, when the provisional determination of the person's behavior continues for a duration, the behavior inference device 300 officially determines that the person's behavior is a fall.

[0129] In this manner, in this embodiment, the behavior of an object is estimated based on the distance between two different skeletal joint points of the object and the distance between two skeletal joint points that differ from the two skeletal joint points by at least one skeletal joint point. Therefore, falls can be detected regardless of how the object falls or the orientation of the body. This improves the robustness of the accuracy of fall detection.

[0130] The configuration of the behavior estimation system 100 described above is a main configuration described in order to explain the features of the above embodiment, but is not limited to the above configuration and can be modified in various ways within the scope of the claims. Furthermore, configurations that are included in general behavior estimation systems are not excluded.

[0131] For example, in the above embodiment, a case where a person's joint points are detected by machine learning has been described, but the present invention is not limited to this case. For example, a configuration may be adopted in which a person's joint points are detected by pattern matching using feature points.

[0132] Furthermore, the means and methods for performing the various processes in the behavior estimation system 100 described above can be realized by either a dedicated hardware circuit or a programmed computer. The program may be provided by a computer-readable recording medium such as a USB memory or a DVD (Digital Versatile Disc)-ROM, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is typically transferred and stored in a storage unit such as a hard disk. The program may also be provided as standalone application software, or may be incorporated as a function into the software of a server or other device.

[0133] This application is based on Japanese Patent Application No. 2023-200543, filed on November 28, 2023, the disclosure of which is incorporated by reference in its entirety.

[0134] 100 Behavior estimation system, 200 Imaging device, 201 Bus, 210 Control unit, 220 Communication unit, 230 Camera, 300 Behavior estimation device, 301 CPU, 302 RAM, 303 ROM, 304 Auxiliary storage unit, 305 Communication I / F, 306 Operation display unit, 307 Alarm, 310 Image acquisition unit, 320 Skeleton detection unit, 330 Distance information calculation unit, 340 Tracking processing unit, 350 Primary behavior determination unit, 360 Secondary behavior determination unit, 370 Notification unit, 380 UI control unit.

Claims

1. A behavior estimation device comprising: an image acquisition unit that acquires an image of an object; a skeleton detection unit that detects skeleton information including information on a plurality of skeleton joint points of the object based on the image; a distance information calculation unit that calculates, based on the skeleton information, first distance information regarding a distance between two different skeletal joint points of the object and second distance information regarding a distance between two skeletal joint points that differ from the two skeletal joint points by at least one skeletal joint point; and a behavior estimation unit that estimates behavior of the object based on the first distance information and the second distance information.

2. The behavior estimation device of claim 1, wherein the image acquisition unit acquires an image of a person as the object, the distance information calculation unit calculates first distance information for an upper body of the person included in the image and calculates the second distance information for a lower body of the person, and the behavior estimation unit determines that the behavior of the person is a specific behavior when, for the upper body and lower body of the person, inter-joint point distances between two skeletal joint points of the person have decreased relative to respective predetermined reference distances.

3. The behavior estimation device according to claim 2, wherein the behavior estimation unit includes: a primary behavior determination unit that provisionally determines that the behavior of the person is the specific behavior when the inter-joint point distance has shrunk to a predetermined ratio or less of the predetermined reference distance; and a secondary behavior determination unit that formally determines that the behavior of the person is the specific behavior when the provisional determination that the behavior of the person is the specific behavior continues for a predetermined first time period.

4. The behavior estimation device according to claim 3, wherein the primary behavior determination unit provisionally determines that the behavior of the person is the specific behavior if the time it takes for the inter-joint point distance to reduce to the specified ratio with respect to the specified reference distance is less than or equal to a specified second time.

5. The behavior estimation device of claim 3, wherein the primary behavior determination unit determines that the behavior of the person is not the specific behavior when the inter-joint point distance has not reduced to the specified ratio with respect to the specified reference distance, and the secondary behavior determination unit cancels the formal determination that the behavior of the person is the specific behavior when the state in which the primary behavior determination unit has determined that the behavior of the person is not the specific behavior continues for a specified third time period after formally determining that the behavior of the person is the specific behavior.

6. The behavior estimation device according to claim 2, wherein the behavior estimation unit determines that the behavior of the person is the specific behavior based on a ratio, an amount of change, or a rate of change of the inter-joint point distance relative to the specified reference distance.

7. The behavior inference device according to any one of claims 1 to 6, further comprising a tracking processing unit that continuously holds the same ID information for the same object based on skeletal information of the object.

8. A behavior estimation device as described in any one of claims 1 to 6, wherein the skeleton detection unit estimates a degree of certainty for the skeleton information of the object, and the distance information calculation unit calculates the first distance information and the second distance information when the skeleton information has a degree of certainty equal to or greater than a predetermined value.

9. A behavior estimation device as described in any one of claims 1 to 6, further comprising: a display device that displays a video captured by an imaging device that captures the object and outputs a video consisting of a plurality of images; and a display control unit that controls the display device, wherein the skeleton detection unit analyzes the video captured by the image capture unit to calculate the skeletal information of the object, and the display control unit causes the display device to display the skeletal information of the object as an image together with the video.

10. The behavior estimation device as described in claim 9, further comprising a marking processing unit that superimposes and displays figures and / or characters on the image to distinguish the object whose behavior has been estimated by the behavior estimation unit from other objects other than the object on the image.

11. The behavior estimation device of claim 9, wherein the image acquisition unit acquires an image of a person as the object, and the display control unit performs mosaic or blurring processing at the position of the person in the video based on position information contained in the person's skeletal information so as to prevent the person from being identified as an individual.

12. The behavior inference device according to claim 11, wherein the display control unit superimposes and displays a figure representing skeletal information of the person on the image of the person on which the mosaic processing or blurring processing has been performed.

13. A behavior estimation device as described in any one of claims 1 to 6, further comprising a behavior notification unit, which, when the behavior estimation unit determines that the behavior of the object is a specific behavior, notifies an alarm that outputs an alarm by at least one of sound and light of the specific behavior of the object.

14. A behavior estimation device as described in any one of claims 1 to 6, further comprising a behavior notification unit, which, when the behavior estimation unit determines that the behavior of the object is a specific behavior, notifies the specific behavior of the object by transmitting information via a network.

15. A behavior estimation device as claimed in any one of claims 1 to 6, further comprising an effective area input unit for inputting an effective area in which an estimation result of a person's behavior is valid, wherein the image acquisition unit acquires an image of a person as the object, and the behavior estimation unit validates the estimation result of the person's behavior when position coordinates of at least one skeletal joint point of the person's legs, head, and torso are within the effective area.

16. The behavior inference device according to claim 15, further comprising: a display device that displays a video captured by an imaging device that captures the person and outputs a video consisting of a plurality of images; and a display control unit that causes the display device to display a figure indicating the effective area together with the video captured by the imaging device.

17. A behavior estimation device as claimed in any one of claims 1 to 6, further comprising an invalid area input unit for inputting an invalid area in which an estimation result of a person's behavior is invalid, wherein the image acquisition unit acquires an image of a person as the object, and the behavior estimation unit invalidates the estimation result of the person's behavior when position coordinates of at least one skeletal joint point of the person's legs, head, and torso are within the invalid area.

18. The behavior inference device according to claim 17, further comprising: a display device that displays a video captured by an imaging device that captures the person and outputs a video consisting of a plurality of images; and a display control unit that causes the display device to display a figure indicating the invalid area together with the video captured by the imaging device.

19. A behavior estimation method comprising: a step (a) of acquiring an image of an object; a step (b) of detecting skeletal information including information on a plurality of skeletal joint points of the object based on the image; a step (c) of calculating, based on the skeletal information, first distance information regarding a distance between two different skeletal joint points of the object and second distance information regarding a distance between two skeletal joint points that differ from the two skeletal joint points by at least one skeletal joint point; and a step (d) of estimating behavior of the object based on the first distance information and the second distance information.

20. A behavior estimation program for causing a computer to execute a process including the steps of: (a) acquiring an image of an object; (b) detecting skeletal information including information on a plurality of skeletal joint points of the object based on the image; (c) calculating, based on the skeletal information, first distance information regarding a distance between two different skeletal joint points of the object and second distance information regarding a distance between two skeletal joint points that differ from the two skeletal joint points by at least one skeletal joint point; and (d) estimating behavior of the object based on the first distance information and the second distance information.

Images