Abnormal condition detection device and abnormal condition detection method

Abstract

To provide a system for easily and highly accurately detecting abnormal conditions of a person captured in an image. [Solution] In the abnormal condition detection system, detection unit 11 detects a person and a specific item captured in an image by imaging unit 10. Lying position determination unit 12 determines whether or not the person is lying down, and if a specific item is present in the image, item contact determination unit 13 determines whether or not the person is lying down on the specific item, and abnormal condition determination unit 14 determines that an abnormal condition exists except when the person is lying down on the specific item, and notification unit 15 issues a warning if an abnormal condition is determined.

Description

[Technical Field]

[0001] The present invention relates to an abnormal condition detection device and an abnormal condition detection method. [Background technology]

[0002] There are systems for visually inspecting surveillance cameras and reporting abnormalities, such as when a relative living far away watches over an elderly person living alone or when a caregiver at a nursing home watches over a person requiring care to ensure that they do not collapse. Related technologies include, for example, a technology for detecting an abnormal state in which a person in a video has fallen due to a fall or the like, which detects the fact that the person has fallen based on the coordinates and depth changes of the person's skeleton (see, for example, Patent Document 1). Another known technology estimates a person's lying state based on the relationship between the coordinate values of the person's skeleton, but determines that the person is in an abnormal state if there is no movement for a certain period of time (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6525179 [Patent Document 2] Japanese Patent Application Publication No. 2022-16979 Summary of the Invention [Problem to be solved by the invention]

[0004] The invention described in Patent Document 1 makes it possible to estimate behavior with high accuracy by identifying and capturing postures based on the coordinates and depth variations of each of the multiple skeletons of a person captured in time-series images. However, this invention cannot detect, for example, a person who is slowly crouching down and unable to move.

[0005] In addition, the deep learning model needs to learn the movements of falls, but the movements of people when they fall vary greatly depending on the time and situation, making it difficult to train the model to learn all possible patterns of falls.

[0006] In addition, in order to determine a fall based on the movement of the posture at a certain speed or acceleration, it is necessary to make inferences at the same speed as the image speed (FPS) during learning, and high processing capabilities are required of the computer.

[0007] The need to grasp abnormal states is expected to be required in the homes of the elderly living alone, nursing facilities, etc. This includes cases where children living away worry about their elderly parents living alone, or cases where caregivers in nursing facilities are monitored by helpers from another location. In this case, from the perspective of privacy, it is preferable to complete the processing on the edge computer without transferring data to the cloud in order to visualize daily life. However, when processing the invention described in Patent Document 1 on an edge device instead of a cloud system, since there are limitations in processing capabilities, stable inference may be difficult, and in the worst case, there is also a possibility of shutdown.

[0008] The invention described in Patent Document 2 estimates the lying position state from the relationship of the coordinate values of the human skeleton, but determines an abnormal state based on the state of no movement for a certain period of time. However, when this method is directly applied to monitoring the daily life of the elderly living alone or in nursing facilities, a normal sleeping state will also be determined as an abnormal state, resulting in the inconvenient result of determining as abnormal the sleeping that occupies a lot of time in a day.

[0009] The present invention has been made in view of such circumstances, and its object is to provide an abnormal state detection system, an abnormal state detection method, and an abnormal state detection device that can detect an abnormal state of a person with low equipment, low power, and high accuracy while taking privacy into consideration.

Means for Solving the Problems

[0010] In order to solve the above problems, one aspect of the present invention provides an abnormal state detection system and an abnormal state detection device including: a detection unit that detects a person and a specific article based on an image captured by at least one imaging unit that captures an image of a monitoring location which is a location to be monitored; a lying position determination unit that determines whether a person detected by the detection unit is in a lying position based on the image data of the person; an article contact determination unit that determines whether the lying position is in contact with the specific article based on the spatial relationship between the position data of the person and the position data of the specific article to determine the contact state between the person and the article; an abnormal state determination unit that determines that it is an abnormal state when it is determined by the lying position determination unit that the person is in a lying position and it is determined by the article contact determination unit that there is no contact; and a notification unit that issues a warning when the abnormal state determination unit determines that it is an abnormal state.

[0011] According to another aspect of the present invention, there is provided an abnormal state detection program and an abnormal state detection method corresponding to the above abnormal state detection system.

Advantages of the Invention

[0012] According to the abnormal state detection system of the present invention, an abnormal state can be estimated and detected with low equipment, low power, and high accuracy, and operation considering privacy is possible.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Embodiments for Carrying Out the Invention

[0014] Hereinafter, an abnormal state detection system according to an embodiment of the present invention will be described with reference to FIGS. 1-7. In this embodiment, for ease of understanding, a person is simply displayed with only the skeleton

[0015] (Abnormal State Detection System) FIG. 1 is a functional block diagram of an abnormal state detection system according to an embodiment of the present invention. The abnormal state detection system includes an imaging unit 10 that captures at least one time-series image, and an abnormal state detection device 100

[0016] The imaging unit 10 is a surveillance camera installed indoors or outdoors, captures an image of a surveillance location that is the location to be monitored, attaches various information at the time of imaging to the captured image, and transmits it to the abnormal information detection device 100. Here, the various information includes at least camera identification information for identifying the surveillance camera, imaging time of the image, shooting attributes such as shooting speed, etc. It can be imaged with any camera device such as a general camera, an infrared camera, a wide-angle camera, a stereo camera, etc. Also included are devices that measure the distance and shape to an object using laser light such as LIDAR

[0017] The abnormal state detection device 100 may be an edge device, a personal computer, or a cloud computer that can communicate with the imaging unit 10. The abnormal state detection device 100 includes a control unit 20, a storage unit 30, and a notification unit 15

[0018] The storage unit 30 stores various data used by the control unit 20. The storage unit 30 is composed of, for example, storage media such as HDD, flash memory, RAM, ROM, etc., or any combination of these storage media. The storage unit 30 includes an imaging information storage unit 16, a software storage unit 17, a person data storage unit 18, and an article data storage unit 19.

[0019] The imaging information storage unit 16 stores an imaging identification ID - imaging location correspondence table. The imaging identification ID is uniquely numbered corresponding to the imaging camera, and imaging location names such as bedroom, living room, nursing room, etc. are managed in a table. The imaging identification ID is managed to smoothly determine an abnormal state when imaging is performed by a plurality of cameras.

[0020] The software storage unit 17 stores an object detection (DETECTION) program, a skeleton estimation (POSE ESTIMATION) program, an object tracking (TRACKING) program, a segmentation (SEGMENTATION) program, and an age estimation program used by the control unit 20. These programs may be open source or user - made machine learning models.

[0021] The person data storage unit 18 manages the person data table shown in FIG. 2 and stores information related to persons. The person data table manages at least an imaging identification ID, an imaging time, a person identification ID, a lying position state flag, an article contact flag, an abnormal state flag, bounding box coordinates, skeleton coordinates, segmentation mask information, age information, a notification start time, and a re-notification time. The person identification ID is an ID numbered as a result of object tracking by the detection unit 11 described later. The lying position state flag is a flag set when the lying position state is determined by the lying position determination unit 12 described later. The article contact flag is a flag set when it is determined by the article contact determination unit 13 described later that the person is in contact with a specific article. The abnormal state flag is a flag set by the abnormal state determination unit 14 described later when the lying position state flag is set and the article contact flag is not set. The bounding box coordinates are coordinate values on the detected image as a result of object detection by the detection unit 11 described later. In addition to two-dimensional XY coordinates, when depth information is taken into account, it becomes three-dimensional information of the depth. The skeleton coordinates are coordinate values on the estimated image as a result of skeleton estimation by the detection unit 11 described later. In addition to two-dimensional XY coordinates, when depth information is taken into account, it becomes three-dimensional information of the depth. The segmentation mask information is pixel unit class (in this case, person) determination information of the estimated image as a result of segmentation by the article contact determination unit 13 described later. It includes coordinate information of pixels where a person is reflected on the image. The age information is the estimated age information of the person as a result of age estimation by the abnormal state determination unit 14 described later. The notification start time is managed by the notification unit 15 described later as the imaging time when it is first determined as abnormal in units of the imaging identification ID and the person identification ID. The re-notification time is set by the notification unit 15 described later at the time of re-notification. The setting methods for both the notification start time and the re-notification time will be described later.

[0022] The article data storage unit 19 manages the article data table shown in FIG. 3 and stores information regarding articles. The article data table manages at least an imaging identification ID, an imaging time, an article identification ID, bounding box coordinates, and segmentation mask information. The article identification ID is an ID numbered as a result of object tracking by the detection unit 11 described later. The bounding box coordinates are coordinate values on the detected image as a result of object detection by the detection unit 11 described later. In addition to two-dimensional XY coordinates, when depth information is taken into account, it becomes three-dimensional information of the depth. The segmentation mask information is pixel unit class (in this case, article) determination information of the estimated image as a result of segmentation by the object contact determination unit 13 described later. Coordinate information of pixels where articles are reflected on the image is included.

[0023] The control unit 20 is a CPU that executes the software of the abnormal state detection device 100 and comprehensively controls each part of the abnormal state detection device 100. The control unit 20 includes a detection unit 11, a lying position determination unit 12, an article contact determination unit 13, and an abnormal state determination unit 14.

[0024] The detection unit 11 detects a person Z and an article W in the image based on the image captured by at least one imaging unit 10 that captures the monitoring location. In the present embodiment, the detection unit 11 uses an object detection (DETECTION) program, a pose estimation (POSE ESTIMATION) program, and an object tracking (TRACKING) program stored in the software storage unit 17 to detect the person Z and the article W in the image.

[0025] In the object detection program, one or more persons or articles shown in the time-series images are detected as bounding box information. The bounding box information is information for each object detected in the image, and manages the class ID (representing the type of person or article), coordinate values, and estimation confidence. The coordinate values are coordinates with the combination of the maximum and minimum values of the area occupied by the object in the horizontal and vertical directions of the image as vertices, and the bounding box forms a rectangle. Also, in addition to the two-dimensional XY coordinates, the coordinate values become three-dimensional information with depth information added when estimating depth information. In the skeleton estimation program, using a deep learning model, the skeleton positions are predicted from the input video, and the coordinates of the head (top of the head, chin), neck, shoulders (left and right), elbows (left and right), wrists (left and right), waist (left and right), femur (left and right), knees (left and right), ankles (left and right), eyes (left and right), nose, and ears (left and right) can be predicted. These skeleton positions can also handle three-dimensional information that takes into account depth information, not just two-dimensional information.

[0026] In the object tracking program, the movement of a specific object is tracked in a continuous video, and a unique ID is assigned to the object to be tracked. The unique ID can maintain consistency between frames and is managed as the person identification ID in the person data table or the article identification ID in the article data table.

[0027] Conventional techniques for determining abnormal states such as falls based on movements of postures at a certain speed or acceleration require inference at the same speed as the image speed (FPS) during learning, and high processing capabilities are required of the computer. When the amount of arithmetic processing increases rapidly, the power consumption increases, and the temperature of the arithmetic device rises, as in the case of a large number of imaging persons, especially edge devices have limitations in processing capabilities, so smooth inference processing may become difficult or shutdown may occur.

[0028] However, by adopting a method of grasping the abnormal state based on the positional relationship between the static state and a specific article instead of grasping it by the dynamic changes in the posture of a person, the determination performance does not deteriorate even if the number of images per unit time of the image data is significantly reduced. Therefore, by measuring the temperature and power consumption of the arithmetic unit and performing reduction of the number of images per unit time in the detection unit 11 when it exceeds a certain threshold value, stable operation can be ensured due to a reduction in the load on the arithmetic unit, compression of power consumption, and a decrease in temperature.

[0029] In the lying position determination unit 12, based on the bounding box information detected by the object detection (DETECTION) program of the detection unit 11, it is determined whether the person is in the lying position state. In the case of the lying position state, the lying position state flag in the person data table is set. This bounding box information includes a label indicating the type of object such as a person, a bed, or a sofa, coordinates and depth information indicating the position within the image, and the estimated reliability.

[0030] Based on the coordinate values of the bounding box of the label of the person, the side length is calculated. When the side length in the horizontal direction is longer than the side length in the vertical direction by a certain threshold value or more, it can be determined that the person is in the lying position state.

[0031] Regarding a human, each part of the face such as the skeleton, eyes, and nose is also managed as landmark information together with the bounding box within the ROI (Region of Interest) detected by the object detection program. Usually, except when a person is lying face down, they are either standing upright or sitting on a chair. In this case, since the upper body is in a posture close to vertical, the lying position state can also be determined, for example, by the inclination of the line segment between the midpoints of both shoulders and the midpoints of both waists.

[0032] In addition to the cases of standing upright or sitting, a person may sometimes take various postures in daily life. For example, when in the posture of picking up garbage lying on the floor, if only the upper body is judged, it may be judged as the lying position state. In order to avoid misjudgment in this case, it is also preferable not to judge it as the lying position state in the posture of picking up garbage by comparing the midpoints of both shoulders, both waists, and the coordinates of the ankles, and determining that the former is at a higher position than the latter.

[0033] Moreover, in the case of determining the lying position state by the lying position determination unit, as shown in FIG. 6(B), when the person Z7 is falling toward the imaging direction, the shape of the bounding box V10 of the person does not become horizontally long. Also, the inclination of the line segment between the midpoints of both shoulders and the midpoints of both waists becomes an angle close to perpendicular, and an accurate lying position state determination cannot be made. Also in this case, by taking into account the bounding box and the depth information of both shoulders and both waists, and calculating the shape of the three-dimensional bounding box and the inclination of the line segment between the midpoints of both shoulders and the midpoints of both waists, a more accurate lying position state determination becomes possible.

[0034] The article contact determination unit 13 determines whether a person in the lying position state is in contact with a specific article based on the mutual positional relationship between the person in the lying position state and the specific article when it is determined by the lying position determination unit 12 that there is a person in the lying position state. When contact is determined, the article contact flag in the person data table is set. As the specific article, articles such as a bed, a sofa, and a futon on which it is natural for a person to lie horizontally are targeted. In the prior art, there is a case where an abnormal state is defined as a case where the same state continues for a certain period of time after a fall, but in a residence or facility where daily life such as a single elderly person's home or a nursing facility is carried out, normal bedtime is determined to be abnormal, which is inappropriate, and an abnormal determination based on the positional relationship between a person and an object can provide a more accurate abnormal determination.

[0035] FIG. 4 is an explanatory diagram of the usage state of the abnormal state detection system. For example, in FIG. 4, as persons, Z1 and Z2 are detected, and as specific articles, beds W1 and W2 are detected. The determination as to whether the lying position state is being performed on the specific article is made based on the positions of the bounding box, contour, and skeleton of the person and the positions of the bounding box and contour of the specific article.

[0036] In FIG. 4, three cases (A), (B), and (C) are illustrated. Among them, it is determined that the persons Z1, Z2, and Z3 are in the lying position, and at the same time, the beds W1 and W2 of specific articles are detected. In case (A), since Z1 is on W1, no abnormality is determined. In case (B), although there is a bed for a specific article, since Z2 is not on W2, an abnormality is determined. In case (C), since there is no specific article in the image of Z3, an abnormality is determined.

[0037] The positional relationship between a person and a specific article is determined by calculating the IOU (INTERSECTION OVER UNION) with the sum of the areas of the respective bounding boxes as the denominator and the area of the overlapping part as the numerator. Theoretically, when it is other than 0 (in practice, a threshold value including errors can also be set), it is determined that they are in contact. In case (A) of FIG. 4, since the bounding box V2 is included in V1, the IOU exceeds 0. In case (B), since there is no overlap between the bounding boxes V3 and V4, the IOU becomes 0.

[0038] Also, for a person, in addition to the bounding box, the skeleton information can also be used as an index representing the position. For example, when the coordinate value of the midpoint of the line segment connecting the midpoints of both shoulders and both waists is within the coordinate values of the bounding box of a specific article, it can be determined that the person is in a lying position on the specific article. When the coordinate value of the midpoint of the line segment connecting the midpoints of both shoulders and both waists is a numerical value within the range of the maximum and minimum values of the coordinate values of the bounding box in both the vertical and horizontal directions, a contact determination can be made.

[0039] FIG. 5 is an explanatory diagram of a case where a specific article is not facing forward in the usage state of the abnormality detection system. When determining the contact situation between a person and a specific article, if the image captured by the imaging unit 10 is taken at a position facing the specific article, as shown in Fig. 5(A), the bounding box V6 almost exactly coincides with the location of the specific article, enabling highly accurate determination. However, if the image cannot be captured at a position facing the specific article, as shown in Fig. 5(B), the bounding box V7 indicates a region wider than the specific article, and there may be an incorrect determination that a person in a lying position at a location different from the specific article is in contact with it.

[0040] However, it is also conceivable that the image captured by the imaging unit 10 cannot be taken at a position facing the specific article. In this case, in the article contact determination unit 13, it is also possible to determine whether the lying position state is being performed on the specific article by using the segmentation program stored in the software storage unit 17 based on the contact state between the region of the person and the specific article. In the segmentation algorithm, it is determined which detection object each image region in the image belongs to in terms of pixels. As a result, the detected person and article are grasped as if they were masks in the shape of a painted picture. By analyzing the contour of this mask, it becomes possible to grasp the contact situation between the two by calculating the distance between the contours of the two. Also, it is possible to grasp that they are in contact by confirming that there is no background between the masks of the two.

[0041] Fig. 6 is an explanatory diagram of the usage state of the abnormal state detection system in a case where three-dimensional coordinates are required. However, as in the case of Fig. 6(A), when a person is lying behind a specific article as seen from the camera, the bounding box V9 of the person is included in the bounding box V8 of the specific article, and in the 2D image, a case where they are in contact is assumed even though they are not actually in contact. To improve the accuracy in this case, it is possible to determine the contact situation in 3D coordinates taking into account depth information for the bounding boxes and skeleton information of the person and the article. Depth information can be used as a known technique not only by methods measured from a stereo camera or methods using laser light such as LIDAR, but also DEPTH ESTIMATION that estimates depth from an image using machine learning with a single camera.

[0042] Note that in this example, a single case of a person and a specific article is taken up, but for cases where there are multiple of each, the contact situation between each person and all specific articles can be determined by determining the contact situation between each person and all specific articles for each person and repeating this operation for all persons. If no contact is recognized between a person and any specific article, it will be determined as an abnormal state.

[0043] The abnormal state determination unit 14 determines that it is an abnormal state when the lying position determination unit 12 determines the lying position and the article contact determination unit 13 does not determine contact.

[0044] In the abnormal state determination unit 14, using the age estimation program stored in the software storage unit 17, the age of the person is estimated, and when the age is a specific age, it is also possible not to issue a warning. For example, it is conceivable that a child generally rolls on the floor and plays, but it is not preferable to issue a warning in this case. Therefore, it is also possible not to issue a warning for a person below a certain age estimated by the age estimation algorithm. The estimated age information is set in the age information of the person data table, and when no warning is issued, an abnormal state flag is not set. Also, it is preferable to reduce the computer processing load by not performing this estimation when the age is set in the immediately preceding person data table for each imaging identification ID and person identification ID, and inheriting the age of the immediately preceding table.

[0045] When the abnormality determination unit 14 determines an abnormality, the notification unit 15 issues a warning. As notification methods, warning display on a display, voice notification, display by an alarm lamp, notification by email, etc. can be considered.

[0046] When determining an abnormal state inside the edge computer, unlike cloud computing that transmits images through a communication line, the risk of image leakage decreases. When imaging daily life, more privacy-conscious operation is required, but by adopting this method, a mechanism favored by users can be provided. Also, when it is desired to confirm an abnormal state with an image, only the image determined to be in an abnormal state is notified, and other images are not notified, enabling a response that takes utmost consideration of privacy.

[0047] In the notification unit 15, when an abnormality flag is set, the notification start time of the previous table is carried over. However, when the notification start time of the previous table is not set, or when it is the first data, the imaging time of the data is set as the notification start time. This process is performed in units of imaging identification ID and person identification ID. Also, when the abnormality flag is not set, the notification start time is not carried over.

[0048] When an abnormal state continues, continuously sending emails would result in sending a large number of emails, which is not practical. Therefore, smooth notification can be achieved by re-notifying at regular intervals. For example, re-notify when a certain period of time has elapsed, and include the notification start time and re-notification time from the person data table in the email to accurately notify the continuation status of the abnormal state. Record the re-notification time in the person data table at the time of re-notification.

[0049] In addition, when doing abdominal muscle exercises while lying on the floor, etc., in some cases, it is also conceivable to repeatedly alternate between the lying position and a non-lying position within a short period of time. If there is a desire to avoid notification in such a case, the first notification may not be performed, and only the re-notification after a certain period of time may be the target of notification.

[0050] (Abnormal state detection process) FIG. 7 is a flowchart showing the detection process of an abnormal state. Subsequently, the detection of an abnormal state by the abnormal state detection system will be described using the flowchart of FIG. 7.

[0051] Step S1: First, the imaging unit 10 acquires an image captured by at least one imaging unit that captures the monitoring location, which is the location to be monitored, and transmits it to the abnormal state detection device 100 together with shooting attributes such as camera identification information, the imaging time of the image, and the shooting speed.

[0052] Step S2: In the detection unit 11, based on the acquired camera identification information, the detection unit 11 refers to the imaging identification ID - imaging location correspondence table stored in the imaging information storage unit 16 to determine the name of the monitoring location of the imaging unit 10. Based on the image captured by the imaging unit 10, using the object detection (DETECTION) program and the object tracking (TRACKING) program stored in the software storage unit 17, a person and a specific article in the image are detected as bounding box information. At this time, the detection unit 11 assigns a person identification ID to the person extracted from the image and an article identification ID to the specific article.

[0053] Step S3: In the lying position determination unit 12, using the object detection (DETECTION) program stored in the software storage unit 17, a determination is made as to whether it is a lying position state based on the bounding box coordinates. Also, using the pose estimation (POSE ESTIMATION) program, the pose of the person in the image is estimated, and a determination is made as to whether it is a lying position state based on the pose coordinates. As a result, a lying position state flag is set.

[0054] In addition to two-dimensional coordinates, the bounding box coordinate values and skeleton coordinate values can also use three-dimensional coordinates including depth information to improve the determination accuracy. Based on this result, the imaging identification ID, imaging time, person identification ID, lying position state flag, bounding box coordinates, and skeleton coordinates are set in the person data table managed by the person data management unit 18.

[0055] Step S4: According to the lying position state flag, if the flag is set (Step S4 - YES), go to Step 5 ; if not (Step S4 - NO), go to Step 9.

[0056] Step S5: Next, in the article contact determination unit 13, when it is determined that there is a person in the lying position state, it is determined whether the person and a specific article are in contact based on the mutual position relationship between the person in the lying position state and the specific article. If there are multiple specific articles in the image, the contact situation with all specific articles is checked. If it is determined that there is contact with any specific article, the article contact flag in the person data table is set. The position of the person is determined by the coordinate values of the bounding box and the skeleton, and the position of the specific article is determined by the coordinate values of the bounding box. When there is an overlap in the coordinate values of both, the contact determination is made. These coordinate values can be used for two-dimensional determination on the XY axis, but the determination accuracy can also be improved by three-dimensional determination including depth information. Also, by using the segmentation program stored in the software storage unit 17 to create a segmentation mask for the person and the specific article, the positions of the person and the specific article can be determined and the contact determination can be made. In the article contact determination unit 13, the imaging identification ID, imaging time, article identification ID, bounding box coordinates, and segmentation mask information of the article data storage unit 19 are set.

[0057] Step S6: After that, check whether there are other specific articles in the image. If there are (Step S6 - YES), go to Step S9; if not (Step S6 - NO), go to Step S7.

[0058] Step S7: When the lying position status flag in the person data table of the person data storage unit 18 is on and the article contact flag is off, set the abnormal status flag as being in an abnormal state.

[0059] Step S8: When there is data with the abnormal status flag on among the data of the most recent imaging time in the person data table of the person data storage unit 18, the notification unit 15 notifies the abnormal state. When notifying by email, privacy of the person being imaged can be considered by limiting it to the images for which the abnormal determination has been made.

[0060] Step S9: If another person is shown in the image (Step S9 - YES), return to Step S3, and if not (Step S9 - NO), end the process.

[0061] Using the age estimation program stored in the software storage unit 17, after estimating the age of a person, the estimation result is managed as age group information in the person data table, and infants, children, etc. can also be excluded from the notification targets.

[0062] Also, when the abnormal state continues and notifications are made at regular time intervals, the notification start time and re - notification time in the person data table can be notified, and the continuation status of the abnormal state can be notified in detail. The notification start time is set to the imaging time of the data when the immediately preceding data has not been determined to be abnormal, in units of the imaging identification ID and the person identification ID. The re - notification time is set to the imaging time of the notification data.

[0063] Also, if it is desired to omit notifications of abnormal states that repeat in a short period of time, it is also possible to exclude the first notification from the targets.

[0064] As described above, the abnormal state detection system according to the embodiment of the present invention includes a detection unit 11, a lying position determination unit 12, an article contact determination unit 13, an abnormal state determination unit 14, and a notification unit 15. The detection unit 11 detects a person and a specific article based on an image captured by at least one imaging unit 10 that captures an image of a monitoring location which is the location to be monitored. The lying position determination unit 12 determines whether the person detected by the detection unit 11 is in a lying position based on the image data of the person. The article contact determination unit 13 determines the contact state between the person and the article based on the spatial relationship between the position data of the person and the position data of the specific article, whether the lying position state is being performed in contact with the specific article. The abnormal state determination unit 14 determines that it is an abnormal state when it is determined by the lying position determination unit 12 that the person is in a lying position state and it is determined by the article contact determination unit 13 that there is no contact.

[0065] Thereby, the abnormal state detection system according to the embodiment of the present invention determines the lying position state of the person and determines the contact state between the person and the article based on the spatial relationship with the position data of the specific article, so that the abnormal state can be determined with low equipment, low power, and high precision. The abnormal state detection system can also determine an abnormal state even when the person is slowly huddled. In addition, it does not misjudge a normal state of lying in bed or the like as an abnormal state.

[0066] Further, in the embodiment of the present invention, the lying position determination unit 12 uses either or both of the bounding box coordinate values and the skeleton estimation coordinate values of the person for the lying position determination, and the article contact determination unit 13 uses either or both of the bounding box coordinate values and the skeleton estimation coordinate values of the person and the bounding box coordinate values of the article for the article contact determination. Thereby, the abnormal state detection system according to the embodiment of the present invention can perform highly accurate lying position determination and article contact determination, and can accurately detect an abnormal state.

[0067] In addition, in the embodiment of the present invention, the article contact determination unit 13 estimates a pixel-level region for a person and a specific article using a segmentation algorithm, and determines whether or not the two are in contact based on the estimation result. As a result, the abnormal state detection system according to the embodiment of the present invention can prevent misjudgment in cases where the bounding box indicates a region wider than the actual location of the article, such as when the specific article is not facing the captured image.

[0068] In addition, in the embodiment of the present invention, depth information is taken into account in either or both of the determination of the lying position state in the lying position determination unit 12 and the determination of whether the lying position state in the article contact determination unit 13 is in contact with a specific article. As a result, the abnormal state detection system according to the embodiment of the present invention can improve the determination accuracy of the abnormal state by grasping the inclination of the human body in the depth direction of imaging that cannot be grasped by two-dimensional information and the contact state between the person and the specific article.

[0069] In addition, in the embodiment of the present invention, the notification unit 15 does not issue a warning when the person is of a specific age estimated by the age estimation algorithm. As a result, the abnormal state detection system according to the embodiment of the present invention can prevent misdetection of infants and children rolling on the floor and playing as an abnormal state, and can improve the abnormal state determination accuracy.

[0070] In addition, in the embodiment of the present invention, when the processing related to the image is not transferred to the cloud computer but is completed within the edge device and the image is notified by the notification unit 15, a notification method considering privacy is provided by limiting it to the image determined to be an abnormal state.

[0071] As a result, the abnormal state detection system according to the embodiment of the present invention can provide an abnormal state detection service that captures daily life and matches the needs of monitoring systems with high privacy protection requirements.

[0072] In addition, in the embodiment of the present invention, DetectionIn section 11, by adjusting the number of images per unit time according to the measured power consumption and temperature of the computer device, stable operation of the computer is achieved. As a result, the abnormal state detection system according to the embodiment of the present invention can avoid the destabilization of arithmetic processing due to an increase in the power consumption of the arithmetic device and the base temperature even when the number of objects such as people and specific articles in the captured image increases and the amount of arithmetic operations for computer determination increases, and can enable stable detection of abnormal states.

[0073] Further, the abnormal state detection device 100 according to the embodiment of the present invention includes a detection unit 11, a lying position determination unit 12, an article contact determination unit 13, an abnormal state determination unit 14, and a notification unit 15. As a result, the abnormal state detection device 100 according to the embodiment of the present invention can detect abnormal states with low equipment, low power, and high accuracy.

[0074] Further, in the abnormal state detection program and the abnormal state detection method according to the embodiment of the present invention, a detection unit 11 detects a person and a specific article based on an image captured by at least one imaging unit 10 that images a monitoring location that is a location to be monitored, a lying position determination unit 12 determines whether or not a person detected by the detection unit 11 is in a lying position based on the image data of the person, an article contact determination unit 13 determines the contact state between the person and the article based on the spatial relationship between the position data of the person and the position data of the specific article as to whether or not the lying position is being performed in contact with the specific article, an abnormal state determination unit 14 determines that it is an abnormal state when it is determined by the lying position determination unit 12 that the person is in a lying position and it is determined by the article contact determination unit 13 that there is no contact, and a notification unit 15 issues a warning when the abnormal state determination unit 14 determines that it is an abnormal state. As a result, the abnormal state detection program and the abnormal information detection method according to the embodiment of the present invention have the same effects as the above-described abnormal state detection system.

[0075] In addition, in the embodiment of the present invention, an example has been described in which the control unit 20 performs AI calculations using an object detection (DETECTION) program, a skeleton estimation (POSE ESTIMATION) program, an object tracking (TRACKING) program, a segmentation (SEGMENTATION) program, and an age estimation program stored in the software storage unit 12. However, the present invention is not limited to this. For example, the position of a human body part may be extracted by connecting to a website capable of executing AI processing online via a network.

[0076] Further, the present invention is also applicable to a program and method corresponding to the processing performed by the detection unit 11, the lying position determination unit 12, the article contact determination unit 13, the abnormal state determination unit 14, and the notification unit 15, and a recording medium storing the program. In the case of a recording medium, the program will be installed in a computer or the like. Here, the recording medium storing the program may be a non-transitory recording medium. Examples of non-transitory recording media include, but are not limited to, CD-ROMs.

[0077] As described above, the embodiments of the present invention have been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and designs and the like within the scope not departing from the gist of the present invention are also included.

Explanation of Reference Numerals

[0078] 1 Abnormal state detection system 10 Imaging unit 11 Detection unit 12 Lying position determination unit 13 Article contact determination unit 14 Abnormal state determination unit 15 Notification unit 16 Imaging information storage unit 17 Software storage unit 18 Person data storage unit 19 Article data storage unit 20 Control unit 30 Storage unit 100 Abnormal state detection device

Claims

1. A detection unit that detects a person and a specific article based on an image captured by at least one imaging unit that images a monitoring location which is a location to be monitored; A lying position determination unit that determines whether or not a person detected by the detection unit is in a lying position based on the image data of the person; An article contact determination unit that determines a contact state between a person and an article based on a spatial relationship between the position data of the person and the position data of the specific article as to whether or not the lying position is being performed in contact with the specific article; An abnormal state determination unit that determines that it is an abnormal state when it is determined by the lying position determination unit that the person is in a lying position and it is determined by the article contact determination unit that there is no contact; An informing unit that issues a warning when the abnormal state determination unit determines an abnormal state, and The article contact determination unit estimates a region in pixel units for a person and a specific article using a segmentation algorithm, and determines whether or not the two are in contact based on the estimation result, Taking into account depth information in both the determination of the lying position state by the lying position determination unit and the determination by the article contact determination unit as to whether or not the lying position state is being performed in contact with the specific article An abnormal state detection system characterized by the above.

2. In the abnormal state detection system according to Claim 1, Either or both of the bounding box coordinate values and the skeleton estimation coordinate values of the person are used for the determination by the lying position determination unit, and either or both of the bounding box coordinate values and the skeleton estimation coordinate values of the person and the bounding box coordinate values of the article are used for the determination by the article contact determination unit An abnormal state detection system characterized by the above.

3. In the abnormal state detection system according to Claim 1 or 2, The informing unit does not issue a warning when the person is of a specific age estimated by an age estimation algorithm An abnormal state detection system characterized by the above.

4. In the abnormal state detection system according to Claim 1 or 2, Processing related to the image is not transferred to a cloud computer, but is completed within an edge device, and when the informing unit informs an image, it is limited to an image determined to be an abnormal state An abnormal state detection system characterized by the above.

5. In the abnormal state detection system according to Claim 1 or 2, In the detection unit, adjusting the number of images per unit time according to the measured power consumption and temperature of the computer device An abnormal state detection system characterized by the above.

6. A detection unit that detects a person and a specific article based on an image captured by at least one imaging unit that images a monitoring location which is a location to be monitored; A lying position determination unit that determines whether or not a person detected by the detection unit is in a lying position based on the image data of the person; An article contact determination unit that determines a contact state between the person and the article based on a spatial relationship between the position data of the person and the position data of the specific article as to whether or not the lying position is being performed in contact with the specific article; An abnormal state determination unit that determines that it is an abnormal state when it is determined by the lying position determination unit that the person is in a lying position and it is determined by the article contact determination unit that there is no contact; And a notification unit that issues a warning when the abnormal state determination unit determines an abnormal state. The article contact determination unit estimates a region in pixel units for the person and the specific article using a segmentation algorithm, and determines whether or not the two are in contact based on the estimation result. Taking into account depth information in both the determination of the lying position state by the lying position determination unit and the determination of whether or not the lying position state is being performed in contact with the specific article by the article contact determination unit. An abnormal state detection device characterized by the above.

7. A computer, Detection means for detecting a person and a specific article based on an image captured by at least one imaging unit that images a monitoring location which is a location to be monitored; Lying position determination means for determining whether or not a person detected by the detection means is in a lying position based on the image data of the person; Article contact determination means for determining a contact state between the person and the article based on a spatial relationship between the position data of the person and the position data of the specific article as to whether or not the lying position is being performed in contact with the specific article; Abnormal state determination means for determining that it is an abnormal state when it is determined by the lying position determination means that the person is in a lying position and it is determined by the article contact determination means that there is no contact; And causing it to function as notification means for issuing a warning when the abnormal state determination means determines an abnormal state. The article contact determination means estimates a region in pixel units for the person and the specific article using a segmentation algorithm, and determines whether or not the two are in contact based on the estimation result. An abnormal state detection program characterized by taking into account depth information in both the determination of the lying position state by the lying position determination means and the determination by the article contact determination means as to whether the lying position state is in contact with the specific article.

8. A step in which a detection unit detects a person and a specific article based on an image captured by at least one imaging unit that captures an image of a monitoring location that is a location to be monitored; A step in which a lying position determination unit determines whether the person detected by the detection unit is in a lying position state based on the image data of the person; A step in which an article contact determination unit determines the contact state between the person and the article based on the spatial relationship between the position data of the person and the position data of the specific article as to whether the lying position state is in contact with the specific article; A step in which an abnormal state determination unit determines that it is an abnormal state when it is determined by the lying position determination unit that it is in a lying position state and it is determined by the article contact determination unit that there is no contact; A step in which a notification unit issues a warning when the abnormal state determination unit determines that it is an abnormal state, comprising The article contact determination unit estimates a pixel-level region for the person and the specific article using a segmentation algorithm, and determines whether or not the two are in contact based on the estimation result. Taking into account depth information in both the determination of the lying position state by the lying position determination unit and the determination by the article contact determination unit as to whether the lying position state is in contact with the specific article characterizes an abnormal state detection method.

Images