Method for detecting abnormal state through image analysis

Abstract

The present invention may provide a method for detecting an abnormal state through image analysis, comprising: an extraction step of extracting continuous frames from an image stream; an information collection step of detecting and tracking an object from the extracted continuous frames using artificial intelligence, and estimating a pose of the object to collect object information about the object; a data extraction step of extracting geometric data of the object from the object information; a preprocessing step of preprocessing the extracted geometric data; and a fall determination step of determining whether the object has fallen on the basis of the geometric data. According to the above, the accuracy of fall detection can be greatly improved by analyzing various geometric data points.

Description

Method for detecting abnormal conditions through image analysis

[0001] The present invention relates to a method for detecting an abnormal condition through image analysis, and more specifically, to a method for detecting an abnormal condition through image analysis that can accurately recognize whether or not a person has fallen by utilizing a person's posture angle data and reduce the error rate.

[0002] Recently, with the advent of an aging society and an increasing proportion of the population aged 65 and older in the total population, research on health care and health management for the elderly is being actively conducted.

[0003] In particular, research on systems that detect falls in the elderly is being conducted from various perspectives, including sensor-based fall detection systems and complex event processing-based fall detection systems.

[0004] Here, the sensor-based fall detection system determines whether an elderly person has fallen by utilizing various sensors such as an acceleration sensor, a smartphone, and a pressure sensor installed on the floor. In addition, there is also a method of implementing a rule-based fall detection methodology using a complex event processing system.

[0005] However, conventional fall detection systems have limitations in accurately detecting falls in complex environments and have high error rates because they cannot detect all falls or rely on simple motion detection or basic image processing technology.

[0006] The present invention aims to provide a method for detecting abnormal conditions through image analysis, which can accurately recognize falls using artificial intelligence based on detected human posture angle data, improve various machine learning models, and reduce the error rate.

[0007] The present invention can provide an abnormality detection method through image analysis, characterized in that it includes an extraction step of extracting continuous frames from an image stream; an information collection step of detecting and tracking an object from the extracted continuous frames using artificial intelligence, estimating the pose of the object, and collecting object information about the object; a data extraction step of extracting geometric data of the object from the object information; a preprocessing step of preprocessing the extracted geometric data; and a fall determination step of determining whether the object has fallen based on the geometric data.

[0008] The above information collection step can track the object, distinguish it as a person or an object, and assign a unique ID to the person, thereby collecting object information including the unique ID for the object, location information, and coordinate information of the body part.

[0009] In the above data extraction step, the geometric data of the object may include the object's center of mass, the object's shoulder-nose angle, the object's torso angle, the hip angle, the shoulder angle, the leg angle, and the nose-ankle angle.

[0010] Here, the fall determination step may include a step of determining whether the object is lying down through the geometric data, and a step of determining that the object has fallen if the fall conditions are satisfied when the object is lying down.

[0011] Here, the fall condition may include whether the object is in a lying position continuously for a set period of time or longer, whether the object is in a lying position for a set number of consecutive frames, and whether the transition speed from a standing state to a lying state is a set speed.

[0012] The method for detecting abnormal conditions through image analysis according to the present invention can significantly improve the accuracy of fall recognition and also reduce the error rate by analyzing posture angle data such as the center of mass of the detected person, shoulder-nose angle, and waist angle.

[0013] In addition, the method for detecting abnormal conditions through image analysis according to the present invention can detect changes in a person's posture and position in real time by analyzing a video stream from a CCTV camera, and if a fall is suspected, it can determine whether a fall has occurred through additional analysis and take appropriate measures.

[0014] In addition, the method for detecting abnormal conditions through image analysis according to the present invention enables more accurate and efficient monitoring, can provide an effective method for elderly care, and can significantly improve emergency response capabilities.

[0015] In addition, the method for detecting abnormal conditions through image analysis according to the present invention can provide an effective fall detection solution that can be implemented in various environments to improve safety and emergency response, and can provide a high level of accuracy in fall detection by integrating machine learning algorithms and real-time data processing, thereby serving as a useful tool for mitigating risks associated with unexpected accidents.

[0016] Figure 1 is a flow chart showing a method for detecting an abnormal condition through image analysis according to an embodiment of the present invention.

[0017] FIG. 2 is a drawing showing examples of various possible locations and sizes of objects detected in a method for detecting an abnormal state through image analysis according to an embodiment of the present invention.

[0018] FIG. 3 and FIG. 4 are drawings showing the importance of geometric data in a method for detecting an abnormal state through image analysis according to an embodiment of the present invention.

[0019] Hereinafter, a preferred embodiment of the present invention will be described with reference to the attached drawings to enable a more specific understanding of the present invention.

[0020] The present invention is susceptible to various modifications and embodiments. Specific embodiments are illustrated and described in detail in the drawings. However, this is not intended to limit the present invention to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present invention.

[0021] While terms such as "first" and "second" may be used to describe various components, these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component." The term "and / or" includes a combination of multiple related items described herein or any of multiple related items described herein.

[0022] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0023] The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0024] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0025] Hereinafter, with reference to the attached drawings, preferred embodiments of the present invention will be described in more detail. In order to facilitate an overall understanding in describing the present invention, identical reference numerals are used for identical components in the drawings, and redundant descriptions of identical components are omitted.

[0026] Hereinafter, a method for detecting an abnormal condition through image analysis according to an embodiment of the present invention will be examined.

[0027] First, in the present invention, the abnormal condition may include a fall, and below, a method for detecting a fall will be examined.

[0028] In the method for detecting an abnormal condition through image analysis according to an embodiment of the present invention, it is preferable to extract and observe continuous frames (multiple frames) from an image stream, because detecting a fall by relying on a single frame may result in inaccuracy.

[0029] Here, the process of extracting continuous frames can be performed by recording a continuous video stream in real time from a video device such as a CCTV camera and then converting it into multiple frames.

[0030] When continuous frames are extracted in this way, object information about the object can be collected, such as detecting and tracking an object from the extracted continuous frames using artificial intelligence, estimating the pose of the object to identify the movement of the object, and extracting the direction and position of the body parts of the object (person).

[0031] Here, object detection and tracking can be processed using a deep learning model called YOLOv8, an object detection algorithm that takes efficiency and accuracy into account, which can significantly reduce processing time.

[0032] Meanwhile, the above-mentioned deep learning model can perform object detection, object estimation, and pose estimation as described above.

[0033] First, when a deep learning model is given a frame, it tries to recognize and locate the exact location of the (human) object in the frame, and can detect the object by identifying all instances of the human object and drawing a bounding box around each individual object.

[0034] By detecting human objects within a frame and marking them with bounding boxes, each object is tracked as an individual instance. A deep learning model can then assign a unique ID to each detected human object, enabling it to track its movement across multiple frames. Object tracking is required for multiple-fall detection.

[0035] At this time, the deep learning model can assign a unique ID to each object, such as "person1", "person2", etc., when there are multiple person objects in the frame.

[0036] Meanwhile, the purpose of tracking each object in this way is to distinguish between multiple people and objects and to determine which objects have fallen and which have not.

[0037] Additionally, the deep learning model can estimate poses by identifying all visible body parts of each object after obtaining the unique ID and location of each person-object.

[0038] The above-mentioned pose estimation can be estimated by recognizing body parts, which can be called “key points,” and the deep learning model can generate body part coordinates (x, y) for each key point as part of the output.

[0039] Here, key points can include nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0040] As described above, pose estimation can be performed by having the model identify 17 body parts of each human object known as key points, calculate the (x, y) coordinates of the corresponding parts, and classify the pose of the human object into a standing pose and a lying pose through the geometric information of each object.

[0041] As described above, object detection, tracking, and pose estimation using a deep learning model are for collecting object information about the object, and such object information may include a list of unique IDs, location information, and keypoint coordinate information generated by assigning a unique ID to each object found in an input frame.

[0042] Once object information for each object is collected in this way, geometric data can be extracted from the object information (keypoint coordinates), and this information can be used to classify the posture of the human object into a 'standing posture' and a 'lying posture'.

[0043] Meanwhile, adopting the approach of inputting keypoint coordinates into an AI model to classify and then assigning each human object to either a "standing" or "lying" posture can lead to overfitting and bias in the model, which may mean that the model may not perform optimally on unseen data sets at production time.

[0044] This can be confirmed even in separate frames where the human object is in the same falling posture, as shown in Fig. 2, but the position and size of the object may nevertheless differ in each frame.

[0045] In other words, when a model is used to determine whether a person is standing or lying down based on data including ID, location, and keypoints as described above, there is a risk that the model may overfit and its efficiency for unseen data may decrease if the location and size of the person differ depending on the frame. Therefore, it is desirable to increase the adaptability of the model by normalizing the input by calculating various geometric points.

[0046] Therefore, in order to resolve these deformations and increase similarity, it is necessary to preprocess keypoint information and normalize it into geometric data.

[0047] This preprocessing step can be taken by extracting a series of geometric data from pose coordinates to normalize the input data regardless of the change in the object bounding box size and location, and then evaluating, testing and comparing the classification results to effectively normalize the keypoint data regardless of the location and size of the bounding box.

[0048] Meanwhile, the geometric data of the object may include the object's center of mass, the object's shoulder-nose angle, the object's torso angle, the object's hip angle, the object's shoulder angle, the object's leg angle, and the object's nose-ankle angle.

[0049] First, the center of mass of an object represents the location where the mass of the human object is concentrated, representing the average position of all keypoints. This center of mass is the point within the human object where the entire mass can be considered concentrated. By combining mass and position, this center of mass can represent the average position of all individual keypoints.

[0050] The following mathematical formula 1 is a mathematical formula for calculating the center of mass (Cx, Cy) of an object.

[0051] Mathematical formula 1

[0052]

[0053] Here, and represents the sum of all x and y values ​​for all keypoints respectively, and n represents the total number of detected keypoints.

[0054] Next, the shoulder-nose angle of the object can be calculated as the angle between three points, specifically the left shoulder, nose, and right shoulder, expressed as vectors in 2D space, as in Equation 2.

[0055] First, A, B, and C can be defined as follows.

[0056]

[0057] And, vector and The angle θ between them can be calculated as in mathematical equation 2.

[0058] Mathematical formula 2

[0059]

[0060]

[0061]

[0062] Next, the torso angle can be calculated as the angle between the torso and a vertical reference line based on coordinates specified for the midpoint of the nose and buttocks. The torso angle is useful for assessing the body's upward orientation and can be calculated using Equation 3.

[0063] Mathematical formula 3

[0064]

[0065] Here, and angle between The calculation process is as follows.

[0066]

[0067]

[0068] The hip angle can be obtained by calculating the angle between the left and right hips and a horizontal reference line based on the given coordinates, and can measure the orientation or alignment of the hips with respect to the horizontal axis of the coordinate system.

[0069] The calculation formula and process for this hip angle are as follows.

[0070]

[0071] Here, and angle between The calculation process is as follows.

[0072]

[0073]

[0074]

[0075]

[0076]

[0077]

[0078] The shoulder angle can be obtained by calculating the angle between the left shoulder and the right shoulder and the horizontal reference line based on the given coordinates, and the direction or alignment of the shoulder can be measured based on the horizontal axis of the coordinate system, and the calculation formula and calculation process are as follows.

[0079]

[0080] Here, and angle between The calculation process is as follows.

[0081]

[0082]

[0083]

[0084]

[0085]

[0086] The leg angle can be obtained by calculating the angle between the upper and lower parts (ankle-knee-hip) of the left and right legs based on the given coordinates, and the calculation formula and calculation process are as follows.

[0087]

[0088]

[0089]

[0090]

[0091]

[0092] The nose-ankle angle can be obtained by calculating the angle between a vector pointing from the nose to the midpoint of the ankle and a vertical reference line based on the given coordinates. The direction or alignment of this vector relative to the vertical axis of the coordinate system can be measured. Furthermore, this angle helps standardize keypoint data, making standing pose classification more robust and less sensitive to variations across individual frames. The calculation formula and process are as follows.

[0093]

[0094]

[0095]

[0096] As described above, for each frame and all individual objects within that frame, the pose can be classified as standing or lying based on the geometric data extracted in the previous step.

[0097] At this point, multiple classifiers can be trained using the input data, including support vector machines, random forests, decision trees, gradient boosting, and multilayer perceptrons, and these classifiers can be fine-tuned using specific hyperparameters as follows:

[0098] Here, the final decision on fall detection can be based on the following steps. First, pose classification can be performed based on a single frame. The pose classification can then be used to classify each human pose into one of two classes (standing or lying down). Since machine learning is used as the backbone, a numerical threshold value representing the pose classification for each geometric data point cannot be determined. Therefore, the importance of each geometric data point can be indicated as shown in Figures 3 and 4.

[0099] Afterwards, the AI ​​model (classifier) ​​can move on to the important step of classifying each individual's posture into standing and lying postures and then determining whether or not they have fallen.

[0100] Accordingly, if the person's posture in the current frame is not classified as lying down, the model simply moves on to analyze the next frame, and if a lying down posture is detected in the current frame, the system (model) can determine a fall based on the following three criteria.

[0101] That is, the above-described fall judgment determines whether the object is standing, and if it is determined that the object is standing, it determines whether the object is lying down through the geometric data, and if the object is lying down and the object satisfies the set fall conditions, it can be determined that the object has fallen.

[0102] At this time, the above-mentioned fall condition may include whether the object is in a lying position continuously for a set time or longer (time threshold), whether the object is in a lying position for a set number of consecutive frames (number of frames), and whether the transition speed from a standing state to a lying state is a set speed (posture change speed).

[0103] At this time, if these three judgment criteria are met, it can be determined that there was a fall, and if not, it can be concluded that there was no fall in order to repeat the task in the next frame.

[0104] Meanwhile, in the above judgment criteria, the time threshold is the minimum time for which a person must be classified as lying down before a fall is detected, and the setting time can be set to various values, such as approximately 5 seconds or 10 seconds.

[0105] Additionally, in the above judgment criteria, the number of frames may be chosen to define fall detection based on the number of consecutive frames classified as a "lying posture" by the user instead of a time-based criterion, and the threshold for the number of frames may be set to a specific number of frames, such as 200 or 500, depending on the user's needs.

[0106] Additionally, the posture change speed in the above judgment criteria can be configured by measuring in frames or seconds how quickly a person changes from a standing position to a lying position, and this criterion can help distinguish between controlled lying down and sudden falls based on a user-defined rate of change.

[0107] Meanwhile, the present invention inputs the preprocessed geometric data described above into a machine learning classifier such as a random forest, support vector machine, decision tree, gradient boosting, or deep learning model to initially determine whether a person is standing or lying down.

[0108] As described above, first, unlike the existing method that relies on a machine learning algorithm for fall activity detection, the present invention directly uses general human posture data to train a machine learning or deep learning model, thereby independently classifying each human object in the frame as falling or not falling.

[0109] In addition, unlike the existing approaches based on fall detection based on single frame analysis, which tend to be inefficient and lower the system accuracy, the present invention first improves the generalization of input data by utilizing geometric data extracted from general pose information before training various classifiers, thereby determining whether the current pose of each frame is standing or lying down, and then comparing it with a predefined criterion to confirm a fall, thereby preventing a momentary judgment based on a single frame, and by considering multiple frames before completing fall detection, the detection performance can be greatly improved and the accuracy of fall detection can be increased.

[0110] While the present invention has been described with reference to the embodiments illustrated in the drawings, these are merely exemplary, and those skilled in the art will appreciate that various modifications and equivalent alternative embodiments are possible. Therefore, the true scope of technical protection of the present invention should be determined by the technical spirit of the appended claims.

[0111] The present invention can be applied to health-related industries.

Claims

1. An extraction step for extracting continuous frames from an image stream; An information collection step of detecting and tracking an object from the sequential frames extracted using artificial intelligence, estimating the pose of the object, and collecting object information about the object; A data extraction step for extracting geometric data of the object from the object information; A method for detecting an abnormal condition through image analysis, characterized by including a fall judgment step for determining whether the object has fallen based on the geometric data above.

2. In paragraph 1, The above information collection step is: A method for detecting an abnormal condition through image analysis, characterized in that the method tracks the object, distinguishes it as a person and an object, assigns a unique ID to the person, and collects object information including the unique ID for the object and location information and coordinate information of the body part.

3. In paragraph 1, In the above data extraction step, the geometric data of the object is An abnormality detection method through image analysis, characterized in that it includes the object's center of mass, the object's shoulder-nose angle, the body angle, the hip angle, the shoulder angle, the leg angle, and the nose-ankle angle.

4. In paragraph 1, The above fall judgment step is, A step of determining whether the object is lying down through the above geometric data, A method for detecting an abnormal condition through image analysis, characterized in that it includes a step of determining that the object has fallen and fallen if the fall conditions are satisfied when the object is lying down.

5. In paragraph 4, The above fall conditions are: Whether the above object is in a lying position for a longer period of time than the set time, Whether the above object is in a lying position for a set number of consecutive frames, An abnormal condition detection method using image analysis, characterized in that it includes whether the transition speed from a standing state to a lying state is a set speed.

Images