Fall detection method and device based on human posture estimation

Abstract

This invention provides a fall detection method and apparatus based on human body posture estimation. [Solution] The method includes: using a neural network to recognize key points on a human body in an image to obtain key points and bounding boxes of the human body's skeleton; estimating and classifying the posture of the human body based on the obtained key points and bounding boxes to obtain a classified posture; arranging the classified postures in chronological order to form a posture sequence; determining whether the human body is lying on its side based on the posture sequence; and, when the human body is lying on its side, determining the direction of the human body's lateral position in the posture sequence based on the image, determining the direction of the human body's lateral position and the posture sequence, and determining whether the human body has fallen in that direction.

Description

Technical Field

[0001] The present invention relates to the technical field of motion recognition.

Background Art

[0002] Fall down detection is a technology that estimates the posture of the human body and detects and analyzes movements to recognize and predict falls. Currently, the mainstream fall down detection methods can be classified into wearable device-based, environmental sensor-based, vision-based posture estimation, etc. The vision-based fall down detection method has already been proven effective in dealing with falls. Some advanced vision-based fall down detection methods utilizing neural networks have shown excellent detection effects for falls in video clips.

Summary of the Invention

Problems to be Solved by the Invention

[0003] However, the inventors have discovered the following. That is, the current fall down detection method using vision-based human body posture estimation is difficult to be widely applied due to the complexity of algorithms or models, or due to hardware limitations.

[0004] In order to address at least one of the above technical problems, an object of the present invention is to provide a fall down detection method and device based on human body posture estimation that can reduce the complexity of fall down detection, improve the performance of fall down detection, and ensure a better fall down detection effect.

Means for Solving the Problems

[0005] According to one aspect of an embodiment of the present invention, a fall down detection device based on human body posture estimation is provided, which includes: An acquisition unit that uses a neural network to recognize key points of a human body in an image to obtain the key points and bounding boxes of the human body's skeleton; A classification unit that performs estimation and classification of the human body's posture based on the acquired key points and boundary frame to obtain a classified posture; A first judgment unit that arranges the aforementioned classified postures in chronological order to form a posture sequence and determines whether the human body is in a lying-down position based on the posture sequence; and The system includes a second determination unit that, when the human body is in a lying-down position, determines the direction of the lying-down position of the human body in the posture sequence based on the image, and determines whether the human body has fallen in the direction based on the direction of the lying-down position of the human body and the posture sequence.

[0006] According to another aspect of the embodiments of the present invention, a fall detection method based on human body posture estimation is provided, which is, Using a neural network, keypoint recognition is performed on the human body in the image to obtain the keypoints and bounding box of the human body's skeleton; Based on the acquired keypoints and boundary frame, estimation and classification of the human body's posture are performed to obtain a classified posture; The aforementioned classified postures are arranged in chronological order to form a posture sequence, and it is determined whether the human body is in a lying-down position based on the posture sequence; and The method includes determining the direction of the body's lateral position in the posture sequence based on the image when the body is lying on its side, and determining whether the body has fallen in the direction based on the direction of the body's lateral position and the posture sequence.

[0007] According to yet another aspect of the embodiments of the present invention, an electronic device is provided which includes a memory and a processor connected to the memory, wherein the memory stores a computer program, and the processor is configured to execute the computer program to realize the aforementioned tipping detection method. [Effects of the Invention]

[0008] The advantageous effects of the embodiments of the present invention are at least as follows: by arranging the classified human body postures in chronological order to form a posture sequence, and by determining the human body's fall and the direction of the fall based on the posture sequence, it is possible to reduce the complexity of fall detection, improve the performance of fall detection, and ensure a better fall detection effect. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows a method for detecting tipping over in an embodiment of the present invention. [Figure 2] This figure shows the key points and boundary frame of the human skeleton in an embodiment of the present invention. [Figure 3] This figure shows the posture sequence in an embodiment of the present invention. [Figure 4] This figure shows a human body lying on its side in different directions according to an embodiment of the present invention. [Figure 5] This figure shows the calculation of the ratio of the upper body to the lower body in an embodiment of the present invention. [Figure 6] This figure shows a human body falling forward in an embodiment of the present invention. [Figure 7] This figure shows a human body falling to the side or backward in an embodiment of the present invention. [Figure 8] This is a flowchart for tipping detection in an embodiment of the present invention. [Figure 9] This figure shows a tipping detection device according to an embodiment of the present invention. [Figure 10] This figure shows an electronic device in an embodiment of the present invention. [Modes for carrying out the invention]

[0010] Hereinafter, preferred embodiments for carrying out the present invention will be described in detail with reference to the attached drawings. Note that these embodiments are merely illustrative and do not limit the present invention.

[0011] In an embodiment of the present invention, the object of motion recognition or fall detection may be human bodies of various age groups. For example, it may be the elderly, children, the elderly and / or caregivers, children and / or guardians, etc. Note that the present invention is not limited thereto, and the object to be detected may be a human body or other moving object having biological characteristics, a machine having no biological characteristics, etc.

[0012] <Embodiment of the first aspect> The present invention provides a fall detection method based on human body pose estimation. FIG. 1 is a diagram showing the fall detection method in an embodiment of the present invention. As shown in FIG. 1, the method includes the following steps, that is, Step 101: Use a neural network to recognize key points of the human body in the image to obtain the key points and boundary frames of the human body skeleton; Step 102: Perform estimation and classification on the pose of the human body based on the obtained key points and the boundary frames to obtain a classified pose (the pose after classification); Step 103: Arrange the classified poses in chronological order to form a pose sequence, and based on the pose sequence, determine whether the human body is in a lying state; and Step 104: When the human body is in a lying state, determine the lying direction of the human body in the pose sequence based on the image, and based on the lying direction of the human body and the pose sequence, determine whether the human body has fallen in the direction.

[0013] According to the above embodiment, by arranging the classified human body poses in chronological order to form a pose sequence, and determining the fall and fall direction of the human body based on the pose sequence, the complexity of fall detection can be reduced, the performance of fall detection can be improved, and a better fall detection effect can also be ensured.

[0014] Note that the above Figure 1 is only for exemplarily explaining the embodiments of the present invention, and the present invention is not limited thereto. For example, the execution order between each step can be appropriately adjusted, or several steps can be appropriately increased or decreased. A person skilled in the art can make appropriate modifications based on the above content without being limited to the description of Figure 1 above.

[0015] Figure 2 is a diagram showing the key points and bounding boxes of the human body skeleton in an embodiment of the present invention.

[0016] In some embodiments, in step 101, the shoulders, hips, knees, feet, hands, etc. of the human body are recognized as key points of the human body, and a bounding box including all the key points can be obtained. Alternatively, other joint points of the human body can be selected as key points for recognition. As shown in Figure 2, for example, the feet, knees, hips, shoulders, etc. of the human body are selected as key points of the human body, and a bounding box 201 that may include these key points is obtained.

[0017] In some embodiments, in step 102, based on the obtained key points and bounding box, an estimation and classification are performed on the posture of the human body to obtain a classified posture. For example, the classified posture may be stand, walk, bend, sit, squat, lie, up body, down body, etc. Among them, up body means that the lower body of the human body is hidden or only the upper body of the human body can be seen in the image, and down body means that the upper body of the human body is hidden or only the lower body of the human body can be seen in the image. For each classified posture, it may be obtained based only on the current frame of the classified posture, that is, it may not depend on the frames before and after the current frame. As shown in Figure 2, based on the obtained key points and bounding box 201 of the human body, the posture of the human body in the image is estimated as crawl, and the classified posture of the image can be obtained as crawl.

[0018] Figure 3 shows the posture sequence in an embodiment of the present invention.

[0019] In some embodiments, step 103 arranges the classified poses in chronological order to form a pose sequence. As shown in Figure 3, the classified poses are placed into the pose sequence in chronological order, and these classified poses may be various types of classified poses as described above. For the same human body pose sequence, if the human body or its classified pose is not detected within a given frame, the classified pose for that frame is set to "None".

[0020] In some embodiments, classified poses are placed into corresponding pose sequences in chronological order, and when the buffer size of the pose sequence is exceeded, the pose furthest from the current frame is deleted using a "first-in, first-out" method. Taking Figure 3 as an example, when classified pose 301 is placed into the pose sequence, the buffer size of the pose sequence is exceeded. At this point, the classified pose 302 furthest from the frame to which the current classified pose 301 belongs is deleted using a "first-in, first-out" method, that is, the classified pose 302 that was placed earliest (first) in the current pose sequence is deleted.

[0021] In some embodiments, one pose sequence stores the classified pose of one person, and the buffer size of one pose sequence may be set according to the actual situation, and the buffer size may be a fixed value or a variable value, for example, the buffer size may be related to the number of frames per second (frame rate) (FPS, Frames Per Second) of the video.

[0022] In some embodiments, step 103 determines whether the human body is in a lying position based on the posture sequence.

[0023] When the posture sequence satisfies the first condition, it is determined that the human body is in a lying-down position. The first condition is that among the N postures closest to the current time in the posture sequence, the number of final postures (ending postures) is greater than the first threshold, where N is a positive integer.

[0024] In some embodiments, the first condition may further include the following: that, in the posture sequence, the number of final postures among the N postures closest to the current time is M2 or greater, and these N postures include only a few specific postures, in which case it is determined that the human body is in a lying-down position.

[0025] If either of the two cases described above occurs, the human body is considered to be in a recumbent position.

[0026] For example, the posture after a fall may be set as the final posture, and depending on the classification criteria, the final posture may be lying down (lie), crawling (crawl), etc. In this invention, the explanation will be given with lying down as the final posture, and the explanation will be given using the example that the human body is in a lying-down state when the first condition is satisfied.

[0027] In some embodiments, the N poses include only a few specific poses, which may be "lie," "None," "up body," "down body," etc., but a reasonable specific pose may be set according to the actual needs.

[0028] In the above embodiment, N, M1, and M2 are all positive integers, and reasonable values ​​may be set depending on the actual usage scenario.

[0029] When it is determined that the human body is lying on its side, the direction of the body's lateral position is further determined.

[0030] In some embodiments, determining the direction of the human body lying down in the final position based on the image may include the following: When a person is lying on their side with their head facing forward (for example, towards the head of a bed), the direction of the person's lateral position is determined to be forward-facing; When a person is lying on their side with their head facing to the side (for example, to the side of a bed), the direction of the person's lateral position is determined to be lateral; and When a person is lying on their side with their head facing backward (for example, towards the foot of the bed), the direction of the person's lying position is determined to be backward.

[0031] Figure 4 shows an embodiment of the present invention in which a human body is lying on its side in different directions.

[0032] In some examples,

[0033]

number

[0034]

number

[0035] for example,

[0036]

number

[0037] As shown in Figure 4, due to the aforementioned angle range,

[0038]

number

[0039]

number

[0040]

number

[0041]

number

[0042] In addition to the angle range described above, other angle ranges may be set according to the needs of the specific scenario to determine the direction of the human body lying down. Furthermore, instead of dividing the direction of the human body lying down into three types, it is also possible to divide the direction of lying down into multiple other types according to the specific needs, and set the angle range according to the specific needs to determine the corresponding direction of lying down.

[0043] In some embodiments, when the human body is in a lying position, it is determined whether the human body satisfies a second condition, which is defined as follows: that, starting from the N+1th posture closest to the current time in the posture sequence, or the last final posture in the first direction out of N postures, a window of length W1 is adopted and the body slides in the first direction, and the number n of intermediate postures found from the postures that enter the window is greater than the second threshold.

[0044] The window slides off when the human body satisfies the second condition, where n is a positive integer, the length of W1 is greater than the length of one frame in the posture sequence, and the first direction represents the direction away from the current time in the posture sequence.

[0045] Taking Figure 3 as an example, a window 303 whose length is greater than the length of one frame is adopted and slid in the first direction X to find an intermediate posture. For example, if N is 3, and window 303 starts sliding from the fourth classified posture in the first direction, or the last final posture in the first direction, the sliding of window 303 stops when the number of intermediate postures that fit within the window is greater than the second threshold, at which point the human body satisfies the second condition.

[0046] In the above-described embodiment, the intermediate posture includes sitting, squatting, or bending forward.

[0047] The following explains how to determine if a person has fallen forward when they are lying on their side with their body facing forward.

[0048] In several embodiments, when the direction of the human body lying on its side was forward, it was determined whether the human body satisfied the third condition, and the third condition expressed the following: When the human body satisfies the second condition described above, in the posture sequence, M3 starting postures can be found from W2 consecutive postures among the classified postures in the first direction of the window described above; and When the human body does not satisfy the second condition described above, in the posture sequence, it is possible to find M3 starting postures from W2 consecutive postures, which are either the N+1 postures closest to the current time, or the classification posture in the first direction of the last final posture in the first direction among the N postures. Of these, the ratio of the upper body to the lower body (half-body ratio) of the three initial postures described above satisfies the third threshold, and the third threshold is related to the average half-body ratio of the final posture, and M3 and W2 are all positive integers, and W2 is greater than M3.

[0049] Taking Figure 3 as an example, the human body satisfies the third condition when the second condition is met, that is, when there is an intermediate posture in window 303 in which the second threshold is met, and it is possible to find M3 starting postures from W2 consecutive postures in the first direction X of window 303. The human body satisfies the third condition when the human body does not satisfy the second condition, that is, when there are not enough intermediate postures in window 303, and it is possible to find the final posture closest to the first direction in the first direction of the fourth posture closest to the current time, or from the three postures closest to the current time, and then find M3 starting postures from W2 consecutive postures in the first direction of that final posture.

[0050] Figure 5 shows the calculation of the ratio of the upper body to the lower body in an embodiment of the present invention.

[0051] The ratio of the length of the upper body to the lower body in an image of the human body can be obtained by calculating the average distance from the center point of the neck or shoulder to both buttocks, and the average of the sum of the distance from both buttocks to the knees and the distance from the knees to the ankles, as shown in Figure 5. For example, the ratio of the length of the upper body to the lower body of a human body can be calculated using the following formula.

[0052]

number

[0053] In the above-described embodiment, the relative positions of key points in the human body can be calculated by comparing the ratio of the upper and lower body, allowing for the distinction between several easily confused specific postures, such as "lying down" and "standing" / "walking." This enables the evaluation of the reliability of the posture, thereby further improving the accuracy of fall detection.

[0054] In some embodiments, the starting posture includes standing or walking.

[0055] In the above-described embodiment, the starting posture may be sitting, squatting, or bending forward, in addition to standing and walking. However, sitting, squatting, or bending forward in this case does not indicate that the human body is actually in that state, but may be an approximate movement that occurred when the human body lost balance before falling. In such cases, intermediate movements should not be considered, and only the starting and final postures should be considered; that is, the second condition is not considered.

[0056] In several embodiments, when the direction of the human body lying on its side is forward, it is determined whether the human body satisfies the fourth condition, and the fourth condition represents the following: The first condition is that, of the N postures in the posture sequence, the number of lateral lying postures facing forward is greater than the number of lateral lying postures facing other directions.

[0057] Of these, the first condition is satisfied when the number of final postures out of N postures is greater than the first threshold. Taking Figure 3 as an example, in the posture sequence shown in Figure 3, for example, if N is 3, and the number of forward-facing lateral postures of the body among the three postures closest to the current time is greater than the number of lateral postures in other directions, then the body satisfies the fourth condition.

[0058] In several embodiments, when the direction of the human body lying on its side is forward, it is determined whether the human body satisfies the fifth condition, and the fifth condition represents the following: The first condition is that when the human body satisfies the first condition, the proportion of N postures in a posture sequence whose corresponding boundary frames are non-flattened exceeds the fourth threshold.

[0059] Of these, the first condition is that the number of final postures out of N postures is greater than the first threshold. Taking Figure 3 as an example, in the posture sequence shown in Figure 3, for example, if N is 3, and the proportion of the three postures closest to the current time in which the boundary frames corresponding to the three postures are non-flattened exceeds the fourth threshold, then the human body satisfies the fifth condition.

[0060] Among these, the meaning of a non-flattened boundary frame is as follows: a boundary frame can be considered non-flattened when its height is greater than its width, and the difference between its height and width is greater than a pre-set threshold. Taking Figure 5 as an example, if the height of the boundary frame 501 in the image shown in Figure 5 is greater than its width, and the difference between its height and width is greater than a pre-set threshold, then the boundary frame 501 is a non-flattened boundary frame.

[0061] In the example above, the boundary frame may be a frame that includes the entire human body or all key points. As shown in Figure 2, the boundary frame 201 includes all key points of the human body, and in some special cases, the boundary frame does not have to include all key points of the human body. For example, when the human body loses its center of gravity and extends its arms, the arms affect the height and width of the boundary frame, so the boundary frame does not have to include the arms or key points on the arms.

[0062] In the above embodiment, the fourth threshold may be a different value pre-set according to the needs of different scenarios.

[0063] In some embodiments, when the direction of the human body's lateral position is forward, and the human body satisfies the third, fourth, and fifth conditions described above, it is determined that the human body has fallen forward.

[0064] Figure 6 shows how a forward fall of the human body occurs in an embodiment of the present invention.

[0065] As shown in Figure 6, when the human body satisfies all of the first through fifth conditions, it is determined that a forward transfer of the human body has occurred. Of these, the second condition may be ignored if the starting posture is sitting, squatting, or bending forward. Furthermore, all of the third through fifth conditions presuppose that "the human body is in a lying position," which implicitly includes the first condition. Therefore, in order to determine that a forward inversion of the human body has occurred, at least the third, fourth, and fifth conditions must be satisfied.

[0066] Furthermore, even if the direction of the human body's lateral position is forward, if the human body does not satisfy all of the third, fourth, and fifth conditions mentioned above, it may be determined whether the human body has fallen forward based on other conditions.

[0067] In several embodiments, when the direction of the human body's lateral position is forward, and the human body does not satisfy all of the third, fourth, and fifth conditions, it is determined whether the human body satisfies the sixth condition, and the sixth condition represents the following: When the human body satisfies the second condition, in the posture sequence, M4 starting postures can be found from W3 consecutive postures among the classified postures in the first direction of the intermediate posture; and When the human body does not satisfy the second condition, in the posture sequence, it is possible to find M4 starting postures from W3 consecutive postures among the classified postures in the first direction of the final posture. Of these, W3 and M4 are both positive integers, and W3 is greater than M4.

[0068] Taking Figure 3 as an example, the sixth condition represents the following: when the human body satisfies the second condition, that is, when there is an intermediate posture in window 303 that satisfies the second threshold, M4 starting postures can be found from W3 consecutive postures among the classified postures in the first direction X of window 303. Also, when the human body does not satisfy the second condition, that is, when the number of intermediate postures in window 303 does not satisfy the second threshold, M4 starting postures can be found from W3 consecutive postures among the classified postures in the first direction of the final posture, that is, among the classified postures in the first direction of the three postures closest to the current time. For example, when W3 is equal to 6 and M4 is equal to 3, 3 starting postures can be found from the 6 consecutive postures mentioned above.

[0069] In the above-described embodiment, the reliability of the starting posture can be determined by the posture score, or by the keypoint score and relative position.

[0070] In several embodiments, when the direction of the human body's lateral position is forward, and the human body does not satisfy all of the third, fourth, and fifth conditions, it is determined whether the human body satisfies the seventh condition, and the seventh condition represents the following: The tipping angle θ > D1, where D1 is a pre-set angle threshold based on the actual situation.

[0071] In the above embodiment, the tipping angle θ corresponds to the starting position that satisfies the sixth condition described above.

[0072]

number

[0073]

number

[0074]

number

[0075] In some embodiments, it is determined that a forward inversion of the human body has occurred when the direction of the human body's lateral position is forward and the human body satisfies all of the sixth and seventh conditions.

[0076] In the above embodiment, when the human body falls forward onto its side, the forward inversion can be determined by whether the human body satisfies the third, fourth, and fifth conditions, or by whether the human body satisfies the sixth and seventh conditions. It is sufficient if either one of these conditions is satisfied.

[0077] The following explains how to determine whether a person has fallen when they are lying on their side or backward.

[0078] In several embodiments, when the direction of the human body lying on its side was either to the side or backward, it was determined whether the human body satisfied the eighth condition, which is defined as follows: When the human body satisfies the second condition, in the posture sequence, M five starting postures can be found from W four consecutive postures among the classified postures in the first direction of the intermediate posture; and When the human body does not satisfy the second condition, in the posture sequence, it is possible to find M5 starting postures from W4 consecutive postures among the classified postures in the first direction of the final posture. Of these, W4 and M5 are all positive integers, and W4 is greater than M5.

[0079] In several embodiments, when the direction of the human body's lateral position is either lateral or posterior, it is determined whether the human body satisfies the ninth condition, which is defined as follows: The tipping angle θ > D2, where D2 is a pre-set angle threshold based on the actual situation.

[0080] Figure 7 shows a human body lying on its side in an embodiment of the present invention.

[0081] As shown in Figure 7, taking the example of a lateral tilt of the human body, the starting posture that satisfies the eighth condition corresponds to

[0082]

number

[0083]

number

[0084]

number

[0085]

number

[0086]

number

[0087]

number

[0088] In the above-described embodiment, the eighth and ninth conditions for determining whether the human body has fallen to the side or backward, and the sixth and seventh conditions for determining whether the human body has fallen forward, are all the same except that the preset numerical values ​​W4, M5 and W3, M4, and the setting of posture confidence can be set independently based on specific parameters such as video clarity, camera shooting angle, and camera shooting range (wide angle) in the actual situation. For other specific details of the eighth and ninth conditions, refer to the sixth and seventh conditions.

[0089] In some embodiments, when the direction of the human body's lateral position is to the side or backward, and the human body satisfies all eighth and ninth conditions, it is determined that a lateral or backward tilt of the human body has occurred.

[0090] As shown in Figure 7, when the human body satisfies all of the first, second, eighth, and ninth conditions, it is determined that a forward fall of the human body has occurred. Furthermore, when the starting posture is sitting, squatting, or bending forward, the consideration of the second condition may be omitted. Also, the eighth and ninth conditions all presuppose that "the human body is in a lying position," that is, they implicitly include the first condition. Therefore, in order to determine that a lateral or backward fall of the human body has occurred, at least the eighth and ninth conditions must be satisfied.

[0091] In some embodiments, a detection status mark is set on the human body to indicate whether fall detection is currently required for that human body, and the detection mark may include "falling," "skipping," and "checking."

[0092] When a person is marked as "fallen" by Mercury, the fall detection is skipped directly. Regarding "detection skip," if the system determines that the person is lying down but cannot determine that the person has fallen, the fall detection is skipped. "Detection skip" is an optional marking, meaning that the detection markings may include only two types: "fallen" and "detection required."

[0093] In some embodiments, when the attitude sequence has W5 attitudes closest to the current time and W6 or more starting attitudes, the status indicator is reset to "Detection Required", and both W5 and W6 are positive integers, with W5 being greater than W6.

[0094] Taking Figure 3 as an example, when W5 is equal to N, and in the attitude sequence, there are W6 or more starting attitudes in the N attitudes closest to the current time, the status marking is reset to "Detection Required", and of these, N and W6 are all positive integers, and N is greater than W6.

[0095] In the embodiments described above, in some cases, the reliability of posture may be further introduced to make additional decisions. For example, based on the conditions described above, the reliability of W5 postures may be calculated, a predetermined threshold may be set, and the status indicator may be reset to "Detection Required" if the threshold is satisfied.

[0096] In embodiments of the present invention, a corresponding confidence level may be introduced for all initial, intermediate, and final postures participating in fall detection to make further decisions. The confidence level of a posture can be calculated using a posture score or a keypoint score, for example, by using the relative position of keypoints, or by calculating the relative position of keypoints for several easily confused specific postures, for example, by calculating the ratio of the upper body to the lower body. Furthermore, the confidence level of a posture may be determined by a boundary frame detector and different posture classifier models, but the present invention is not limited to this. Figure 8 is a flowchart of fall detection in an embodiment of the present invention.

[0097] As shown in Figure 8, after obtaining the posture sequence, it is first determined whether it is necessary to perform fall detection on the human body. If the human body is marked as "detection required," it is determined whether the human body is lying on its side and in what direction based on the first condition, and then a fall is determined for the human body based on the different directions of lying on its side. When the human body is lying on its side facing forward, it is determined whether the human body satisfies all of the third, fourth, and fifth conditions. If all are satisfied, it is determined that the human body has fallen forward. If the human body does not satisfy all of the third, fourth, and fifth conditions, it is further determined whether the human body satisfies the sixth and seventh conditions. If all are satisfied, it is similarly determined that the human body has fallen forward. Furthermore, when the human body is lying on its side or backward, it is determined whether the human body satisfies the eighth and ninth conditions. If they are satisfied, it is determined that the human body has fallen on its side or backward.

[0098] In fall detection, if the human body is found to be unable to satisfy any one of the above conditions, that is, if the human body is determined to be in a lying position (i.e., the first condition is satisfied), but it is not possible to determine whether a fall has occurred, the human body is marked with "detection skipped".

[0099] In the above embodiment, when it is determined that a fall has occurred in the human body, or when the human body is marked as "detection skipped", it is further determined whether a marking reset is necessary for the human body. If necessary, the marking on the human body is reset to "detection required", and fall detection is performed again. If not necessary, a new posture sequence is acquired, and fall detection is performed on that posture sequence.

[0100] In the embodiments described above, one posture sequence can be used to perform fall detection on only one human body, but the present invention is not limited thereto. When it is necessary to perform fall detection on multiple human bodies, multiple posture sequences may be set up and the fall detection flow shown in Figure 8 may be performed simultaneously.

[0101] Table 1 illustrates some of the effects of the embodiments of the present invention and shows the performance when the methods of the embodiments of the present invention are applied to different detectors and classifiers.

[0102] [Table 1] As shown in Table 1, test 1 represents performing fall detection on a human body using Method 1 (e.g., using a human body detector model (YOLOX), a human body posture estimation model (CPN), and an original (conventional) classifier trained on the key points of CPN) and an original fall detection method; test 2 represents performing fall detection on a human body using Method 2 (e.g., using a human body detector model (YOLOX), a human body posture estimation model (CPN), and an original classifier trained on the key points of CPN) and a fall detection method described in an embodiment of the present invention; test 3 represents performing fall detection on a human body using Method 3 (e.g., using a human body posture estimation model (trtpose) and a posture classifier trained on the key points of trtpose) and a fall detection method described in an embodiment of the present invention; and test 4 represents performing detection on a human body using Method 4 (e.g., using a human body detector model (YOLOX), a human body posture estimation model (CPN), and an improved posture classifier) ​​and a fall detection method described in an embodiment of the present invention. Furthermore, "recall" represents the recall rate, i.e., the probability of a fall being effectively detected within the data, and "precision" represents the accuracy of the fall detection. NVR1, NVR2, NVR3, and NVR4 each represent four long videos of approximately 20 minutes (duration) containing multiple falls. The videos include a series of consecutive actions such as walking, falling, and getting up, allowing for thorough testing of the fall detection effectiveness of the present invention.

[0103] As can be seen from Table 1, the accuracy of tipping detection is effectively improved after using the method described in the embodiments of the present invention for different detectors and classifiers.

[0104] Tables 2 and 3 illustrate several other effects of the embodiments of the present invention, showing the impact of the methods in the embodiments of the present invention on the performance of tip-over detection.

[0105] [Table 2]

[0106] [Table 3] Using the YOLOV3 detector model as an example, Table 2 shows the performance of the fall detection system for different video files using the original fall detection method, and Table 3 shows the performance of the fall detection system for different video files using the fall detection method shown in the embodiment of the present invention. As can be seen from Tables 2 and 3, the fall detection method shown in the embodiment of the present invention improves the performance of fall detection while having little impact on the execution speed and storage of the test system (Actlyzer).

[0107] Although only steps or processes relating to the present invention have been described above, the present invention is not limited thereto. The tipping detection method may further include other steps or processes, and the specific details of these steps or processes can be found in the prior art. Furthermore, although several structures of the tipping detection model have been described above as illustrative examples of embodiments of the present invention, the present invention is not limited to these structures, and appropriate modifications may be made to these structures, and any method of carrying out these modifications is included within the scope of embodiments of the present invention.

[0108] The embodiments described above are for illustrative purposes to illustrate embodiments of the present invention, but the present invention is not limited thereto, and appropriate modifications may be made based on the embodiments described above. For example, the embodiments described above can be used individually, or a combination of several of the embodiments described above can be used.

[0109] As can be seen from the above-described embodiment, by arranging the classified human body postures in chronological order to form a posture sequence, and then determining the human body's fall and the direction of the fall based on the posture sequence, the complexity of fall detection can be reduced, the performance of fall detection can be improved, and a better fall detection effect can be ensured.

[0110] <Example of the second aspect> This invention provides a fall detection device based on human body posture estimation, and the description of the same content as in the first embodiment is omitted here.

[0111] Figure 9 shows a tipping detection device in an embodiment of the present invention. As shown in Figure 9, the tipping detection device 900 includes the following, namely, Acquisition unit 901: Using a neural network, it recognizes keypoints in the human body in the image and acquires the keypoints and boundary frame of the human body's skeleton; Classification unit 902: Based on the acquired key points and boundary frame, it performs estimation and classification on the posture of the human body to obtain a classified posture; First judgment unit 903: Arranges the classified postures in chronological order to form a posture sequence, and determines whether the human body is in a lying-down position based on the posture sequence; and Second judgment unit 904: When the human body is in a lying-down position, it determines the direction of the lying-down position of the human body in the posture sequence based on the image, and determines whether the human body has fallen over in the direction based on the direction of the lying-down position of the human body and the posture sequence.

[0112] In some embodiments, when the posture sequence satisfies a first condition, the first judgment unit 903 determines that the human body is in a lying-down position, where the first condition is that the number of final postures among the N postures closest to the current time in the posture sequence is greater than a first threshold, and N is a positive integer.

[0113] In some embodiments, the final posture indicates that the classified posture is recumbent or prone.

[0114] In some embodiments, the determination of the direction of the human body lying down by the second judgment unit 904 includes the following: Based on the aforementioned image, the direction of the human body lying on its side in the final posture is determined, If the human body is lying on its side with its head facing forward, it is determined that the direction of the human body's lateral position is lying on its side facing forward; When the head of the human body is lying on its side, it is determined that the direction of the human body's lateral position is to the side; and When the human body is lying on its side with its head facing backward, it is determined that the direction of the human body's lying position is backward.

[0115] In some embodiments, when the human body is in a recumbent position, the second judgment unit 904 determines whether the human body satisfies a second condition, the second condition being that the number of intermediate postures n found from the postures entering the window is greater than a second threshold, when the human body satisfies the second condition, the sliding of the window stops, in which case n is a positive integer, the length of W1 is greater than the length of one frame in the posture sequence, and the first direction is the direction away from the current time in the posture sequence.

[0116] In some embodiments, the intermediate posture includes sitting, squatting, or bending forward.

[0117] In some embodiments, when the direction of the human body lying on its side is forward, the second judgment unit 904 determines whether the human body satisfies the third condition, of which the third condition represents the following: When the human body satisfies the second condition, in the posture sequence, M3 starting postures can be found from W2 consecutive postures among the classified postures in the first direction of the window; and When the human body does not satisfy the second condition, in the posture sequence, it is possible to find M3 starting postures from W2 consecutive postures, which is the N+1 posture closest to the current time, or the classification posture in the first direction of the last final posture in the first direction among the N postures. Of these, the ratio of the upper body to the lower body in the three initial postures satisfies the third threshold. The third threshold is related to the average half-body ratio in the final posture. Of these, W2 and M3 are all positive integers, and W2 is greater than M3.

[0118] In some embodiments, the starting posture includes standing or walking.

[0119] In some embodiments, when the direction of the human body lying on its side is forward, the second judgment unit 904 determines whether the human body satisfies the fourth condition, of which the fourth condition represents the following: When the human body satisfies the first condition, the number of forward-facing lateral positions of the human body among the N positions in the posture sequence is greater than the number of lateral positions in other directions.

[0120] In several embodiments, when the direction of the human body lying on its side is forward, the second judgment unit 904 determines whether the human body satisfies the fifth condition, of which the fifth condition represents the following: When the human body satisfies the first condition, the proportion of the N postures in the posture sequence in which the boundary frames corresponding to the N postures are non-flattened exceeds the fourth threshold.

[0121] In some embodiments, the second judgment unit 904 determines that the human body has fallen forward when the direction of the human body's lateral position is forward and the human body satisfies all of the third, fourth, and fifth conditions.

[0122] In some embodiments, when the direction of the human body's lateral position is forward-facing and the human body does not satisfy all of the third, fourth, and fifth conditions, the second judgment unit 904 determines whether the human body satisfies the sixth condition, of which the sixth condition represents the following: When the human body satisfies the second condition, in the posture sequence, it is possible to find M4 starting postures from W3 consecutive postures among the classified postures in the first direction of the intermediate posture; and When the human body does not satisfy the second condition, in the posture sequence, it is possible to find M4 starting postures from W3 consecutive postures among the classified postures in the first direction of the final posture. Of these, W3 and M4 are all positive integers, and W3 is greater than M4.

[0123] In some embodiments, when the direction of the human body's lateral position is forward and the human body does not satisfy all of the third, fourth, and fifth conditions, the second judgment unit 904 determines whether the human body satisfies the seventh condition, of which the seventh condition represents the following: The tipping angle θ > D1, where D1 is a pre-set angle threshold based on the actual situation.

[0124] In some embodiments, the second judgment unit 904 determines that the human body has fallen forward when the direction of the human body's lateral position is forward and the human body satisfies all of the sixth and seventh conditions.

[0125] In some embodiments, when the direction of the human body lying down is to the side or to the rear, the second judgment unit determines whether the human body satisfies the eighth condition, of which the eighth condition represents the following: When the human body satisfies the second condition, in the posture sequence, M5 starting postures can be found from W4 consecutive postures among the classified postures in the first direction of the intermediate posture; and When the human body does not satisfy the second condition, in the posture sequence, it is possible to find M5 starting postures from W4 consecutive postures among the classified postures in the first direction of the final posture. Of these, W4 and M5 are all positive integers, and W4 is greater than M5.

[0126] In some embodiments, when the direction of the human body lying down is to the side or to the rear, the second judgment unit 904 determines whether the human body satisfies the ninth condition, of which the ninth condition represents the following: The tipping angle θ > D2, where D2 is a pre-set angle threshold based on the actual situation.

[0127] In some embodiments, when the direction of the human body's lateral position is either lateral or backward, and the human body satisfies all of the sixth and seventh conditions, the second judgment unit 904 determines that the human body has fallen to the side or backward.

[0128] In some embodiments, the apparatus further includes the following: Marking unit 905: By setting (providing) detection status marks on the human body, it indicates whether it is currently necessary to perform fall detection on the human body, and among these, the detection status marks include fall, detection skip, and detection required.

[0129] In some embodiments, when there are six or more starting attitudes in the attitude sequence that are closest to the current time, the marking unit 905 resets the status mark as needing detection. Of these, W5 and W6 are all positive integers, and W5 is greater than W6.

[0130] Although only the individual components or modules relating to the present invention have been described above, the present invention is not limited to these. The tipping detection device 900 may further include other components or modules, and the specific details of these components or modules can be found in the relevant technologies.

[0131] For convenience, Figure 9 only shows the connection relationships or signal directions between each component or module; however, various related technologies such as bus connections may be used so that those skilled in the art can understand them. Note that the above-mentioned components or modules may be implemented by hardware such as processors and memory units, but the embodiments of the present invention are not limited to these.

[0132] The embodiments described above are for illustrative purposes to illustrate embodiments of the present invention, but the present invention is not limited thereto, and appropriate modifications may be made based on the embodiments described above. For example, the embodiments described above can be used individually, or a combination of several of the embodiments described above can be used.

[0133] As can be seen from the above-described embodiment, by arranging the classified human body postures in chronological order to form a posture sequence, and then determining the human body's fall and the direction of the fall based on the posture sequence, the complexity of fall detection can be reduced, the performance of fall detection can be improved, and a better fall detection effect can be ensured.

[0134] <Example of the third side> The present invention provides an electronic device, which includes a tip-over detection device 900 described in the second aspect embodiment, the contents of which are hereby combined. The electronic device may be, for example, a computer, server, workstation, laptop computer, smartphone, etc., but the embodiments of the present invention are not limited to these.

[0135] Figure 10 shows an electronic device in an embodiment of the present invention. As shown in Figure 10, the electronic device 1000 may include a processor (e.g., a central processor CPU) 1010 and a memory unit 1020, the memory unit 1020 being connected to the central processor 1010. The memory unit 1020 can store various data, store information processing programs, and execute these programs under the control of the processor 1010.

[0136] In some embodiments, the functions of the tipping detection device 900 are integrated into the processor 1010. In some of these embodiments, the processor 1010 is configured to implement the tipping detection method described in the embodiment on the first side.

[0137] In some embodiments, the tipping detection device 900 and the processor 1010 are arranged separately. For example, the tipping detection device 900 may be configured as a chip connected to the processor 1010, and the functions of the tipping detection device 900 may be realized by controlling the processor 1010.

[0138] For example, the processor 1010 is configured to perform the following control: it uses a neural network to recognize keypoints in the human body in an image and obtains keypoints and bounding frames of the human body's skeleton; it estimates and classifies the posture of the human body based on the obtained keypoints and bounding frames and obtains classified postures; it arranges the classified postures in chronological order to form a posture sequence, and determines whether the human body is lying on its side based on the posture sequence; and, when the human body is lying on its side, it determines the direction of the human body's lateral position in the posture sequence based on the image, and determines whether the human body has fallen in that direction based on the direction of the human body's lateral position and the posture sequence.

[0139] Furthermore, as shown in Figure 10, the electronic device 1000 may further include an input / output (I / O) device 1030, a display unit 1040, and the like. Since the functions of these components are the same as in the prior art, a detailed explanation is omitted here. Note that the electronic device 1000 does not need to include all the components shown in Figure 10. Also, the electronic device 800 may further include components not shown in Figure 10; for these, please refer to related technologies.

[0140] In embodiments of the present invention, a computer-readable program is further provided, wherein when the program is executed on an electronic device, the program causes the computer to execute the tipping detection method described in the first embodiment on the electronic device.

[0141] In embodiments of the present invention, a storage medium storing a computer-readable program is further provided, wherein the computer-readable program causes a computer to execute the tipping detection method described in the first embodiment using an electronic device.

[0142] Furthermore, the above-described apparatus and method may be implemented by software or hardware, or by a combination of hardware and software. The present invention further relates to a computer-readable program as described below, that is, the program, when executed by a logic component, causes the logic component to implement the above-described apparatus or component, or to the logic component to implement the above-described various methods or steps. The logic component may be, for example, an FPGA (Field Programmable Gate Array), a microprocessor, or a processor used in a computer. The present invention further relates to a storage medium storing the above-described program, for example, a hard disk, a magnetic disk, an optical hard disk, a DVD, or a flash memory.

[0143] Furthermore, one or more combinations of the functional blocks shown in the drawings and / or one or more combinations of functional blocks may be implemented as a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic component, discrete gate or transistor logic component, discrete hardware assembly or any other suitable combination for performing the functions described herein. Also, one or more combinations of the functional blocks shown in the drawings and / or one or more combinations of functional blocks may further be configured as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors connected to a DSP by communication or any other combination of configurations.

[0144] Furthermore, the above-mentioned embodiments and other details are further disclosed as follows:

[0145] (Note 1) A fall detection method based on human posture estimation, Using a neural network, keypoint recognition is performed on the human body in the image to obtain the keypoints and bounding box of the human body's skeleton; Based on the acquired keypoints and boundary frame, estimation and classification of the human body's posture are performed to obtain a classified posture; The aforementioned classified postures are arranged in chronological order to form a posture sequence, and it is determined whether the human body is in a lying-down position based on the posture sequence; and The method includes determining the direction of the lying position of the human body in the posture sequence based on the image when the human body is lying on its side, and determining whether the human body has fallen in the direction based on the direction of the lying position of the human body and the posture sequence.

[0146] (Note 2) The method described in Appendix 1, When the direction of the human body lying on its side is forward, and the human body does not satisfy all of the third, fourth, and fifth conditions, it is determined whether the human body satisfies the sixth condition, and of these, the sixth condition represents the following: When the human body satisfies the second condition, in the posture sequence, M4 starting postures can be found from W3 consecutive postures among the classified postures in the first direction of the intermediate posture; and When the human body does not satisfy the second condition, in the posture sequence, it is possible to find M4 starting postures from W3 consecutive postures among the classified postures in the first direction of the final posture. Among these, W3 and M4 are all positive integers, and W3 is greater than M4.

[0147] (Note 3) The method described in Appendix 1, When the direction of the human body lying down is forward, and the human body does not satisfy all of the third, fourth, and fifth conditions, it is determined whether the human body satisfies the seventh condition, and of these, the seventh condition represents the following: The tipping angle θ > D1, where D1 is a pre-set angle threshold based on the actual situation.

[0148] (Note 4) The method described in Appendix 1, This method determines that a forward inversion of the human body has occurred when the direction of the human body's lateral position is forward, and the human body satisfies both of the sixth and seventh conditions.

[0149] (Note 5) The method described in Appendix 1, When the direction of the human body lying down is either to the side or to the rear, it is determined whether the human body satisfies the sixth condition, and of which the sixth condition represents the following: When the human body satisfies the second condition, in the posture sequence, among the classification postures in the first direction of the intermediate posture, a continuous W4 The ability to find M5 starting postures from one posture; and When the human body does not satisfy the second condition, in the posture sequence, it is possible to find M5 starting postures from W4 consecutive postures among the classified postures in the first direction of the final posture. Among these, W4 and M5 are all positive integers, and W4 is greater than M5.

[0150] (Note 6) The method described in Appendix 1, When the direction of the human body lying down is either to the side or to the rear, it is determined whether the human body satisfies the seventh condition, and of which the seventh condition represents the following: The tipping angle θ > D2, where D2 is a pre-set angle threshold based on the actual situation.

[0151] (Note 7) The method described in Appendix 1, When the direction of the human body's lateral position is either to the side or to the rear, and the human body satisfies both the sixth and seventh conditions, it is determined that the human body has fallen to the side or to the rear.

[0152] (Note 8) The method described in Appendix 1, further, This includes providing a detection status mark on the human body to indicate whether it is currently necessary to perform fall detection on the said human body. Among these, the detection status marks include those indicating overturning, detection skipping, and detection required.

[0153] (Note 9) The method described in Appendix 1, In the posture sequence, if there are six or more starting postures in the five postures W closest to the current time, the state mark is detected and reset. Among these, W5 and W6 are both positive integers, and W5 is greater than W6.

[0154] (Note 10) It is an electronic device, The electronic device includes a processor and a memory device, and the processor is configured to implement the tipping detection method described in any one of the appendices 1 to 9.

[0155] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and any modification to the present invention that does not deviate from the spirit of the invention falls within the technical scope of the present invention.

Claims

1. A device for detecting falls based on human posture estimation, An acquisition unit that uses a neural network to recognize keypoints in a human body image and acquires the keypoints and bounding box of the human body's skeleton; A classification unit that performs estimation and classification of the human body posture based on the acquired key points and boundary frame to obtain a classified posture; A first determination unit that arranges the aforementioned classified postures in chronological order to form a posture sequence, and determines whether the human body is in a lying-down position based on the posture sequence; and An apparatus comprising a second determination unit that, when the human body is in a lying-down position, determines the direction of the lying-down position of the human body in the posture sequence based on the image, and determines whether the human body has fallen in the direction based on the direction of the lying-down position of the human body and the posture sequence.

2. The apparatus according to claim 1, When the posture sequence satisfies the first condition, the first judgment unit determines that the human body is in a lying-down position. The first condition refers to the fact that in the posture sequence, the number of final postures among the N postures closest to the current time is greater than the first threshold. The apparatus wherein N is a positive integer, and the final posture refers to the recumbent or prone position of the classified posture.

3. The apparatus according to claim 2, The second judgment unit determines the direction of the human body lying down, Based on the aforementioned image, the direction of the human body lying on its side in the final posture is determined, When the aforementioned human body is lying on its side with its head facing forward, the direction of the human body's lateral position is determined to be forward; When the human body is lying on its side with its head turned to the side, the direction of the human body's lateral position is determined to be to the side; and A device that includes determining that the direction of the human body's lateral position is backward when the human body is lying on its side with its head facing backward.

4. The apparatus according to claim 3, When the human body is in a lying-down position, the second judgment unit determines whether the human body satisfies the second condition. The second condition is that, in the attitude sequence, the N+1 attitude closest to the current time, or the last final attitude in the first direction among the N attitudes, has a length of W. １ This refers to the case where the object slides in the first direction using the window, and the number of intermediate postures n found from the posture entering the window is greater than the second threshold. When the aforementioned human body satisfies the second condition, the sliding of the window stops. N is a positive integer, and W １ The device wherein the length of is greater than the length of one frame in the posture sequence, the first direction refers to the direction away from the current time in the posture sequence, and the intermediate posture includes sitting, squatting, or bending forward.

5. The apparatus according to claim 4, When the direction of the human body lying on its side is forward, the second judgment unit determines whether the human body satisfies the third, fourth, and fifth conditions. The third condition mentioned above is: When the human body satisfies the second condition, in the posture sequence, among the classified postures in the first direction of the window, a continuous W ２ From individual posture M ３ The ability to find the starting position of each individual; and When the human body does not satisfy the second condition, in the posture sequence, the N+1 posture closest to the current time, or the classification posture in the first direction of the last final posture in the first direction among the N postures, is a continuous W ２ From individual posture M ３ This refers to being able to find the starting position for each individual. Said M ３ The ratio of the upper body to the lower body in the initial posture satisfies the third threshold, and the said third threshold is related to the average half-body ratio in the final posture. The aforesaid W ２ and the aforesaid M ３ are both positive integers, the aforesaid W ２ is greater than the aforesaid M ３ and the starting posture includes standing or walking. The fourth condition means that when the human body satisfies the first condition, the number of forward-facing lateral positions of the human body among the N positions in the posture sequence is greater than the number of lateral positions in other directions. The fifth condition refers to the device in which, when the human body satisfies the first condition, the proportion of the non-flattened boundary frames corresponding to the N postures in the posture sequence exceeds the fourth threshold.

6. The apparatus according to claim 5, A device in which, when the direction of the human body lying on its side is forward, and the human body satisfies all of the third, fourth, and fifth conditions, the second judgment unit determines that the human body has fallen forward.

7. The apparatus according to claim 4, When the direction of the human body lying down is to the side or to the rear, the second judgment unit determines whether the human body satisfies the eighth and ninth conditions. The eighth condition mentioned above is, When the human body satisfies the second condition, in the posture sequence, among the classification postures in the first direction of the intermediate posture, a continuous W ４ From individual posture M ５ The ability to find the starting position of each individual; and When the human body does not satisfy the second condition, in the posture sequence, among the classified postures in the first direction of the final posture, a continuous W ４ From individual posture M ５ This refers to being able to find the starting position for each individual. The aforementioned W ４ and the M ５ All of these are positive integers, and the aforementioned W ４ is the aforementioned M ５ Larger than, The nine conditions mentioned above are: tilt angle θ > D ２ This refers to the aforementioned D ２ This is a device where the angle threshold is pre-set according to the actual situation.

8. The apparatus according to claim 7, A device in which, when the direction of the human body lying down is to the side or to the rear, and the human body satisfies all of the eighth and ninth conditions, the second judgment unit determines that the human body has fallen to the side or to the rear.

9. The apparatus according to claim 1, The system further includes a marking unit that indicates whether it is currently necessary to perform fall detection on the human body by providing a detection status mark on the human body. The detection status marks include those for tipping over, detection skip, and detection required, in the device.

10. A method for detecting falls based on human posture estimation, Using a neural network, keypoint recognition is performed on the human body in the image to obtain the keypoints and bounding box of the human body's skeleton; Based on the acquired key points and boundary frame, estimation and classification of the human body's posture are performed to obtain a classified posture; The aforementioned classified postures are arranged in chronological order to form a posture sequence, and it is determined whether the human body is in a lying-down position based on the posture sequence; and A method comprising determining the direction of the lying position of the human body in the posture sequence based on the image when the human body is in a lying position, and determining whether the human body has fallen in the direction based on the direction of the lying position of the human body and the posture sequence.

Images