A fall detection method, device, electronic device and storage medium

Abstract

The embodiments of the present invention relate to a fall detection method, device, electronic device and storage medium, which are applied to the field of computer vision and image processing technology. The method comprises: detecting key point sets and ground contact points in a target image corresponding to a preset number of target parts, wherein the target image includes at least one target object; constructing a stripped contour of the ground contact part through a ground contact contour stripping algorithm based on the key point set corresponding to the ground contact part; extracting ground contact part features based on the stripped contour of the ground contact part; and performing fall detection on the target object based on the ground contact part features. Thus, the key parts of the human body touching the ground are directly found, and the ground contact parts are striped and segmented for instance for fall detection, which has better recognition and generalization, introduces the aspect ratio auxiliary feature, reduces missed detection caused by missing key fall parts, improves sensitivity, introduces the median width, and reduces misjudgment caused by open hands.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of computer vision and image processing, and in particular to a fall detection method, device, electronic device, and storage medium. Background Art

[0002] Existing fall detection algorithms are mainly divided into fall detection based on wearable devices, fall detection based on vision (RGB or RGBD camera) and fall detection based on ambient sound according to different sensors. With the development of deep learning technology and edge computing in recent years, vision-based fall detection algorithms have attracted more and more attention from academia and industry due to their non-contact characteristics and accurate performance.

[0003] There are currently two types of vision-based fall detection methods: 1) end-to-end fall detection methods based on target detection; 2) posture estimation methods based on human key point detection. The end-to-end fall detection method based on target detection uses the current fast YOLO series of target detection algorithms to achieve end-to-end fall detection through binary classification. This method is relatively simple and easy to implement. However, since the classification is end-to-end, if there is no large amount of training data, it is difficult to ensure stability in practical applications. The posture estimation method based on human key point detection can usually divide the human posture into 16-25 key points, and can use graph convolution or YOLOv8-pose to achieve fall action recognition. Compared with the end-to-end detection algorithm, the key point-based detection method is more complicated, but more interpretable. For surveillance cameras, the above key point detection method has fewer key point misses. However, for mobile robots, especially robots with lower "height", their viewing angle is lower and their sight is easily blocked by objects such as the human body or furniture, which poses a higher challenge to fall detection. Summary of the invention

[0004] In view of this, in order to solve the above-mentioned technical problems or part of the technical problems, the embodiments of the present invention provide a fall detection method, device, electronic device and storage medium, which directly finds the key parts of the human body touching the ground, performs strip-shaped instance segmentation on the touching parts, and introduces aspect ratio auxiliary features to further perform fall detection, so as to solve the problems of high data volume requirements, poor interpretability, or missed detection in existing vision-based fall detection methods.

[0005] In a first aspect, an embodiment of the present invention provides a fall detection method, comprising:

[0006] Detecting key point sets corresponding to a preset number of target parts in a target image and ground contacting parts of the target parts, wherein the target image includes at least one target object;

[0007] Based on the key point set corresponding to the ground contact part, constructing a striped contour of the ground contact part by using a ground contact contour striping algorithm;

[0008] Extracting ground contact part features based on the strip-shaped contour of the ground contact part;

[0009] Performing fall detection on the target object based on the ground contact part feature;

[0010] When the number of the detected target parts does not reach the preset number, collecting a human body contour key point set of the target object;

[0011] Constructing a human body contour based on the human body contour key point set, and calculating the median height-to-width ratio of the human body contour;

[0012] Whether the target object falls is determined based on the median height-to-width ratio.

[0013] In a possible implementation, the method further includes:

[0014] Connecting the human body contour key points in the human body contour key point set end to end in sequence to construct a human body contour;

[0015] Based on the human body contour key point set, a human body mask is obtained by using a contour filling function in the OpenCV library;

[0016] A horizontal line is constructed with the middle value of the height of the human body contour, and the width of the longest connected line segment where the horizontal line intersects with the human body mask is used as the human body width;

[0017] The median height-to-width ratio of the human body contour is calculated based on the median height value of the human body contour and the human body width.

[0018] In a possible implementation, the method further includes:

[0019] Sorting the row coordinates in the human body contour key point set from small to large to obtain a height sequence;

[0020] constructing a horizontal line with the middle value of the height in the height sequence;

[0021] Obtain the intersection line segment between the human body mask and the horizontal line through connected component calculation;

[0022] The width of the longest intersecting line segment is taken as the width of the human body.

[0023] In a possible implementation, the method further includes:

[0024] The median height-to-width ratio of the human body contour is calculated by a first formula, wherein the first formula is:

[0025]

[0026] Among them, F is the median height-to-width ratio of the human body contour, max|y i -y j | is the median height of the human body contour, x r -x l is the width of the human body, (i,j)∈Q, Q is the index set, x l is the column coordinate of the left endpoint of the longest intersecting line segment, x r is the column coordinate of the right endpoint of the longest intersecting line segment.

[0027] In a possible implementation, the method further includes:

[0028] Pre-set the fall judgment threshold;

[0029] If the median height-to-width ratio is less than the fall judgment threshold, it is determined that the target object has fallen.

[0030] In a possible implementation, the method further includes:

[0031] Connecting the key points in the key point set corresponding to the ground contact part in sequence end to end to construct a segmented contour corresponding to the ground contact part;

[0032] Generate a strip corresponding to the segmented contour by using the line drawing function in the OpenCV library;

[0033] The strip contour corresponding to the strip band is extracted by the contour search function in the OpenCV library to obtain the strip contour corresponding to the ground contacting part.

[0034] In a possible implementation, the method further includes:

[0035] Determining a ground contact length based on a strip-shaped profile corresponding to the ground contact part;

[0036] If the ground contact part includes other target parts except the foot, and the ground contact length is greater than the corresponding preset threshold, it is determined that the target object has fallen.

[0037] In a second aspect, an embodiment of the present invention provides a fall detection device, comprising:

[0038] A detection module, configured to detect key point sets corresponding to a preset number of target parts in a target image and ground contact parts of the target parts, wherein the target image includes at least one target object; when the number of the detected target parts does not reach the preset number, collecting a human body contour key point set of the target object;

[0039] A construction module, configured to construct a striped contour of the contact part based on a key point set corresponding to the contact part by a contact contour striping algorithm; construct a human body contour based on the human body contour key point set, and calculate a median height-to-width ratio of the human body contour;

[0040] An extraction module, configured to extract ground contact part features based on the strip-shaped contour of the ground contact part;

[0041] The detection module is further used to perform fall detection on the target object based on the ground contact part characteristics; and determine whether the target object falls based on the median height-to-width ratio.

[0042] In a third aspect, an embodiment of the present invention provides a mobile robot fall detection system, comprising: a camera, an image processing module and a fall detection module; the camera is used to collect an environmental image of the current environment; the image processing module is used to receive the environmental image, and detect a target image with a target object in the environmental image; the fall detection module is used to perform fall detection on the target object in the target image.

[0043] In a fourth aspect, an embodiment of the present invention provides a storage medium, including: the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the fall detection method described in the first aspect above.

[0044] The fall detection scheme provided by the embodiment of the present invention detects the key point sets corresponding to a preset number of target parts in the target image and the ground contact parts in the target parts, wherein the target image includes at least one target object; based on the key point set corresponding to the ground contact part, a striped contour of the ground contact part is constructed by a ground contact contour striping algorithm; the ground contact part features are extracted based on the striped contour of the ground contact part; the target object is subjected to fall detection based on the ground contact part features; when the number of detected target parts does not reach the preset number, the human body contour key point set of the target object is collected; the human body contour is constructed based on the human body contour key point set, and the median height-to-width ratio of the human body contour is calculated; and whether the target object falls is determined based on the median height-to-width ratio. Compared with the existing vision-based fall detection method, it has higher data volume requirements, poor interpretability, or key point detection may have missed detection problems. This solution directly finds the key parts of the human body touching the ground, segments the touching parts into strips, and then performs fall detection, which has better recognition and generalization. It introduces the aspect ratio auxiliary feature to reduce missed detection caused by missing the key falling parts, improves sensitivity, and introduces the median width to reduce misjudgment caused by open hands. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of a flow chart of a fall detection method provided by an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of segmented labeling of a human body fall contour provided by an embodiment of the present invention;

[0047] Figure 3 A bar diagram of a key part of a human body touching the ground provided by an embodiment of the present invention;

[0048] Figure 4 A schematic flow chart of a fall detection method based on the median height-to-width ratio of a human body contour provided by an embodiment of the present invention;

[0049] Figure 5 A schematic diagram of the aspect ratio of the median line of a detection frame and a human body contour provided by an embodiment of the present invention;

[0050] Figure 6 A schematic diagram of the structure of a fall detection device provided by an embodiment of the present invention;

[0051] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] To facilitate understanding of the embodiments of the present invention, specific embodiments will be further explained below in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0054] Figure 1 A flow chart of a fall detection method provided by an embodiment of the present invention is shown as follows: Figure 1 As shown, the method specifically includes:

[0055] S11, detecting key point sets corresponding to a preset number of target parts in a target image and ground contacting parts of the target parts.

[0056] The embodiments of the present invention are mainly applied to a mobile robot fall detection system based on an RGB camera in a home or indoor scene, such as actively detecting a fallen elderly person or child and sending a distress signal.

[0057] Firstly, the mobile robot camera collects video images, analyzes the video images and extracts the target image with the target object, which is the human body.

[0058] Furthermore, a preset number of target parts in the target image are detected, and the key point sets and the ground contact parts of the target parts correspond to them respectively. The preset number can be determined according to the usage scenario or the current brightness. For example, if the usage scenario is the fall detection of people in the home, the preset number can be eight; if the usage scenario is the gym, the preset number can be eighteen or twenty-five joint point detections; for another example, if the current brightness is lower than the threshold, which may cause unclear image acquisition, the preset number can be reduced to four, etc.

[0059] For example, eight target parts of the human body of the target object in the target image are detected, including: the whole body, feet, legs, buttocks, trunk, palms, arms, and head. Different from the existing key point detection, the target parts of the embodiment of the present invention only include eight in total. These eight target parts can be divided into non-contacting (whole body) and seven touching parts (including feet, legs, buttocks, palms, arms, trunk, and head).

[0060] Furthermore, the existing key point collection method can be used to collect the key point sets corresponding to the eight key parts, and the image is analyzed to determine the ground contact part among the eight target parts, and the ground contact part is expressed as a bar, which is then used for instance segmentation.

[0061] S12: constructing a striped contour of the ground contact part by using a ground contact contour striping algorithm based on a key point set corresponding to the ground contact part.

[0062] The key points in the key point set corresponding to the contact part are connected end to end to construct the segmented contour corresponding to the contact part. Then, the strip corresponding to the segmented contour is generated by the line drawing function in the OpenCV library, and the strip contour corresponding to the strip is extracted by the contour search function in the OpenCV library to obtain the strip contour corresponding to the contact part.

[0063] Specifically, Figure 2 As shown in the figure, red indicates the contour of the ground contact, and green indicates the contour of the ground contact. The striped contour of the ground contact part is constructed by the ground contact contour striping algorithm, as shown in Figure 3 As shown. Then, the six strip-shaped parts of the legs, hips, palms, arms, torso, and head touching the ground are segmented and extracted through the training instance segmentation network (for example, YOLOv8-seg).

[0064] Touchdown contour striping algorithm:

[0065] Assume that for image I, the segmented annotations {S1, S2, ..., SN} and the ground contact marks {M1, M2, ..., MN} of the human body contour are marked. Among them, the i-th segmented contour point set Si = [[xi1, yi1], [xi2, yi2], ..., [xik, yik]] ([x, y] represent the column and row coordinates of the image respectively), and the ground contact mark Mi = 1 indicates that the contour Si touches the ground; Mi = 0 indicates that the contour Si does not touch the ground. Without loss of generality, it can be assumed that the length of each segmented contour is the same, which is k points. For example Figure 2 As shown in , the human body contour is divided into multiple segments, including head, torso, arm, palm, hip, non-contacting leg, contacting leg, and foot from left to right. For a given segmented contour {S1, S2, ..., SN}, a closed strip contour is obtained by the contacting contour stripping algorithm, as shown in Figure 3 The Python-based touchdown contour striping algorithm includes the following steps:

[0066] The known input includes image I, annotations {S1, S2, ..., SN} and {M1, M2, ..., MN}, and strip width StripWidth (default 20 pixels)

[0067] Step 1: Connect the segmented contour point sets {S1, S2, ..., SN} end to end and construct the overall contour point set The human body mask MASK=cv2.fillPoly(0*I, [S], color=255) is obtained through the contour filling function in the OpenCV library;

[0068] Step 2: For each segmented contour Si that touches the ground, that is, Mi=1, generate a strip Strip_i=(MASK>0)*cv2.polylines(0*MASK, [Si], False, 255, StripWidth) through the line drawing function in the OpenCV library;

[0069] Step 3: For the strip Strip_i, extract the strip contour C_i through the contour search function in the OpenCV library, that is, C_i = cv2.findContours(Strip_i,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE);

[0070] Step 4: Return the human body silhouette S and the touchdown strip silhouette {C_i}.

[0071] S13: Extracting ground contact part features based on the strip-shaped contour of the ground contact part.

[0072] S14: Perform fall detection on the target object based on the ground contact part features.

[0073] In an embodiment of the present invention, the ground contact part feature includes but is not limited to the ground contact length. The six target parts touching the ground (including legs, palms, arms, torso, buttocks, and head) are used as the basis for judging a fall, and any of these six parts touching the ground is counted as a fall. The ground contact length of the ground contact part is calculated based on the striped contour corresponding to the ground contact part; if the ground contact part includes other target parts except the feet, and the ground contact length is greater than the corresponding preset threshold, it is determined that the target object has fallen. Different target parts can correspond to different preset thresholds. For example, the leg length of an adult female is about 80 cm, and the leg length of an adult male is about 100 cm. The ground contact length threshold of the leg can be preset to 60 cm. When the ground contact length of the leg is greater than 60 cm, it is determined that the target object has fallen.

[0074] S15. When the number of detected target parts does not reach the preset number, a human body contour key point set of the target object is collected.

[0075] In some postures and perspectives, the target parts touching the ground, such as the legs, are easily blocked by the body or other objects, which will increase the probability of missed detection. Therefore, in addition to the seven parts touching the ground, the entire human body contour (such as Figure 2 The middle green line + red line) is used as the entity segmentation of the eighth category. Through the human body contour, the auxiliary feature of the median height-to-width ratio of the human body contour is introduced to reduce the missed detection of falls.

[0076] Specifically, when the number of detected target parts does not reach eight, it indicates that the target parts are blocked. The existing key point collection method can be used to collect the key point set of the human body contour of the target object.

[0077] S16. Constructing a human body contour based on the human body contour key point set, and calculating the median height-to-width ratio of the human body contour.

[0078] S17: Determine whether the target object has fallen based on the median height-to-width ratio.

[0079] The human body contour key points in the human body contour key point set are connected end to end in sequence to construct the human body contour; based on the human body contour key point set, the human body mask is obtained through the contour filling function in the OpenCV library; the horizontal line is constructed with the middle value of the height of the human body contour, and the width of the longest connected line segment where the horizontal line intersects with the human body mask is taken as the human body width; the median height-to-width ratio of the human body contour is calculated based on the middle value of the height of the human body contour and the human body width.

[0080] Among them, a horizontal line is constructed with the median value of the height of the human body contour, and the width of the longest connected line segment where the horizontal line intersects the human body mask is used as the human body width, specifically including: sorting the row coordinates of the key points of the human body contour from small to large to obtain a height sequence; constructing a horizontal line with the median value of the height in the height sequence; obtaining the intersecting line segment of the human body mask and the horizontal line through connected component calculation; and taking the width of the longest intersecting line segment as the human body width.

[0081] Specifically, the human body contour key points in the human body contour key point set are connected end to end to construct the human body contour; based on the human body contour key point set, the human body mask MASK is obtained through the contour filling function in the OpenCV library. Then, a horizontal line is constructed with the middle value of the height of the human body contour, and the width of the longest connected line segment intersecting the horizontal line and the human body mask MASK is used as the human body width. This can avoid false detection that may be caused when the human hand is open, such as Figure 5 shown.

[0082] Assuming that the instance segmentation network Yolo8-seg estimates the human mask MASK, the calculation steps of the aspect ratio auxiliary feature F are as follows:

[0083] Step 1: Use the contour search function in the OpenCV library, i.e., C = cv2.findcontour(mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE), to obtain the contour point set C = {[x1, y1], [x2, y2], …, [xn, yn]};

[0084] Step 2: Sort the height sequence [y1, y2, ..., yn], that is, Y = sort ([y1, y2, ..., yn]), and obtain the median height of the human body contour, ymed = Y [int (n / 2)], where int () represents the rounding function;

[0085] Step 3: Use the connected components to calculate the line segment where MASK intersects with the straight line Y=ymed, and get the left and right endpoints of the longest intersecting line segment (xl,ymed), (xr,ymed), then the width of the human body is W=x r -x l ;

[0086] Step 4: Traverse and calculate the height where the index set Q = {(i, j)|[xi,yi]∈C,[xj,yj]∈C, and x l ≤x i =x j ≤x r};

[0087] Step 5: Calculate the height-to-width ratio of the human body: F = H / W;

[0088] Step 6: If F is less than a preset fall judgment threshold (eg, 1.2), it is judged as a fall.

[0089] like Figure 5 As shown in the figure, the aspect ratio of the dotted detection box is less than 1, which can easily cause false alarms of falls; while the aspect ratio F (green solid line length / red solid line length) based on the median line is around 2, which is higher than the fall judgment threshold of 1.2, so it can be judged that the target object in the figure has not fallen.

[0090] Optionally, if the target object falls multiple times within a preset time, an alarm may be issued to alert other personnel. For example, if the target object falls twice within five seconds, a fall alarm may be issued.

[0091] The fall detection method provided by the embodiment of the present invention detects the key point sets corresponding to a preset number of target parts in the target image and the ground contact parts in the target parts, wherein the target image includes at least one target object; based on the key point set corresponding to the ground contact part, a striped contour of the ground contact part is constructed by a ground contact contour striping algorithm; the ground contact part features are extracted based on the striped contour of the ground contact part; and the target object is subjected to fall detection based on the ground contact part features. Compared with the existing vision-based fall detection methods, which have higher data volume requirements, poor interpretability, or key point detection may have missed detection problems, this method directly searches for the key parts of the human body touching the ground, performs instance segmentation of the ground contact part into strips, and then performs fall detection, which has better recognition and generalization.

[0092] Figure 4 A flow chart of a fall detection method based on the median height-to-width ratio of a human body contour provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the method specifically includes:

[0093] S41, detecting key point sets corresponding to a preset number of target parts in a target image and ground contacting parts of the target parts.

[0094] The embodiments of the present invention are mainly aimed at the situation where the mobile robot is relatively low in height (for example, less than 30 cm), resulting in a low shooting angle and the target part is easily blocked, resulting in missed falls.

[0095] The embodiments of the present invention are mainly applied to a mobile robot fall detection system based on an RGB camera in a home or indoor scene, such as actively detecting a fallen elderly person or child and sending a distress signal.

[0096] Firstly, the mobile robot camera collects video images, analyzes the video images and extracts the target image with the target object, which is the human body.

[0097] Furthermore, eight target parts of the human body of the target object in the target image are detected, including: the whole body, feet, legs, buttocks, trunk, palms, arms, and head. Different from the existing key point detection, the target parts of the embodiment of the present invention only include eight in total. These eight target parts can be divided into non-contacting (whole body) and seven touching parts (including feet, legs, buttocks, palms, arms, trunk, and head).

[0098] Furthermore, the existing key point collection method can be used to collect the key point sets corresponding to the eight key parts, and the image is analyzed to determine the ground contact part among the eight target parts, and the ground contact part is expressed as a bar, which is then used for instance segmentation.

[0099] S42: When the number of the detected target parts does not reach the preset number, a human body contour key point set of the target object is collected.

[0100] In some postures and perspectives, the target parts touching the ground, such as the legs, are easily blocked by the body or other objects, which will increase the probability of missed detection. Therefore, in addition to the seven parts touching the ground, the entire human body contour (such as Figure 2 The middle green line + red line) is used as the entity segmentation of the eighth category. Through the human body contour, the auxiliary feature of the median height-to-width ratio of the human body contour is introduced to reduce the missed detection of falls.

[0101] Specifically, when the number of detected target parts does not reach eight, it indicates that the target parts are blocked. The existing key point collection method can be used to collect the key point set of the human body contour of the target object.

[0102] S43: construct a human body contour based on the human body contour key point set, and calculate the median height-to-width ratio of the human body contour.

[0103] S44: Determine whether the target object has fallen based on the median height-to-width ratio.

[0104] The human body contour key points in the human body contour key point set are connected end to end in sequence to construct the human body contour; based on the human body contour key point set, the human body mask is obtained through the contour filling function in the OpenCV library; the horizontal line is constructed with the middle value of the height of the human body contour, and the width of the longest connected line segment where the horizontal line intersects with the human body mask is taken as the human body width; the median height-to-width ratio of the human body contour is calculated based on the middle value of the height of the human body contour and the human body width.

[0105] Among them, a horizontal line is constructed with the median value of the height of the human body contour, and the width of the longest connected line segment where the horizontal line intersects the human body mask is used as the human body width, specifically including: sorting the row coordinates of the key points of the human body contour from small to large to obtain a height sequence; constructing a horizontal line with the median value of the height in the height sequence; obtaining the intersecting line segment of the human body mask and the horizontal line through connected component calculation; and taking the width of the longest intersecting line segment as the human body width.

[0106] Specifically, the human body contour key points in the human body contour key point set are connected end to end to construct the human body contour; based on the human body contour key point set, the human body mask MASK is obtained through the contour filling function in the OpenCV library. Then, a horizontal line is constructed with the middle value of the height of the human body contour, and the width of the longest connected line segment intersecting the horizontal line and the human body mask MASK is used as the human body width. This can avoid false detection that may be caused when the human hand is open, such as Figure 5 shown.

[0107] Assuming that the instance segmentation network Yolov8-seg estimates the human mask MASK, the calculation steps of the aspect ratio auxiliary feature F are as follows:

[0108] Step 1: Use the contour search function in the OpenCV library, i.e., C = cv2.findcontour(mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE), to obtain the contour point set C = {[x1, y1], [x2, y2], …, [xn, yn]};

[0109] Step 2: Sort the height sequence [y1, y2, ..., yn], that is, Y = sort ([y1, y2, ..., yn]), and obtain the median height of the human body contour, ymed = Y [int (n / 2)], where int () represents the rounding function;

[0110] Step 3: Calculate the line segment where MASK intersects with the straight line Y=ymed through the connected components, and obtain the left and right endpoints of the longest intersecting line segment (xl,ymed), (xr,ymed), then the width of the human body is W=xr-xl;

[0111] Step 4: Traverse and calculate the height where the index set Q = {(i, j)|[xi,yi]∈C,[xj,yj]∈C, and x l ≤x i =x j ≤x r};

[0112] Step 5: Calculate the height-to-width ratio of the human body: F = H / W;

[0113] Step 6: If F is less than a preset fall judgment threshold (eg, 1.2), it is judged as a fall.

[0114] like Figure 5 As shown in the figure, the aspect ratio of the dotted detection box is less than 1, which can easily cause false alarms of falls; while the aspect ratio F (green solid line length / red solid line length) based on the median line is around 2, which is higher than the fall judgment threshold of 1.2, so it can be judged that the target object in the figure has not fallen.

[0115] The advantages of the embodiments of the present invention are: 1. The fall detection based on instance segmentation of the key parts of the bar-shaped touching the ground has better recognition and generalization than the key parts of the point-shaped touching ground. The key parts of the bar-shaped touching the ground contain a mechanism of low height from the ground, which is convenient for network learning to generalize to unknown scenes, and does not require the introduction of additional posture estimation features to judge the fall, which is easy to program. 2. The judgment of the fall posture is explainable. For the person who falls, the key parts that touch the ground and the length of the touchdown can be given. 3. The introduction of the aspect ratio auxiliary feature, on the one hand, reduces the missed detection caused by the omission of the key fall parts and improves the sensitivity, and on the other hand, introduces the median width to reduce the judgment errors caused by the opening of the hands, such as Figure 5 shown.

[0116] The fall detection method based on the median height-to-width ratio of the human body contour provided by the embodiment of the present invention detects the key point sets corresponding to a preset number of target parts in the target image and the ground contact parts of the target parts; when the number of detected target parts does not reach the preset number, collects the key point set of the human body contour of the target object; constructs the human body contour based on the key point set of the human body contour, and calculates the median height-to-width ratio of the human body contour; and determines whether the target object falls based on the median height-to-width ratio. Compared with the existing vision-based fall detection method, the problem of missed detection may occur when the ground contact part of the human body is blocked. This method introduces the auxiliary feature of the height-to-width ratio, which, on the one hand, reduces missed detection caused by the omission of key fall parts and improves sensitivity; on the other hand, introduces the median width to reduce misjudgment caused by open hands.

[0117] Figure 6 A schematic diagram of a fall detection device provided by an embodiment of the present invention is shown in FIG. Figure 6 As shown, specifically including:

[0118] The detection module 601 is used to detect the key point sets corresponding to the preset number of target parts in the target image and the ground contact parts of the target parts; when the number of the detected target parts does not reach the preset number, the key point set of the human body contour of the target object is collected. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0119] The construction module 602 is used to construct a striped contour of the contact part based on the key point set corresponding to the contact part by using a contact contour striping algorithm. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here; construct a human body contour based on the human body contour key point set, and calculate the median height-width ratio of the human body contour.

[0120] The extraction module 603 is used to extract the ground contact part features based on the strip-shaped contour of the ground contact part. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0121] The detection module 601 is further configured to detect the fall of the target object based on the ground contact part feature and determine whether the target object has fallen based on the median height-to-width ratio. Detailed descriptions refer to the relevant descriptions corresponding to the above method embodiments, which will not be repeated here.

[0122] The fall detection device provided in this embodiment can be as follows Figure 6 The fall detection device shown in FIG. 1 may perform the following steps: Figure 1 , 4 All steps of the fall detection method in order to achieve Figure 1 , 4 For details on the technical effects of the fall detection method shown, please refer to Figure 1 , 4 For the sake of brevity, the relevant description is not repeated here.

[0123] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, Figure 7 The electronic device 700 shown includes: at least one processor 701, a memory 702, at least one network interface 704 and other user interfaces 703. The various components in the electronic device 700 are coupled together via a bus system 705. It is understood that the bus system 705 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 705 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 705 is not described in detail. Figure 7 Various buses are labeled as bus system 705.

[0124] The user interface 703 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).

[0125] It can be understood that the memory 702 in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct RAM bus random access memory (DRRAM). The memory 702 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0126] In some implementations, the memory 702 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system 7021 and an application 7022 .

[0127] The operating system 7021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 7022 includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application 7022.

[0128] In the embodiment of the present invention, by calling the program or instruction stored in the memory 702, specifically, the program or instruction stored in the application 7022, the processor 701 is used to execute the method steps provided by each method embodiment, for example, including:

[0129] Detect key point sets corresponding to a preset number of target parts in a target image and ground contact parts in the target parts, wherein the target image includes at least one target object; construct a stripped outline of the ground contact part through a ground contact outline stripping algorithm based on the key point set corresponding to the ground contact part; extract ground contact part features based on the stripped outline of the ground contact part; perform fall detection on the target object based on the ground contact part features; when the number of detected target parts does not reach the preset number, collect a human body contour key point set of the target object; construct a human body contour based on the human body contour key point set, and calculate the median height-to-width ratio of the human body contour; and determine whether the target object falls based on the median height-to-width ratio.

[0130] In one possible implementation, when the number of detected target parts does not reach the preset number, a human body contour key point set of the target object is collected; a human body contour is constructed based on the human body contour key point set, and the median height-to-width ratio of the human body contour is calculated; and whether the target object has fallen is determined based on the median height-to-width ratio.

[0131] In one possible implementation, the human body contour key points in the human body contour key point set are connected end to end in sequence to construct a human body contour; based on the human body contour key point set, a human body mask is obtained by using a contour filling function in an OpenCV library; a horizontal line is constructed using the middle value of the height of the human body contour, and the width of the longest connected line segment where the horizontal line intersects with the human body mask is used as the human body width; and the median height-to-width ratio of the human body contour is calculated based on the middle value of the height of the human body contour and the human body width.

[0132] In one possible implementation, the row coordinates in the human body contour key point set are sorted from small to large to obtain a height sequence; a horizontal line is constructed using the middle value of the height in the height sequence; the intersection line segment of the human body mask and the horizontal line is obtained by connected component calculation; and the width of the longest intersection line segment is used as the human body width.

[0133] In a possible implementation, the median height-to-width ratio of the human body contour is calculated using a first formula, where the first formula is:

[0134]

[0135] Among them, F is the median height-to-width ratio of the human body contour, max|y i -y j | is the median height of the human body contour, x r -x l is the width of the human body, (i,j)∈Q, Q is the index set, x l is the column coordinate of the left endpoint of the longest intersecting line segment, x ris the column coordinate of the right endpoint of the longest intersecting line segment.

[0136] In a possible implementation manner, a fall judgment threshold is preset; if the median height-to-width ratio is smaller than the fall judgment threshold, it is determined that the target object has fallen.

[0137] In one possible implementation, the key points in the key point set corresponding to the contact part are connected end to end in sequence to construct a segmented contour corresponding to the contact part; a strip band corresponding to the segmented contour is generated by a line drawing function in the OpenCV library; and a strip contour corresponding to the strip band is extracted by a contour finding function in the OpenCV library to obtain a strip contour corresponding to the contact part.

[0138] In one possible implementation, the ground contact length is determined based on the strip-shaped contour corresponding to the ground contact part; if the ground contact part includes other target parts except the foot, and the ground contact length is greater than the corresponding preset threshold, it is determined that the target object has fallen.

[0139] The method disclosed in the above embodiment of the present invention can be applied to the processor 701, or implemented by the processor 701. The processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 701 or the instruction in the form of software. The above processor 701 can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to execute, or the hardware and software units in the decoding processor can be executed. The software unit can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 702, and the processor 701 reads the information in the memory 702 and completes the steps of the above method in combination with its hardware.

[0140] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPDevice, DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0141] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0142] The electronic device provided in this embodiment may be Figure 7 The electronic device shown in FIG. 1 may perform the following steps: Figure 1 , 4 All steps of the fall detection method in order to achieve Figure 1 , 4 For details on the technical effects of the fall detection method shown, please refer to Figure 1 , 4 For the sake of brevity, the relevant description is not repeated here.

[0143] The embodiment of the present invention also provides a storage medium (computer-readable storage medium). The storage medium here stores one or more programs. The storage medium may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory may also include a combination of the above-mentioned types of memory.

[0144] When one or more programs in the storage medium can be executed by one or more processors, the fall detection method executed on the electronic device side can be implemented.

[0145] The processor is used to execute the fall detection program stored in the memory to implement the following steps of the fall detection method performed on the electronic device side:

[0146] Detect key point sets corresponding to a preset number of target parts in a target image and ground contact parts in the target parts, wherein the target image includes at least one target object; construct a stripped outline of the ground contact part through a ground contact outline stripping algorithm based on the key point set corresponding to the ground contact part; extract ground contact part features based on the stripped outline of the ground contact part; perform fall detection on the target object based on the ground contact part features; when the number of detected target parts does not reach the preset number, collect a human body contour key point set of the target object; construct a human body contour based on the human body contour key point set, and calculate the median height-to-width ratio of the human body contour; and determine whether the target object falls based on the median height-to-width ratio.

[0147] In one possible implementation, when the number of detected target parts does not reach the preset number, a human body contour key point set of the target object is collected; a human body contour is constructed based on the human body contour key point set, and the median height-to-width ratio of the human body contour is calculated; and whether the target object has fallen is determined based on the median height-to-width ratio.

[0148] In one possible implementation, the human body contour key points in the human body contour key point set are connected end to end in sequence to construct a human body contour; based on the human body contour key point set, a human body mask is obtained by using a contour filling function in an OpenCV library; a horizontal line is constructed using the middle value of the height of the human body contour, and the width of the longest connected line segment where the horizontal line intersects with the human body mask is used as the human body width; and the median height-to-width ratio of the human body contour is calculated based on the middle value of the height of the human body contour and the human body width.

[0149] In one possible implementation, the row coordinates in the human body contour key point set are sorted from small to large to obtain a height sequence; a horizontal line is constructed using the middle value of the height in the height sequence; the intersection line segment of the human body mask and the horizontal line is obtained by connected component calculation; and the width of the longest intersection line segment is used as the human body width.

[0150] In a possible implementation, the median height-to-width ratio of the human body contour is calculated using a first formula, where the first formula is:

[0151]

[0152] Among them, F is the median height-to-width ratio of the human body contour, max|y i -y j | is the median height of the human body contour, x r -x l is the width of the human body, (i,j)∈Q, Q is the index set, x l is the column coordinate of the left endpoint of the longest intersecting line segment, x ris the column coordinate of the right endpoint of the longest intersecting line segment.

[0153] In a possible implementation manner, a fall judgment threshold is preset; if the median height-to-width ratio is smaller than the fall judgment threshold, it is determined that the target object has fallen.

[0154] In one possible implementation, the key points in the key point set corresponding to the contact part are connected end to end in sequence to construct a segmented contour corresponding to the contact part; a strip band corresponding to the segmented contour is generated by a line drawing function in the OpenCV library; and a strip contour corresponding to the strip band is extracted by a contour finding function in the OpenCV library to obtain a strip contour corresponding to the contact part.

[0155] In one possible implementation, the ground contact length is determined based on the strip-shaped contour corresponding to the ground contact part; if the ground contact part includes other target parts except the foot, and the ground contact length is greater than the corresponding preset threshold, it is determined that the target object has fallen.

[0156] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0157] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0158] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A fall detection method, characterized in that: include: Detecting key point sets corresponding to a preset number of target parts in a target image and ground contacting parts of the target parts, wherein the target image includes at least one target object; Based on the key point set corresponding to the ground contact part, constructing a striped contour of the ground contact part by using a ground contact contour striping algorithm; Extracting a ground contact part feature based on the strip-shaped contour of the ground contact part; the ground contact part feature is a ground contact length of the ground contact part; Performing fall detection on the target object based on the ground contact part feature; When the number of detected target parts does not reach the preset number, a human body contour key point set of the target object is collected. ; Constructing a human body contour based on the human body contour key point set, and calculating the median height-to-width ratio of the human body contour; Determining whether the target object has fallen based on the median height-to-width ratio; The step of constructing a human body contour based on the human body contour key point set and calculating the median height-to-width ratio of the human body contour comprises: Connecting the human body contour key points in the human body contour key point set end to end in sequence to construct a human body contour; Based on the human body contour key point set, a human body mask is obtained by using a contour filling function in the OpenCV library; A horizontal line is constructed with the middle value of the height of the human body contour, and the width of the longest connected line segment where the horizontal line intersects with the human body mask is used as the human body width; The median height-to-width ratio of the human body contour is calculated by a first formula, wherein the first formula is: in, is the median height-to-width ratio of the human body contour, is the width of the human body, , Q is the index set, is the column coordinate of the left endpoint of the longest intersecting line segment, is the column coordinate of the right endpoint of the longest intersecting line segment, , are the row coordinates of the human body contour pixels; the index set Q={(i,j)|[xi,yi] , [xj,yj] ,and }; The performing fall detection on the target object based on the ground contact part feature includes: Determining a ground contact length based on a strip-shaped profile corresponding to the ground contact part; If the ground contact part includes other target parts except the foot, and the ground contact length is greater than the corresponding preset threshold, it is determined that the target object has fallen.

2. The method according to claim 1, characterized in that The step of constructing a horizontal line based on the middle value of the height of the human body contour and taking the longest connected line segment width where the horizontal line intersects with the human body mask as the human body width includes: Sorting the row coordinates in the human body contour key point set from small to large to obtain a height sequence; constructing a horizontal line with the middle value of the height in the height sequence; Obtain the intersection line segment between the human body mask and the horizontal line through connected component calculation; The width of the longest intersecting line segment is taken as the width of the human body.

3. The method according to claim 1, characterized in that The determining whether the target object falls based on the median height-to-width ratio includes: Pre-set the fall judgment threshold; If the median height-to-width ratio is less than the fall judgment threshold, it is determined that the target object has fallen.

4. The method according to claim 1, characterized in that: The step of constructing a stripped contour of the ground contact part by using a ground contact contour stripping algorithm based on a key point set corresponding to the ground contact part comprises: Connecting the key points in the key point set corresponding to the ground contact part in sequence end to end to construct a segmented contour corresponding to the ground contact part; Generate a strip corresponding to the segmented contour by using the line drawing function in the OpenCV library; The strip contour corresponding to the strip band is extracted by the contour search function in the OpenCV library to obtain the strip contour corresponding to the ground contacting part.

5. A fall detection device, characterized in that: include: A detection module is used to detect key point sets corresponding to a preset number of target parts in a target image and ground contact points of the target parts, wherein the target image includes at least one target object; when the number of detected target parts does not reach the preset number, collect the key point sets of the human body contour of the target object ; A construction module, configured to construct a striped contour of the contact part based on a key point set corresponding to the contact part by a contact contour striping algorithm; construct a human body contour based on the human body contour key point set, and calculate a median height-to-width ratio of the human body contour; An extraction module, configured to extract a ground contact part feature based on the strip-shaped contour of the ground contact part; the ground contact part feature is a ground contact length of the ground contact part; The detection module is further used to perform fall detection on the target object based on the ground contact part characteristics; and determine whether the target object falls based on the median height-to-width ratio; The step of constructing a human body contour based on the human body contour key point set and calculating the median height-to-width ratio of the human body contour comprises: Connecting the human body contour key points in the human body contour key point set end to end in sequence to construct a human body contour; Based on the human body contour key point set, a human body mask is obtained by using a contour filling function in the OpenCV library; A horizontal line is constructed with the middle value of the height of the human body contour, and the width of the longest connected line segment where the horizontal line intersects with the human body mask is used as the human body width; The median height-to-width ratio of the human body contour is calculated by a first formula, wherein the first formula is: in, is the median height-to-width ratio of the human body contour, is the width of the human body, , Q is the index set, is the column coordinate of the left endpoint of the longest intersecting line segment, is the column coordinate of the right endpoint of the longest intersecting line segment, , are the row coordinates of the human body contour pixels; the index set Q={(i,j)|[xi,yi] , [xj,yj] ,and }; The performing fall detection on the target object based on the ground contact part feature includes: Determining a ground contact length based on a strip-shaped profile corresponding to the ground contact part; If the ground contact part includes other target parts except the foot, and the ground contact length is greater than the corresponding preset threshold, it is determined that the target object has fallen.

6. A mobile robot fall detection system, using the fall detection method according to any one of claims 1 to 4, characterized in that: include: Camera, image processing module and fall detection module; The camera is used to collect an environmental image of the current environment; The image processing module is used to receive the environment image and detect a target image with a target object in the environment image; The fall detection module is used to perform fall detection on the target object in the target image.

7. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the fall detection method according to any one of claims 1 to 4.

Images