Fall behavior detection method and device based on patrol robot, medium and equipment

Abstract

The application discloses a kind of based on patrol robot's fall behavior detection method, comprising: control patrol robot moves in preset place, and the image signal of the preset place is collected;Image information is detected for fall behavior, and the human body information of suspected fall state is obtained;The key point detection is carried out to the human body information of suspected fall state, and the human body key point information and its confidence are obtained;According to the filtering logic of preestablished, the human body key point information and its confidence are screened, and fall behavior person is obtained.The application effectively solves the problems of missing detection and false detection, complex deployment and high detection cost in the prior art when detecting fall behavior.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, medium and equipment for detecting fall behavior based on patrol robots. Background Technology

[0002] Due to declining physical functions, elderly people often have difficulty getting up on their own if they fall while walking in parks or nursing homes. Therefore, management of public places such as parks and nursing homes, for the safety of the elderly, aim to promptly detect falls and provide assistance.

[0003] To address the need for timely detection of falls among the elderly in public places such as parks and nursing homes, existing technologies primarily employ three methods for fall detection. The first method uses deep learning-based human target detection and key point detection to detect falls. However, this method detects too many human states, leading to significant missed and false positives. The second method uses deep learning-based pedestrian re-identification, human key point detection, and motion capture technology to detect falls, but this method suffers from complex deployment and high costs. The third method uses deep learning and LiDAR-based 3D point cloud human detection technology to detect falls. However, this method has expensive imaging sensors and misses a large number of people in a fallen state. Summary of the Invention

[0004] This invention provides a method, apparatus, medium, and equipment for detecting fall behavior based on patrol robots, in order to solve the problems of missed detections and false detections, complex deployment, and high detection costs in existing technologies when detecting fall behavior.

[0005] A method for detecting fall behavior based on a patrol robot, the method comprising:

[0006] Control the patrol robot to move within a predetermined area and collect image signals from the predetermined area;

[0007] The image information is used to detect fall behavior and obtain information about a human body in a suspected fall state.

[0008] Key point detection is performed on the human body information suspected of being in a fall state to obtain human body key point information and its confidence level;

[0009] The key information of the human body and its confidence level are filtered according to the preset filtering logic to obtain the person who fell.

[0010] Optionally, the step of detecting fall behavior in the image information to obtain human information of a suspected fall includes:

[0011] The image information is downsampled, and each pixel in the downsampled image information is preprocessed to obtain the image to be detected.

[0012] The image to be detected is input into a preset deep learning object detection network for detection to obtain a human body in a suspected fall state and its target bounding box.

[0013] Based on the human body and its target bounding box, the corresponding image region is extracted from the image information as the human body information in a suspected fall state.

[0014] Optionally, the step of performing key point detection on the human body information of the suspected fall state to obtain human body key point information and its confidence level includes:

[0015] Preprocess the information of the human body suspected of falling;

[0016] The preprocessed human body information is input into the preset key point detection framework Alphapose for key point detection to obtain human body key point information and its confidence level.

[0017] Optionally, the backbone network of the key point detection framework Alphapose adopts a ResNet18 network, uses the PixelShuffle operator during training, and uses the deconvolution operator during inference.

[0018] During the training phase, the human body key point information is divided into a first type of human body key point information and a second type of human body key point information. The first type of human body key point information is obtained through annotation, and the second type of human body key point information is obtained by calculating the median value based on the position of the specified key point in the first type of human body key point information.

[0019] Optionally, the step of filtering the key human body information and its confidence level according to a preset filtering logic to obtain the fall victim includes:

[0020] The target key points and their confidence levels are obtained from the human body key point information and their confidence levels. The target key points include head key points, right ankle key points, left ankle key points, midpoint between the two shoulders, and midpoint between the two hips.

[0021] The confidence level of the target key point is compared with a preset confidence threshold.

[0022] If the confidence level of at least one target key point is greater than or equal to the preset confidence threshold, then the confidence levels of the head key point and the midpoint between the two shoulders are compared with the preset confidence threshold respectively.

[0023] If the confidence level of any one of the head key point and the midpoint between the two shoulders is greater than or equal to the preset confidence threshold, compare the confidence level of the midpoint between the two hips with the preset confidence threshold.

[0024] When the confidence level of the midpoint between the two hips is greater than or equal to the preset confidence threshold, the cosine value of the upper body is calculated based on the midpoint between the two shoulders, the midpoint between the two hips, and the head key point. The cosine value of the left leg is calculated based on the comparison result of the left ankle key point with the preset confidence threshold. The cosine value of the right leg is calculated based on the comparison result of the right ankle key point with the preset confidence threshold. The upper body cosine value, right leg cosine value, and left leg cosine value are used to filter and obtain the fall behavior.

[0025] When the confidence level of the midpoint between the two hips is less than the preset confidence threshold, the cosine values ​​of the right side and the left side of the whole body are calculated based on the midpoint between the two shoulders, the key point of the head, the key point of the right ankle, and the key point of the left ankle. The cosine values ​​of the left side and the right side of the whole body are used to filter and identify the person who fell.

[0026] Optionally, the step of calculating the upper body cosine value based on the midpoint between the two shoulders, the midpoint between the two hips, and the head key point; calculating the left leg cosine value based on the comparison result of the left ankle key point with a preset reliability threshold; calculating the right leg cosine value based on the comparison result of the right ankle key point with a preset reliability threshold; and filtering to obtain fall-causing individuals based on the upper body cosine value, right leg cosine value, and left leg cosine value includes:

[0027] If the confidence level of the midpoint between the two shoulders is greater than or equal to the preset confidence threshold, then the cosine of the angle between the line connecting the midpoint between the two hips and the midpoint between the two shoulders and the horizontal line is calculated as the cosine value of the upper body.

[0028] If the confidence level of the midpoint between the two shoulders is less than the preset confidence threshold, then the head key point is detected. When the confidence level of the head key point is greater than or equal to the preset confidence threshold, the cosine of the angle between the line connecting the midpoint between the two hips and the head key point and the horizontal line is calculated as the upper body cosine value. When the confidence level of the head key point is less than the preset confidence threshold, the specified value is obtained as the upper body cosine value.

[0029] When the confidence level of the left ankle key point is greater than or equal to the preset confidence threshold, the cosine of the angle between the line connecting the midpoint of the two hips and the left ankle key point and the horizontal line is calculated as the cosine value of the left leg; otherwise, the specified value is obtained as the cosine value of the left leg.

[0030] When the confidence level of the right ankle key point is greater than or equal to the preset confidence threshold, the cosine of the angle between the line connecting the midpoint of the two hips and the right ankle key point and the horizontal line is calculated as the cosine value of the right leg; otherwise, the specified value is obtained as the cosine value of the right leg.

[0031] Compare the cosine value of the upper body with the first cosine value, compare the cosine value of the left leg with the second cosine value, compare the cosine value of the right leg with the second cosine value; and / or, compare the cosine value of the upper body with the first cosine value, compare the cosine value of the left leg with the third cosine value, and compare the cosine value of the right leg with the third cosine value.

[0032] When the cosine value of the upper body is less than or equal to the first cosine value, the cosine value of the left leg is greater than or equal to the second cosine value, and the cosine value of the right leg is greater than or equal to the second cosine value, or when the cosine value of the upper body is less than or equal to the first cosine value, the cosine value of the left leg is less than or equal to the third cosine value, and the cosine value of the right leg is less than or equal to the third cosine value, the human body information indicates that the person is not a person who has fallen; otherwise, the human body information indicates that the person has fallen.

[0033] Optionally, the step of calculating the right-side cosine value and the left-side cosine value of the whole body based on the midpoint between the two shoulders, the key point of the head, the key point of the right ankle, and the key point of the left ankle, and filtering to obtain the fall victim based on the left-side cosine value and the right-side cosine value includes:

[0034] If the confidence level of the midpoint between the two shoulders is greater than or equal to the preset confidence threshold, when the confidence level of the left ankle key point is greater than the preset confidence threshold, the cosine of the angle between the line connecting the midpoint between the two shoulders and the left ankle key point and the horizontal line is calculated as the cosine value of the left side of the whole body; otherwise, a specified value is obtained as the cosine value of the left side of the whole body. When the confidence level of the right ankle key point is greater than the preset confidence threshold, the cosine of the angle between the line connecting the midpoint between the two shoulders and the right ankle key point and the horizontal line is calculated as the cosine value of the right side of the whole body; otherwise, a specified value is obtained as the cosine value of the right side of the whole body.

[0035] If the confidence level of the midpoint between the two shoulders is less than the preset confidence threshold, then the head keypoint is detected. When the confidence level of the head keypoint is greater than or equal to the preset confidence threshold, if the confidence level of the left ankle keypoint is greater than the preset confidence threshold, the cosine of the angle between the line connecting the head keypoint and the left ankle keypoint and the horizontal line is calculated as the cosine value of the left side of the whole body; otherwise, a specified value is obtained as the cosine value of the left side of the whole body. When the confidence level of the right ankle keypoint is greater than the preset confidence threshold, the cosine of the angle between the line connecting the head keypoint and the right ankle keypoint and the horizontal line is calculated as the cosine value of the right side of the whole body; otherwise, a specified value is obtained as the cosine value of the right side of the whole body.

[0036] Compare the cosine value of the right side of the whole body with the fourth cosine value, and compare the cosine value of the left side of the whole body with the fourth cosine value;

[0037] When the cosine value of the right side of the body is less than the fourth cosine value, and the cosine value of the left side of the body is less than the fourth cosine value, the human body information indicates that the person is not a person who has fallen; otherwise, the human body information indicates that the person has fallen.

[0038] A method for detecting fall behavior based on a patrol robot, the device comprising:

[0039] The acquisition module is used to control the patrol robot to move in a preset location and to acquire image signals of the preset location;

[0040] The fall detection module is used to detect fall behavior in the image information and obtain human information in a suspected fall state.

[0041] The key point detection module is used to detect key points in the human body information of the suspected fall state, and obtain human body key point information and its confidence level;

[0042] The filtering module is used to filter the key human body information and its confidence level according to preset filtering logic to identify the person who fell.

[0043] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the fall detection method based on a patrol robot as described above.

[0044] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the fall detection method based on a patrol robot as described above.

[0045] This invention, through its embodiments, controls a patrol robot to move within a preset location and collects image signals from that location. It then performs fall detection on the image information to obtain information on individuals suspected of falling. Focusing only on individuals suspected of falling, it filters out false positives by combining key points. The system then performs key point detection on the suspected fallen individuals to obtain their key point information and confidence levels. Following a preset filtering logic, it selects the individuals who have fallen, effectively reducing false positives and false negatives. This method avoids obscured normal walking individuals and detects fallen individuals whose upper bodies are obscured. Furthermore, the equipment is low-cost and easy to deploy. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart of a fall detection method based on a patrol robot provided in an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of the layout of key points of the human body provided in an embodiment of the present invention;

[0049] Figure 3This is a schematic diagram of the key point detection results of human body information in a suspected fall state provided by an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of the structure of a fall detection device based on a patrol robot provided in an embodiment of the present invention;

[0051] Figure 5 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] The fall detection method based on a patrol robot provided in this invention divides fall detection into two steps. In the first step, only people in a suspected fall state are considered, without detecting people in all states. When training the deep learning network, the fall state is treated as the target, and only information about people in a fall state is detected to reduce false positives and false negatives. In the second step, based on the information about people in a suspected fall state, logical filtering is performed using key points of the human body to filter out falsely detected states from the first step, thereby improving the accuracy of fall detection. Experiments show that this embodiment, by deliberately labeling people in a fall state during training, can not only avoid occluded normally walking people but also detect occluded upper body bodies in a fall state. Furthermore, the superimposed logical filtering effectively solves the problems of false positives and false negatives, complex deployment, and high detection costs in existing technologies for fall detection.

[0054] The following provides a detailed description of the fall detection method based on a patrol robot provided in this embodiment. Figure 1 As shown, the fall detection method based on patrol robots includes:

[0055] Step S101: Control the patrol robot to move in a preset location and collect image signals of the preset location;

[0056] Step S102: Detect fall behavior in the image information to obtain information about a human body in a suspected fall state;

[0057] Step S103: Perform key point detection on the human body information of the suspected fall state to obtain human body key point information and its confidence level;

[0058] Step S104: The key information of the human body and its confidence level are filtered according to the preset filtering logic to obtain the person who fell.

[0059] The patrol robot is equipped with a camera, for example, on its head. The designated locations primarily refer to public places where fall detection is required, including but not limited to hospitals, parks, and nursing homes.

[0060] As an example, in step S101, the patrol robot is activated and patrols the preset location along a fixed route. At the same time, it turns on its camera to collect image signals in real time and stores them on its own storage device.

[0061] The fall detection refers to identifying whether a fall occurs in the image signal. The detected fall is recorded as a suspected fall.

[0062] As an example, in step S102, this embodiment pre-trains a deep learning network that only detects human information in a fall state. By using the human fall state as the training target, the data samples are labeled only with the human body in a fall state. Then, the pre-set deep learning network is trained using data samples with deliberately labeled fall states, thereby obtaining a deep learning target detection network specifically for detecting fall behavior. By detecting only human information in a fall state, rather than detecting all states, such as standing or sitting, it is beneficial for subsequent accurate judgment.

[0063] The key point detection refers to identifying specific key points in the information of the human body suspected of falling.

[0064] As an example, in step S103, if the human body information indicating a suspected fall is output in step S102, this embodiment uses a specified key point detection framework to detect key points in the human body information indicating a suspected fall. Optionally, the key point detection framework is preferably the human pose recognition Alphapose.

[0065] The filtering logic refers to the logic of filtering out false positives from the human information of suspected fall behavior and obtaining the judgment logic of the person who fell.

[0066] As an example, in step S104, this embodiment determines whether the key points and confidence level corresponding to the human information of the suspected fall behavior meet the preset logic. If they meet the logic, the suspected fall behavior is considered to be a false detection state target. If they do not meet the logic, the suspected fall behavior is considered to be a real fall behavior, and a person who has fallen is generated.

[0067] As can be seen, the fall detection method based on patrol robots provided by this invention effectively expands the detection range by leveraging the mobility and image acquisition capabilities of the patrol robot, and is easy and simple to deploy. Compared to detecting human bodies in all states, such as standing, sitting, and fallen, the filtering logic of this invention is simple and easy to implement. Compared to existing technologies that rely on pedestrian re-identification, key point detection, and motion capture detection for fall detection, this invention has moderate complexity, is simpler to deploy, and has lower deployment costs. This embodiment only requires a single camera, resulting in low-cost imaging sensors and significantly improved detection accuracy.

[0068] In one embodiment, step S102, which involves detecting a fall in the image information to obtain information about a person in a suspected fall state, includes:

[0069] Step S1021: Downsample the image information, and preprocess each pixel in the downsampled image information to obtain the image to be detected;

[0070] Step S1022: Input the image to be detected into a preset deep learning object detection network for detection to obtain a human body in a suspected fall state and its target box;

[0071] Step S1023: Extract the corresponding image region from the image information based on the human body and its target bounding box, as the human body information of the suspected fallen state.

[0072] The image to be detected is the input parameter of the deep learning object detection network, which is obtained by downsampling and preprocessing the image information.

[0073] As an example, in step S1021, since the original image information captured by the camera on the patrol robot is 1920*1080 in size, this embodiment first performs downsampling, downsampling the 1920*1080 image information to the target size of 768*448, and then divides the pixel value of each pixel by 255 to obtain the image to be detected.

[0074] The deep learning object detection network is used to detect fall behavior. This embodiment uses a single-class YOLOv8 object detection network to detect a human body in a fall state. To improve inference speed, the YOLOv8 object detection network is improved as follows:

[0075] The ReLU activation function is used instead of the SiLU activation function in the original YOLOv8.

[0076] A combination of operators, including deconvolution, batch normalization, and ReLU activation, is used to replace the upsample operator in the original YOLOv8.

[0077] In this embodiment, the sample data used when training the deep learning object detection network is self-collected and self-labeled data samples.

[0078] As an example, in step S1022, the image to be detected is input into a preset deep learning target detection network to detect the fall state, so as to complete the preliminary detection of the fall state.

[0079] As an example, in step S1023, if information about a person in a fallen state is detected, a region of interest (ROI) image of the size of the target bounding box is extracted from the original image information and recorded as information about a person in a suspected fallen state. If no information about a person in a fallen state is detected, the detection ends.

[0080] Optionally, the deep learning object detection network ultimately outputs information about a human body in a fallen state, including human ROI image data and bounding boxes of the fallen human body. The ROI image data format is [N,3,h,w], where N is the number of ROIs, 3 is the number of ROI image channels, h is the height of the ROI image, and w is the width of the ROI image. The bounding box format is [x1,y1,x2,y2,id], where x1 and y1 represent the coordinates of the top-left corner of the box, x2 and y2 represent the coordinates of the bottom-right corner of the box, and id represents the category of the object within the box; id = 0 indicates "fallen human body".

[0081] This embodiment only detects human information in a fallen state, and specifically labels the fallen human body for model training. It can not only avoid the normally walking human body that is obscured, but also detect the upper body of the fallen human body that is obscured. Compared with the existing technology that detects human bodies in all states, such as standing, sitting, and fallen, this embodiment effectively reduces false detections and false negatives, and is simpler to deploy and easier to implement.

[0082] In one embodiment, step S103, which involves performing key point detection on the human body information of the suspected fall state to obtain human key point information and its confidence level, includes:

[0083] Step S1031: Preprocess the information of the human body in the suspected fall state;

[0084] Step S1032: Input the preprocessed human body information into the preset key point detection framework Alphapose to perform key point detection and obtain human body key point information and its confidence level.

[0085] The preprocessing refers to converting the human information of the suspected fall into input parameters adapted to the key point detection framework Alphapose.

[0086] As an example, in step S1031, the human ROI image data in the suspected fall state is preprocessed, including scaling to a preset size, subtracting a preset channel mean by channel, and finally multiplying each pixel value by a preset ratio. Optionally, the preset size is, for example, 256*192, transforming the human ROI image data from [N,3,h,w] to [N,3,256,192]. The preset channel mean is [103.53,116.535,122.40], and the preset ratio is 0.0039215686. In this way, the human ROI image in the suspected fall state is normalized to a Gaussian distribution with a mean of 0 and a variance of 1, thereby accelerating training convergence.

[0087] In this embodiment, the human body key point information includes a total of 15 points, and their layout is as follows: Figure 2 As shown, the key points are as follows: 1 represents the head, 2 represents the left shoulder, 3 represents the right shoulder, 4 represents the left elbow, 5 represents the right elbow, 6 represents the left wrist, 7 represents the right wrist, 8 represents the left hip, 9 represents the right hip, 10 represents the left knee, 11 represents the right knee, 12 represents the left ankle, 13 represents the right ankle, 14 represents the midpoint between the shoulders, and 15 represents the midpoint between the hips.

[0088] The Alphapose keypoint detection framework uses a ResNet18 network as its backbone network, employing the PixelShuffle operator during training and a deconvolution operator during inference. To improve the speed and accuracy of human keypoint detection, this embodiment adapts the Alphapose framework, using a ResNet18 residual network as the backbone network. The PixelShuffle operator in Alphapose remains unchanged during training but is replaced with an equivalent Deconvolution operator during inference.

[0089] To ensure robustness during the inference phase, this embodiment categorizes the human body keypoint information into two types during the training phase: a first type and a second type. The first type of keypoint information is obtained through annotation, while the second type is obtained by calculating the median value based on the positions of specified keypoints within the first type. Optionally, for keypoints labeled 1-15, keypoints labeled 1-13 can be obtained through annotation and designated as the first type. Keypoints labeled 14 and 15 are obtained through calculation and designated as the second type. Specifically, label 14 represents the midpoint between the two shoulders, calculated using the median value of the right shoulder keypoint (labeled 3) and the left shoulder keypoint (labeled 2); label 15 represents the midpoint between the two hips, calculated using the median value of the right hip keypoint (labeled 9) and the left hip keypoint (labeled 8).

[0090] The confidence level is the criterion for judging whether someone has fallen. The higher the confidence level, the greater the probability of falling; conversely, the lower the confidence level, the lower the probability of falling.

[0091] As an example, in step S1032, this embodiment inputs the preprocessed human body information into a preset keypoint detection framework, Alphapose, for keypoint detection. The Alphapose framework ultimately outputs the keypoints of the human body information in the suspected fall state. The data format for the 15 human keypoints is [15, x, y, conf], where 15 represents the number of keypoints, x and y represent the coordinates of the human keypoints on the image, and conf represents the confidence level of the human keypoints. In this embodiment, keypoints with a confidence level (conf) greater than 0.5 are selected for logical filtering. For ease of understanding, Figure 3 The key point detection results for human information in a suspected fall state provided in this embodiment.

[0092] This embodiment specifically optimizes the Alphapose framework for patrol robot deployment platforms, and selects key points at appropriate locations. Alphapose is retrained on a dataset composed of multiple merged datasets, which not only improves the key point detection speed but also the detection accuracy, effectively supporting subsequent logical filtering.

[0093] In one embodiment, step S104, namely, filtering the key human body information and its confidence level according to a preset filtering logic to obtain the person who fell, includes:

[0094] Step S1041: Obtain target key points and their confidence from the human body key point information and their confidence. The target key points include head key point 1, right ankle key point 13, left ankle key point 12, midpoint between the two shoulders 14, and midpoint between the two hips 15.

[0095] Step S1042: Compare the confidence level of the target key point with a preset confidence threshold;

[0096] Step S1043: If the confidence level of at least one target key point is greater than or equal to a preset confidence threshold, then the confidence levels of the head key point and the midpoint between the two shoulders are compared with the preset confidence threshold respectively.

[0097] Step S1044: If the confidence level of any one of the head key point and the midpoint between the two shoulders is greater than or equal to the preset confidence threshold, compare the confidence level of the midpoint between the two hips with the preset confidence threshold.

[0098] Step S1045: When the confidence level of the midpoint between the two hips is greater than or equal to the preset confidence threshold, calculate the cosine value of the upper body based on the midpoint between the two shoulders, the midpoint between the two hips, and the head key point; calculate the cosine value of the left leg based on the comparison result of the left ankle key point with the preset confidence threshold; calculate the cosine value of the right leg based on the comparison result of the right ankle key point with the preset confidence threshold; and filter for fall victims based on the upper body cosine value, right leg cosine value, and left leg cosine value.

[0099] Step S1046: When the confidence level of the midpoint between the two hips is less than the preset confidence threshold, calculate the right cosine value and the left cosine value of the whole body based on the midpoint between the two shoulders, the head key point, the right ankle key point, and the left ankle key point. Then, filter the fall victim based on the left cosine value and the right cosine value of the whole body.

[0100] The target key points are obtained from 15 human body key point information, preferably head key point 1, right ankle key point 13, left ankle key point 12, midpoint between shoulders 14, and midpoint between hips 15. By training the model to detect multiple human body key points, compared with training the model to detect only the target key point, it is beneficial to improve the accuracy of the model in detecting key points and can predict key points more accurately.

[0101] The confidence threshold is used as a criterion for determining whether a corresponding key point is a valid key point. The confidence threshold is preferably 0.5, meaning that when the confidence of a target key point is greater than or equal to 0.5, the target key point is valid and can indicate that the human body is in a fall; otherwise, the target key point is invalid and cannot indicate that the human body is in a fall.

[0102] As an example, in step S1042, this embodiment iterates through the confidence level of each target key point, comparing the head key point 1, right ankle key point 13, left ankle key point 12, midpoint between shoulders 14, and midpoint between hips 15 with a preset confidence threshold. When the confidence levels of the head key point 1, right ankle key point 13, left ankle key point 12, midpoint between shoulders 14, and midpoint between hips 15 are all less than the confidence threshold, it indicates that the suspected fall information is not a real fall, and the information is discarded, moving on to the next suspected fall information.

[0103] If the confidence level of at least one target keypoint is greater than or equal to a preset confidence threshold, then check the confidence levels of head keypoint 1 and midpoint 14 between the shoulders, comparing them with the preset confidence thresholds respectively. If the confidence levels of both head keypoint 1 and midpoint 14 between the shoulders are less than the preset confidence thresholds, discard the human body information and proceed to the next suspected fall human body information. If the confidence level of at least one of head keypoint 1 and midpoint 14 between the shoulders is greater than or equal to the preset confidence threshold, compare the confidence level of midpoint 15 between the hips with the preset confidence threshold.

[0104] This embodiment employs different filtering logics based on the comparison between the confidence level of the midpoint 15 between the two hips and a preset confidence threshold. When the confidence level of the midpoint 15 between the two hips is greater than or equal to the preset confidence threshold, it is considered a valid keypoint, indicating that the hips of the person have fallen. Then, based on the state of the person's legs during the fall, the actual fall behavior is further filtered out. The cosine value of the upper body is calculated based on the midpoint 14 between the two shoulders, the midpoint 15 between the two hips, and the head keypoint 1. The cosine value of the left leg is calculated based on the comparison between the left ankle keypoint 12 and the preset confidence threshold. The cosine value of the right leg is calculated based on the comparison between the right ankle keypoint 13 and the preset confidence threshold. Finally, the upper body cosine value, right leg cosine value, and left leg cosine value are used for filtering, excluding individuals sitting on the ground with their legs extended, or sitting in a chair or squatting, ultimately identifying the person who has fallen.

[0105] When the confidence level of the midpoint 15 between the two hips is less than the preset confidence threshold, it is considered an invalid keypoint, indicating that the hips of the person have not fallen. Further screening is then conducted based on the tilt of the left and right sides of the body to identify the actual fall behavior. The cosine values ​​of the right and left sides of the entire body are calculated based on the midpoint 14 between the two shoulders, the head keypoint 1, the right ankle keypoint 13, and the left ankle keypoint 12. Then, based on these cosine values, the body is filtered to exclude individuals in a bent-over position or those tilting while holding onto an object, such as a leaf, ultimately identifying those who have fallen.

[0106] In one embodiment, step S1045, which involves calculating the cosine value of the upper body based on the midpoint between the two shoulders, the midpoint between the two hips, and the head key point; calculating the cosine value of the left leg based on the comparison result of the left ankle key point with a preset confidence threshold; calculating the cosine value of the right leg based on the comparison result of the right ankle key point with a preset confidence threshold; and filtering based on the upper body cosine value, right leg cosine value, and left leg cosine value to obtain the person who fell, includes:

[0107] Step S451: If the confidence level of the midpoint between the two shoulders is greater than or equal to the preset confidence threshold, then calculate the cosine value of the angle between the line connecting the midpoint between the two hips and the midpoint between the two shoulders and the horizontal line as the upper body cosine value.

[0108] Step S452: If the confidence level of the midpoint between the two shoulders is less than the preset confidence threshold, then detect the head key point. When the confidence level of the head key point is greater than or equal to the preset confidence threshold, calculate the cosine of the angle between the line connecting the midpoint between the two hips and the head key point and the horizontal line as the upper body cosine value. When the confidence level of the head key point is less than the preset confidence threshold, obtain the specified value as the upper body cosine value.

[0109] Step S453: When the confidence level of the left ankle key point is greater than or equal to the preset confidence threshold, calculate the cosine value of the angle between the line connecting the midpoint of the two hips and the left ankle key point and the horizontal line as the cosine value of the left leg; otherwise, obtain the specified value as the cosine value of the left leg.

[0110] Step S454: When the confidence level of the right ankle key point is greater than or equal to the preset confidence threshold, calculate the cosine of the angle between the line connecting the midpoint of the two hips and the right ankle key point and the horizontal line as the cosine value of the right leg; otherwise, obtain the specified value as the cosine value of the right leg.

[0111] Step S455: Compare the cosine value of the upper body with the first cosine value, compare the cosine value of the left leg with the second cosine value, compare the cosine value of the right leg with the second cosine value; and / or, compare the cosine value of the upper body with the first cosine value, compare the cosine value of the left leg with the third cosine value, and compare the cosine value of the right leg with the third cosine value;

[0112] Step S456: When the cosine value of the upper body is less than or equal to the first cosine value, the cosine value of the left leg is greater than or equal to the second cosine value, and the cosine value of the right leg is greater than or equal to the second cosine value, or when the cosine value of the upper body is less than or equal to the first cosine value, the cosine value of the left leg is less than or equal to the third cosine value, and the cosine value of the right leg is less than or equal to the third cosine value, the human body information indicates that the person is not falling; otherwise, the human body information indicates that the person is falling.

[0113] Here, the upper body cosine value body_cos represents the degree of inclination of the upper body torso relative to the horizontal line. Since the head and shoulders are both at the top, in calculating the upper body cosine value, this embodiment selects the calculation object based on the confidence level corresponding to the key point 1 of the head and the midpoint 14 of the shoulders.

[0114] As an example, in steps S451 and S452, the confidence level conf of the middle_shoulder of the two shoulder midpoints 14 is checked. If it is greater than 0.5, it indicates that the two shoulder midpoints 14 are valid. Then, the cosine value of the line connecting the two hip midpoints middle_hip and the two shoulder midpoints middle_shoulder and the horizontal line is calculated and assigned to the variable body_cos. If it is less than 0.5, the confidence level conf of the head keypoint middle_head is checked. If the confidence level conf is greater than 0.5, the cosine value of the line connecting the two hip midpoints middle_hip and the head keypoint middle_head and the horizontal line is calculated and assigned to the variable body_cos. If the confidence level conf of the head keypoint middle_head is less than 0.5, the specified value -100 is obtained and assigned to the variable body_cos.

[0115] The left leg cosine value represents the degree of inclination of the left leg relative to the horizontal line in the lower body. The right leg cosine value represents the degree of inclination of the right leg relative to the horizontal line in the lower body.

[0116] As an example, in step S453, the confidence of the left ankle keypoint left_ankle is checked. If it is greater than 0.5, it means that the left ankle keypoint is valid. Then, the cosine value of the line connecting the midpoint between the two hips, the left ankle keypoint left_ankle, and the horizontal line is calculated and assigned to the variable left leg cosine value leg_left_cos. If it is less than 0.5, the specified value -100 is obtained and assigned to the variable left leg cosine value leg_left_cos.

[0117] As an example, in step S454, the confidence level conf of the right ankle keypoint right_ankle is checked. If it is greater than 0.5, it means that the right ankle keypoint is valid. Then, the cosine value of the line connecting the midpoint middle_hip of the two hips, the right ankle keypoint right_ankle, and the horizontal line is calculated and assigned to the variable leg cosine value leg_right_cos. If it is less than 0.5, the specified value -100 is assigned to the variable leg cosine value leg_right_cos.

[0118] The first cosine value, used as the criterion for determining whether the upper body is upright and the hips are on the ground, is preferably 0.1. The second cosine value, used as the criterion for determining whether the left or right leg is extended to the left, is preferably 0.9. The third cosine value, used as the criterion for determining whether the left or right leg is extended to the right, is preferably 0.2.

[0119] As an example, in steps S455 and S456, this embodiment determines whether the cosine values ​​of the upper body (body_cos), left leg (leg_left_cos), and right leg (leg_right_cos) satisfy the following conditions: (body_cos != -100 && body_cos <= 0.1) and (leg_left_cos >= 0.9 || leg_right_cos >= 0.9);

[0120] If the conditions are met, it indicates that the person suspected of falling is sitting on the ground with both legs extended to the left, not in a falling state. The information for this person is discarded, and the process moves on to the next person suspected of falling. Additionally, it is determined whether the cosine values ​​of the upper body (body_cos), left leg (leg_left_cos), and right leg (leg_right_cos) satisfy the following conditions: (body_cos != 100 && body_cos <= 0.1) and (leg_left_cos >= 0.9 || leg_right_cos >= 0.9).

[0121] If the condition is met, it means that the person suspected of falling is sitting on the ground with both legs extended to the right, and is not in a falling state. The information of this person is discarded, and the process moves on to the next person suspected of falling.

[0122] In one embodiment, step S1046, namely, calculating the cosine values ​​of the right side and the left side of the body based on the midpoint between the two shoulders, the key point of the head, the key point of the right ankle, and the key point of the left ankle, and filtering to obtain the person who fell, includes:

[0123] Step S461: If the confidence level of the midpoint between the two shoulders is greater than or equal to a preset confidence threshold, when the confidence level of the left ankle key point is greater than the preset confidence threshold, calculate the cosine of the angle between the line connecting the midpoint between the two shoulders and the left ankle key point and the horizontal line as the cosine value of the left side of the whole body; otherwise, obtain a specified value as the cosine value of the left side of the whole body. When the confidence level of the right ankle key point is greater than the preset confidence threshold, calculate the cosine of the angle between the line connecting the midpoint between the two shoulders and the right ankle key point and the horizontal line as the cosine value of the right side of the whole body; otherwise, obtain a specified value as the cosine value of the right side of the whole body.

[0124] Step S462: If the confidence level of the midpoint between the two shoulders is less than the preset confidence threshold, then detect the head key points. When the confidence level of the head key points is greater than or equal to the preset confidence threshold, if the confidence level of the left ankle key point is greater than the preset confidence threshold, calculate the cosine of the angle between the line connecting the head key point and the left ankle key point and the horizontal line as the cosine value of the left side of the whole body; otherwise, obtain a specified value as the cosine value of the left side of the whole body. When the confidence level of the right ankle key point is greater than the preset confidence threshold, calculate the cosine of the angle between the line connecting the head key point and the right ankle key point and the horizontal line as the cosine value of the right side of the whole body; otherwise, obtain a specified value as the cosine value of the right side of the whole body.

[0125] Step S463: Compare the cosine value of the right side of the whole body with the fourth cosine value, and compare the cosine value of the left side of the whole body with the fourth cosine value;

[0126] Step S464: When the cosine value of the right side of the body is less than the fourth cosine value and the cosine value of the left side of the body is less than the fourth cosine value, the human body information indicates that the person is not a person who has fallen; otherwise, the human body information indicates that the person has fallen.

[0127] Wherein, the left-side cosine value body_leg_left_cos represents the degree of inclination of the left side of the human body relative to the horizontal line. The right-side cosine value body_leg_right_cos represents the degree of inclination of the right side of the human body relative to the horizontal line. Since the head and both shoulders are located at the top, this embodiment selects the calculation object based on the confidence level corresponding to the key point 1 of the head and the midpoint 14 of the two shoulders when calculating the left-side and right-side cosine values.

[0128] As an example, in step S461, the confidence level conf of the middle point 'middle_shoulder' between the two shoulders is checked. If it is greater than 0.5, it means that the middle point 'middle_shoulder' between the two shoulders is valid. Then, the confidence level conf of the left ankle keypoint 'left_ankle' is checked. If it is greater than 0.5, it means that the left ankle keypoint is valid. The cosine value of the line connecting the middle point 'middle_shoulder' between the two shoulders and the left ankle keypoint 'left_ankle' and the horizontal line is calculated and assigned to the variable body_leg_left_cos. If it is less than 0.5, the specified value -100 is assigned to the variable body_leg_left_cos. Check the confidence score conf of the right ankle keypoint right_ankle. If it is greater than 0.5, it means that the right ankle keypoint is valid. Then calculate the cosine of the line connecting the midpoint middle_shoulder of the two shoulders, the right ankle keypoint right_ankle, and the horizontal line, and assign it to the variable body_leg_right_cos. If it is less than 0.5, assign the specified value -100 to the variable body_leg_right_cos.

[0129] As an example, in step S462, if the confidence score conf of the midpoint between the two shoulders (middle_shoulder) is less than 0.5, it means that the midpoint between the two shoulders (middle_shoulder) is invalid; continue to check the confidence score conf of the head keypoint (middle_head). If it is greater than 0.5, it means that the head keypoint (middle_head) is valid. Then check the confidence score conf of the left ankle keypoint (left_ankle). If it is greater than 0.5, calculate the cosine value of the line connecting the head keypoint (middle_head) and the left ankle keypoint (left_ankle) and the horizontal line, and assign it to the variable body_leg_left_cos. If it is less than 0.5, assign the specified value -100 to the variable body_leg_left_cos. Check the confidence score conf of the right ankle keypoint right_ankle. If it is greater than 0.5, calculate the cosine of the line connecting the head keypoint middle_head and the right ankle keypoint right_ankle and the horizontal line, and assign it to the variable body_leg_right_cos. If it is less than 0.5, assign the specified value -100 to the variable body_leg_right_cos.

[0130] The fourth cosine value, used as a criterion for judging whether a person is tilting or bending over, is preferably 0.7.

[0131] As an example, in step S463, this embodiment determines whether the following condition is met based on the cosine value of the left side of the body (body_leg_left_cos) and the cosine value of the right side of the body (body_leg_right_cos) obtained in step S461 or S462: (body_leg_left_cos < 0.7 && body_leg_left_cos != -100)||

[0132] If (body_leg_right_cos<0.7&&body_leg_right_cos!=-100), it means that the human body suspected of falling is in a bent-over or tilted state, not a falling state. The human body information is discarded and the process moves on to the next human body suspected of falling.

[0133] After steps S1045 and S1046, the information of the human body in the fall state is finally obtained and output, including the target box, which is in the format [x1,y1,x2,y2,id], where x1 and y1 represent the coordinates of the upper left corner of the box, x2 and y2 represent the coordinates of the lower right corner of the box, and id represents the category of the target in the box. id is 0, which means "the person who fell".

[0134] In summary, this embodiment uses filtering logic to filter human information that appears to be in a fallen state. Compared to existing technologies that detect all human states, including but not limited to standing, sitting, and fallen, this embodiment has the advantages of simpler filtering rules and easier implementation. Compared to existing technologies that perform pedestrian re-identification, key point detection, and motion capture detection, this embodiment has moderate complexity, is simpler to deploy, and has lower deployment costs. Using only a single camera, it not only has a low-cost imaging sensor but also high detection accuracy.

[0135] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0136] In one embodiment, the present invention also provides a fall detection device based on a patrol robot, which corresponds one-to-one with the fall detection method based on a patrol robot described in the above embodiments. Figure 4 As shown, the fall detection device based on the patrol robot includes a data acquisition module 41, a fall detection module 42, a key point detection module 43, and a filtering module 44. Detailed descriptions of each functional module are as follows:

[0137] The acquisition module 41 is used to control the patrol robot to move in a preset location and acquire image signals of the preset location;

[0138] The fall detection module 42 is used to detect fall behavior in the image information and obtain human information in a suspected fall state.

[0139] Key point detection module 43 is used to perform key point detection on the human body information of the suspected fall state, and obtain human body key point information and its confidence level;

[0140] The filtering module 44 is used to filter the key information of the human body and its confidence level according to the preset filtering logic to obtain the person who fell.

[0141] Optionally, the fall detection module 42 includes:

[0142] The preprocessing unit is used to downsample the image information and preprocess each pixel in the downsampled image information to obtain the image to be detected.

[0143] The fall detection unit is used to input the image to be detected into a preset deep learning object detection network for detection, and to obtain the human body and its target box in a suspected fall state.

[0144] The cropping unit is used to crop the corresponding image region from the image information based on the human body and its target box, as human body information in a suspected fall state.

[0145] Optionally, the key point detection module 43 includes:

[0146] The preprocessing unit is used to preprocess the human information of the suspected fall state;

[0147] The key point detection unit is used to input the preprocessed human body information into the preset key point detection framework Alphapose to perform key point detection and obtain human body key point information and its confidence level.

[0148] Optionally, the backbone network of the key point detection framework Alphapose adopts a ResNet18 network, uses the PixelShuffle operator during training, and uses the deconvolution operator during inference.

[0149] During the training phase, the human body key point information is divided into a first type of human body key point information and a second type of human body key point information. The first type of human body key point information is obtained through annotation, and the second type of human body key point information is obtained by calculating the median value based on the position of the specified key point in the first type of human body key point information.

[0150] Optionally, the filtering module 44 includes:

[0151] The acquisition unit is used to acquire target key points and their confidence levels from the human body key point information and their confidence levels. The target key points include head key points, right ankle key points, left ankle key points, midpoint between the two shoulders, and midpoint between the two hips.

[0152] The first comparison unit is used to compare the confidence level of the target key point with a preset confidence threshold.

[0153] The second comparison unit is used to compare the confidence of the head key point and the midpoint between the two shoulders with the preset confidence threshold respectively if the confidence of at least one target key point is greater than or equal to the preset confidence threshold.

[0154] The third comparison unit is used to compare the confidence level of the midpoint between the two hips with the preset confidence threshold if the confidence level of any one of the head key points and the midpoint between the two shoulders is greater than or equal to the preset confidence threshold.

[0155] The first screening unit is used to calculate the upper body cosine value based on the midpoint between the two hips, the midpoint between the two shoulders, the midpoint between the two hips, and the head key point when the confidence level of the midpoint between the two hips is greater than or equal to the preset confidence threshold. It also calculates the left leg cosine value based on the comparison result of the left ankle key point with the preset confidence threshold and the right leg cosine value based on the comparison result of the right ankle key point with the preset confidence threshold. The unit then filters for fall victims based on the upper body cosine value, right leg cosine value, and left leg cosine value.

[0156] The second screening unit is used to calculate the right cosine value and the left cosine value of the whole body based on the midpoint of the two hips, the head key point, the right ankle key point, and the left ankle key point when the confidence of the midpoint of the two shoulders, the head key point, the right ankle key point, and the left ankle key point. The unit then filters the people who have fallen based on the left cosine value and the right cosine value.

[0157] Optionally, the first filtering unit includes:

[0158] The upper body cosine value calculation subunit is used to calculate the cosine value of the angle between the line connecting the midpoints of the two shoulders and the horizontal line if the confidence level of the midpoint between the two shoulders is greater than or equal to the preset confidence threshold. If the confidence level of the midpoint between the two shoulders is less than the preset confidence threshold, the head key points are detected. When the confidence level of the head key points is greater than or equal to the preset confidence threshold, the cosine value of the angle between the line connecting the midpoints of the two hips and the head key points and the horizontal line is calculated as the upper body cosine value. When the confidence level of the head key points is less than the preset confidence threshold, a specified value is obtained as the upper body cosine value.

[0159] The left leg cosine value calculation subunit is used to calculate the cosine value of the angle between the line connecting the midpoint of the two hips and the left ankle key point and the horizontal line when the confidence of the left ankle key point is greater than or equal to the preset confidence threshold. Otherwise, it obtains the specified value as the left leg cosine value.

[0160] The right leg cosine value calculation subunit is used to calculate the cosine value of the angle between the line connecting the midpoint of the two hips and the right ankle key point and the horizontal line when the confidence of the right ankle key point is greater than or equal to the preset confidence threshold. Otherwise, it obtains the specified value as the right leg cosine value.

[0161] The comparison subunit is used to compare the cosine value of the upper body with the first cosine value, compare the cosine value of the left leg with the second cosine value, compare the cosine value of the right leg with the second cosine value; and / or compare the cosine value of the upper body with the first cosine value, compare the cosine value of the left leg with the third cosine value, and compare the cosine value of the right leg with the third cosine value;

[0162] The filtering subunit is configured to determine that the human body information is not a person who has fallen when the cosine value of the upper body is less than or equal to the first cosine value, the cosine value of the left leg is greater than or equal to the second cosine value, the cosine value of the right leg is greater than or equal to the second cosine value, or when the cosine value of the upper body is less than or equal to the first cosine value, the cosine value of the left leg is less than or equal to the third cosine value, and the cosine value of the right leg is less than or equal to the third cosine value; otherwise, the human body information is a person who has fallen.

[0163] Optionally, the second filtering unit includes:

[0164] The first whole-body cosine value calculation subunit is used to calculate the cosine of the angle between the line connecting the midpoint of the two shoulders and the left ankle key point and the horizontal line as the whole-body left cosine value if the confidence level of the midpoint of the two shoulders is greater than or equal to a preset confidence threshold, and if the confidence level of the left ankle key point is greater than the preset confidence threshold, as the preset confidence threshold; otherwise, a specified value is obtained as the whole-body left cosine value.

[0165] The second whole-body cosine value calculation subunit is used to detect head key points if the confidence level of the midpoint between the two shoulders is less than a preset confidence threshold. When the confidence level of the head key points is greater than or equal to the preset confidence threshold, if the confidence level of the left ankle key point is greater than the preset confidence threshold, the cosine value of the angle between the line connecting the head key point and the left ankle key point and the horizontal line is calculated as the whole-body left cosine value; otherwise, a specified value is obtained as the whole-body left cosine value. When the confidence level of the right ankle key point is greater than the preset confidence threshold, the cosine value of the angle between the line connecting the head key point and the right ankle key point and the horizontal line is calculated as the whole-body right cosine value; otherwise, a specified value is obtained as the whole-body right cosine value.

[0166] The comparison subunit is used to compare the cosine value of the right side of the whole body with the fourth cosine value, and to compare the cosine value of the left side of the whole body with the fourth cosine value.

[0167] The filtering subunit is used to determine that the human body information is not a person who has fallen when the cosine value of the right side of the whole body is less than the fourth cosine value and the cosine value of the left side of the whole body is less than the fourth cosine value; otherwise, the human body information is a person who has fallen.

[0168] Specific limitations regarding the fall detection device based on patrol robots can be found in the limitations of the fall detection method based on patrol robots mentioned above, and will not be repeated here. Each module in the aforementioned fall detection device based on patrol robots can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in memory as software, so that the processor can call and execute the corresponding operations of each module.

[0169] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a fall detection method based on a patrol robot.

[0170] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0171] Control the patrol robot to move within a predetermined area and collect image signals from the predetermined area;

[0172] The image information is used to detect fall behavior and obtain information about a human body in a suspected fall state.

[0173] Key point detection is performed on the human body information suspected of being in a fall state to obtain human body key point information and its confidence level;

[0174] The key information of the human body and its confidence level are filtered according to the preset filtering logic to obtain the person who fell.

[0175] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0176] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0177] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for detecting fall behavior based on a patrol robot, characterized in that, include: Control the patrol robot to move within a predetermined area and collect image signals from the predetermined area; The image information is used to detect fall behavior and obtain information about a human body in a suspected fall state. The method involves performing keypoint detection on the human body information suspected of being in a fall state to obtain keypoint information and its confidence level. Specifically, this includes: preprocessing the human body information suspected of being in a fall state; inputting the preprocessed human body information into a preset keypoint detection framework, Alphapose, for keypoint detection to obtain keypoint information and its confidence level. The backbone network of the Alphapose keypoint detection framework uses a ResNet18 network, employing the PixelShuffle operator during training and deconvolution during inference. During the training phase... The human body key point information is divided into a first type of human body key point information and a second type of human body key point information. The first type of human body key point information is obtained through annotation, and the second type of human body key point information is obtained by calculating the median value based on the position of the specified key point in the first type of human body key point information. The human body key point information includes head key point, left shoulder key point, right shoulder key point, left elbow key point, right elbow key point, left wrist key point, right wrist key point, left hip key point, right hip key point, left knee key point, right knee key point, left ankle key point, right ankle key point, midpoint between the two shoulders, and midpoint between the two hips. The system filters the human body key point information and its confidence level according to a preset filtering logic to identify the person who fell. Specifically, this includes: obtaining target key points and their confidence levels from the human body key point information and their confidence levels; the target key points include the head key point, right ankle key point, left ankle key point, midpoint between the two shoulders, and midpoint between the two hips; comparing the confidence level of the target key points with a preset confidence threshold; if the confidence level of at least one target key point is greater than or equal to the preset confidence threshold, then comparing the confidence levels of the head key point and the midpoint between the two shoulders with preset confidence thresholds respectively; if the confidence level of either the head key point or the midpoint between the two shoulders is greater than or equal to the preset confidence threshold, then comparing the confidence level of the midpoint between the two hips with the preset confidence threshold. A reliability threshold is set. When the confidence level of the midpoint between the two hips is greater than or equal to the preset reliability threshold, the cosine value of the upper body is calculated based on the midpoint between the two shoulders, the midpoint between the two hips, and the head key point. The cosine value of the left leg is calculated based on the comparison result of the left ankle key point with the preset reliability threshold. The cosine value of the right leg is calculated based on the comparison result of the right ankle key point with the preset reliability threshold. The upper body cosine value, right leg cosine value, and left leg cosine value are used to filter for identifying fall-prone individuals. When the confidence level of the midpoint between the two hips is less than the preset reliability threshold, the right side cosine value and the left side cosine value of the whole body are calculated based on the midpoint between the two shoulders, the head key point, the right ankle key point, and the left ankle key point. The left side cosine value and the right side cosine value of the whole body are used to filter for identifying fall-prone individuals.

2. The fall detection method based on a patrol robot as described in claim 1, characterized in that, The step of detecting fall behavior from the image information to obtain human information in a suspected fall state includes: The image information is downsampled, and each pixel in the downsampled image information is preprocessed to obtain the image to be detected. The image to be detected is input into a preset deep learning object detection network for detection to obtain a human body in a suspected fall state and its target bounding box; Based on the human body and its target bounding box, the corresponding image region is extracted from the image information as the human body information in a suspected fall state.

3. The fall detection method based on a patrol robot as described in claim 1, characterized in that, The method involves calculating the cosine value of the upper body based on the midpoint between the two shoulders, the midpoint between the two hips, and the key point of the head; calculating the cosine value of the left leg based on the comparison result of the left ankle key point with a preset reliability threshold; calculating the cosine value of the right leg based on the comparison result of the right ankle key point with a preset reliability threshold; and filtering based on the cosine values ​​of the upper body, right leg, and left leg to identify individuals who have fallen. If the confidence level of the midpoint between the two shoulders is greater than or equal to the preset confidence threshold, then the cosine of the angle between the line connecting the midpoint between the two hips and the midpoint between the two shoulders and the horizontal line is calculated as the cosine value of the upper body. If the confidence level of the midpoint between the two shoulders is less than the preset confidence threshold, then the head key point is detected. When the confidence level of the head key point is greater than or equal to the preset confidence threshold, the cosine of the angle between the line connecting the midpoint between the two hips and the head key point and the horizontal line is calculated as the upper body cosine value. When the confidence level of the head key point is less than the preset confidence threshold, the specified value is obtained as the upper body cosine value. When the confidence level of the left ankle key point is greater than or equal to the preset confidence threshold, the cosine of the angle between the line connecting the midpoint of the two hips and the left ankle key point and the horizontal line is calculated as the cosine value of the left leg; otherwise, the specified value is obtained as the cosine value of the left leg. When the confidence level of the right ankle key point is greater than or equal to the preset confidence threshold, the cosine of the angle between the line connecting the midpoint of the two hips and the right ankle key point and the horizontal line is calculated as the cosine value of the right leg; otherwise, the specified value is obtained as the cosine value of the right leg. Compare the cosine value of the upper body with the first cosine value, compare the cosine value of the left leg with the second cosine value, compare the cosine value of the right leg with the second cosine value; and / or, compare the cosine value of the upper body with the first cosine value, compare the cosine value of the left leg with the third cosine value, and compare the cosine value of the right leg with the third cosine value. When the cosine value of the upper body is less than or equal to the first cosine value, the cosine value of the left leg is greater than or equal to the second cosine value, and the cosine value of the right leg is greater than or equal to the second cosine value, or when the cosine value of the upper body is less than or equal to the first cosine value, the cosine value of the left leg is less than or equal to the third cosine value, and the cosine value of the right leg is less than or equal to the third cosine value, the human body information indicates that the person is not a person who has fallen; otherwise, the human body information indicates that the person has fallen.

4. The fall detection method based on a patrol robot as described in claim 1, characterized in that, The method involves calculating the cosine values ​​of the right and left sides of the body based on the midpoint between the two shoulders, the key point of the head, the key point of the right ankle, and the key point of the left ankle. The resulting list of individuals who have fallen is then filtered based on these cosine values. If the confidence level of the midpoint between the two shoulders is greater than or equal to the preset confidence threshold, when the confidence level of the left ankle key point is greater than the preset confidence threshold, the cosine of the angle between the line connecting the midpoint between the two shoulders and the left ankle key point and the horizontal line is calculated as the cosine value of the left side of the whole body; otherwise, a specified value is obtained as the cosine value of the left side of the whole body. When the confidence level of the right ankle key point is greater than the preset confidence threshold, the cosine of the angle between the line connecting the midpoint between the two shoulders and the right ankle key point and the horizontal line is calculated as the cosine value of the right side of the whole body; otherwise, a specified value is obtained as the cosine value of the right side of the whole body. If the confidence level of the midpoint between the two shoulders is less than the preset confidence threshold, then the head keypoint is detected. When the confidence level of the head keypoint is greater than or equal to the preset confidence threshold, if the confidence level of the left ankle keypoint is greater than the preset confidence threshold, the cosine of the angle between the line connecting the head keypoint and the left ankle keypoint and the horizontal line is calculated as the cosine value of the left side of the whole body; otherwise, a specified value is obtained as the cosine value of the left side of the whole body. When the confidence level of the right ankle keypoint is greater than the preset confidence threshold, the cosine of the angle between the line connecting the head keypoint and the right ankle keypoint and the horizontal line is calculated as the cosine value of the right side of the whole body; otherwise, a specified value is obtained as the cosine value of the right side of the whole body. Compare the cosine value of the right side of the whole body with the fourth cosine value, and compare the cosine value of the left side of the whole body with the fourth cosine value; When the cosine value of the right side of the body is less than the fourth cosine value, and the cosine value of the left side of the body is less than the fourth cosine value, the human body information indicates that the person is not a person who has fallen; otherwise, the human body information indicates that the person has fallen.

5. A fall detection device based on a patrol robot, characterized in that, The device includes: The acquisition module is used to control the patrol robot to move in a preset location and to acquire image signals of the preset location; The fall detection module is used to detect fall behavior in the image information and obtain human information in a suspected fall state. A keypoint detection module is used to detect keypoints in the human body information suspected of being in a fall state, and to obtain human keypoint information and its confidence level. Specifically, this includes: preprocessing the human body information suspected of being in a fall state; inputting the preprocessed human body information into a preset keypoint detection framework, Alphapose, for keypoint detection, and obtaining human keypoint information and its confidence level. The backbone network of the Alphapose keypoint detection framework uses a ResNet18 network, employing the PixelShuffle operator during training and the deconvolution operator during inference. During the training phase, the human body key point information is divided into a first type of human body key point information and a second type of human body key point information. The first type of human body key point information is obtained through annotation, and the second type of human body key point information is obtained by calculating the median value based on the position of the specified key points in the first type of human body key point information. The human body key point information includes head key points, left shoulder key points, right shoulder key points, left elbow key points, right elbow key points, left wrist key points, right wrist key points, left hip key points, right hip key points, left knee key points, right knee key points, left ankle key points, right ankle key points, midpoint between the two shoulders, and midpoint between the two hips. The filtering module is used to filter the human body key point information and its confidence level according to a preset filtering logic to obtain the person who fell. Specifically, it includes: obtaining target key points and their confidence levels from the human body key point information and their confidence levels; the target key points include head key points, right ankle key points, left ankle key points, midpoint between the two shoulders, and midpoint between the two hips; comparing the confidence levels of the target key points with preset confidence thresholds; if the confidence level of at least one target key point is greater than or equal to the preset confidence threshold, then comparing the confidence levels of the head key point and the midpoint between the two shoulders with preset confidence thresholds respectively; if the confidence level of either the head key point or the midpoint between the two shoulders is greater than or equal to the preset confidence threshold, then comparing the confidence levels of the midpoint between the two hips. Reliability and a preset reliability threshold; when the confidence level of the midpoint between the two hips is greater than or equal to the preset reliability threshold, the cosine value of the upper body is calculated based on the midpoint between the two shoulders, the midpoint between the two hips, and the head key point; the cosine value of the left leg is calculated based on the comparison result of the left ankle key point with the preset reliability threshold; the cosine value of the right leg is calculated based on the comparison result of the right ankle key point with the preset reliability threshold; and the upper body cosine value, right leg cosine value, and left leg cosine value are used to filter for identifying fall-prone individuals; when the confidence level of the midpoint between the two hips is less than the preset reliability threshold, the right side cosine value and the left side cosine value of the whole body are calculated based on the midpoint between the two shoulders, the head key point, the right ankle key point, and the left ankle key point; and the left side cosine value and the right side cosine value of the whole body are used to filter for identifying fall-prone individuals.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the fall detection method based on a patrol robot as described in any one of claims 1 to 4.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the fall detection method based on a patrol robot as described in any one of claims 1 to 4.

Images