Indoor fall detection method and device based on dual-camera image stitching, computer program and storage medium

Abstract

The invention discloses an indoor fall detection method and device based on dual-camera image splicing, a computer program and a storage medium, and relates to the technical field of computer vision, in particular to the technical field of action recognition based on computer vision. The detection method comprises the following steps: S1, detecting a human body target and a position thereof in an image at a current moment in video data of a main monitoring area; s2, tracking the position change of the human body target according to the video data of the main monitoring area; s3, judging whether the current position of the human body target enters a leak detection area or not; s4, extracting feature points of the human body target in the image at the current moment in the video data of the main monitoring area and the missing detection area, and splicing the feature points to obtain a complete human body target; and S5, extracting skeleton points of the human body target based on an OpenPose model, and judging that a fall event occurs. The method is suitable for the technical fields of health monitoring and medical services.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to the field of action recognition technology based on computer vision. Background Art

[0002] With the improvement of medical conditions and the improvement of living standards, the proportion of the elderly population in the world continues to rise. According to data from the United Nations, it is estimated that by 2050, the number of people aged 60 and above in the world will reach 2.1 billion. Therefore, how to effectively care for the elderly and prevent and reduce falls has become a key area of ​​concern in society. Falls can cause a series of health problems, such as fractures, head injuries and even death. Especially for the elderly, their physical functions are declining, their injury recovery time is longer, and the risk of sequelae is significantly increased. In addition, falls may also lead to serious psychological problems, reduced independence in life and increased medical expenses.

[0003] Current fall detection can be divided into three main categories:

[0004] 1. Wearable sensor-based fall detection technology: This type of detection technology uses the accelerometer, gyroscope and other sensors integrated in the device worn by the user to collect the wearer's movements and postures, and analyzes the collected data in real time to determine whether the wearer has fallen. This type of detection technology can monitor the movement status of users who wear these devices correctly in real time, thereby detecting the user's fall status.

[0005] Disadvantages of this type of technology: users must wear the device correctly to realize the detection function. If they do not wear it, detection cannot be realized. If they wear it but the wearing position is not in the required position, or a device is worn loosely, the detection data will be inaccurate, which will lead to misjudgment or even invalid judgment, affecting the timeliness and accuracy of fall detection.

[0006] 2. Fall detection technology based on environmental sensors. This type of detection technology mainly relies on external devices such as infrared sensors, vibration sensors and acoustic sensors to collect data from the surrounding environment, and then analyzes and infers the occurrence of fall events based on these data.

[0007] The disadvantage of this type of detection technology is that external sensors are easily interfered by environmental factors, such as changes in light, changes in surrounding vibrating objects, or other background noise changes, which may lead to false alarms or missed detections, reducing the reliability of detection.

[0008] 3. The fall detection technology based on computer vision is a relatively advanced technology compared with the first two detection technologies. This detection technology is based on computer vision technology. This detection technology uses the real-time images captured by the camera as the detection input data, and uses image processing technology to analyze the real-time images to obtain the detection results, which can effectively determine the occurrence of fall events. The advantages of this technology include: non-invasive, high accuracy, real-time monitoring and low cost. It is suitable for simultaneous detection of the activity status of all people in a certain area, especially for monitoring the activity status of the elderly, which significantly improves the safety and quality of life of the elderly in their daily lives. This type of detection technology has gradually become an important choice in the field of fall detection for the elderly.

[0009] Disadvantages of this type of detection technology: The detection area is limited by the camera installation position and angle. Certain specific areas cannot be captured by the camera. These areas are called missed detection areas, and the activity status of people in these areas cannot be detected. If the detection target is active near the boundary between this area and the detection area, the entire person or part of the body will be out of the camera's monitoring range due to its position, resulting in incomplete or impossible extraction of the detection target, which in turn affects the judgment of the fall event. The missed detection areas in traditional single-camera monitoring are as follows: Figure 1 As shown in the figure, the gray shaded area at the bottom is the area that is prone to missed detection. When a human target appears in the gray area of ​​the camera's field of view, due to the limitation of the camera's field of view, usually only the upper body of the human target can be captured, and the full body image cannot be fully obtained. In this case, it is impossible to judge a fall based on the movement characteristics of the upper body of the human target, thereby reducing the accuracy of fall detection.

[0010] In traditional fall detection methods, the circumscribed rectangle (or ellipse), center position or bone point height of the human body are often used as the basis for determination, while the influence of time factors is ignored. This approach can easily misjudge "lying down" and other similar fall behaviors as fall events. Summary of the invention

[0011] In order to solve the shortcomings of a single camera having a blind spot and the traditional computer vision-based fall detection method not considering the time factor, the present invention proposes the following solution:

[0012] An indoor fall detection method based on dual-camera image stitching, the method is implemented based on two cameras, wherein a main monitoring camera (C1) collects video data of a main monitoring area in a monitoring area, and a supplementary monitoring camera (C2) collects video data of a missed detection area in the monitoring area, the missed detection area is located below the main monitoring camera (C1), the missed detection area is superimposed with the main monitoring area to form a monitoring area, and the missed detection area partially overlaps with the main monitoring area, and the detection method comprises the following steps:

[0013] Real-time acquisition of video data collected by two cameras;

[0014] S1, a step of detecting a human target and its position in the image at the current moment in the video data of the main monitoring area;

[0015] S11, if no human target is detected, return to S1 to continue detection;

[0016] S12: If a human target and its position are detected, steps S2 to S5 are executed in a loop:

[0017] S2, a step of tracking the position change of the human target according to the video data of the main monitoring area;

[0018] S3, a step of determining whether the current position of the human target enters a missed detection area;

[0019] S31, if entering the missed detection area, proceed to step S4;

[0020] S32: If the missed detection area has not been entered, proceed to step S5;

[0021] S4, extracting feature points of human targets in the image at the current moment in the video data of the main monitoring area and the missed detection area, and splicing the feature points to obtain a complete human target;

[0022] S5. Extract the skeleton points of the human target based on the OpenPose model and determine the steps of the fall event.

[0023] Furthermore, step S1 further includes: using the OpenCV library in Python to read the video data of the main monitoring area, using YOLO v 5 algorithm detects the human target in the image at the current moment in the video data and obtains the detection result. The detection result is to draw a bounding box of the human target on the image. The position information of the human target includes: the upper left corner coordinates (X1, Y1), the lower right corner coordinates (X2, Y2) and the center point coordinates (X center , Y center ):

[0024] X center =(X1+X2) / 2,

[0025] Y center =(Y1+Y2) / 2,

[0026] Among them, X center Indicates the horizontal coordinate of the center point of the human body bounding box, Y center Indicates the ordinate of the center point of the human body bounding box.

[0027] Further, step S2 includes:

[0028] S201, prediction: creating a predicted trajectory according to the current image of the main monitoring area image, and dividing the predicted trajectory into a confirmed state and an unconfirmed state by using Kalman filter prediction, wherein the confirmed state is added to the cascade matching of S302, and the unconfirmed state is added to the intersection-and-union matching of S302;

[0029] S202, Detection: Use YOLO v 5. The algorithm detects the human target in the image of the main monitoring area and obtains the detection result. The detection result is cascade matched with the confirmed state result obtained in S301 to obtain unmatched track 1, unmatched detection result 1 and matched track 1. The unmatched track 1, unmatched detection result 1 and the unconfirmed state result obtained in S301 are matched by intersection and union ratio to obtain unmatched track 2, unmatched detection result 2 and matched track 2. The unmatched detection result 2 is used as a new track and incorporated into the predicted track of S301.

[0030] S203, screening: using the DeepSort algorithm to divide the unmatched track 2 in S302 into confirmed and unconfirmed states.

[0031] If it is an unconfirmed state, deleting the unconfirmed state data;

[0032] If it is in the confirmation state, the maximum mismatch frame number is determined, and the maximum mismatch frame number is set to 30 frames:

[0033] FPS Unmatched rracks2 >30,

[0034] Deleting the confirmed data,

[0035] FPS Unmatched Tracks2 <30,

[0036] The unmatched trajectory 2 in the confirmed state is incorporated into the predicted trajectory in S301;

[0037] Among them, FPS Unmatched Tracks2 Indicates the number of frames that did not match track 2.

[0038] S204, update: use Kalman filtering to update the matched trajectory 1 and the matched trajectory 2 in S302, and merge the updated matched trajectory 1 and the matched trajectory 2 into the predicted trajectory in S301;

[0039] S205, cyclic comparison: after the predicted trajectory is detected, screened and updated, Kalman filter prediction is performed again, and the prediction result is compared with the result of the previous round of Kalman filter prediction to achieve continuous tracking of the human target.

[0040] Further, step S3 includes: judging whether the human target has entered the missed detection area according to the human target and its position obtained in step S1; when it is detected that the center point coordinates of the human body boundary box appear in the lower 1 / 4 area of ​​the main monitoring area, or the vertical coordinates of the lower right corner coordinates of the human body boundary box are 0, it is determined that the target has entered the missed detection area, and detecting the human target and its position in the image at the current moment in the video data of the missed detection area.

[0041] Furthermore, step S4 includes: extracting feature points in the image using the SIFT algorithm, describing the extracted feature points, creating feature point sets for the main monitoring area and the supplementary monitoring area, matching the feature points in the feature point sets, correcting the matched feature points, and obtaining a complete human target.

[0042] Further, the S5 includes: extracting skeleton points of the human target using the OpenPose model, and setting the position of the nose in the skeleton points as the 0 point:

[0043] First, eliminate the impact of the distance from the camera:

[0044]

[0045] Where i represents the image at the current moment, i-1 represents the image at the previous moment, and i≥2;

[0046] y i 0 Indicates the vertical coordinate of point 0 in the image at the current moment;

[0047] h i Represents the height of the human body bounding box in the image at the current moment;

[0048] L i Indicates the relative movement distance of point 0 in the image at the current moment and the image at the previous moment;

[0049] Second, detect possible fall events:

[0050]

[0051] Among them, F i The indicator variable indicating whether a fall event occurs in the i-th frame, 1 indicates a fall, 0 indicates no fall, and T indicates the threshold of the moving distance;

[0052] Finally, determine whether a fall event has occurred:

[0053]

[0054] Among them, Fall i A variable that comprehensively determines whether a fall event has occurred. Indicates that within 30 consecutive frames, there are more than and including 20 frames of F i The value is 1.

[0055] Based on the same inventive concept, the present invention also provides an indoor fall detection device based on dual-camera image stitching, the device comprising:

[0056] A module for detecting human targets and their positions in the image at the current moment in the video data of the main monitoring area;

[0057] A module for tracking the position change of the human target according to the video data of the main monitoring area;

[0058] A module for determining whether the current position of the human target enters a missed detection area;

[0059] A module for extracting feature points of a human target in the image at the current moment from the video data of the main monitoring area and the missed detection area, and splicing the feature points to obtain a complete human target;

[0060] A module that extracts skeleton points of human targets based on the OpenPose model and determines when a fall event occurs.

[0061] Based on the same inventive concept, the present invention also provides a computer storage medium for storing a computer program. When the computer program is read by a computer, the computer executes the method described in the present invention.

[0062] Based on the same inventive concept, the present invention also provides a computer, including a processor and a storage medium. When the processor reads the computer program stored in the storage medium, the computer executes the method described in the present invention.

[0063] Based on the same inventive concept, the present invention also provides a computer program product, which is a computer program. When the computer program is read, the method described in the present invention is implemented.

[0064] Beneficial Effects

[0065] (1) The present invention provides an indoor fall detection method based on dual-camera image stitching, which realizes all-round monitoring of the indoor environment by combining dual-camera image stitching technology, and overcomes the problem of missed detection existing in a single camera. The overlapping monitoring areas of the two cameras enable the cameras to capture more motion information, especially the complete image of the human target. The complete human target image after stitching can more accurately judge the fall event, avoiding the misjudgment or missed judgment caused by the field of view limitation of a single camera.

[0066] (2) The present invention provides an indoor fall detection method based on dual-camera image stitching, which uses Kalman filtering and YOLOv5 algorithms to efficiently track the motion trajectory of human targets and determine whether the human targets leave the main monitoring area based on their motion characteristics. The DeepSort algorithm is further combined to screen and update the motion trajectory, thereby improving the stability and accuracy of target tracking and ensuring the real-time and effectiveness of fall judgment.

[0067] (3) The present invention provides an indoor fall detection method based on dual-camera image stitching. By setting precise missed detection conditions: the lower 1 / 4 area of ​​the camera field of view, it is determined whether the human target has entered the missed detection area. When the target enters the area, the second camera is immediately started for supplementary monitoring, thereby ensuring the continuity and comprehensiveness of the monitoring range. This effectively makes up for the problem that the traditional single-camera technology cannot fully cover the monitoring area, and improves the reliability and stability of the fall detection system.

[0068] (4) The present invention provides an indoor fall detection method based on dual-camera image stitching, which performs SIFT feature point detection, feature point matching and image correction on the images of camera 1 and camera 2 respectively. The stable feature points in the image can be accurately extracted, and the image stitching problem caused by the different viewing angles of the dual cameras is solved through feature point matching and image correction, and a complete human target image is obtained, thereby improving the accuracy of the detection system and avoiding detection omissions caused by differences in viewing angles.

[0069] (5) The present invention is an indoor fall detection method based on dual-camera image stitching. By using the OpenPose model to extract 18 human skeleton points, the nose is set as the 0 point, and the displacement of the head is calculated. The human body movement characteristics can be captured accurately. When falling, the posture of the human body will change rapidly, usually from an upright state to a horizontal state, and the height change of the head is the most obvious. After excluding the influence of the camera distance, the displacement can accurately reflect the vertical movement of the human body and avoid misjudgment of "lying down" and other similar fall behaviors. When a fall occurs, the fall can be reasonably judged by detecting the movement characteristics of the head in the video.

[0070] (6) The present invention effectively solves the blind spot problem in traditional fall detection methods and improves the accuracy of fall event judgment through technical means such as dual-camera image stitching, YOLOv5 target detection, Kalman filter tracking, SIFT feature point matching, and OpenPose key point extraction. By comprehensively analyzing head movement, human target integrity, and continuous frame features, it can accurately detect fall events, reduce misjudgments and missed judgments, and thus provide a safer living environment for the elderly.

[0071] The indoor fall detection method based on dual-camera image stitching of the present invention is applicable to technical fields such as health monitoring and medical services.

[0072] Figure Description

[0073] Figure 1 The traditional single camera monitoring area mentioned in the background technology has missed detection areas.

[0074] Figure 2 This is a flow chart of an indoor fall detection method based on dual-camera image stitching described in Implementation Method 1.

[0075] Figure 3 Schematic diagram of the human body bounding box of the YOLOv5 detection result described in Implementation Method 2.

[0076] Figure 4 This is the target tracking flow chart of the DeepSort algorithm described in Implementation Method 3.

[0077] Figure 5 It is a flowchart of the SIFT algorithm described in the fifth implementation mode.

[0078] Figure 6 This is a schematic diagram of three installation positions of dual cameras in implementation mode 7. Figure symbols: C1 represents camera 1, and C2 represents camera 2.

[0079] Figure 7 This is a schematic diagram of the missed fall detection area in Implementation 4. In the figure, the left side shows that the coordinates of the center point of the human body boundary box appear in the lower 1 / 4 area of ​​the video screen, and the right side shows that the vertical coordinate of the lower right corner of the human body boundary box is 0.

[0080] Figure 8 It is an image after the video data collected by camera 1 and camera 2 in implementation mode 7 is processed. In the figure, the left side is an image after the video data of the main monitoring area collected by camera 1 is processed, and the right side is an image after the video data of the supplementary monitoring area collected by camera 2 is processed.

[0081] Fig. 9 This is a schematic diagram of feature point matching in Implementation Example 7. In the figure, the upper part is the processed image of the video data of the main monitoring area collected by camera 1, and the lower part is the processed image of the video data of the supplementary monitoring area collected by camera 2. The red and green lines in the middle are feature points.

[0082] Fig.10 It is the splicing result after the feature points are matched as described in the seventh implementation mode. Implementation

[0084] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with this figure. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0085] Implementation Method 1

[0086] Combination Figure 2 This embodiment is described. This embodiment provides an indoor fall detection method based on dual-camera image stitching. The method is implemented based on two cameras, wherein a main monitoring camera (C1) collects video data of a main monitoring area in a monitoring area, and a supplementary monitoring camera (C2) collects video data of a missed detection area in the monitoring area. The missed detection area is located below the main monitoring camera (C1). The missed detection area is superimposed with the main monitoring area to form a monitoring area, and the missed detection area partially overlaps with the main monitoring area. The detection method comprises the following steps:

[0087] Real-time acquisition of video data collected by two cameras;

[0088] S1, a step of detecting a human target and its position in the image at the current moment in the video data of the main monitoring area;

[0089] S11, if no human target is detected, return to S1 to continue detection;

[0090] S12: If a human target and its position are detected, steps S2 to S5 are executed in a loop:

[0091] S2, a step of tracking the position change of the human target according to the video data of the main monitoring area;

[0092] S3, a step of determining whether the current position of the human target enters a missed detection area;

[0093] S31, if entering the missed detection area, proceed to step S4;

[0094] S32: If the missed detection area has not been entered, proceed to step S5;

[0095] S4, extracting feature points of human targets in the image at the current moment in the video data of the main monitoring area and the missed detection area, and splicing the feature points to obtain a complete human target;

[0096] S5. Extract the skeleton points of the human target based on the OpenPose model and determine the steps of the fall event.

[0097] This implementation method uses dual cameras to achieve image stitching, expands the monitoring area, and avoids the problem of blind spots in the field of view of a single camera. The image acquisition areas of the two cameras overlap with each other, ensuring that complete image data can be obtained in a timely and accurate manner regardless of how the target position changes, significantly improving the coverage and accuracy of fall detection. Through the collaboration of dual cameras, not only the fault tolerance of the system is enhanced, but also the monitoring of large-scale scenes becomes more effective.

[0098] Implementation Method 2

[0099] This embodiment further limits the indoor fall detection method based on dual-camera image stitching described in Embodiment 1. Figure 3 In this embodiment, step S1 further includes: using the OpenCV library in Python to read the video data of the main monitoring area, using the YOLOv5 algorithm to detect the human target in the image at the current moment in the video data, and obtaining a detection result, wherein the detection result is to draw a bounding box of the human target on the image, and the position information of the human target includes: the upper left corner coordinates (X1, Y1), the lower right corner coordinates (X2, Y2) and the center point coordinates (X center , Y center ):

[0100] X center =(X1+X2) / 2,

[0101] Y center =(Y1+Y2) / 2,

[0102] Among them, X center Indicates the horizontal coordinate of the center point of the human body bounding box, Y center Indicates the ordinate of the center point of the human body bounding box.

[0103] Based on implementation method 1, combined with YOLO v The 5 algorithm accurately identifies the position and bounding box of human targets, effectively improving the detection accuracy of human targets. By calculating the center point coordinates, the target position changes can be accurately tracked, further enhancing the recognition effect of fall events and avoiding the problem of missed detection caused by camera viewing angles and angle limitations. This method enhances the adaptability of human target detection and can cope with complex indoor environments and different target posture changes.

[0104] Implementation Method 3

[0105] This embodiment further limits the indoor fall detection method based on dual-camera image stitching described in Embodiment 1. Figure 4 To illustrate this embodiment, further, step S2 includes:

[0106] S201, prediction: creating a predicted trajectory according to the current image of the main monitoring area image, and dividing the predicted trajectory into a confirmed state and an unconfirmed state by using Kalman filter prediction, wherein the confirmed state is added to the cascade matching of S302, and the unconfirmed state is added to the intersection-and-union matching of S302;

[0107] S202, Detection: Use YOLO v 5. The algorithm detects the human target in the image of the main monitoring area and obtains the detection result. The detection result is cascade matched with the confirmed state result obtained in S301 to obtain unmatched track 1, unmatched detection result 1 and matched track 1. The unmatched track 1, unmatched detection result 1 and the unconfirmed state result obtained in S301 are matched by intersection and union ratio to obtain unmatched track 2, unmatched detection result 2 and matched track 2. The unmatched detection result 2 is used as a new track and incorporated into the predicted track of S301.

[0108] S203, screening: using the DeepSort algorithm to divide the unmatched track 2 in S302 into confirmed and unconfirmed states.

[0109] If it is an unconfirmed state, deleting the unconfirmed state data;

[0110] If it is in the confirmation state, the maximum mismatch frame number is determined, and the maximum mismatch frame number is set to 30 frames:

[0111] FPS Unmatched Tracks2 >30,

[0112] Deleting the confirmed data,

[0113] FPS Unmatched rracks2 <30,

[0114] The unmatched trajectory 2 in the confirmed state is incorporated into the predicted trajectory in S301;

[0115] Among them, FPS Unmatched Tracks2 Indicates the number of frames that did not match track 2.

[0116] S204, update: use Kalman filtering to update the matched trajectory 1 and the matched trajectory 2 in S302, and merge the updated matched trajectory 1 and the matched trajectory 2 into the predicted trajectory in S301;

[0117] S205, cyclic comparison: after the predicted trajectory is detected, screened and updated, Kalman filter prediction is performed again, and the prediction result is compared with the result of the previous round of Kalman filter prediction to achieve continuous tracking of the human target.

[0118] By combining Kalman filtering and the YOLOv5 algorithm, target tracking and prediction are optimized, and the motion trajectory and state changes of human targets can be judged more accurately. The DeepSort algorithm is further used to filter the matching results, eliminating uncertain trajectories, avoiding erroneous fall judgments, improving the robustness and accuracy of tracking and detection, and ensuring real-time and reliability. The prediction and tracking capabilities of this method not only improve the detection accuracy of fall events, but also enhance the system's response speed to emergencies.

[0119] Implementation Method 4

[0120] The present embodiment is a further limitation of the indoor fall detection method based on dual-camera image stitching described in embodiment 1. Furthermore, step S3 includes: judging whether the human target has entered the missed detection area according to the human target and its position obtained in step S1; when it is detected that the center point coordinates of the human body boundary box appear in the lower 1 / 4 area of ​​the main monitoring area, or the vertical coordinates of the lower right corner coordinates of the human body boundary box are 0, it is determined that the target has entered the missed detection area; and detecting the human target and its position in the image at the current moment in the video data of the missed detection area.

[0121] This implementation solves the problem of missed detection caused by the camera's viewing angle limitation by determining whether the target has entered the missed detection area and enabling supplementary cameras for supplementary monitoring when necessary. After starting the second camera, it can ensure all-round monitoring of human targets without blind spots, thereby improving the comprehensiveness and accuracy of fall detection. Through this dynamic adjustment of monitoring method, it is possible to always maintain efficient monitoring effects in complex environments, improving the safety and reliability of the overall system.

[0122] Implementation Method 5

[0123] This embodiment further limits the indoor fall detection method based on dual-camera image stitching described in Embodiment 1. Figure 5 To illustrate this embodiment, further, step S4 includes: using SIFT algorithm to extract feature points in the image, and describing the extracted feature points, creating feature point sets for the main monitoring area and the supplementary monitoring area, matching the feature points in the feature point sets, and correcting the matched feature points to obtain a complete human target.

[0124] By using the SIFT algorithm to extract and match the feature points of the main monitoring area and the supplementary monitoring area, the image stitching effect is further optimized to ensure that the stitched image can maintain high quality and no distortion in any scene, thereby improving the clarity and detection accuracy of target recognition and ensuring accurate judgment of fall events. This method improves the precision of image processing, can achieve more accurate target stitching and recognition in complex scenes, and enhances the system's adaptability to different environments.

[0125] Implementation Method 6

[0126] This implementation manner further limits the indoor fall detection method based on dual-camera image stitching described in the first implementation manner. Further, the S5 includes: extracting skeleton points of the human target using the OpenPose model, and setting the position of the nose in the skeleton points as the 0 point:

[0127] First, eliminate the impact of the distance from the camera:

[0128]

[0129] Where i represents the image at the current moment, i-1 represents the image at the previous moment, and i≥2;

[0130] y i 0 Indicates the vertical coordinate of point 0 in the image at the current moment;

[0131] h i Represents the height of the human body bounding box in the image at the current moment;

[0132] L i Indicates the relative movement distance of point 0 in the image at the current moment and the image at the previous moment;

[0133] Second, detect possible fall events:

[0134]

[0135] Among them, F i The indicator variable indicating whether a fall event occurs in the i-th frame, 1 indicates a fall, 0 indicates no fall, and T indicates the threshold of the moving distance;

[0136] Finally, determine whether a fall event has occurred:

[0137]

[0138] Among them, Fall i A variable that comprehensively determines whether a fall event has occurred. Indicates that within 30 consecutive frames, there are more than and including 20 frames of F i The value is 1.

[0139] By extracting key human skeleton points, especially the nose position, through the OpenPose model, the interference caused by distance changes is eliminated, and the dynamic changes of the human body can be detected more accurately. Combined with the fall event judgment method of more than 20 frames within 30 frames, it ensures that the judgment of the fall event does not only rely on a single frame of data, but integrates multi-frame information, reduces the possibility of misjudgment, and improves the accuracy and reliability of fall detection. This method greatly improves the stability and accuracy of detection through comprehensive judgment of multi-frame data, which plays an important role in reducing false alarms.

[0140] Implementation Method 7

[0141] This embodiment provides an example, which is used to explain the above-mentioned embodiments 1 to 6. The specific example is as follows:

[0142] The present invention proposes an indoor fall detection method based on dual-camera image stitching. For indoor environments, dual cameras are used for joint judgment of fall detection. Camera 1 is responsible for obtaining the main field of view, and camera 2 is used to supplement the blind field of view below camera 1. After reading the video data of camera 1, the Deep Sort algorithm is used to track the human target. When the target ID is detected to appear at the boundary of the field of view of camera 1, the image captured by camera 1 and the image captured by camera 2 are spliced ​​through the SIFT algorithm to obtain an image of the complete human target. The image is input into the OpenPose model for skeleton point extraction, and the ratio of the displacement difference of the key points of the human face in multiple consecutive frames to the height of the human target frame in the previous frame is counted to determine whether a fall event has occurred.

[0143] To achieve the above object, the present invention proposes an indoor fall detection method based on dual-camera image stitching, which is implemented by the following steps. The fall detection method flow is as follows: Figure 2 As shown:

[0144] Arrange cameras 1 and 2 indoors.

[0145] The present invention aims at the blind spot problem of indoor cameras and designs a dual-camera system for fall detection and judgment. The camera blind spot refers to the area in the monitoring system that cannot be captured by the camera due to the camera's viewing angle, installation position or other factors. Each camera has a specific field of view, which refers to the image range that the camera can capture, usually expressed as an angle. Areas beyond this field of view will form a blind spot.

[0146] The blind spots can be reduced by properly arranging the cameras. The three arrangements of dual cameras are as follows: Figure 6As shown: In arrangement A, when the human target is located under the blind spot of camera 1 and stands close to camera 2, it is very likely that the complete human target cannot be extracted at this time. In arrangement B, the area can be well covered, and the dual cameras need to be installed separately at high places and turned on at the same time for fall detection. In arrangement C, the dual cameras are uniformly installed at a high place, where camera 1 is the main camera, responsible for monitoring the situation in the main area of ​​the room, and camera 2 performs supplementary monitoring on the blind spot of camera 1. The installation angle of camera 2 needs to be different from that of camera 1, and it is more toward the blind spot of camera 1. When no human target is detected entering the blind spot below camera 1, camera 2 can be put into standby mode. This embodiment selects arrangement C, which is convenient for providing power to the two cameras and is convenient for installation and wiring; considering the height of the person, the target is unlikely to appear in the blind spot above camera 1, and arrangement C can cover the activity range of the target in the room.

[0147] S1, uses YOLOv5 for human target detection and dynamically adjusts the frame rate.

[0148] The OpenCV library in Python is used to read the video data of camera 1, and the YOLOv5 algorithm is selected as the human target detector. The detection results of the human target are displayed in the form of an image. The bounding box of the human body is drawn on the original image. The coordinates of the upper left corner (X1, Y1), the lower right corner (X2, Y2) and the center point (X center , Y center ):

[0149] X center =(X1+X2) / 2, (1)

[0150] Y center =(Y1+Y2) / 2, (2)

[0151] Among them, X center Indicates the horizontal coordinate of the center point of the human body bounding box, Y center Indicates the ordinate of the center point of the human body bounding box.

[0152] Camera 1 dynamically adjusts the frame rate according to scene changes, thereby reducing storage costs and optimizing bandwidth usage. The frame rate of indoor cameras is usually between 15FPS and 30FPS, of which 15FPS is suitable for basic monitoring, can capture most activities, and is suitable for scenes that do not require high details. 30FPS is the standard frame rate for indoor surveillance cameras, which can provide smooth video streaming and clearly capture the movements of people and objects.

[0153] When camera 1 detects a human body in the picture, the frame rate will increase to 30FPS to ensure that a clear picture is captured; when there is no human body in the picture, the frame rate will decrease to 15FPS, thereby effectively reducing data storage requirements and network bandwidth consumption.

[0154] S2, uses Deepsort algorithm for human target tracking.

[0155] The present invention selects the DeepSort multi-target tracking algorithm to track human targets in video sequences, ensuring that human targets are continuously tracked in complex environments. The DeepSort target tracking algorithm flow chart is as follows Figure 4 shown.

[0156] The main steps of the DeepSort algorithm are:

[0157] Further, step S2 includes:

[0158] S201, prediction: creating a predicted trajectory according to the current image of the main monitoring area image, and dividing the predicted trajectory into a confirmed state and an unconfirmed state by using Kalman filter prediction, wherein the confirmed state is added to the cascade matching of S302, and the unconfirmed state is added to the intersection-and-union matching of S302;

[0159] S202, detection: using YOLO v 5. The algorithm detects the human target in the image of the main monitoring area and obtains the detection result. The detection result is cascade matched with the confirmed state result obtained in S301 to obtain unmatched track 1, unmatched detection result 1 and matched track 1. The unmatched track 1, unmatched detection result 1 and the unconfirmed state result obtained in S301 are matched by intersection and union ratio to obtain unmatched track 2, unmatched detection result 2 and matched track 2. The unmatched detection result 2 is used as a new track and incorporated into the predicted track of S301.

[0160] S203, screening: using the DeepSort algorithm to divide the unmatched track 2 in S302 into confirmed and unconfirmed states.

[0161] If it is an unconfirmed state, deleting the unconfirmed state data;

[0162] If it is in the confirmation state, the maximum mismatch frame number is determined, and the maximum mismatch frame number is set to 30 frames:

[0163] FPS Unmatched Tracks2 >30,

[0164] Deleting the confirmed data,

[0165] FPS Unmatched Tracks2 <30,

[0166] The unmatched trajectory 2 in the confirmed state is incorporated into the predicted trajectory in S301;

[0167] Among them, FPS Unmatched Tracks2 Indicates the number of frames that did not match track 2.

[0168] S204, update: use Kalman filtering to update the matched trajectory 1 and the matched trajectory 2 in S302, and merge the updated matched trajectory 1 and the matched trajectory 2 into the predicted trajectory in S301;

[0169] S205, cyclic comparison: after the predicted trajectory is detected, screened and updated, Kalman filter prediction is performed again, and the prediction result is compared with the result of the previous round of Kalman filter prediction to achieve continuous tracking of the human target.

[0170] S3, judging whether the human body leaves the screen. If no complete human body target is detected, execute S4; otherwise, directly enter S5.

[0171] Acquiring a complete and effective human target is a necessary condition for fall detection. When the target appears near the blind spot of camera 1, the camera often cannot capture the complete human image due to its limited field of view. At this time, the monitoring screen can only capture part of the human body, and the lower part of the video monitoring is defined as the missed area of ​​fall detection.

[0172] The human body bounding box and the coordinates of the center point of the bounding box output by YOLOv5 are used to determine whether the target has entered the missed detection area. When the coordinates of the center point of the human body bounding box are detected to appear in the lower area of ​​the video screen, and the coordinates of the lowest point of the bounding box are very close to the screen border, the human body is determined to be in the missed detection area.

[0173] Camera 2 serves as a supplement to the blind spot vision of Camera 1. When it is determined that the target enters the fall detection missed area, Camera 2 will be activated.

[0174] like Figure 7 As shown, due to the different field of view angles and installation positions of different cameras and the different heights of the detected human body, there will be differences in the missed detection area of ​​fall detection. In order to adapt to the fall detection of the present invention, when the coordinates of the center point of the bounding box are detected to appear in the lower 1 / 4 area of ​​the video screen, or the ordinate of the lower boundary of the bounding box is 0, it is determined that the target has entered the missed detection area and the human body has left the screen.

[0175] S4, use SIFT algorithm to extract features for image stitching.

[0176] The present invention uses SIFT algorithm to stitch the images captured by camera 1 and camera 2 to obtain a complete human target image. The core idea of ​​SIFT algorithm is to extract key feature points in the image and generate invariant feature descriptors to achieve image matching and other tasks. The main steps of SIFT algorithm are as follows: Figure 5 The SIFT algorithm is used to extract feature points in the main monitoring area image and the supplementary monitoring area image, and the extracted feature points are described to create a feature point set of the main monitoring area and the supplementary monitoring area, and the feature points in the feature point set are matched, and the matched feature points are corrected to obtain a complete image.

[0177] When camera 1 detects that a target enters the blind spot, camera 1 cannot obtain the complete human target, which will affect the judgment of fall detection. Therefore, camera 2 needs to be enabled to supplement the field of view. SIFT feature point detection is performed on the images of camera 1 and camera 2 respectively, and high-quality matching points are saved. Finally, these feature points are used for image stitching. The images collected by camera 1 and camera 2 are as follows: Figure 8 As shown; the schematic diagram of feature point matching is shown in Fig. 9 ; The result of stitching the two images from camera 1 and camera 2 is as follows Fig.10 shown.

[0178] S5, uses the skeleton key points extracted by the Openpose model to perform fall detection.

[0179] The present invention uses the OpenPose model for posture estimation. OpenPose extracts the skeleton points of the human target including the nose. For the image spliced ​​by the dual cameras, OpenPose itself has integrated the function of human body detection. OpenPose can directly detect the key points of the human body from the input image and perform posture estimation.

[0180] Usually, when a person falls, the head moves a long distance from the ground. The present invention determines a fall by calculating the displacement of the key point 0. Since the distance of the human body from the camera will affect the size of the calculated displacement, the ratio of the displacement difference of the frames before and after 0 to the height of the human body frame in the previous frame is calculated to eliminate the influence of the distance from the camera. 0 Indicates the vertical coordinate of point 0 in the i-th frame, h i Indicates the height of the human body circumscribed rectangle in the i-th frame, L i It represents the relative moving distance of point 0 in the i-th frame, as shown in formula (5):

[0181]

[0182] A threshold T is set to detect possible fall events, where 1 indicates that a fall event has occurred and 0 indicates that a fall event has not occurred, as shown in formula (6).

[0183]

[0184] Taking the commonly used fall detection dataset URFD as an example, we determine the threshold T and calculate L for 30 sets of fall video sequences and 40 sets of fall-like video sequences. i When falling occurs, there will be obvious weightlessness and acceleration before the human body falls to the ground. When the human body is about to fall to the ground, the height of the human body target frame will be shortened, and the displacement of the face will increase. i It can well describe the characteristics of falling. When the threshold T is selected as 0.04, it can well distinguish between falling and non-falling.

[0185] From the perspective of the movement characteristics of a fall, a fall usually occurs within 1 to 3 seconds, during which the body falls rapidly to the ground. i It reflects the degree of head movement between the previous and next frames by continuously detecting the F of the human target in the video sequence of 30 frames before and after. i The value is used to determine whether a fall event has occurred, as shown in formula (7). When there are more than and including 20 frames of F in the detected video sequence within 30 consecutive frames, i The value is 1, indicating that a fall event has occurred; otherwise, no fall event has occurred.

[0186]

[0187] Parameter definition:

[0188] F i is an indicator variable indicating whether a fall event occurs in the i-th frame;

[0189] Fall i is a variable that indicates the comprehensive judgment of whether a fall event occurred;

[0190] The technical solution provided by the present invention is further described in detail through the above specific implementation modes in order to highlight the advantages and benefits of the technical solution provided by the present invention. However, the above specific implementation modes are not intended to limit the present invention. Any reasonable modification and improvement of the present invention, combination of implementation modes and equivalent substitution within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0191] Those skilled in the art will appreciate that the above are only preferred embodiments of the present invention, and the various embodiments of the present disclosure and / or the features described in the claims may be combined or combined in various ways, even if such combinations or combinations are not explicitly described in the present disclosure. It is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art may still modify the technical solutions described in the aforementioned embodiments, or perform equivalent substitutions on some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

[0192] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An indoor fall detection method based on dual-camera image stitching, characterized in that: The method is implemented based on two cameras, wherein a main monitoring camera (C1) collects video data of a main monitoring area in a monitoring area, and a supplementary monitoring camera (C2) collects video data of a missed detection area in the monitoring area, wherein the missed detection area is located below the main monitoring camera (C1), and the missed detection area is superimposed with the main monitoring area to form a monitoring area, and the missed detection area partially overlaps with the main monitoring area. The detection method comprises the following steps: Real-time acquisition of video data collected by two cameras; S1, a step of detecting a human target and its position in the image at the current moment in the video data of the main monitoring area; S11, if no human target is detected, return to S1 to continue detection; S12: If a human target and its position are detected, steps S2 to S5 are executed in a loop: S2, a step of tracking the position change of the human target according to the video data of the main monitoring area; S3, a step of determining whether the current position of the human target enters a missed detection area; S31, if entering the missed detection area, proceed to step S4; S32: If the missed detection area has not been entered, proceed to step S5; S4, extracting feature points of a human target in the image at the current moment from the video data of the main monitoring area and the missed detection area, and splicing the feature points to obtain a complete human target; S5. Extract the skeleton points of the human target based on the OpenPose model and determine the steps of the fall event.

2. The method according to claim 1, characterized in that Step S1 further includes: using the OpenCV library in Python to read the video data of the main monitoring area, using the YOLOv5 algorithm to detect the human target in the image at the current moment in the video data, and obtaining the detection result, wherein the detection result is to draw a bounding box of the human target on the image, and the position information of the human target includes: the upper left corner coordinates (X1, Y1) of the bounding box, the lower right corner coordinates (X2, Y2) and the center point coordinates (X center , Y center ): X center =(X1+X2) / 2, Y center =(Y1+Y2) / 2, Among them, X center Indicates the horizontal coordinate of the center point of the human body bounding box, Y center Indicates the ordinate of the center point of the human body bounding box.

3. The method according to claim 1, characterized in that Step S2 includes: S301, prediction: creating a predicted trajectory according to the current image of the main monitoring area image, and dividing the predicted trajectory into a confirmed state and an unconfirmed state by using Kalman filter prediction, wherein the confirmed state is added to the cascade matching of S302, and the unconfirmed state is added to the intersection-and-union matching of S302; S302, detection: Use the YOLOv5 algorithm to detect human targets in the image of the main monitoring area to obtain a detection result, and perform cascade matching on the detection result and the confirmed state result obtained in S301 to obtain unmatched trajectory 1, unmatched detection result 1 and matched trajectory 1, wherein the obtained unmatched trajectory 1, unmatched detection result 1 and the unconfirmed state result obtained in S301 are matched by intersection and union ratio to obtain unmatched trajectory 2, unmatched detection result 2 and matched trajectory 2, wherein the unmatched detection result 2 is used as a newly added trajectory and incorporated into the predicted trajectory of S301; S303, screening: using the DeepSort algorithm to divide the unmatched track 2 in S302 into confirmed and unconfirmed states. If it is an unconfirmed state, deleting the unconfirmed state data; If it is in the confirmation state, the maximum mismatch frame number is determined, and the maximum mismatch frame number is set to 30 frames: FPS Unmatched Tracks2 >30, Deleting the confirmed data, FPS Unmatched Tracks2 <30, The unmatched trajectory 2 in the confirmed state is incorporated into the predicted trajectory in S301; Among them, FPS Unmatched Tracks2 Indicates the number of frames that did not match track 2. S304, update: use Kalman filtering to update the matched trajectory 1 and the matched trajectory 2 in S302, and merge the updated matched trajectory 1 and the matched trajectory 2 into the predicted trajectory of S301; S305, cyclic comparison: after the predicted trajectory is detected, screened and updated, Kalman filter prediction is performed again, and the prediction result is compared with the result of the previous round of Kalman filter prediction to achieve continuous tracking of the human target.

4. The method according to claim 1, characterized in that Step S3 includes: judging whether the human target has entered the missed detection area according to the human target and its position obtained in step S1; when it is detected that the center point coordinates of the human body boundary box appear in the lower 1 / 4 area of ​​the main monitoring area, or the vertical coordinates of the lower right corner coordinates of the human body boundary box are 0, it is judged that the target has entered the missed detection area, and detecting the human target and its position in the image at the current moment in the video data of the missed detection area.

5. The method according to claim 1, characterized in that Step S4 includes: extracting feature points in the image using the SIFT algorithm, describing the extracted feature points, creating feature point sets for the main monitoring area and the supplementary monitoring area, matching the feature points in the feature point sets, correcting the matched feature points, and obtaining a complete human target.

6. The method according to claim 1, characterized in that The S5 comprises: extracting skeleton points of a human target using the OpenPose model, and setting the position of the nose in the skeleton points as the 0 point: First, eliminate the impact of the distance from the camera: Where i represents the image at the current moment, i-1 represents the image at the previous moment, and i≥2; y i 0 Indicates the vertical coordinate of point 0 in the image at the current moment; h i Represents the height of the human body bounding box in the image at the current moment; L i Indicates the relative movement distance of point 0 in the image at the current moment and the image at the previous moment; Second, detect possible fall events: Among them, F i An indicator variable indicating whether a fall event occurs in the i-th frame, 1 indicates a fall, 0 indicates no fall, and T indicates the threshold of the moving distance; Finally, determine whether a fall event has occurred: Among them, Fall i A variable that comprehensively determines whether a fall event has occurred. Indicates that within 30 consecutive frames, there are more than and including 20 frames of F i The value is 1.

7. An indoor fall detection device based on dual-camera image stitching, characterized in that: The device comprises: A module for detecting human targets and their positions in the image at the current moment in the video data of the main monitoring area; A module for tracking the position change of the human target according to the video data of the main monitoring area; A module for determining whether the current position of the human target enters a missed detection area; Extracting feature points of human targets in the image at the current moment from the video data of the main monitoring area and the missed detection area, and splicing the feature points to obtain a module of a complete human target; A module that extracts skeleton points of human targets based on the OpenPose model and determines when a fall event occurs.

8. A computer storage medium for storing a computer program, characterized in that: When the computer program is read by a computer, the computer executes the method according to any one of claims 1 to 6.

9. A computer, comprising a processor and a storage medium, characterized in that: When the processor reads the computer program stored in the storage medium, the computer executes the method according to any one of claims 1 to 6.

10. A computer program product, being a computer program, characterized in that When the computer program is read, the method according to any one of claims 1 to 6 is implemented.

Images