Personnel falling detection method and system based on human body key points and time sequence characteristics

Abstract

The invention provides a human body key point and time sequence feature-based personnel falling detection method and system. The method comprises the following steps of: sequentially performing frame extraction, shielding detection, generative completion and enhanced super-resolution processing on video stream data; a dynamic anchor frame adaptive YOLOv8 algorithm is adopted to detect a personnel target, and information such as boundary frame coordinates is output; tracking a target through an improved BoT-SORT algorithm to obtain an action track and target frame data; human body key points are detected based on a YOLOv8-Pose algorithm, and low-confidence key points are complemented in combination with kinematics constraints; the multi-dimensional features are fused and input into a TCN-Bi-LSTM mixed model, and a tumble detection result is output; the method improves the detection precision and the anti-interference capability through multi-link optimization, can effectively deal with shielding and target offset scenes, guarantees the monitoring stability, and is suitable for safety monitoring of various public and private scenes.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and safety monitoring technology, specifically to a method and system for detecting human falls based on key points and temporal characteristics of the human body. Background Technology

[0002] With the accelerating aging of global societies, home safety issues for the elderly are becoming increasingly prominent, with falls being one of the main risk factors leading to injury, disability, and even death among the elderly. Timely and accurate fall detection and early warning systems can provide rapid assistance to the elderly, significantly reducing the serious consequences of falls. Therefore, the development of related technologies has significant social value and application prospects.

[0003] Currently, mainstream fall detection technologies fall into two main categories: one is wearable sensor-based detection technology, which relies on devices such as accelerometers and gyroscopes to collect human motion data and analyze changes in posture to determine if a fall has occurred. However, it has a significant drawback: it requires users to wear the device continuously, and elderly people often do not use it due to forgetting to wear it, discomfort, or resistance, resulting in a significant reduction in detection coverage and reliability. The other category is computer vision detection technology based on video images. This technology uses a fixed camera to collect video data and utilizes algorithms such as target detection and motion recognition to achieve contactless fall detection, avoiding the limitations of wearable devices and becoming a current research hotspot.

[0004] However, existing visual fall detection methods still have many technical shortcomings in practical applications: video streams are easily affected by factors such as scene occlusion and image blurring, resulting in poor quality of the basic data for subsequent detection and tracking; the anchor boxes of personnel target detection algorithms are mostly fixed, which has poor adaptability to personnel targets of different sizes and limits detection accuracy; target tracking algorithms are easily affected by target intersection and brief occlusion, resulting in trajectory drift and loss of tracking, and insufficient trajectory stability; human key point detection is prone to generating low-confidence key points, and existing completion methods lack human kinematic constraints, resulting in poor integrity of skeletal key point sequences; action feature extraction focuses on a single dimension and does not fully integrate temporal features, which easily confuses falling with similar actions such as bending over to pick up objects or sitting down, ultimately making it difficult for the accuracy and anti-interference ability of fall detection to meet the needs of real-world scenarios. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a method for detecting falls based on key human body points and temporal characteristics, comprising: The video stream data of the area to be monitored is acquired, and the video stream data is sequentially subjected to frame extraction, occlusion detection, generative completion and enhanced super-resolution processing to obtain a preprocessed high-definition video stream; The dynamic anchor box adaptive YOLOv8 algorithm is used to detect people in the preprocessed high-definition video stream, and the bounding box coordinates, category information and confidence scores of the target people are output. Based on the bounding box coordinates, category information, and confidence level of the target person, an improved BoT-SORT algorithm is used to track the target person and obtain the target person's movement trajectory and human target box data. Based on the target person's movement trajectory and human target bounding box data, the YOLOv8-Pose algorithm is used to detect human key points of the target person, and combined with preset human kinematic constraints, low-confidence key points are adaptively completed to obtain the skeletal key point sequence of the target person. Basic coordinate features, dynamic features, and temporal features are extracted from the skeletal key point sequence and fused together. The fused features are then input into a pre-constructed fine-grained action recognition model, which outputs the fall detection results of the target person in the detection area. The fine-grained action recognition model is based on a hybrid feature fusion structure of temporal convolutional network TCN and bidirectional long short-term memory network Bi-LSTM, and is constructed through joint training with cross-entropy loss and contrastive loss.

[0006] Optionally, the step of sequentially performing frame extraction, occlusion detection, generative completion, and enhanced super-resolution processing on the video stream data to obtain a preprocessed high-definition video stream includes: The video stream data is frame extracted, and the extracted image frames are initially denoised using a median filter to obtain denoised video stream data; the highlight regions in the denoised video stream data are identified by adaptive threshold segmentation; and the highlight regions are subjected to Gamma correction to obtain initial processed data. The occlusion detection network based on the Transformer attention masking mechanism is used to perform occlusion detection on the initial processed data to obtain the occluded region in the region to be monitored. The occluded region is filled in using a generative adversarial network that incorporates prior knowledge of human pose, resulting in filled image data. The generator of the generative adversarial network adopts a U-Net structure, and the discriminator adopts a convolutional neural network structure. Through adversarial training, the texture and brightness of the filled region are kept consistent with those of the unoccluded region. The completed image data is processed using an enhanced super-resolution algorithm to obtain a pre-processed high-definition video stream.

[0007] Optionally, the step of using the dynamic anchor box adaptive YOLOv8 algorithm to perform personnel target detection on the preprocessed high-definition video stream, and outputting the bounding box coordinates, category information, and confidence score of the target personnel, includes: Multi-scale image features are extracted from the preprocessed high-definition video stream through a backbone network; wherein, the backbone network is constructed based on CSPDarknet, with a new context feature enhancement branch added to extract global context features and fuse them with local features element by element; Based on the multi-scale image features, an anchor frame adapted to the preprocessed high-definition video stream is generated; Based on the anchor frame, the multi-scale image features are complementaryly fused through the Neck feature fusion network to obtain an enhanced multi-scale feature map; wherein, the Neck feature fusion network is constructed based on the fusion structure of the path aggregation network PAN and the feature pyramid network FPN; The enhanced multi-scale feature map is detected by the Head detection head, and the bounding box coordinates, category information and confidence scores of candidate targets in the monitored area are output. Based on the bounding box coordinates, category information, and confidence level of the candidate targets, a non-maximum suppression algorithm is used to filter the candidate targets, and the bounding box coordinates, category information, and confidence level of the target personnel are output.

[0008] Optionally, the step of tracking the target person using an improved BoT-SORT algorithm based on the bounding box coordinates, category information, and confidence level to obtain the target person's movement trajectory and human bounding box data includes: Based on the bounding box coordinates, category information, and confidence level of the target person, the position features, appearance features, and motion trend features of the target person are extracted; the motion trend features include x-direction velocity, y-direction velocity, rate of change of velocity, and motion direction angle; Based on the location characteristics of the target personnel, construct an IoU metric matrix; Based on the appearance characteristics of the target person, construct an appearance feature similarity matrix; Based on the movement trend characteristics of the target personnel, a time correlation matrix is ​​constructed; The IoU metric matrix, appearance feature similarity matrix, and temporal correlation matrix are fused using a spatiotemporal attention mechanism to obtain a fused feature matrix; Based on the fused feature matrix, the Hungarian algorithm is used to associate the current frame detection box of the target person with the existing trajectory; Based on the associated trajectory, combined with the mean confidence level and occlusion rate of the target person, a trajectory quality score is calculated, and high-quality trajectories with a trajectory quality score not lower than a preset threshold are selected. Based on the high-quality trajectory, output the movement trajectory of the target person in consecutive frames and the human target bounding box data of the corresponding frames.

[0009] Optionally, the key points of the human body are the core skeletal joints of the human body, including the tip of the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. The human kinematic constraints include one or more of the following: joint angle constraints, joint length constraints, and motion continuity constraints.

[0010] Optionally, the step of extracting basic coordinate features, dynamic features, and temporal features from the skeletal keypoint sequence and fusing them includes: Generate the coordinates of the bone joints based on the sequence of bone key points; The coordinates of the skeletal joints are flattened to obtain the basic coordinate feature vector; Dynamic features are extracted from the coordinates of the skeletal joints to form a motion feature vector; wherein, the dynamic features include the angle change rate of multiple joints, the change in the height of the human body's center of gravity, and the body tilt angle; Temporal features are extracted from the coordinates of the skeletal joints to form a temporal feature vector; wherein, the temporal features include the duration of the action and the curvature of the motion trajectory of each key point; The basic coordinate feature vector, motion feature vector, and time feature vector are concatenated and fused.

[0011] Optionally, the construction process of the fine-grained action recognition model includes: The pre-acquired sample dataset is divided into a training set, a validation set, and a test set according to a preset ratio; wherein, the sample dataset contains five types of action samples: falling, bending over to pick up an object, sitting down, squatting, and walking normally, and the number of samples in each type meets the model training requirements. A hybrid structure of temporal convolutional network and bidirectional long short-term memory network is constructed. The temporal convolutional network contains multiple convolutional blocks, and the bidirectional long short-term memory network contains multiple hidden layers. Set the total loss function to be the sum of cross-entropy loss and the comparative loss with preset weights; Based on the training set, the Adam optimizer is used to train the model, the number of training iterations and the initial learning rate are set, and the learning rate is decayed at a preset interval. The model hyperparameters are adjusted using the validation set. Training stops when the validation set accuracy does not improve for a preset number of consecutive rounds, and the optimal model is saved as a fine-grained action recognition model.

[0012] Based on the same inventive concept, this invention also provides a fall detection system for individuals based on key points and temporal characteristics of the human body, comprising: The data preprocessing module is used to acquire video stream data of the area to be monitored, and sequentially perform frame extraction, occlusion detection, generative completion and enhanced super-resolution processing on the video stream data to obtain a preprocessed high-definition video stream. The target detection module is used to perform personnel target detection on the preprocessed high-definition video stream using the dynamic anchor box adaptive YOLOv8 algorithm, and outputs the bounding box coordinates, category information and confidence score of the target personnel; The target tracking module is used to track the target person using an improved BoT-SORT algorithm based on the bounding box coordinates, category information, and confidence level of the target person, thereby obtaining the target person's movement trajectory and human target box data. The key point detection module is used to detect human key points of the target person based on the target person's movement trajectory and human target box data, using the YOLOv8-Pose algorithm, and adaptively completes low-confidence key points by combining preset human kinematic constraints to obtain the skeletal key point sequence of the target person. The fall detection module is used to extract basic coordinate features, dynamic features and time series features from the skeletal key point sequence and fuse them. The fused features are then input into a pre-built fine-grained action recognition model, and the fall detection results of the target person in the detection area are output. The fine-grained action recognition model is based on a hybrid feature fusion structure of temporal convolutional network TCN and bidirectional long short-term memory network Bi-LSTM, and is constructed through joint training with cross-entropy loss and contrastive loss.

[0013] Optionally, the data preprocessing module includes: The frame extraction submodule is used to extract frames from the video stream data, and to perform preliminary denoising on the extracted image frames using a median filter to obtain denoised video stream data; to identify the highlight regions in the denoised video stream data through adaptive threshold segmentation; and to perform Gamma correction on the highlight regions to obtain initial processed data. The occlusion detection submodule is used to perform occlusion detection on the initial processed data using an occlusion detection network built based on the Transformer attention masking mechanism, so as to obtain the occluded region in the region to be monitored. The region completion submodule is used to complete the occluded region using a generative adversarial network that integrates prior knowledge of human pose, so as to obtain the completed image data. The generator of the generative adversarial network adopts a U-Net structure, and the discriminator adopts a convolutional neural network structure. Through adversarial training, the texture and brightness of the completed region are kept consistent with those of the non-occluded region. The super-resolution processing submodule is used to perform super-resolution processing on the completed image data using an enhanced super-resolution algorithm to obtain a pre-processed high-definition video stream.

[0014] Optionally, the target detection module includes: The feature extraction submodule is used to extract multi-scale image features from the preprocessed high-definition video stream through the backbone network; wherein, the backbone network is built based on CSPDarknet, and a new context feature enhancement branch is added to extract global context features and fuse them with local features element by element; An anchor frame generation submodule is used to generate anchor frames adapted to the preprocessed high-definition video stream based on the multi-scale image features. The feature fusion submodule is used to perform complementary fusion of the multi-scale image features based on the anchor boxes through the Neck feature fusion network to obtain an enhanced multi-scale feature map; wherein, the Neck feature fusion network is constructed based on the fusion structure of the path aggregation network PAN and the feature pyramid network FPN; The feature detection submodule is used to detect the enhanced multi-scale feature map through the Head detection head, and output the bounding box coordinates, category information and confidence of the candidate targets in the region to be monitored; The target filtering submodule is used to filter the candidate targets using a non-maximum suppression algorithm based on the bounding box coordinates, category information, and confidence level of the candidate targets, and outputs the bounding box coordinates, category information, and confidence level of the target personnel.

[0015] Optionally, the target tracking module includes: The trend prediction submodule is used to extract the location features, appearance features, and motion trend features of the target person based on the bounding box coordinates, category information, and confidence level of the target person; the motion trend features include x-direction velocity, y-direction velocity, rate of change of velocity, and motion direction angle; The metric matrix construction submodule is used to construct an IoU metric matrix based on the location characteristics of the target personnel. The appearance matrix construction submodule is used to construct an appearance feature similarity matrix based on the appearance features of the target person. The time matrix construction submodule is used to construct a time correlation matrix based on the movement trend characteristics of the target personnel; The matrix fusion submodule is used to fuse the IoU metric matrix, appearance feature similarity matrix and temporal correlation matrix through a spatiotemporal attention mechanism to obtain a fused feature matrix; The trajectory association submodule is used to associate the current frame detection box of the target person with the existing trajectory based on the fused feature matrix and using the Hungarian algorithm. The trajectory evaluation submodule is used to calculate the trajectory quality score based on the associated trajectory, combined with the mean confidence level and occlusion rate of the target person, and to filter high-quality trajectories whose trajectory quality score is not lower than a preset threshold. The trajectory output submodule is used to output the movement trajectory of the target person in consecutive frames and the human target bounding box data of the corresponding frames based on the high-quality trajectory.

[0016] Optionally, the key points of the human body are the core skeletal joints of the human body, including the tip of the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. The human kinematic constraints include one or more of the following: joint angle constraints, joint length constraints, and motion continuity constraints.

[0017] Optionally, the fall detection module includes: The joint coordinate generation submodule is used to generate bone joint coordinates based on the bone key point sequence. The flattening submodule is used to flatten the coordinates of the skeletal joint points to obtain the basic coordinate feature vector. The feature vector generation submodule is used to extract dynamic features from the coordinates of the skeletal joints and form motion feature vectors; wherein, the dynamic features include the angle change rate of multiple joints, the change of the height of the human body's center of gravity, and the body tilt angle; The time vector generation submodule is used to extract temporal features from the coordinates of the skeletal joints and form a time feature vector; wherein, the temporal features include the duration of the action and the curvature of the motion trajectory of each key point; The vector splicing submodule is used to splice and fuse the basic coordinate feature vector, motion feature vector, and time feature vector.

[0018] Optionally, the system further includes: a model building module, specifically including: The dataset partitioning submodule is used to divide the pre-acquired sample dataset into training set, validation set and test set according to a preset ratio; wherein, the sample dataset contains five types of action samples: falling, bending over to pick up objects, sitting down, squatting and normal walking, and the number of samples in each type meets the model training requirements. The architecture building submodule is used to build a hybrid structure of temporal convolutional network and bidirectional long short-term memory network. The temporal convolutional network contains multiple convolutional blocks, and the bidirectional long short-term memory network contains multiple hidden layers. The loss setting submodule is used to set the total loss function to the sum of cross-entropy loss and the comparison loss with preset weights; The parameter setting submodule is used to train the model using the Adam optimizer based on the training set, set the number of training iterations and the initial learning rate, and decay the learning rate at preset intervals. The model validation submodule is used to adjust the model hyperparameters using the validation set. When the accuracy of the validation set does not improve for a preset number of consecutive rounds, training is stopped and the optimal model is saved as a fine-grained action recognition model.

[0019] In another aspect, the present invention also provides an electronic device, comprising: at least one processor and a memory; the memory and the processor are connected via a bus; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, a method for detecting human falls based on key points and temporal features of the human body, as described above, is implemented.

[0020] In another aspect, the present invention also provides a computer device readable storage medium having an executable program stored thereon, wherein when the executable program is executed, it implements the aforementioned method for detecting human falls based on key points and temporal characteristics of the human body.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention provides a method and system for detecting falls based on human body key points and temporal features, comprising: acquiring video stream data of the area to be monitored; sequentially performing frame extraction, occlusion detection, generative completion, and enhanced super-resolution processing on the video stream data to obtain a preprocessed high-definition video stream; using a dynamic anchor box adaptive YOLOv8 algorithm to detect human targets in the preprocessed high-definition video stream, outputting the bounding box coordinates, category information, and confidence level of the target person; using an improved BoT-SORT algorithm to track the target person based on the bounding box coordinates, category information, and confidence level of the target person, obtaining the target person's movement trajectory and human body target box data; using the YOLOv8-Pose algorithm to detect human body key points in the target person based on the target person's movement trajectory and human body target box data, and combining preset human kinematic constraints to adaptively complete low-confidence key points, obtaining the skeletal key point sequence of the target person; extracting basic coordinate features, dynamic features, and temporal features from the skeletal key point sequence and fusing them, and inputting the fused features into a pre-constructed fine-grained action recognition model. The system outputs the fall detection results of the target person in the detection area. The fine-grained action recognition model is based on a hybrid feature fusion structure of a temporal convolutional network (TCN) and a bidirectional long short-term memory network (Bi-LSTM), and is constructed through joint training with cross-entropy loss and contrast loss. This invention effectively improves the quality of basic video data and solves interference caused by occlusion and image blur by sequentially performing frame extraction, occlusion detection, generative completion, and enhanced super-resolution processing on the video stream. The use of a dynamic anchor box adaptive YOLOv8 algorithm for personnel target detection improves adaptability to personnel of different sizes and increases target detection accuracy. The improved BoT-SORT algorithm for target tracking enhances the continuity and stability of the trajectory, reducing trajectory drift and loss of tracking. Adaptive completion of low-confidence keypoints using preset human kinematic constraints helps ensure the integrity and accuracy of the skeletal keypoint sequence. The fusion and extraction of basic coordinates, dynamic and temporal features from the skeletal keypoint sequence and input into the TCN and Bi-LSTM hybrid model improves the recognition and differentiation of falls and similar actions, thus increasing the fall detection accuracy. Attached Figure Description

[0022] Figure 1 A flowchart illustrating a method for detecting human falls based on key points and temporal features provided by this invention; Figure 2 A schematic diagram of the overall framework of a method for detecting human falls based on key points and temporal features provided by the present invention. Figure 3A schematic diagram of the structural composition of a human fall detection system based on key points and temporal features provided by the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device provided by the present invention. Detailed Implementation

[0023] This invention proposes a method, system, device, and medium for detecting human falls based on key points and temporal characteristics of the human body. The specific embodiments of this invention will be further described in detail below with reference to the accompanying drawings.

[0024] Example 1: This invention provides a method for detecting falls by human body based on key points and temporal features, as illustrated in the flowchart below. Figure 1 As shown, it includes: Step 1: Obtain video stream data of the area to be monitored, and sequentially perform frame extraction, occlusion detection, generative completion and enhanced super-resolution processing on the video stream data to obtain a pre-processed high-definition video stream; Step 2: Use the dynamic anchor box adaptive YOLOv8 algorithm to detect people in the preprocessed high-definition video stream, and output the bounding box coordinates, category information and confidence score of the target people; Step 3: Based on the bounding box coordinates, category information, and confidence level of the target person, the improved BoT-SORT algorithm is used to track the target person and obtain the target person's movement trajectory and human target box data; Step 4: Based on the target person's movement trajectory and human target bounding box data, the YOLOv8-Pose algorithm is used to detect human key points of the target person, and combined with preset human kinematic constraints, the low-confidence key points are adaptively completed to obtain the skeletal key point sequence of the target person. Step 5: Extract basic coordinate features, dynamic features, and temporal features from the skeletal key point sequence and fuse them. Input the fused features into a pre-built fine-grained action recognition model and output the fall detection results of the target person in the detection area. The fine-grained action recognition model is based on a hybrid feature fusion structure of temporal convolutional network TCN and bidirectional long short-term memory network Bi-LSTM, and is constructed through joint training with cross-entropy loss and contrastive loss.

[0025] In one implementation, the process of sequentially performing frame extraction, occlusion detection, generative completion, and enhanced super-resolution processing on the video stream data in step 1 above to obtain a preprocessed high-definition video stream may include: The video stream data is frame extracted, and the extracted image frames are initially denoised using a median filter to obtain denoised video stream data; the highlight regions in the denoised video stream data are identified by adaptive threshold segmentation; and the highlight regions are subjected to Gamma correction to obtain initial processed data. The occlusion detection network based on the Transformer attention masking mechanism is used to perform occlusion detection on the initial processed data to obtain the occluded region in the region to be monitored. The occluded region is filled in using a generative adversarial network that incorporates prior knowledge of human pose, resulting in filled image data. The generator of the generative adversarial network adopts a U-Net structure, and the discriminator adopts a convolutional neural network structure. Through adversarial training, the texture and brightness of the filled region are kept consistent with those of the unoccluded region. The completed image data is processed using an enhanced super-resolution algorithm to obtain a pre-processed high-definition video stream; In this implementation, video stream data can be retrieved from the video stream management platform from the camera video stream data of the target monitoring area (such as the living room, bedroom, bathroom, etc. in an elderly person's home), obtain the unique number and video stream address (such as RTSP protocol address) corresponding to each camera, perform unified parsing and processing on the video stream with the camera number, and establish a mapping relationship between the video stream and the monitoring area to facilitate the subsequent location of abnormal events.

[0026] The core objective of the preprocessing stage is to eliminate video quality defects and scene interference, providing high-quality image input for subsequent detection modules. The specific steps are as follows: 1. Frame extraction and preliminary denoising: For example, the OpenCV video processing library can be used to extract the video stream frame by frame to obtain a continuous sequence of image frames. A 3×3 median filter is used to traverse the image pixels to suppress salt-and-pepper noise and random noise. For overexposed areas, an adaptive threshold segmentation algorithm is used to identify highlight areas. The brightness and contrast of the area are adjusted by Gamma correction to suppress reflections and avoid loss of detail in highlight areas.

[0027] 2. Occlusion Detection: An attention-based masking mechanism based on Transformer can be introduced to construct an occlusion detection network. This network takes initially denoised image frames as input, extracts global and local features of the image through a Transformer encoder, uses a self-attention mechanism to locate the boundary between the human body region and the background region, and then identifies areas where the human body is occluded by furniture (such as sofas and tables), doors, windows, or other people. Binarization is then used to generate an occlusion mask (occluded areas are marked as 1, and unoccluded areas are marked as 0), accurately labeling the location and extent of the occlusion.

[0028] 3. Occlusion Completion: An occlusion completion model can be built based on Generative Adversarial Networks (GANs). This model consists of two parts: a generator and a discriminator. The generator can employ a U-Net network structure, taking as input a pre-denoised image frame and an occlusion mask. It combines prior knowledge of human pose (such as the connection relationships of human skeletal joints and the distribution patterns of key points in common poses) to predict and generate the human contour and skeletal key points in the occluded region. In this implementation, the generator learns the pose features of a large number of occluded human images to ensure that the human structure in the completed region conforms to physiological movement patterns. Discriminator: A convolutional neural network (CNN) structure can be used to distinguish the completed image from the real unoccluded image. The completion effect of the generator can be optimized through adversarial training to make the texture and brightness of the completed area consistent with the unoccluded area and avoid obvious completion traces. The completion process is as follows: The region corresponding to the occlusion mask is input into the generator to generate a completed partial image of the human body. The completed region is then seamlessly stitched with the non-occluded region of the original image using an image fusion algorithm to obtain a human body image without occlusion interference. 4. Enhanced Super-Resolution Processing: An enhanced super-resolution model can be constructed by optimizing the traditional Real-ESRGAN super-resolution algorithm and incorporating occlusion masking information. For example: For non-occluded areas: the original super-resolution logic of Real-ESRGAN can be used to recover high-frequency details of the image through deep learning models, thereby improving resolution and clarity; For the occluded region after completion: Texture consistency constraints can be added during the super-resolution process. By calculating the texture features (such as gradient and edge direction) of the completed region and the surrounding non-occluded region, the output weights of the super-resolution model can be adjusted to ensure that the texture details of the completed region are naturally connected with the surrounding region, avoiding texture breakage or inconsistency. 5. Preprocessing output: The image frames after occlusion detection, completion, and enhanced super-resolution processing are recombined to form a high-definition, occlusion-free, and detail-rich video stream, which is then passed to subsequent pedestrian target detection. This implementation accurately locates occluded regions using a Transformer attention masking mechanism. Combined with a GAN network incorporating prior knowledge of human pose, it achieves reasonable completion of occluded areas. Furthermore, an enhanced super-resolution algorithm incorporating occlusion masks improves image clarity, simultaneously resolving the superimposed interference of blur and occlusion. This mechanism ensures that the preprocessed image is both clear and complete, providing defect-free input for subsequent detection modules and addressing the core pain point of keypoint loss and decreased detection accuracy caused by occlusion in existing technologies.

[0029] In one implementation, step 2 above, which uses the dynamic anchor box adaptive YOLOv8 algorithm to detect people in the preprocessed high-definition video stream and outputs the bounding box coordinates, category information, and confidence score of the target people, may include: Multi-scale image features are extracted from the preprocessed high-definition video stream through a backbone network; wherein, the backbone network is constructed based on CSPDarknet, with a new context feature enhancement branch added to extract global context features and fuse them with local features element by element; Based on the multi-scale image features, an anchor frame adapted to the preprocessed high-definition video stream is generated; Based on the anchor frame, the multi-scale image features are complementaryly fused through the Neck feature fusion network to obtain an enhanced multi-scale feature map; wherein, the Neck feature fusion network is constructed based on the fusion structure of the path aggregation network PAN and the feature pyramid network FPN; The enhanced multi-scale feature map is detected by the Head detection head, and the bounding box coordinates, category information and confidence scores of candidate targets in the monitored area are output. Based on the bounding box coordinates, category information, and confidence level of the candidate targets, a non-maximum suppression algorithm is used to filter the candidate targets, and the bounding box coordinates, category information, and confidence level of the target personnel are output. In this implementation, the original YOLOv8 CSPDarknet-53 backbone network is retained. This network reduces the amount of computation through cross-stage local connectivity (CSP) structure while enhancing feature extraction capabilities. Furthermore, a new context feature enhancement branch is added. After the feature output of the 3rd, 4th, and 5th layers of CSPDarknet-53, a global average pooling layer and a 1×1 convolutional layer are respectively connected to extract the global context features of the image (such as the relative positional relationship between the human body and the environment). The global features and local features are added and fused element by element, thereby improving the feature recognition of small target pedestrians and avoiding missed detections due to small target size and weak features. In this implementation, the specific process of dynamically generating anchor frames can be as follows: During the model startup phase, pedestrian target statistics can be performed on the first 100 preprocessed image frames to extract the width (w) and height (h) of each pedestrian target box and construct a target size dataset. The K-means clustering algorithm is used to perform cluster analysis on the target size dataset (e.g., the number of clusters can be set to 9) to obtain an initial set of anchor boxes that fits the pedestrian size distribution in the current video stream. During model operation, the target size distribution can be re-statistically analyzed every 50 frames, the cluster centers can be updated, and the size and proportion of the anchor boxes can be dynamically adjusted to replace the fixed anchor boxes of traditional YOLOv8. To address the size difference between pedestrians in the foreground (large target size) and the background (small target size), the dynamically generated anchor boxes are divided into three scales: large, medium, and small, each corresponding to a different level of feature map (large-scale anchor boxes correspond to shallow feature maps, and small-scale anchor boxes correspond to deep feature maps), ensuring that pedestrian targets of different sizes can be accurately matched. In this implementation, the Neck feature fusion network adopts the PAN-FPN structure of YOLOv8. It achieves the fusion of multi-scale features through the top-down feature pyramid (FPN) and the bottom-up path aggregation network (PAN). It performs upsampling, downsampling and concatenation processing on the feature maps of different levels output by the backbone network, so that the model has both strong semantic information and high-resolution features, thereby improving the detection accuracy of overlapping targets and small targets.

[0030] In this implementation, the head detection head adopts the decoupled detection head structure of YOLOv8, which separates the classification task and the regression task, and outputs the target's class confidence and bounding box coordinates through independent convolutional layers; the classification branch outputs the class probabilities of pedestrians and non-pedestrians, and the regression branch outputs the coordinates (x1, y1, x2, y2) and size information of the target box; In this implementation, when optimizing the loss function, Focal Loss can be introduced to replace the traditional Cross-Entropy Loss function for classification tasks. By setting the focusing parameter γ (e.g., γ=2), the weight of easily classified samples (such as background regions) is reduced, while the weight of difficult-to-classify samples (such as small pedestrian targets and overlapping pedestrian edge regions) is increased, thus solving the sample imbalance problem. For the regression task, the CIoU loss function is used, which comprehensively considers the overlap of the target boxes, the distance between the center points, and the aspect ratio to improve the accuracy of bounding box regression.

[0031] In the above implementation, the preprocessed high-definition video stream is input frame by frame into the dynamic anchor box adaptive YOLOv8 model. Features are extracted through the backbone network, and the detection results are obtained through dynamic anchor box matching, feature fusion, and detection head inference. The detection results are then filtered using the non-maximum suppression (NMS) algorithm. A confidence threshold (e.g., the confidence threshold conf can be set to 0.5) and an IOU threshold (e.g., the IOU threshold iou can be set to 0.45) are set to remove redundant detection boxes with high overlap and retain the effective detection boxes with the highest confidence. Finally, if a pedestrian target is detected, the bounding box coordinates (x1, y1, x2, y2), category information (label="person"), confidence score (conf), and target size (w=x2-x1, h=y2-y1) of each target are output. If no pedestrian target is detected, no pedestrian label is output and no further processing is performed. This implementation addresses the poor adaptability of traditional YOLOv8 fixed anchor boxes by proposing a dynamic anchor box generation mechanism. This mechanism dynamically adjusts the size and proportion of anchor boxes by statistically analyzing the size distribution of pedestrian targets in the video stream in real time, using a K-means clustering algorithm to adaptively adapt to changes in target size in near / far-view and single / multi-person scenarios. A new contextual feature enhancement branch is added, fusing global and local features to improve the feature recognition of small pedestrian targets. Furthermore, a focus loss function is introduced to address the sample imbalance problem, significantly improving the detection recall rate of small and overlapping targets.

[0032] In one implementation, step 3 above, which uses an improved BoT-SORT algorithm to track the target person based on the bounding box coordinates, category information, and confidence level, to obtain the target person's movement trajectory and human target box data, may include: Based on the bounding box coordinates, category information, and confidence level of the target person, the position features, appearance features, and motion trend features of the target person are extracted; the motion trend features include x-direction velocity, y-direction velocity, rate of change of velocity, and motion direction angle; Based on the location characteristics of the target personnel, construct an IoU metric matrix; Based on the appearance characteristics of the target person, construct an appearance feature similarity matrix; Construct a time correlation matrix based on the movement trend characteristics of the target personnel; The IoU metric matrix, appearance feature similarity matrix, and temporal correlation matrix are fused using a spatiotemporal attention mechanism to obtain a fused feature matrix; Based on the fused feature matrix, the Hungarian algorithm is used to associate the current frame detection box of the target person with the existing trajectory; Based on the associated trajectory, combined with the mean confidence level and occlusion rate of the target person, a trajectory quality score is calculated, and high-quality trajectories with a trajectory quality score not lower than a preset threshold are selected. Based on the high-quality trajectory, output the movement trajectory of the target person in consecutive frames and the human target bounding box data of the corresponding frames; In this implementation, by adding motion trend feature extraction, spatiotemporal attention association, and trajectory quality evaluation mechanisms to the original BoT-SORT algorithm, the tracking stability in scenarios with multiple targets crossing and brief occlusion can be improved. The specific design can be as follows: 1. Input data preprocessing: Receive the detection results output by the pedestrian target detection model, including the target box coordinates (x1, y1, x2, y2), category information (label), confidence score (conf), and target size (w, h). Standardize the format of the detection results and convert them into an input format that the tracker can recognize.

[0033] 2. Multi-dimensional feature extraction: Appearance feature extraction: The Transformer feature embedding network of BoT-SORT is used to crop the human image region corresponding to each detection box and input it into the Transformer encoder to extract appearance feature vectors (512 dimensions) to distinguish different pedestrian targets; Location feature extraction: The target bounding box coordinates (x1, y1, x2, y2) are converted into center coordinates (cx, cy) and width and height (w, h), and a 4-dimensional location feature vector is constructed. Motion trend feature extraction: A new motion feature extraction module has been added to analyze the tracking trajectory of each target in the first 5 frames, calculate the target's motion direction (such as horizontal, vertical, and diagonal) and velocity change rate (the ratio of the current frame velocity to the previous frame velocity), and construct a 6-dimensional motion trend feature vector (including parameters such as x-direction velocity, y-direction velocity, velocity change rate, and motion direction angle).

[0034] 3. Spatiotemporal attention association mechanism: Spatial association calculation: Calculate the IOU (Intersection over Union) between the current frame detection box and the existing trajectory, and construct the IOU metric matrix to measure the spatial overlap between the detection box and the trajectory; Temporal correlation calculation: Based on motion trend feature vectors, the cosine similarity algorithm is used to calculate the consistency of motion trends between the current frame detection box and the existing trajectory, and a temporal correlation matrix is ​​constructed; Attention Fusion: Through the spatiotemporal attention mechanism of Transformer, the appearance feature similarity matrix, IOU metric matrix and temporal correlation matrix are weighted and fused. The weight coefficients are adaptively learned through model training (initial weights are set to 0.5 for appearance features, 0.3 for IOU and 0.2 for motion trend) to generate a comprehensive correlation matrix, replacing the traditional single-dimensional matching standard. Trajectory association: The Hungarian algorithm is used to perform optimal matching on the comprehensive association matrix, and the detection box of the current frame is associated with the existing trajectory to achieve continuous tracking of the target identity.

[0035] 4. Trajectory quality assessment: The trajectory quality scoring metric is defined, comprising three core parameters: ① Mean confidence score (the average confidence score of the detection boxes in the last 10 frames, weighted at 0.4); ② Position stability (the variance of the target center coordinates in the last 10 frames, weighted at 0.3); ③ Occlusion rate (the percentage of occluded frames in the last 10 frames, weighted at 0.3). The trajectory quality score is calculated as follows: 0.4 × Mean confidence score + 0.3 × (1 - Normalized position stability value) + 0.3 × (1 - Occlusion rate), with a score range of 0-1. Trajectory filtering: Set a quality score threshold (0.6). Trajectories with a score ≥ 0.6 are judged as high-quality trajectories, retained and updated; trajectories with a score < 0.6 are judged as low-quality trajectories (may have identity confusion or tracking drift) and discarded to avoid interfering with the subsequent key point detection process. 5. Trajectory Optimization and Recovery: Online trajectory smoothing: The Kalman filter algorithm is used to optimize high-quality trajectories. When the trajectory changes abruptly (the distance between the center coordinates of adjacent frames is >20 pixels) or the accumulated error is large, a virtual intermediate point is generated by the detection result of the current frame and the trajectory position of the previous 3 frames to assist the Kalman filter in state prediction and reduce the accumulated error. Trajectory recovery strategy: When a trajectory is lost due to a brief occlusion, the appearance features, motion trend features, and last appearance location information of the trajectory are retained. In the next 5 frames, a matching target is continuously searched in the vicinity of the lost location (within a radius of 50 pixels). If a detection box with an appearance feature similarity of ≥0.8 and consistent motion trend is found, the trajectory is recovered; if no matching target is found within 5 frames, the trajectory is deleted. In this implementation, the specific process for target tracking can be as follows: 1. Initialization: In the first frame of detection results, assign a unique identifier (person ID, starting from 1 and incrementing) to each pedestrian target, and initialize the trajectory of each target (including position, size, speed, appearance features and other state information). 2. Inter-frame update: For each subsequent frame of detection results, perform multi-dimensional feature extraction, spatiotemporal attention association, trajectory quality assessment and trajectory optimization operations, update the state information of the existing trajectory, and maintain the consistency of the target ID; 3. Valid trajectory filtering: Check the number of consecutive frames of the trajectory. When the ID of a target remains stable for N consecutive frames (N is a natural number between 20 and 30, and the default is 25 frames) and the trajectory quality score is ≥0.6, it is determined to be a valid trajectory. Extract the human target bounding box data corresponding to the ID in the N frames and pass it to the human key point detection process. 4. Output information: Output the tracking results for each valid trajectory, including the target bounding box coordinates (x1, y1, x2, y2), personnel ID, trajectory quality score, and number of consecutive frames.

[0036] This implementation adds motion trend feature extraction to the BoT-SORT algorithm, constructing a multi-dimensional association system of appearance features, space, IOU, and motion trends. By fusing the multi-dimensional association matrix through a spatiotemporal attention mechanism, it helps solve the trajectory breakage problem caused by multi-target intersection and brief occlusion. Furthermore, by designing trajectory quality evaluation indicators, low-quality trajectories are downweighted or discarded, which can avoid identity confusion and tracking drift from interfering with subsequent processes. By optimizing trajectory smoothing and recovery strategies, it helps improve the continuity and stability of tracking.

[0037] In one implementation, the process of detecting human key points in step 4 above may specifically include: Receive valid trajectory data output by the target tracking module, namely, the human target bounding box image (cropped to contain only a single human body) corresponding to the same person ID in N consecutive frames (N=30). Using the original YOLOv8-Pose network structure and CSPNet as the backbone network, deep features are extracted from human target bounding box images to obtain multi-scale feature maps; feature fusion is performed through the Neck network (PAN-FPN) to enhance the expressive power of key point features. The head detection head outputs the probability distribution heatmap, coordinate offset, scale, and orientation information for each keypoint. The heatmap reflects the probability distribution of the keypoint in the image, the offset indicates the precise offset of the keypoint relative to the peak position of the heatmap, and the scale and orientation are used to correct the scale and orientation of the keypoint. By combining the original image coordinates and offset, the initial coordinates (x, y) of each keypoint are calculated, and the confidence score (range 0-1) of the keypoint is determined based on the peak probability of the heatmap. The higher the confidence score, the more reliable the keypoint detection.

[0038] For example, the key points of the human body mentioned above can be the core skeletal joints of the human body, including the tip of the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. The human kinematic constraints may include one or more of the following: joint angle constraints, joint length constraints, and motion continuity constraints. Specifically, the process of adaptively completing low-confidence keypoints in this implementation method may include: Set a confidence threshold (0.5) and judge the confidence of each key point: High confidence key point: confidence ≥ 0.5, marked as "valid key point", and retain its original coordinates; Low confidence key point: confidence < 0.5, combined with occlusion mask information, marked as "suspected missing / occluded key point", which needs to be completed. Extract the keypoint coordinate sequence of the person's ID from the first 10 frames to construct a temporal trajectory database. For each suspected missing keypoint, analyze its variation pattern in the temporal trajectory: for example, calculate the coordinate change trend of the keypoint in the last 10 frames (such as displacement in the x and y directions, and movement speed); analyze the angle change pattern of the corresponding joints (for example, when the left elbow keypoint is missing, calculate the historical angle range of left shoulder-left elbow-left wrist, usually the elbow joint bending angle is 0-140°); Based on the physiological movement patterns of the human body, kinematic constraint equations are set for each joint. For example, joint angle constraints define the range of joint angles formed by adjacent keypoints (such as shoulder-elbow-wrist, hip-knee-ankle). For instance, the shoulder joint angle (left shoulder-right shoulder-right elbow) can range from 120° to 180°, and the knee joint angle (left hip-left knee-left ankle) can range from 0° to 160°. Joint length constraints are set based on human body proportions, defining the distance range between adjacent keypoints (such as the distance from shoulder to elbow being approximately 1 / 5 of the height, and the distance from elbow to wrist being approximately 1 / 6 of the height). The height of the human body is estimated by the target bounding box size, thereby determining the range of joint lengths. Motion continuity constraints stipulate that the displacement of the same keypoint in adjacent frames must not exceed 1 / 10 of the human body height (to avoid jumping movements that do not conform to physical laws). A constraint-based interpolation algorithm is used to complete low-confidence keypoints. For example, based on the temporal trajectory trend of the keypoint, preliminary completed coordinates are generated. The preliminary completed coordinates are substituted into the kinematic constraint equations to verify whether the constraints of joint angle, length and motion continuity are satisfied. If the constraints are satisfied, the coordinates are determined as the final coordinates after completion. If the constraints are not satisfied, the coordinate values ​​are adjusted by the gradient descent algorithm until all constraints are satisfied. Finally, the coordinates (x, y) of the 17 completed skeletal key points and the confidence score are integrated to form a complete skeletal key point sequence of N consecutive frames (N=30), which is then passed to the fall action recognition process. In this implementation, a completion scheme based on temporal reference and kinematic constraints is proposed to address the problem of missing key points caused by blurring and occlusion. By analyzing the temporal trajectory of key points in previous frames and combining the constraints of human joint angles, lengths and motion continuity, low-confidence key points are completed, generating a complete skeletal key point sequence that conforms to physiological laws. This helps to overcome the limitations of traditional single-frame detection, avoid the failure of action recognition due to missing key points, and improve the accuracy of action recognition for incomplete human bodies.

[0039] In one implementation, step 5 above, which involves extracting basic coordinate features, dynamic features, and temporal features from the skeletal keypoint sequence and fusing them, may include: Generate the coordinates of the bone joints based on the sequence of bone key points; The coordinates of the skeletal joints are flattened to obtain the basic coordinate feature vector; Dynamic features are extracted from the coordinates of the skeletal joints to form a motion feature vector; wherein, the dynamic features include the angle change rate of multiple joints, the change in the height of the human body's center of gravity, and the body tilt angle; Temporal features are extracted from the coordinates of the skeletal joints to form a temporal feature vector; wherein, the temporal features include the duration of the action and the curvature of the motion trajectory of each key point; The basic coordinate feature vector, motion feature vector, and time feature vector are concatenated and fused.

[0040] In this implementation, the construction process of the fine-grained action recognition model may include: The pre-acquired sample dataset is divided into a training set, a validation set, and a test set according to a preset ratio; wherein, the sample dataset contains five types of action samples: falling, bending over to pick up an object, sitting down, squatting, and walking normally, and the number of samples in each type meets the model training requirements. A hybrid structure of temporal convolutional network and bidirectional long short-term memory network is constructed. The temporal convolutional network contains multiple convolutional blocks, and the bidirectional long short-term memory network contains multiple hidden layers. Set the total loss function to be the sum of cross-entropy loss and the comparative loss with preset weights; Based on the training set, the Adam optimizer is used to train the model, the number of training iterations and the initial learning rate are set, and the learning rate is decayed at a preset interval. The model hyperparameters are adjusted using the validation set. Training stops when the validation set accuracy does not improve for a preset number of consecutive rounds, and the optimal model is saved as a fine-grained action recognition model. Specifically, in this implementation, a fine-grained action recognition model based on multi-dimensional feature fusion and comparative learning can be constructed to accurately distinguish between falls and easily confused normal actions. The specific process is as follows: Using the completed sequence of 17 skeletal key points as input, three types of core features are extracted to comprehensively depict the dynamic characteristics of human movement: 1. Basic coordinate features: The coordinates (x, y) of 17 key points in 30 frames are flattened to obtain a 30×17×2=1020-dimensional basic coordinate feature vector, which reflects the spatial position changes of the key points; 2. Dynamic characteristics: Joint angle change rate: Calculate the angle change of each joint (shoulder-elbow-wrist, hip-knee-ankle, etc.) between adjacent frames, divide by the frame interval time (default 33ms) to obtain the joint angle change rate (unit: ° / ms). The angle change rates of 12 joints are extracted to form a 30×12=360-dimensional feature vector. Changes in body center of gravity height: Based on the coordinates of the left and right hip key points (11,12) and the left and right shoulder key points (5,6), the weighted average method is used to calculate the body center of gravity coordinates (cx=(x5+x6+x11+x12) / 4, cy=(y5+y6+y11+y12) / 4). The center of gravity y-coordinate (height) sequence of 30 frames is extracted, and the change in center of gravity height and the rate of change of adjacent frames are calculated to form a 30×2=60-dimensional feature vector. Body tilt angle: With the center of gravity of the human body as the origin, construct a human coordinate system, calculate the angle (tilt angle) between the torso (shoulder-hip line) and the vertical direction, extract the tilt angle sequence of 30 frames, and form a 30-dimensional feature vector; 3. Temporal characteristics: Action duration: Count the number of frames during which the center of gravity drops (the number of frames during a fall is usually 5-10, while the number of frames during a bend is only 2-3). Curvature of key point motion trajectory: Curve fitting is performed on the 30-frame coordinate sequence of each key point to calculate the curvature change of the trajectory. The curvature of the key point trajectory of the falling action is usually greater than that of the normal action. Temporal feature vector: Integrates parameters such as action duration and trajectory curvature of each key point to form an 80-dimensional temporal feature vector; By concatenating the basic coordinate features, dynamic features, and temporal features, a multi-dimensional action feature vector of 1020+360+60+30+80=1550 dimensions is obtained and input into the pre-built fine-grained action recognition model.

[0041] Specifically, the construction process of the aforementioned fine-grained action recognition model may include: A hybrid structure of temporal convolutional network (TCN) and bidirectional long short-term memory network (Bi-LSTM) is adopted; The TCN layer can contain 3 convolutional blocks, each consisting of a 1×3 temporal convolutional layer, a batch normalization layer, and a ReLU activation function, used to extract the local temporal dependencies of feature vectors and capture short-term dynamic changes of actions. Bi-LSTM layers can contain two hidden layers (each with 256 hidden units) to capture long-term dependencies of feature vectors and uncover long-term trends in actions. The local features output by the TCN layer are concatenated with the long-term features output by the Bi-LSTM layer, and the feature dimension is reduced to 256 dimensions through a fully connected layer to obtain the final fused feature vector.

[0042] A Softmax classifier is constructed to classify actions into 5 categories: falling, bending over to pick up an object, sitting down, squatting, and walking normally, replacing the traditional binary classification mode and achieving fine-grained action recognition. Furthermore, a contrastive loss is introduced during model training to enhance the feature discrimination of easily confused actions: the feature distance between similar action sample pairs (such as two falling samples) is shortened, while the feature distance between dissimilar action sample pairs (such as falling and bending samples) is lengthened; the total loss function = cross-entropy loss (classification loss) + 0.3 × contrastive loss (feature discrimination loss), and the model parameters are optimized through joint training to improve the clarity of the classification boundary.

[0043] The dynamic threshold adjustment and output process can be as follows: The fall detection threshold is adaptively adjusted based on scene complexity, which is quantified by three parameters: ① mean trajectory quality score (average quality score of all valid trajectories); ② number of people (number of pedestrians detected in the current frame); ③ illumination intensity (average brightness value of the image). Scene complexity score = 0.4 × (1 - mean trajectory quality score) + 0.3 × normalized value of number of people + 0.3 × (1 - normalized value of illumination intensity), with a score range of 0-1. For low-complexity scenarios (score < 0.3): the decision threshold can be set to 0.6, and the threshold can be lowered to improve the recall rate; For medium-complexity scenarios (0.3 ≤ score < 0.7): the decision threshold can be set to 0.7 to balance recall and precision; For highly complex scenarios (score ≥ 0.7): the decision threshold can be set to 0.8 to increase the false positive rate.

[0044] The fused feature vector is input into a fine-grained classifier to obtain the predicted probability of each action. If the predicted probability of the "falling" category is greater than or equal to the dynamic threshold, the "falling" label and predicted probability are output; if the probability of the "falling" category is less than the dynamic threshold, the corresponding normal action category (such as "bending over to pick up an object" or "sitting down") and predicted probability are output, and it is judged as "normal".

[0045] This approach breaks through the traditional recognition model that relies solely on key point coordinates. It integrates multi-dimensional dynamic features such as joint angle change rate, center of gravity height change, and body tilt angle to comprehensively characterize the dynamic characteristics of movements. It constructs a fine-grained classifier to divide movements into 5 categories, accurately distinguishing between falling and easily confused movements such as bending over and sitting down. It also introduces contrastive learning training to increase the feature distance of easily confused movements, thereby improving the clarity of classification boundaries. Furthermore, it designs a dynamic threshold adjustment mechanism to adaptively adapt to scenarios of different complexities.

[0046] To further improve the reliability of the detection results, a cross-modal auxiliary verification module is added to support the collaborative verification of visual features and audio / environmental context features. The specific design can be as follows: The microphone linked to the camera collects the ambient audio signal. The sampling rate can be set to 16kHz and the quantization bit depth can be 16bit. The Mel frequency cepstral coefficient (MFCC) algorithm is used to extract features from the audio signal, resulting in a 13-dimensional MFCC feature vector. Combined with short-time energy and zero-crossing rate, a 30-dimensional audio feature vector is constructed. Train an audio classifier to identify three types of audio: "impact sound" (the sound produced when a person falls and collides with the ground), "call for help" (the sound of an elderly person calling for help after falling), and "environmental noise," and output the classification results and confidence scores. The sensor module collects environmental information of the monitoring area, including ground flatness (pre-defined as "flat", "with steps", and "with carpet") and obstacle distribution (pre-modeled and marked by the camera). Environmental parameters are quantified into numerical features (e.g., flat = 0, steps = 1, carpet = 2) to construct an environmental context feature vector. When the vision module outputs a "fall down" indicator, cross-modal verification is initiated: If audio data exists, the audio is classified as "impact sound" or "call for help" with a confidence level ≥ 0.7, and the verification is successful, confirming the "fall" event. If the audio is classified as "environmental noise", then the environmental context features are considered. If the environment is "with steps" or "with many obstacles", the "fall" is confirmed; otherwise, "suspected fall" is indicated. If no audio data is available, based on the contextual features of the environment, if the environment is characterized by "steps", "many obstacles" or "slippery ground", then "fall" is confirmed; otherwise, "suspected fall" is indicated. When the vision module outputs a "normal" label, if the audio module detects an "impact sound" or a "cry for help", a secondary vision detection is triggered (re-analyzing the most recent 5 frames of images) to avoid missed detections. Based on the cross-modal verification results, the final judgment result ("fall", "suspected fall", "normal") is output and pushed to the video management platform to trigger the corresponding early warning mechanism (such as sending a text message reminder to the guardian or activating the on-site sound and light alarm). This approach proposes a cross-modal collaborative verification scheme that integrates vision, audio, and environmental context. It identifies fall-related impact sounds and cries for help through audio features, and performs secondary verification of visual detection results by combining environmental context features (ground flatness, obstacle distribution), forming a multi-dimensional verification closed loop. This process can be flexibly adapted to different scenarios (with or without audio equipment), effectively reducing the false positive and false negative rates of single-modal detection, thereby improving the versatility and reliability of the solution.

[0047] In summary, this invention addresses the technical problems of existing personnel fall detection technologies, such as interference from complex environments including video blurring, sudden changes in lighting, and human occlusion; insufficient detection capability for small and overlapping targets; trajectory breakage and identity confusion issues in target tracking; high misjudgment rate for easily confused actions; and a lack of scene adaptation and multimodal collaboration capabilities. To address these issues, this invention proposes a personnel fall detection method based on human key points and temporal features. The overall framework flowchart is shown below. Figure 2As shown, video stream data of the area to be monitored is acquired, and data preprocessing is performed to optimize video quality and information integrity. Accurate detection of human targets is achieved using the YOLOv8 algorithm, and the detected targets are continuously tracked using the BoT-SORT algorithm. Valid tracking targets are filtered by judging whether the same target can be tracked for 30 consecutive frames. For targets that meet the judgment criteria, the YOLOv8-Pose algorithm is used to extract their skeletal key points. The key point information is then input into a key point-based fall action recognition algorithm (i.e., the fine-grained action recognition model mentioned above) for action feature analysis. Finally, a comprehensive judgment step outputs the detection result of a fall or normal movement. This invention effectively addresses the superimposed effects of complex environmental factors, ensuring performance even under harsh conditions. It acquires clear and complete human target information; adaptively adapts to pedestrian targets of different sizes, improves the recall and accuracy of pedestrian detection in distant small targets and multi-person overlapping scenes, enhances the stability and continuity of target tracking, and solves the problems of trajectory breakage and identity confusion caused by multi-target intersection and brief occlusion, thereby achieving continuous and stable tracking of pedestrian targets; at the same time, it repairs the missing key points caused by blurring or occlusion through an adaptive completion mechanism, generates a complete skeletal key point sequence that conforms to the laws of human movement, deeply mines the dynamic features of falling actions to accurately distinguish falling from normal actions such as bending over, sitting down, and squatting, reduces the misjudgment rate of easily confused actions, and can also dynamically adapt to home scenes of different complexities. It can optionally integrate cross-modal information such as audio and environmental context to further improve detection robustness.

[0048] Example 2: Based on the same inventive concept, this invention also provides a fall detection system for human bodies based on key points and temporal characteristics, as shown in the schematic diagram below. Figure 3 As shown, it includes: The data preprocessing module is used to acquire video stream data of the area to be monitored, and sequentially perform frame extraction, occlusion detection, generative completion and enhanced super-resolution processing on the video stream data to obtain a preprocessed high-definition video stream. The target detection module is used to perform personnel target detection on the preprocessed high-definition video stream using the dynamic anchor box adaptive YOLOv8 algorithm, and outputs the bounding box coordinates, category information and confidence score of the target personnel; The target tracking module is used to track the target person using an improved BoT-SORT algorithm based on the bounding box coordinates, category information, and confidence level of the target person, thereby obtaining the target person's movement trajectory and human target box data. The key point detection module is used to detect human key points of the target person based on the target person's movement trajectory and human target box data, using the YOLOv8-Pose algorithm, and adaptively completes low-confidence key points by combining preset human kinematic constraints to obtain the skeletal key point sequence of the target person. The fall detection module is used to extract basic coordinate features, dynamic features and time series features from the skeletal key point sequence and fuse them. The fused features are then input into a pre-built fine-grained action recognition model, and the fall detection results of the target person in the detection area are output. The fine-grained action recognition model is based on a hybrid feature fusion structure of temporal convolutional network TCN and bidirectional long short-term memory network Bi-LSTM, and is constructed through joint training with cross-entropy loss and contrastive loss.

[0049] In one implementation, the data preprocessing module may include: The frame extraction submodule is used to extract frames from the video stream data, and to perform preliminary denoising on the extracted image frames using a median filter to obtain denoised video stream data; to identify the highlight regions in the denoised video stream data through adaptive threshold segmentation; and to perform Gamma correction on the highlight regions to obtain initial processed data. The occlusion detection submodule is used to perform occlusion detection on the initial processed data using an occlusion detection network built based on the Transformer attention masking mechanism, so as to obtain the occluded region in the region to be monitored. The region completion submodule is used to complete the occluded region using a generative adversarial network that integrates prior knowledge of human pose, so as to obtain the completed image data. The generator of the generative adversarial network adopts a U-Net structure, and the discriminator adopts a convolutional neural network structure. Through adversarial training, the texture and brightness of the completed region are kept consistent with those of the non-occluded region. The super-resolution processing submodule is used to perform super-resolution processing on the completed image data using an enhanced super-resolution algorithm to obtain a pre-processed high-definition video stream.

[0050] In one implementation, the target detection module may include: The feature extraction submodule is used to extract multi-scale image features from the preprocessed high-definition video stream through the backbone network; wherein, the backbone network is built based on CSPDarknet, and a new context feature enhancement branch is added to extract global context features and fuse them with local features element by element; An anchor frame generation submodule is used to generate anchor frames adapted to the preprocessed high-definition video stream based on the multi-scale image features. The feature fusion submodule is used to perform complementary fusion of the multi-scale image features based on the anchor boxes through the Neck feature fusion network to obtain an enhanced multi-scale feature map; wherein, the Neck feature fusion network is constructed based on the fusion structure of the path aggregation network PAN and the feature pyramid network FPN; The feature detection submodule is used to detect the enhanced multi-scale feature map through the Head detection head, and output the bounding box coordinates, category information and confidence of the candidate targets in the region to be monitored; The target filtering submodule is used to filter the candidate targets using a non-maximum suppression algorithm based on the bounding box coordinates, category information, and confidence level of the candidate targets, and outputs the bounding box coordinates, category information, and confidence level of the target personnel.

[0051] In one implementation, the target tracking module may include: The trend prediction submodule is used to extract the location features, appearance features, and motion trend features of the target person based on the bounding box coordinates, category information, and confidence level of the target person; the motion trend features include x-direction velocity, y-direction velocity, rate of change of velocity, and motion direction angle; The metric matrix construction submodule is used to construct an IoU metric matrix based on the location characteristics of the target personnel. The appearance matrix construction submodule is used to construct an appearance feature similarity matrix based on the appearance features of the target person. The time matrix construction submodule is used to construct a time correlation matrix based on the movement trend characteristics of the target personnel; The matrix fusion submodule is used to fuse the IoU metric matrix, appearance feature similarity matrix and temporal correlation matrix through a spatiotemporal attention mechanism to obtain a fused feature matrix; The trajectory association submodule is used to associate the current frame detection box of the target person with the existing trajectory based on the fused feature matrix and using the Hungarian algorithm. The trajectory evaluation submodule is used to calculate the trajectory quality score based on the associated trajectory, combined with the mean confidence level and occlusion rate of the target person, and to filter high-quality trajectories whose trajectory quality score is not lower than a preset threshold. The trajectory output submodule is used to output the movement trajectory of the target person in consecutive frames and the human target bounding box data of the corresponding frames based on the high-quality trajectory.

[0052] For example, the key points of the human body can be the core skeletal joints of the human body, including the tip of the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. The human kinematic constraints include one or more of the following: joint angle constraints, joint length constraints, and motion continuity constraints.

[0053] In one implementation, the fall recognition module may include: The joint coordinate generation submodule is used to generate bone joint coordinates based on the bone key point sequence. The flattening submodule is used to flatten the coordinates of the skeletal joint points to obtain the basic coordinate feature vector. The feature vector generation submodule is used to extract dynamic features from the coordinates of the skeletal joints and form motion feature vectors; wherein, the dynamic features include the angle change rate of multiple joints, the change of the height of the human body's center of gravity, and the body tilt angle; The time vector generation submodule is used to extract temporal features from the coordinates of the skeletal joints and form a time feature vector; wherein, the temporal features include the duration of the action and the curvature of the motion trajectory of each key point; The vector splicing submodule is used to splice and fuse the basic coordinate feature vector, motion feature vector, and time feature vector.

[0054] In one implementation, the system may further include: a model building module, which may specifically include: The dataset partitioning submodule is used to divide the pre-acquired sample dataset into training set, validation set and test set according to a preset ratio; wherein, the sample dataset contains five types of action samples: falling, bending over to pick up objects, sitting down, squatting and normal walking, and the number of samples in each type meets the model training requirements. The architecture building submodule is used to build a hybrid structure of temporal convolutional network and bidirectional long short-term memory network. The temporal convolutional network contains multiple convolutional blocks, and the bidirectional long short-term memory network contains multiple hidden layers. The loss setting submodule is used to set the total loss function to the sum of cross-entropy loss and the comparison loss with preset weights; The parameter setting submodule is used to train the model using the Adam optimizer based on the training set, set the number of training iterations and the initial learning rate, and decay the learning rate at preset intervals. The model validation submodule is used to adjust the model hyperparameters using the validation set. When the accuracy of the validation set does not improve for a preset number of consecutive rounds, training is stopped and the optimal model is saved as a fine-grained action recognition model.

[0055] Example 3: like Figure 4As shown, the present invention also provides an electronic device, which may be a computer device, a microcontroller device, a smart mobile device, etc. The electronic device in this embodiment may include a processor, a memory, a transceiver component, etc. The memory, processor, and transceiver component are connected via a bus; the memory can be used to store executable programs, and an exemplary executable program may include instructions; the processor is used to execute the instructions stored in the memory. The memory can also be used to store data, which can be accessed and / or modified when instructions are executed.

[0056] The processor may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, and it is suitable for implementing one or more instructions. Specifically, it is suitable for loading and executing one or more instructions in the storage medium to realize the corresponding method flow or corresponding function, so as to realize the steps of the personnel fall detection method based on human body key points and temporal characteristics in the above embodiments.

[0057] Example 4: Based on the same inventive concept, this invention also provides a readable storage medium, specifically an electronic device readable storage medium (Memory). This readable storage medium is a memory device within an electronic device used to store programs and data. It is understood that the storage medium here can include both built-in storage media within the electronic device and extended storage media supported by the electronic device. The storage medium provides storage space, which stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more executable programs (including program code). It should be noted that the storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. Loading and executing one or more instructions stored in the storage medium by the processor can implement the steps of the personnel fall detection method based on human key points and temporal characteristics in the above embodiments.

[0058] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0059] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0060] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0061] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading the present invention, they can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, but these changes, modifications or equivalent substitutions are all within the scope of protection of the claims pending approval.

Claims

1. A method for detecting falls by human body based on key points and temporal features, characterized in that, include: The video stream data of the area to be monitored is acquired, and the video stream data is sequentially subjected to frame extraction, occlusion detection, generative completion and enhanced super-resolution processing to obtain a preprocessed high-definition video stream; The dynamic anchor box adaptive YOLOv8 algorithm is used to detect people in the preprocessed high-definition video stream, and the bounding box coordinates, category information and confidence scores of the target people are output. Based on the bounding box coordinates, category information, and confidence level of the target person, an improved BoT-SORT algorithm is used to track the target person and obtain the target person's movement trajectory and human target box data. Based on the target person's movement trajectory and human target bounding box data, the YOLOv8-Pose algorithm is used to detect human key points of the target person, and combined with preset human kinematic constraints, low-confidence key points are adaptively completed to obtain the skeletal key point sequence of the target person. Basic coordinate features, dynamic features, and temporal features are extracted from the skeletal key point sequence and fused together. The fused features are then input into a pre-constructed fine-grained action recognition model, which outputs the fall detection results of the target person in the detection area. The fine-grained action recognition model is based on a hybrid feature fusion structure of temporal convolutional network TCN and bidirectional long short-term memory network Bi-LSTM, and is constructed through joint training with cross-entropy loss and contrastive loss.

2. The method as described in claim 1, characterized in that, The process of sequentially performing frame extraction, occlusion detection, generative completion, and enhanced super-resolution processing on the video stream data to obtain a preprocessed high-definition video stream includes: The video stream data is frame extracted, and the extracted image frames are initially denoised using a median filter to obtain denoised video stream data; the highlight regions in the denoised video stream data are identified by adaptive threshold segmentation; and the highlight regions are subjected to Gamma correction to obtain initial processed data. The occlusion detection network based on the Transformer attention masking mechanism is used to perform occlusion detection on the initial processed data to obtain the occluded region in the region to be monitored. The occluded region is filled in using a generative adversarial network that incorporates prior knowledge of human pose, resulting in filled image data. The generator of the generative adversarial network adopts a U-Net structure, and the discriminator adopts a convolutional neural network structure. Through adversarial training, the texture and brightness of the filled region are kept consistent with those of the unoccluded region. The completed image data is processed using an enhanced super-resolution algorithm to obtain a pre-processed high-definition video stream.

3. The method as described in claim 1, characterized in that, The method employs a dynamic anchor box adaptive YOLOv8 algorithm to detect people in the preprocessed high-definition video stream, outputting the bounding box coordinates, category information, and confidence score of the target people, including: Multi-scale image features are extracted from the preprocessed high-definition video stream through a backbone network; wherein, the backbone network is constructed based on CSPDarknet, with a new context feature enhancement branch added to extract global context features and fuse them with local features element by element; Based on the multi-scale image features, an anchor frame adapted to the preprocessed high-definition video stream is generated; Based on the anchor frame, the multi-scale image features are complementaryly fused through the Neck feature fusion network to obtain an enhanced multi-scale feature map; wherein, the Neck feature fusion network is constructed based on the fusion structure of the path aggregation network PAN and the feature pyramid network FPN; The enhanced multi-scale feature map is detected by the Head detection head, and the bounding box coordinates, category information and confidence scores of candidate targets in the monitored area are output. Based on the bounding box coordinates, category information, and confidence level of the candidate targets, a non-maximum suppression algorithm is used to filter the candidate targets, and the bounding box coordinates, category information, and confidence level of the target personnel are output.

4. The method as described in claim 1, characterized in that, The improved BoT-SORT algorithm is used to track the target person based on the bounding box coordinates, category information, and confidence level, obtaining the target person's movement trajectory and human target box data, including: Based on the bounding box coordinates, category information, and confidence level of the target person, the position features, appearance features, and motion trend features of the target person are extracted; the motion trend features include x-direction velocity, y-direction velocity, rate of change of velocity, and motion direction angle; Based on the location characteristics of the target personnel, construct an IoU metric matrix; Based on the appearance characteristics of the target person, construct an appearance feature similarity matrix; Based on the movement trend characteristics of the target personnel, a time correlation matrix is ​​constructed; The IoU metric matrix, appearance feature similarity matrix, and temporal correlation matrix are fused using a spatiotemporal attention mechanism to obtain a fused feature matrix; Based on the fused feature matrix, the Hungarian algorithm is used to associate the current frame detection box of the target person with the existing trajectory; Based on the associated trajectory, combined with the mean confidence level and occlusion rate of the target person, a trajectory quality score is calculated, and high-quality trajectories with a trajectory quality score not lower than a preset threshold are selected. Based on the high-quality trajectory, output the movement trajectory of the target person in consecutive frames and the human target bounding box data of the corresponding frames.

5. The method as described in claim 1, characterized in that, The key points of the human body are the core skeletal joints of the human body, including the tip of the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. The human kinematic constraints include one or more of the following: joint angle constraints, joint length constraints, and motion continuity constraints.

6. The method as described in claim 1, characterized in that, The process of extracting basic coordinate features, dynamic features, and temporal features from the skeletal keypoint sequence and fusing them includes: Generate the coordinates of the bone joints based on the sequence of bone key points; The coordinates of the skeletal joints are flattened to obtain the basic coordinate feature vector; Dynamic features are extracted from the coordinates of the skeletal joints to form a motion feature vector; wherein, the dynamic features include the angle change rate of multiple joints, the change in the height of the human body's center of gravity, and the body tilt angle; Temporal features are extracted from the coordinates of the skeletal joints to form a temporal feature vector; wherein, the temporal features include the duration of the action and the curvature of the motion trajectory of each key point; The basic coordinate feature vector, motion feature vector, and time feature vector are concatenated and fused.

7. The method as described in claim 1, characterized in that, The construction process of the fine-grained action recognition model includes: The pre-acquired sample dataset is divided into a training set, a validation set, and a test set according to a preset ratio; wherein, the sample dataset contains five types of action samples: falling, bending over to pick up an object, sitting down, squatting, and walking normally, and the number of samples in each type meets the model training requirements. A hybrid structure of temporal convolutional network and bidirectional long short-term memory network is constructed. The temporal convolutional network contains multiple convolutional blocks, and the bidirectional long short-term memory network contains multiple hidden layers. Set the total loss function to be the sum of cross-entropy loss and the comparative loss with preset weights; Based on the training set, the Adam optimizer is used to train the model, the number of training iterations and the initial learning rate are set, and the learning rate is decayed at a preset interval. The model hyperparameters are adjusted using the validation set. Training stops when the validation set accuracy does not improve for a preset number of consecutive rounds, and the optimal model is saved as a fine-grained action recognition model.

8. A fall detection system for individuals based on key points and temporal characteristics of the human body, characterized in that, include: The data preprocessing module is used to acquire video stream data of the area to be monitored, and sequentially perform frame extraction, occlusion detection, generative completion and enhanced super-resolution processing on the video stream data to obtain a preprocessed high-definition video stream. The target detection module is used to perform personnel target detection on the preprocessed high-definition video stream using the dynamic anchor box adaptive YOLOv8 algorithm, and outputs the bounding box coordinates, category information and confidence score of the target personnel; The target tracking module is used to track the target person using an improved BoT-SORT algorithm based on the bounding box coordinates, category information, and confidence level of the target person, thereby obtaining the target person's movement trajectory and human target box data. The key point detection module is used to detect human key points of the target person based on the target person's movement trajectory and human target box data, using the YOLOv8-Pose algorithm, and adaptively completes low-confidence key points by combining preset human kinematic constraints to obtain the skeletal key point sequence of the target person. The fall detection module is used to extract basic coordinate features, dynamic features and time series features from the skeletal key point sequence and fuse them. The fused features are then input into a pre-built fine-grained action recognition model, and the fall detection results of the target person in the detection area are output. The fine-grained action recognition model is based on a hybrid feature fusion structure of temporal convolutional network TCN and bidirectional long short-term memory network Bi-LSTM, and is constructed through joint training with cross-entropy loss and contrastive loss.

9. The system as described in claim 8, characterized in that, The data preprocessing module includes: The frame extraction submodule is used to extract frames from the video stream data, and to perform preliminary denoising on the extracted image frames using a median filter to obtain denoised video stream data; to identify the highlight regions in the denoised video stream data through adaptive threshold segmentation; and to perform Gamma correction on the highlight regions to obtain initial processed data. The occlusion detection submodule is used to perform occlusion detection on the initial processed data using an occlusion detection network built based on the Transformer attention masking mechanism, so as to obtain the occluded region in the region to be monitored. The region completion submodule is used to complete the occluded region using a generative adversarial network that integrates prior knowledge of human pose, so as to obtain the completed image data. The generator of the generative adversarial network adopts a U-Net structure, and the discriminator adopts a convolutional neural network structure. Through adversarial training, the texture and brightness of the completed region are kept consistent with those of the non-occluded region. The super-resolution processing submodule is used to perform super-resolution processing on the completed image data using an enhanced super-resolution algorithm to obtain a pre-processed high-definition video stream.

10. The system as described in claim 8, characterized in that, The target detection module includes: The feature extraction submodule is used to extract multi-scale image features from the preprocessed high-definition video stream through the backbone network; wherein, the backbone network is built based on CSPDarknet, and a new context feature enhancement branch is added to extract global context features and fuse them with local features element by element; An anchor frame generation submodule is used to generate anchor frames adapted to the preprocessed high-definition video stream based on the multi-scale image features. The feature fusion submodule is used to perform complementary fusion of the multi-scale image features based on the anchor boxes through the Neck feature fusion network to obtain an enhanced multi-scale feature map; wherein, the Neck feature fusion network is constructed based on the fusion structure of the path aggregation network PAN and the feature pyramid network FPN; The feature detection submodule is used to detect the enhanced multi-scale feature map through the Head detection head, and output the bounding box coordinates, category information and confidence of the candidate targets in the region to be monitored; The target filtering submodule is used to filter the candidate targets using a non-maximum suppression algorithm based on the bounding box coordinates, category information, and confidence level of the candidate targets, and outputs the bounding box coordinates, category information, and confidence level of the target personnel.

Images