Pedestrian tumble detection method based on improved YOLOv8 model and space-time diagram convolution

Abstract

The invention discloses a pedestrian tumble detection method based on an improved YOLOv8 model and space-time diagram convolution. The method comprises the following steps: preprocessing an input video; a target frame and joint point coordinates are predicted by improving YOLOv8n-pose; by using a Byterack algorithm, assigning numbers to different figures by using the coordinates of the detection frame; constructing a data sequence containing target frame information and joint point coordinates according to the figure numbers; performing behavior recognition by improving a space-time diagram convolutional network; the steering engine is controlled to adjust the angle of the camera, and tracking of the falling person is achieved; according to the improved YOLOv8n-pose joint point detection model, local and global features are dynamically fused through learnable parameters, and the joint point detection precision is improved. The improved space-time diagram convolutional network learns difference and similarity topology of joint points at the same time, and the space feature extraction capability is enhanced. According to the improved space-time diagram convolutional network, time features are extracted on multiple time scales, motion features in important time scales are enhanced, and the accuracy of behavior recognition is improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent monitoring technology, and in particular to a pedestrian fall detection method based on an improved YOLOv8 model and spatiotemporal graph convolution. Background Technology

[0002] With the accelerating aging of the population, the health and safety of the elderly are becoming increasingly prominent issues. Data from the Seventh National Population Census shows that my country's population aged 60 and above has reached 264 million. Falls, a common accident among the elderly, can cause serious physical and psychological harm, and even threaten their lives. Therefore, developing a system capable of real-time and accurate detection of falls among the elderly is of significant practical importance.

[0003] Existing fall detection technologies mainly include sensor-based fall detection schemes and computer vision-based fall detection schemes. Sensor-based fall detection schemes mainly rely on wearable sensors and environmental sensors, which have problems such as limited application scenarios and uncomfortable wearing. Computer vision-based fall detection schemes have been widely used because they do not require the deployment of a large number of sensors and are suitable for monitoring in various places. Among them, the key point-based fall detection method has attracted attention because of its advantages such as insensitivity to changes in lighting and good privacy protection. However, existing solutions usually face the following problems: (1) The key point detection network has a large computational load and insufficient inference speed on embedded devices, making it difficult to meet the requirements of real-time monitoring; (2) Traditional spatiotemporal library convolutional networks rely on predefined fixed adjacency matrices, which consume a lot of computational resources; (3) In multi-person scenarios, there is a lack of stable target tracking and key point sequence management mechanisms, which can easily lead to confusion of action information of different people. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the purpose of this invention is to provide a pedestrian fall detection method based on an improved YOLOv8 model and spatiotemporal graph convolution. This invention aims to solve the problems of existing fall detection technologies, such as the large computational load of key point detection networks, insufficient inference speed on embedded devices, difficulty in meeting real-time monitoring requirements, and the reliance of traditional spatiotemporal graph convolutional networks on predefined fixed adjacency matrices, which leads to high computational resource consumption. Furthermore, in multi-person scenarios, there is a lack of stable target tracking and key point sequence management mechanisms, which can easily lead to confusion of action information of different people.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, a pedestrian fall detection method based on an improved YOLOv8 model and spatiotemporal graph convolution includes: Step 1: Use a camera to acquire detection video and preprocess the detection video; Step 2: Based on the MobileNetV4 lightweight feature extraction network and deep convolution, build an improved YOLOv8 model, input the detected video, and obtain the target box and key point coordinates; Step 3: Using the Bytetrack method, assign IDs to different people based on the bounding box coordinates, construct a data sequence containing target bounding box information and key point coordinates based on the person IDs, and continuously update it; Step 4: Establish a behavior recognition model based on a spatiotemporal graph convolutional network, input the data sequence, obtain the behavior categories of different people, and adjust the camera angle according to the behavior category to realize pedestrian fall detection.

[0006] Furthermore, in step 2, the improved YOLOv8 model input to the detection video is converted into a feature map. The MobileNetV4 lightweight feature extraction network of the improved YOLOv8 model is used to extract features from the input feature map. Then, the bidirectional feature fusion network of YOLOv8n-pose is used to effectively integrate feature information at different levels to obtain a multi-scale feature map. The target box and keypoint are predicted on the multi-scale feature map to obtain the coordinates of the target box and keypoint.

[0007] Furthermore, feature extraction from the input feature map is performed using the MobileNetV4 lightweight feature extraction network with an improved YOLOv8 model, including: Employing efficient spatial channel decoupled convolutions to input feature maps Extract features and use pointwise convolution to transform the input feature map. The number of channels is doubled, and then spatial features are extracted using depthwise convolution to obtain the feature map. Then use pointwise convolution to transform the feature map The number of channels is doubled, and then grouped convolution is used to combine the feature maps. The number of channels is grouped according to the number of channels in the output feature map, and an independent convolutional kernel is used to extract features within each group to obtain the feature map. ; feature map With feature map The feature map is obtained by adding elements one by one. The UIB module in the MobileNetV4 lightweight feature extraction network was used to process the feature maps. Further feature extraction is performed to obtain feature maps. Use the SPPF module to process feature maps. Preliminary extraction of global features yields feature maps. .

[0008] Furthermore, a bidirectional feature fusion network based on YOLOv8n-pose is used to effectively integrate feature information from different levels, resulting in multi-scale feature maps including: Using a top-down feature fusion path to process feature maps , , The information is upsampled sequentially and passed up layer by layer, as follows: ; In the formula Representing the Layer feature map ( i =4,3), Upsample indicates an upsampling operation. Then, the improved C2f module is used to extract the feature map. ( i In (=4, 3), more feature information is obtained, occlusion joints are detected, and feature maps are generated. After After convolution, the feature maps are divided into two equal parts based on the number of channels and stored in a list. The last feature map in the list is input into the Bottleneck part of the C2f module, and the result is added to the list. Finally, the feature maps in the list are concatenated and then processed... The convolution yields the feature map. ( i =4,3); where the last element of the list is first processed by convolution in the Bottleneck part, and then channel information, spatial information, local information and global information are fused using the improved MLCA. Utilizing a bottom-up feature fusion path, the feature map is processed. The aggregation operation, which passes information layer by layer, is represented as follows: ; In the formula Representing the Layer feature map ( i =4,5), , Conv represents the convolution operation; Using the improved C2f module from the feature map ( i In (=4, 5), more feature information is obtained, occlusion joints are detected, and feature maps are generated. After After convolution, the feature maps are divided into two equal parts based on the number of channels and stored in a list. The last feature map in the list is input into the Bottleneck part of the C2f module, and the result is added to the list. Finally, the feature maps in the list are concatenated and then processed... Convolution yields multi-scale feature maps ( i =4,5), .

[0009] Furthermore, in step 3, the Bytetrack method is used. When a target first appears, it is assigned a number. Kalman filtering is used to predict the target's position at the next moment based on the target's current coordinates and velocity. The overlap between the predicted position and the actual position is calculated. If the overlap is greater than a threshold, it is considered to be the same target. IDs are assigned to different people using the bounding box coordinates. A data sequence containing target bounding box information and key point coordinates is constructed based on the person ID. The existing data sequence is matched based on the person ID. If there is no data sequence with the corresponding number in the current data sequence, a new data sequence is created to store the key point information of the person. When the person number disappears, the corresponding data sequence is cleared to save memory space. The process is continuously updated.

[0010] Furthermore, in step 4, a behavior recognition model is established based on a spatiotemporal graph convolutional network. The data sequence is input to obtain a hybrid adjacency matrix containing both keypoint difference features and keypoint similarity features. This matrix is ​​then compared with the input features. The Einstein summation convolution operation is performed to extract spatial features and output a spatial feature map. Multi-branch temporal convolution is then performed on the spatial feature map to extract temporal features and output a spatiotemporal feature map. Temporal attention is used to assign weights to each time period of the spatiotemporal feature map to enhance the feature representation ability of the spatiotemporal feature map within important time periods. Finally, the behavior category is output through a fully connected layer to obtain the behavior categories of different people.

[0011] Furthermore, a hybrid adjacency matrix containing joint difference features and joint similarity features is obtained, and then compared with the input features. Perform Einstein summation to complete graph convolution operation, extract spatial features, and output spatial feature maps including: Two independent pointwise convolution operations are used to reduce the number of channels in the input feature map, and then a global average pooling operation is performed on the feature map in the time dimension to obtain two dimensions ( N , C e , V Feature map and Through feature maps and Calculate the feature differences between all relevant node pairs, normalize the difference values ​​to the interval [-1, 1], and obtain the adjacency matrix containing the feature differences of relevant nodes. , represented as: ; In the formula, tanh is the tanh activation function; Two independent point convolution operations are used to reduce the number of channels in the input feature map, and then a reshaping operation is used to fuse the channel and temporal dimension information to obtain the result with dimension (). Feature map And the dimension is ( Feature map Through feature maps and Calculate the similarity between all pairs of related nodes and normalize them to the interval [-1, 1] to obtain the adjacency matrix containing the similarity features of related nodes. , represented as: In the formula, tanh is the tanh activation function; A learnable weighted fusion strategy is used to fuse the adjacency matrix with a predefined adjacency matrix to obtain a hybrid adjacency matrix, which is represented as: ; In the formula, and These are learnable parameters, whose values ​​are obtained through model training. The adjacency matrix is ​​predefined. For hybrid adjacency matrix Input features Perform Einstein summation to implement graph convolution operation and obtain output features. .

[0012] Furthermore, a multi-branch temporal convolution operation is performed on the spatial feature map to extract temporal features, and the output spatiotemporal feature map includes: Features Using point convolution operations to reduce the number of channels, we obtain , , , , , Six feature maps, for feature maps , , , Perform temporal convolution operations with receptive fields of 3, 5, 7, and 9 to extract keypoint change features across different time spans, and then process the feature maps. Without performing any other operations, it provides baseline information for subsequent feature fusion without time smoothing, and modulates the feature map. Max pooling is performed to extract salient features at key time points. Finally, the results from the six branches are concatenated along the channel dimension to obtain the feature map. Adjusting feature maps using point convolution The number of channels is used to achieve feature interaction between channels and obtain feature maps. .

[0013] Furthermore, using temporal attention to assign weights to each time period of the spatiotemporal feature map enhances the feature representation ability of the spatiotemporal feature map within important time periods, including: Adaptive pooling is used to refine the spatiotemporal feature map along the key dimension. Perform global aggregation, reducing the feature dimension from ( N , C , T , V Compress to ( N , C , T 1) Eliminate spatial redundancy while preserving complete temporal features. Focus on features in the temporal dimension by dividing the input spatiotemporal feature map into several non-overlapping time groups, each containing... g For each consecutive frame, a convolutional kernel size of ( g 1) Step size is ( g , 1) Convolution extracts within each time group g The temporal features of the frames are analyzed, and the number of channels is compressed to extract more important feature information. Then, point convolution is used to restore the number of channels of the feature map, resulting in a temporal weight matrix. A repeated interpolation strategy is used, where the weights of each time group are repeated along the time dimension. g This yields a time weight matrix with matching dimensions. and feature map Weights are assigned through element-wise multiplication to obtain the feature map. .

[0014] Furthermore, in step 4, adjusting the camera angle based on the behavior category to achieve pedestrian fall detection includes: Extract the detection bounding box information of the person who fell based on their ID number, and obtain the center coordinates of the person. , represented as: ; In the formula, , These are the x-coordinates of the top-left and bottom-right corners of the detection box, respectively. , These are the ordinate values ​​of the top left and bottom right corners of the detection box, respectively; When the character's center coordinates are not in the middle area of ​​the video, the angle that the servo needs to rotate is represented as: ; In the formula, The x-coordinate of the video center point The vertical coordinate of the video center point; The duty cycle of the PWM, obtained based on the required rotation angle of the servo motor, is expressed as follows: ; The duty cycle is sent to the servo drive module via the I2C bus to generate a PWM control signal with the corresponding duty cycle, which controls the servo to rotate and adjust the camera angle, thus enabling real-time tracking of a fallen person.

[0015] Secondly, a pedestrian fall detection system based on an improved YOLOv8 model and spatiotemporal graph convolution includes: Nvidia's Jetson Orin NX edge computing device was selected as the platform for the software system. Two DS3218 servos were used to control the horizontal and vertical viewing angles of the camera, and a PCA9685 servo drive module was used as the servo driver. The hardware modules were connected, and the camera input a video stream to the edge computing device via USB communication protocol. The edge computing device sent control commands to the PCA9685 via I2C protocol to drive the servos to adjust the camera angle, thus achieving tracking of a fallen person.

[0016] The technical solution adopted in this invention has the following beneficial effects: The improved YOLOv8n-pose joint detection model of this invention has lower computational cost. Considering the difficulty of detecting human joints in complex backgrounds, the improved YOLOv8n-pose joint detection model improves joint detection accuracy by dynamically fusing local and global features through learnable parameters. Considering the complexity of joint topology during motion, the improved spatiotemporal graph convolutional network of this invention simultaneously learns the difference and similarity topologies of joints, enhancing spatial feature extraction capabilities. Considering the importance of temporal features of behavior, the improved spatiotemporal graph convolutional network of this invention extracts temporal features at multiple time scales and enhances motion features within important time scales, improving the accuracy of behavior recognition. Attached Figure Description

[0017] Figure 1 This is the hardware design diagram of the present invention; Figure 2 This is a flowchart of the present invention; Figure 3 This is a structural diagram of the joint detection model of the present invention; Figure 4 It is a diagram of an efficient spatial channel decoupled convolution structure; Figure 5 It is an improved C2f_MLCA ​​structure diagram; Figure 6 This is the MLCA structure diagram. Figure 7 This is a diagram showing the results of joint detection; Figure 8This is a diagram of the improved spatiotemporal graph convolutional network structure of the present invention; Figure 9 It is a hybrid adaptive graph convolutional structure graph; Figure 10 This is a diagram of a multi-branch temporal convolution structure; Figure 11 This is a structural diagram of the duration-based attention mechanism; Figure 12 This is a diagram illustrating the effectiveness of pedestrian detection and tracking systems. Detailed Implementation

[0018] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] This invention provides a pedestrian fall detection method based on an improved YOLOv8 model and spatiotemporal graph convolution. The detection process of this method is as follows: Figure 2 As shown, it includes: Step 1: Acquire detection video using a camera and preprocess the video; input the camera's video stream to the Jetson Orin NX edge computing device using the USB communication protocol to acquire the detection video, preprocess the video, and use OpenCV to adjust the video resolution to the desired size. .

[0020] In this embodiment, step 2 involves establishing an improved YOLOv8 model based on the MobileNetV4 lightweight feature extraction network and deep convolutions. The detected video input is used to obtain the bounding boxes and keypoint coordinates. The structure diagram of the improved YOLOv8n-pose model is shown below. Figure 3 As shown, the improved YOLOv8 model is based on YOLOv8n-pose, replacing the feature extraction network with the lightweight MobileNetV4 feature extraction network to reduce the model's computational cost. An efficient spatial-channel decoupled convolution (ESC) is designed and replaces the standard convolutions in the MobileNetV4 lightweight feature extraction network and the YOLOv8n-pose feature fusion network, further reducing computational cost while enhancing feature extraction capabilities. A depthwise convolution is introduced into the dual branches of Mixed Local Channel Attention (MLCA) to improve its feature extraction capabilities, and the improved MLCA attention is embedded into the C2f module of the neck region to enhance the accuracy of joint detection.

[0021] Step 2 specifically includes: Step 2.1: Extract features from the input feature map using the improved MobileNetV4 lightweight feature extraction network, including the following steps 2.1.1~2.1.3: Step 2.1.1: Use the efficient spatial channel decoupled convolution proposed in this invention to pair the input feature map. Feature extraction, efficient spatial channel decoupling convolutional structure diagram as shown in the figure. Figure 4 As shown, the steps include the following: 2.1.1.1 to 2.1.1.3: Step 2.1.1.1: Use pointwise convolution to transform the input feature map The number of channels is doubled to provide a rich feature set for subsequent operations. Then, spatial features are extracted using depthwise convolution to obtain the feature map. .

[0022] Step 2.1.1.2: Use pointwise convolution to transform the feature map. The number of channels is doubled, and then group convolution is used to combine the feature maps. The number of channels is grouped according to the number of channels in the output feature map, and an independent convolutional kernel is used to extract features within each group to obtain the feature map. .

[0023] Step 2.1.1.3: Transfer the feature map With feature map The feature map is obtained by adding elements one by one. .

[0024] Step 2.1.2: Use the UIB module in the MobileNetV4 lightweight feature extraction network to process the feature map. Further feature extraction is performed to obtain feature maps. .

[0025] Step 2.1.3: Use the SPPF module to process the feature map Preliminary extraction of global features yields feature maps. .

[0026] Step 2.2: Use the YOLOv8n-pose bidirectional feature fusion network to effectively integrate feature information from different levels, obtaining three feature maps of different sizes. This includes steps 2.2.1 to 2.2.4: Step 2.2.1: Apply a top-down feature fusion path to the feature map obtained in Step 2.1. , , The information is upsampled layer by layer and passed up to the next layer in sequence. The formula is as follows: In the formula Representing the Layer feature map ( i =4,3), Upsample indicates an upsampling operation.

[0027] Step 2.2.2: Use the improved C2f module to extract features from the feature map ( i To obtain more feature information from (=4, 3), occlusion joints are detected. The structure diagram is as follows: Figure 5 As shown, feature map After After convolution, the feature maps are divided into two equal parts based on the number of channels and stored in a list. The last feature map in the list is input into the Bottleneck part of the C2f module, and the result is added to the list. Finally, the feature maps in the list are concatenated and then processed... The convolution yields the feature map. ( i =4, 3). The last element of the list undergoes convolution in the Bottleneck region, and then improved MLCA fuses channel, spatial, local, and global information to enhance joint detection accuracy. The improved MLCA structure diagram is shown below. Figure 6 The design includes the following steps 2.2.2.1 to 2.2.2.5: Step 2.2.2.1: Apply depthwise convolution to the input feature map to extract spatial feature information and obtain the feature map. .

[0028] Step 2.2.2.2: Then, feature maps are extracted using GAP and LAP respectively. The global and local features are used to obtain the feature map. and .

[0029] Step 2.2.2.3: Process the feature map and One-dimensional convolution is performed along the channel dimension to fuse information from different feature channels, resulting in a feature map. and .

[0030] Step 2.2.2.4: Transfer the feature map and Attention weights are obtained through fusion The formula is as follows: In the formula and For learnable branch fusion parameters, and with Constraints prevent the weight of one branch from increasing indefinitely and completely ignoring information from another branch.

[0031] Step 2.2.2.5: Attention weights from step 2.2.2.4 Dynamically adjust the input feature map A new feature map is obtained. And add it to the list in the C2f module.

[0032] Step 2.2.3: Utilize a bottom-up feature fusion path to process the feature map obtained in Step 2.2.2. The aggregation operation passes information layer by layer. The formula is as follows: In the formula Representing the Layer feature map ( i =4,5), , Conv represents the convolution operation.

[0033] Step 2.2.4: Use the improved C2f module to extract features from the feature map ( i To obtain more feature information from points (=4, 5), we can detect occluded joints. Feature map After After convolution, the feature maps are divided into two equal parts based on the number of channels and stored in a list. The last feature map in the list is input into the Bottleneck part of the C2f module, and the result is added to the list. Finally, the feature maps in the list are concatenated and then processed... The convolution yields the feature map. ( i =4,5), .

[0034] Step 2.3: Process the feature map obtained in Step 2.2 Perform bounding box and keypoint prediction, and output the bounding box and keypoint coordinates. The result is shown in the figure below. Figure 7 As shown.

[0035] To demonstrate the advantages of this invention in the keypoint detection task, keypoint detection was performed on the COCO2017 KeyPoints test set and compared with the original network. The detection results are shown in Table 1. While reducing GFLOPS by 1.4, P(%) increased by 1.8% and AP@50(%) increased by 1.7%. Therefore, the improved keypoint detection network of this invention is suitable for running on edge computing devices.

[0036] Table 1 Comparison of Joint Detection In this embodiment, step 3, using the Bytetrack method, assigns IDs to different individuals based on the bounding box coordinates, constructs a data sequence containing target bounding box information and keypoint coordinates based on the individual IDs, and continuously updates it; specifically including: Step 3.1: Using the Bytetrack algorithm, assign IDs to different individuals based on the bounding box coordinates. When a target appears for the first time, assign it a number. Use Kalman filtering to predict its position at the next moment based on its current coordinates and velocity. Calculate the overlap between the predicted and actual positions; if the overlap is greater than a threshold, they are considered the same target.

[0037] Step 3.2: Construct a data sequence containing target bounding box information and keypoint coordinates based on the character ID. It is continuously updated. Based on the character's ID, it matches the existing data sequence. If there is no corresponding data sequence in the current data sequence, a new data sequence is created to store the character's key information. When the character's ID disappears, the corresponding data sequence is also cleared to save memory space.

[0038] In this embodiment, step 4 involves establishing a behavior recognition model based on a spatiotemporal graph convolutional network. The data sequence is input to obtain the behavior categories of different individuals, and the camera angle is adjusted according to the behavior category to achieve pedestrian fall detection. Specifically, the behavior recognition model is based on the spatiotemporal graph convolutional network, with a Mixed Adaptive Graph Convolution (MAGCN) module designed to replace the spatiotemporal graph convolutional module in the network, improving the network's feature extraction capability in the spatial dimension. Multi-branch Temporal Convolution (MTBN) is used to replace the temporal convolution in the network, improving the network's temporal feature extraction capability. A Temporal Length Attention (TLA) mechanism is designed and added to layers 5 and 8 of the spatiotemporal graph convolutional network to enhance the network's feature extraction capability in local time intervals.

[0039] When the data sequence reaches 100 frames in length, it is input into an improved spatiotemporal graph convolutional network, which outputs the behavior categories of different characters. The structure diagram of the improved spatiotemporal graph convolutional network is shown below. Figure 8 As shown, it includes the following steps 4.1 to 4.4.

[0040] Step 4.1: Use hybrid adaptive graph convolution to obtain a hybrid adjacency matrix containing keypoint difference features and keypoint similarity features, and then combine it with the input features. Performing Einstein summation completes the graph convolution operation, extracting spatial features. A hybrid adaptive graph convolution structure diagram is shown below. Figure 9 As shown, this includes the following steps 4.1.1 to 4.1.4: Step 4.1.1: Obtain the adjacency matrix containing the differences in node characteristics. This includes the following steps: 4.1.1.1 to 4.1.1.2: Step 4.1.1.1: Reduce the number of channels in the input feature map using two independent point convolution operations, and then perform global average pooling on the feature map in the time dimension to obtain two dimensions ( N , C e , V Feature map and , Step 4.1.1.2: Through feature maps and Calculate the feature differences between all pairs of relevant nodes and normalize the difference values ​​to the interval [-1, 1]. The formula is as follows: In the formula, tanh is the tanh activation function.

[0041] Step 4.1.2: Obtain the adjacency matrix containing the similarity features of relevant nodes. This includes the following steps: 4.1.2.1 to 4.1.2.2: Step 4.1.2.1: Use two independent point convolution operations to reduce the number of channels in the input feature map, and then use a reshaping operation to fuse the channel and temporal dimension information to obtain the following dimensions ( Feature map And the dimension is ( Feature map , Step 4.1.2.2: Through feature maps and Calculate the similarity between all pairs of related nodes and normalize it to the interval [-1, 1]. The formula is as follows: In the formula, tanh is the tanh activation function.

[0042] Step 4.1.3: Using a learnable weighted fusion strategy, the adjacency matrices obtained in steps 6.1.1 and 6.1.2 are fused with the predefined adjacency matrix to obtain a hybrid adjacency matrix. The calculation formula is as follows: In the formula, and These are learnable parameters, whose values ​​are obtained through model training. This is a predefined adjacency matrix.

[0043] Step 4.1.4: For the hybrid adjacency matrix Input features Perform Einstein summation to implement graph convolution operation and obtain output features. .

[0044] Step 4.2: Process the feature map obtained in Step 4.1 Perform a multi-branch temporal convolution operation to extract temporal features. The multi-branch temporal convolution structure diagram is shown below. Figure 10 As shown, it includes the following steps 6.2.1 to 6.2.3: Step 4.2.1: For Using point convolution operations to reduce the number of channels, we obtain , , , , , Six feature maps.

[0045] Step 4.2.2: Process the feature map , , , Perform temporal convolution operations with receptive fields of 3, 5, 7, and 9 to extract keypoint variation features across different time spans. Then, process the feature maps... No other operations are performed, providing unsmoothed baseline information for subsequent feature fusion. (Regarding the feature map...) Max pooling is performed to extract salient features at key time points. Finally, the results from the six branches are concatenated along the channel dimension to obtain the feature map. .

[0046] Step 4.2.3: Adjust the feature map using point convolution. The number of channels is used to achieve feature interaction between channels and obtain feature maps. .

[0047] Step 4.3: Use duration-based attention for feature maps Weights are assigned to each time period to enhance the feature map's ability to represent features within important time periods. The duration-based attention structure diagram is shown below. Figure 11 As shown, it includes the following steps 4.3.1 to 4.3.3: Step 4.3.1: Perform adaptive pooling (Adapool) to globally aggregate the feature map along the keypoint dimension, reducing the feature dimension from ( N , C , T , V Compress to ( N , C , T , 1), to eliminate spatial redundancy information and retain complete temporal features, so that subsequent calculations can focus on features in the time dimension.

[0048] Step 4.3.2: Divide the input feature map into several non-overlapping time groups along the time dimension. Each group contains g 1 consecutive frames. The kernel size used is ( g 1) Step size is ( g , 1) Convolution extracts within each time group g The temporal features of the frames are analyzed, and the number of channels is compressed to extract more important feature information. Then, point convolution is used to restore the number of channels of the feature map, resulting in a temporal weight matrix. .

[0049] Step 4.3.3: Using a repeated interpolation strategy, the weights of each time group are repeated along the time dimension. g This yields a time weight matrix with matching dimensions. and feature map Weights are assigned through element-wise multiplication to obtain the feature map. .

[0050] Step 4.4: Feature Map The behavior category is output after the fully connected layer.

[0051] To demonstrate the advantages of this invention in behavior recognition tasks, behavior recognition was performed on the NTU-RGB+D 60 test set and compared with the original network. The detection results are shown in Table 2. While reducing GFLOPS by 1.60, TOP1-acc(%) increased by 1.59% and TOP5-acc(%) increased by 0.16%. Therefore, the improved behavior recognition network of this invention has high recognition accuracy and low computational cost.

[0052] Table 1 Comparison of Behavior Recognition Step 4.5: If the output behavior category in step 4.4 is "falling behavior", then control the servo motor to adjust the camera angle to track the fallen person, including the following steps 4.5.1~4.5.4: Step 4.5.1: Extract the detection box information of the person who fell based on their ID number, and calculate the center coordinates of the person. The formula is as follows: In the formula, , These are the x-coordinates of the top-left and bottom-right corners of the detection box, respectively. , These are the ordinate values ​​of the top left and bottom right corners of the detection box, respectively.

[0053] Step 4.5.2: If the character's center coordinates are not in the center area of ​​the video, further calculate the angle the servo needs to rotate. The formula is as follows: In the formula, The x-coordinate of the video center point The vertical coordinate is the center point of the video.

[0054] Step 4.5.3: Calculate the PWM duty cycle based on the required rotation angle of the servo motor. The calculation formula is shown below. Step 4.5.4: The duty cycle is sent to the PCA9685 servo drive module via the I2C bus to generate a PWM control signal with the corresponding duty cycle. This controls the servo to rotate and adjust the camera angle, achieving real-time tracking of the fallen person. Real-time tracking is as follows: Figure 12 As shown, Figure 12 This image shows the effect of fall behavior detection and fall tracking.

[0055] In the pedestrian fall detection method based on the improved YOLOv8 model and spatiotemporal graph convolution, the model still needs to be trained. First, the improved YOLOv8n-pose network is trained using the COCO2017 KeyPoints dataset. 56,880 video action samples from the NTU-RGB+D60 dataset are selected. The trained improved YOLOv8n-pose network is used to extract keypoints, forming skeleton sequences in groups of 100 frames. The processed skeleton sequences are then used to train the improved spatiotemporal graph convolutional network.

[0056] Then, a keypoint data storage and update module is designed using the Bytetrack algorithm. The Bytetrack algorithm uses Kalman filtering to predict the target's position at the next moment based on the current target's coordinates and velocity information, and calculates the overlap between the predicted and actual positions. If the overlap exceeds a set threshold, the target is considered to be the same target. Keypoint sequences for different targets are established based on their target numbers.

[0057] Finally, based on the relationship between the rotation angle of the servo motor and the pulse width of the PWM control signal, a servo motor control system was designed and trained to achieve pedestrian fall detection.

[0058] Secondly, a pedestrian fall detection system based on an improved YOLOv8 model and spatiotemporal graph convolution is presented, with the system design diagram shown below. Figure 1 As shown, the system includes an edge computing device, memory, processor, camera, servo motors, and servo motor drivers. Nvidia's Jetson Orin NX edge computing device is selected as the platform for the software system. Two DS3218 servos are used to control the horizontal and vertical viewing angles of the camera, and a PCA9685 servo motor driver module is used as the servo motor driver. All hardware modules are connected. The camera inputs a video stream to the edge computing device via USB communication protocol, and the edge computing device sends control commands to the PCA9685 via I2C protocol to drive the servos to adjust the camera angle, thus enabling the tracking of a fallen person.

[0059] Furthermore, embodiments of the present invention provide a computer device, including an edge computing device, a memory, a processor, a camera, a servo motor, a servo motor driver, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements steps of a pedestrian fall detection method based on an improved YOLOv8 model and spatiotemporal graph convolution.

[0060] This invention provides an edge computing device readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps of a pedestrian fall detection method based on an improved YOLOv8 model and spatiotemporal graph convolution.

[0061] This invention provides a video input device that is connected to an edge computing device via a USB communication protocol. When a computer program is executed by a processor, the video input device provides a real-time video stream to the edge computing device, implementing a pedestrian fall detection method based on an improved YOLOv8 model and spatiotemporal graph convolution.

[0062] This invention provides a camera control device, which is connected to the edge computing device via the I2C communication protocol. When the computer program is executed by the processor, the camera control device implements the steps of a pedestrian fall detection method based on an improved YOLOv8 model and spatiotemporal graph convolution.

[0063] This invention, based on a Jetson Orin NX edge computing device and combined with a PCA9685 driver and a DS3218 servo, constructs a completely independent embedded fall detection system. Compared to systems relying on cloud servers, this invention offers faster response times and active tracking capabilities, effectively solving the problem of blind spots in the field of view of fixed cameras. Considering the limited computing power of edge devices, the improved YOLOv8n-pose joint detection model of this invention has lower computational cost. Recognizing the difficulty in detecting human joints in complex backgrounds, the improved YOLOv8n-pose joint detection model dynamically fuses local and global features through learnable parameters, improving joint detection accuracy. Considering the complexity of joint topology during movement, the improved spatiotemporal graph convolutional network of this invention simultaneously learns the differences and similarities in joint topology, enhancing spatial feature extraction capabilities. Considering the importance of temporal features of behavior, the improved spatiotemporal graph convolutional network of this invention extracts temporal features at multiple time scales and enhances motion features within important time scales, improving the accuracy of behavior recognition.

[0064] In summary, this invention constructs a hardware system consisting of an edge computing device, a camera, a servo motor, a servo motor driver, and a screen, and a software system consisting of a joint detection model, a target tracking network, a behavior recognition network, and a servo motor control system. The joint detection network of this invention, based on YOLOv8n-pose, replaces the feature extraction network with a lightweight feature extraction network; it designs an efficient spatial channel decoupled convolution and replaces the standard convolution in the network to reduce computation while enhancing feature extraction capabilities; it improves the hybrid local channel attention and embeds it into the C2f module of the neck, improving the accuracy of joint detection. The fall behavior recognition network of this invention, based on a spatiotemporal graph convolutional network, replaces the spatiotemporal graph convolutional module in the network with a hybrid adaptive graph convolutional module, improving the network's feature extraction capability in the spatial dimension; it replaces the temporal convolution in the network with a multi-branch temporal convolution, improving the network's temporal feature extraction capability; and it designs a temporal attention mechanism and adds it to the 5th and 8th layers of the spatiotemporal graph convolutional network to enhance the network's feature extraction capability in local time. The software system of this invention is deployed in the hardware system to realize real-time tracking of people who have fallen, which can detect falls of the elderly in real time and accurately, and reduce physical and mental harm to the elderly.

Claims

1. A fall detection method for pedestrians based on an improved YOLOv8 model and spatio-temporal graph convolution, characterized in that, include: Step 1: Use a camera to acquire detection video and preprocess the detection video; Step 2: Based on the MobileNetV4 lightweight feature extraction network and deep convolution, build an improved YOLOv8 model, input the detected video, and obtain the target box and key point coordinates; Step 3: Using the Bytetrack method, assign IDs to different people based on the bounding box coordinates, construct a data sequence containing target bounding box information and key point coordinates based on the person IDs, and continuously update it; Step 4: Establish a behavior recognition model based on a spatiotemporal graph convolutional network, input the data sequence, obtain the behavior categories of different people, and adjust the camera angle according to the behavior category to realize pedestrian fall detection.

2. The fall detection method of pedestrian based on improved YOLOv8 model and spatio-temporal graph convolution according to claim 1, characterized in that, In step 2, the detected video input is converted into a feature map using the improved YOLOv8 model. The MobileNetV4 lightweight feature extraction network of the improved YOLOv8 model is used to extract features from the input feature map. Then, the bidirectional feature fusion network of YOLOv8n-pose is used to effectively integrate feature information at different levels to obtain a multi-scale feature map. The target bounding boxes and key points are predicted from the multi-scale feature map to obtain the coordinates of the target bounding boxes and key points.

3. The pedestrian fall detection method based on the improved YOLOv8 model and spatiotemporal graph convolution as described in claim 2, characterized in that, Feature extraction from the input feature map using the MobileNetV4 lightweight feature extraction network with an improved YOLOv8 model includes: Employing efficient spatial channel decoupled convolutions to input feature maps Extract features and use pointwise convolution to transform the input feature map. The number of channels is doubled, and then spatial features are extracted using depthwise convolution to obtain the feature map. Then use pointwise convolution to transform the feature map The number of channels is doubled, and then grouped convolution is used to combine the feature maps. The number of channels is grouped according to the number of channels in the output feature map, and an independent convolutional kernel is used to extract features within each group to obtain the feature map. ; feature map With feature map The feature map is obtained by adding elements one by one. The UIB module in the MobileNetV4 lightweight feature extraction network was used to process the feature maps. Further feature extraction is performed to obtain feature maps. Use the SPPF module to process feature maps. Preliminary extraction of global features yields feature maps. .

4. The pedestrian fall detection method based on the improved YOLOv8 model and spatiotemporal graph convolution as described in claim 2, characterized in that, The YOLOv8n-pose bidirectional feature fusion network is used to effectively integrate feature information from different levels, resulting in multi-scale feature maps including: Using a top-down feature fusion path to process feature maps , , The information is upsampled sequentially and passed up layer by layer, as follows: ； In the formula Representing the Layer feature map ( i =4,3), Upsample indicates an upsampling operation. Then, the improved C2f module is used to extract the feature map. ( i In (=4, 3), more feature information is obtained, occlusion joints are detected, and feature maps are generated. After After convolution, the feature maps are divided into two equal parts based on the number of channels and stored in a list. The last feature map in the list is input into the Bottleneck part of the C2f module, and the result is added to the list. Finally, the feature maps in the list are concatenated and then processed... The convolution yields the feature map. ( i =4,3); where the last element of the list is first processed by convolution in the Bottleneck part, and then channel information, spatial information, local information and global information are fused using the improved MLCA. Utilizing a bottom-up feature fusion path, the feature map is processed. The aggregation operation, which passes information layer by layer, is represented as follows: ； In the formula Representing the Layer feature map ( i =4,5), , Conv represents the convolution operation; Using the improved C2f module from the feature map ( i In (=4, 5), more feature information is obtained, occlusion joints are detected, and feature maps are generated. After After convolution, the feature maps are divided into two equal parts based on the number of channels and stored in a list. The last feature map in the list is input into the Bottleneck part of the C2f module, and the result is added to the list. Finally, the feature maps in the list are concatenated and then processed... Convolution yields multi-scale feature maps ( i =4,5), .

5. The pedestrian fall detection method based on the improved YOLOv8 model and spatiotemporal graph convolution as described in claim 1, characterized in that, In step 3, the Bytetrack method is used. When a target first appears, it is assigned a number. Kalman filtering is used to predict the target's position at the next moment based on the target's current coordinates and velocity. The overlap between the predicted and actual positions is calculated. If the overlap is greater than a threshold, the target is considered to be the same. IDs are assigned to different people using the bounding box coordinates. A data sequence containing target bounding box information and key point coordinates is constructed based on the person ID. The existing data sequence is matched based on the person ID. If there is no data sequence with the corresponding number in the current data sequence, a new data sequence is created to store the key point information of the person. When the person ID disappears, the corresponding data sequence is cleared to save memory space. The process is continuously updated.

6. The pedestrian fall detection method based on the improved YOLOv8 model and spatiotemporal graph convolution as described in claim 1, characterized in that, In step 4, a behavior recognition model is established based on a spatiotemporal graph convolutional network. The data sequence is input to obtain a hybrid adjacency matrix containing both keypoint difference and keypoint similarity features. This matrix is ​​then compared with the input features. The Einstein summation convolution operation is performed to extract spatial features and output a spatial feature map. Multi-branch temporal convolution is then performed on the spatial feature map to extract temporal features and output a spatiotemporal feature map. Temporal attention is used to assign weights to each time period of the spatiotemporal feature map to enhance the feature representation ability of the spatiotemporal feature map within important time periods. Finally, the behavior category is output through a fully connected layer to obtain the behavior categories of different people.

7. The pedestrian fall detection method based on the improved YOLOv8 model and spatiotemporal graph convolution as described in claim 6, characterized in that, Obtain a hybrid adjacency matrix containing joint difference features and joint similarity features, and then compare it with the input features. Perform Einstein summation to complete graph convolution operation, extract spatial features, and output spatial feature maps including: Two independent pointwise convolution operations are used to reduce the number of channels in the input feature map, and then a global average pooling operation is performed on the feature map in the time dimension to obtain two dimensions ( N , C e , V Feature map and Through feature maps and Calculate the feature differences between all relevant node pairs, normalize the difference values ​​to the interval [-1, 1], and obtain the adjacency matrix containing the feature differences of relevant nodes. , is represented as: ； In the formula, tanh is the tanh activation function; Two independent point convolution operations are used to reduce the number of channels in the input feature map, and then a reshaping operation is used to fuse the channel and temporal dimension information to obtain the result with dimension (). Feature map And the dimension is ( Feature map Through feature maps and Calculate the similarity between all pairs of related nodes and normalize them to the interval [-1, 1] to obtain the adjacency matrix containing the similarity features of related nodes. , is represented as: ； In the formula, tanh is the tanh activation function; A learnable weighted fusion strategy is used to fuse the adjacency matrix with a predefined adjacency matrix to obtain a hybrid adjacency matrix, which is represented as: ； In the formula, and These are learnable parameters, whose values ​​are obtained through model training. The adjacency matrix is ​​predefined. For hybrid adjacency matrix Input features Perform Einstein summation to implement graph convolution operation and obtain output features. .

8. The pedestrian fall detection method based on the improved YOLOv8 model and spatiotemporal graph convolution as described in claim 6, characterized in that, Perform multi-branch temporal convolution on the spatial feature map to extract temporal features and output a spatiotemporal feature map. include: Features Using point convolution operations to reduce the number of channels, we obtain , , , , , Six feature maps, for feature maps , , , Perform temporal convolution operations with receptive fields of 3, 5, 7, and 9 to extract keypoint change features across different time spans, and then process the feature maps. Without performing any other operations, it provides baseline information for subsequent feature fusion without time smoothing, and modulates the feature map. Max pooling is performed to extract salient features at key time points. Finally, the results from the six branches are concatenated along the channel dimension to obtain the feature map. Adjusting feature maps using point convolution The number of channels is used to achieve feature interaction between channels and obtain feature maps. .

9. The pedestrian fall detection method based on the improved YOLOv8 model and spatiotemporal graph convolution as described in claim 6, characterized in that, Using temporal attention to assign weights to each time interval of the spatiotemporal feature map enhances the feature representation capability of the spatiotemporal feature map within important time intervals, including: Adaptive pooling is used to refine the spatiotemporal feature map along the key dimension. Perform global aggregation, reducing the feature dimension from ( N , C , T , V Compress to ( N , C , T 1) Eliminate spatial redundancy while preserving complete temporal features. Focus on features in the temporal dimension by dividing the input spatiotemporal feature map into several non-overlapping time groups, each containing... g For each consecutive frame, a convolutional kernel size of ( g 1) Step size is ( g , 1) Convolution extracts within each time group g The temporal features of the frames are analyzed, and the number of channels is compressed to extract more important feature information. Then, point convolution is used to restore the number of channels of the feature map, resulting in a temporal weight matrix. A repeated interpolation strategy is used, where the weights of each time group are repeated along the time dimension. g This yields a time weight matrix with matching dimensions. and feature map Weights are assigned through element-wise multiplication to obtain the feature map. .

10. The pedestrian fall detection method based on the improved YOLOv8 model and spatiotemporal graph convolution as described in claim 1, characterized in that, In step 4, adjusting the camera angle based on the behavior category to achieve pedestrian fall detection includes: Extract the detection bounding box information of the person who fell based on their ID number, and obtain the center coordinates of the person. , is represented as: ； In the formula, , These are the x-coordinates of the top-left and bottom-right corners of the detection box, respectively. , These are the ordinate values ​​of the top left and bottom right corners of the detection box, respectively; When the character's center coordinates are not in the middle area of ​​the video, the angle that the servo needs to rotate is represented as: ； In the formula, The x-coordinate of the video center point The vertical coordinate of the video center point; The duty cycle of the PWM, obtained based on the required rotation angle of the servo motor, is expressed as follows: ； The duty cycle is sent to the servo drive module via the I2C bus to generate a PWM control signal with the corresponding duty cycle, which controls the servo to rotate and adjust the camera angle, thus enabling real-time tracking of a fallen person.

Images