Skeleton detection and fall detection method based on improved spatio-temporal adaptive graph convolution

Abstract

Disclosed in the present application is a skeleton detection and fall detection method based on improved spatio-temporal adaptive graph convolution. The method comprises the steps of: S1, collecting image data to acquire data of each image frame; S2: using a pre-trained yolov5 target person detection model to detect whether a target person is present in the data of each image frame, and if a target person is present, turning to step S3, and if no target person is present, ending the process; S3: for each detected target person, using a Deepsort target tracking algorithm to perform target tracking to obtain a tracking result, calculating the similarity to obtain the result of target association, and updating trajectory information of each target person; and S4: performing pose recognition on each target person on the basis of the trajectory information, using a spatio-temporal adaptive graph convolutional network to extract a feature vector of a pose, and using a classifier to perform human body behavior classification and recognition, in order to determine whether the target person has experienced a fall incident. The method achieves higher accuracy and robustness.

Description

Skeleton detection and fall detection method based on improved spatiotemporal adaptive graph convolution

[0001] The present application claims priority to the Chinese patent application No. 2023116620681, filed on December 06, 2023, and entitled "Skeleton detection and fall detection method based on improved spatiotemporal adaptive graph convolution", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application belongs to the field of artificial intelligence, and in particular relates to a skeleton detection and fall detection method based on improved spatiotemporal adaptive graph convolution. BACKGROUND

[0003] Existing technologies use computer vision-based methods, which use cameras or depth sensors to capture human movements and postures, and perform fall detection by analyzing key points or skeleton data. However, computer vision methods are sensitive to changes in lighting, occlusion or changes in viewing angle, which can lead to a decrease in detection accuracy, and have significant defects in practical applications.

[0004] Existing technologies use machine learning-based methods, which use machine learning algorithms to identify characteristic patterns of fall behavior by training models. However, traditional machine learning methods often require manual feature extraction and have limited ability to model complex spatiotemporal relationships.

[0005] Existing technologies use sensor-based fall detection algorithms, which mainly rely on sensors such as three-axis accelerometers, gyroscopes, and pressure sensors to detect human motion and locate the human body position to identify fall behavior. Yodpijit et al. used an accelerometer and a gyroscope motion sensor to detect fall behavior, using an artificial neural network (ANN) based algorithm to distinguish between fall behavior and daily behavior. Chen KH et al. used a smartphone equipped with a motion sensor to detect fall behavior. By using the motion sensor in the smartphone and the transmission module, the changes in human motion are calculated and the data is sent to the server to analyze the motion behavior and identify the fall. However, due to the high cost of wearable devices and the discomfort they can cause to the wearer, they do not have good universality, so it is not realistic to detect whether a person who does not wear a sensor device has fallen in a public place. Environment-based methods also rely on sensors like wearable devices. The difference is that they mainly rely on radar or ground sensors to collect current or audio information to determine whether a fall has occurred. Because environmental devices are very sensitive to external noise and other disturbances. Therefore, in a complex multi-person scenario, environment-based methods are obviously not very effective.

[0006] Existing techniques employ fall detection algorithms based on traditional computer vision methods. With the rapid development of computer vision technology, more and more researchers are focusing on the study of behavior recognition. Human behavior recognition is widely used in security monitoring, human-computer interaction and virtual reality. Due to the limitations of the above sensor-based method, fall detection based on computer vision has received increasing attention in recent years. The computer vision-based method does not require sensor devices, only needs to install a camera, and can identify whether a person has fallen through video, and the accuracy is higher than that of the sensor method. Wang et al. proposed a new foreground segmentation model to detect pedestrians, and detected the fall behavior by the contour change of the pedestrian. Zerrouki et al. used a hidden Markov model to identify and classify fall behavior based on the contour shape features of pedestrians. Yu et al. used a hidden Markov model combined with an acceleration sensor, and used a direction calibration algorithm to compensate for errors. Deng Zifeng et al. combined geometric features and constructed a pedestrian bounding matrix to determine the fall behavior in different directions. Fan et al. classified fall behavior by separately calculating dynamic and static features. Miao et al. used an ellipse fitting method to wrap the human body contour with an ellipse, and used geometric features and position information combined with SVM to identify fall behavior and classify different behaviors. However, this traditional method focuses on feature extraction and classification, and is easily affected by noise, changes in lighting and occlusion.

[0007] Due to the above shortcomings of the fall detection method: the sensor-based fall detection method relies on specific sensors, which limits the deployment and use of the system, and requires the installation and configuration of sensor devices in a specific environment. In addition, the detection range and coverage of the sensor-based method are usually limited by the sensing range of the sensor. For example, using a fixed-position accelerometer can only detect fall events within a fixed range, and cannot cover the entire environment. The sensor method also needs to acquire and process sensor data in real time, and analyze and judge it. This requires real-time and stability of the system, and has certain dependence on the quality and accuracy of the sensor data. The fall detection algorithm based on traditional computer vision methods has high requirements for the environment, and requires clear camera view and good lighting conditions to accurately capture and analyze human posture and action information. In complex or harsh environments, the performance of the algorithm may decrease. Computer vision traditional methods usually rely on hand-designed feature extraction methods, which need to design and select suitable features for fall detection tasks. However, this manual feature extraction is often affected by human experience and subjective factors, and cannot fully utilize the potential information of the data.

[0008] The Chinese patent document (CN112966628A) discloses a view angle adaptive multi-target fall detection method based on a graph convolutional neural network, which comprises the following steps: a target detection algorithm is used to detect the human target in each frame of image in the target video source, a pose estimation algorithm is used to extract the key skeleton point data of the human target in each frame of image, when the number of frames of the same human target detected continuously is greater than a preset detection threshold, the extracted key skeleton point data is input into the trained view angle adaptive subnetwork to obtain the view angle adjustment parameter; the key skeleton point data is adjusted in view angle according to the view angle adjustment parameter, and the motion data is calculated according to the key skeleton point data adjusted in view angle; the key skeleton point data adjusted in view angle and the motion data are input into the trained graph convolutional fall recognition main network for fall detection, and the detection result label is output. This method cannot fully utilize the potential information of data.

[0009] Therefore, the application provides a skeleton detection and fall detection method based on improved spatio-temporal adaptive graph convolution, which is suitable for different fall modes and environmental conditions, and has higher accuracy and robustness.

[0010] SUMMARY

[0011] The technical problem to be solved by the present application is to provide a skeleton detection and fall detection method based on improved spatio-temporal adaptive graph convolution, which is suitable for different fall modes and environmental conditions, and has higher accuracy and robustness.

[0012] To solve the above technical problems, the technical scheme adopted by the present application is as follows: the skeleton detection and fall detection method based on improved spatio-temporal adaptive graph convolution, specifically comprising the following steps:

[0013] S1: collecting image data and obtaining each frame of image data;

[0014] S2: using a pre-trained yolov5 target personnel detection model to detect whether there is a target personnel in each frame of image data, if there is a target personnel, turning to step S3, if there is no target personnel, ending;

[0015] S3: for each detected target personnel, using a Deepsort target tracking algorithm for target tracking to obtain a tracking result, calculating a similarity to obtain a target association result, and updating the trajectory information of each target personnel;

[0016] S4: performing pose recognition on each target personnel in combination with the trajectory information, using a spatio-temporal adaptive graph convolution network to extract a feature vector of the pose, using a classifier to classify and recognize the human behavior, and judging whether the target personnel has a fall event.

[0017] By adopting the technical solution, the deep learning method can automatically learn and extract features, adapt to different falling modes and environmental conditions, and perform end-to-end training and optimization from a large amount of data, which makes the falling detection algorithm based on deep learning have higher accuracy and robustness. By applying the adaptive graph convolution network in the ST-GCN falling detection model, the flexibility, adaptability and representation ability of the model are improved, so that the human action recognition and spatio-temporal feature modeling are more accurate. The ability of modeling complex spatio-temporal features is improved, and the flexibility and adaptability of the model are enhanced. The adaptive graph convolution network can dynamically adjust the weights in the convolution operation according to the features and context information of the input data. Through adaptive weight adjustment, the model can weight the features according to different situations, better capturing and emphasizing key spatio-temporal features. The adaptive graph convolution network can also adapt to different environments and scenes, and has certain robustness to factors such as light changes, background noise and occlusion. This makes the falling detection model maintain good performance in different actual application scenarios.

[0018] Preferably, the specific steps of training the yolov5 target personnel detection model in the step S2 are:

[0019] S21: Frame the image data collected in step S1 in a video frame extraction manner to generate a falling data set;

[0020] S22: Perform enhancement processing on the falling data set to obtain an enhanced data set, and divide it into a training set and a test set;

[0021] S23: Build a YOLOV5 algorithm model, input data and train to obtain algorithm model weights, and obtain a yolov5 target personnel detection model. In order to improve the accuracy of model detection, a large number of videos of real scenes of personnel falling and simulated videos of personnel falling are collected, and the videos are frame extracted to generate a falling data set. Common data enhancement methods such as Mosaic data enhancement and adaptive anchor frame calculation are used to enhance the data set and realize the generalization ability of the enhanced model.

[0022] Preferably, the specific steps of the step S3 are:

[0023] S31: Predict the position of the next frame of the target personnel using Kalman filtering, and then perform matching;

[0024] S32: Data association is performed using the Hungarian algorithm, i.e. the position information of the target personnel is obtained by inputting the video into the detection network, and then transmitted to the tracking network of the Deepsort target tracking algorithm for data association and matching of the target personnel in the previous and subsequent frames of the target personnel, so as to obtain a tracking target;

[0025] S33: Reuse Kalman filter to update trajectory to determine tracking results and determine ID.

[0026] Preferably, the similarity in step S32 is calculated using motion information and appearance information of the target person, and Mahalanobis distance is used to determine the correlation between the predicted target person and the detected target person for the motion information. The formula of Mahalanobis distance is:

[0027] wherein d j is the position of the detection frame j; y i is the predicted position of the tracker i; d j -y i represents subtracting the feature vector of the jth data point from the feature vector of the ith data point to obtain a difference vector, and the difference vector represents the difference or distance between the two data points in each feature dimension; (d j -y i ) T represents transposing the difference vector, i.e., changing from a row vector to a column vector, to facilitate matrix operation; S i is the covariance matrix of the detection and prediction positions; and Mahalanobis distance considers the uncertainty of state measurement by calculating the standard deviation between the detection position and the average tracking position.

[0028] When the target is long-term occluded or the viewing angle is jittered, appearance information is introduced to solve the problem of identity switching caused by occlusion through cosine distance. The expression of cosine distance is:

[0029] wherein r j is the eigenvector of the detection frame d j ; is the set of N (N is set to 100) eigenvectors closest to the distance of the tracker i; and R i is the appearance feature vector library.

[0030] Then, linear weighting is used to sum up, and the formula is: c i，j = λd (1) (i, j) + (1-λ)d (2) (i, j);

[0031] wherein λ is a weight parameter, the formula combines Mahalanobis distance and cosine distance, and the contribution of the two is balanced by adjusting the weight coefficient λ; the value range is between [0, 1]; and the function c i，j is used to represent the comprehensive distance or similarity between the ith sample and the jth sample; by adjusting the weight parameter λ, the importance of the two distance measurements can be balanced when combining the distance measurements; and the measurement value c i，j exists in d (1)(i, j) and d (2) (i, j) and d is the N frames with the highest similarity to the target feature of the current frame, and the cosine similarity between the target feature of the current frame and the target feature of each previous frame needs to be calculated, and the N frames with the highest similarity are selected; that is, the model trained is used to extract the features of the jth detection box and the ith tracking box to obtain the appearance feature vector r j and the appearance feature vector r i The kth appearance feature vector of the ith tracking box is stored in the appearance feature library R . i , is the appearance feature cosine similarity between the jth detection box and the ith tracking box, and the value is the minimum cosine distance d (2) (i, j) is 1; wherein, d (2) The smaller the sum of d

[0032] Preferably, the specific steps of the step S4 are:

[0033] S41 Alphapose posture recognition: using the top-down method of the Alphapose model to perform skeleton detection to obtain continuous skeleton frames, and then using the SPPE (single person pose estimation) algorithm to estimate the posture of the detected target personnel to obtain a target personnel skeleton map;

[0034] S42 fall behavior recognition: using a spatiotemporal adaptive graph convolution network to perform feature extraction in the spatial and temporal dimensions to obtain a feature vector of the target personnel;

[0035] S43 human behavior classification and recognition: using a trained classifier to classify and recognize the feature vector of the target personnel, and the classifier will determine whether the input data belongs to the fall category according to the feature vector.

[0036] Preferably, in the step S41, a symmetric space transformation network (SSTN), a pose-guided proposal generator (PGPG), and a parametric pose non-maximum suppression (PPNMS) are added to the Alphapose model to realize skeleton detection; the specific steps are:

[0037] S411 Data Preprocessing: Preprocess the cropped image segments of the target personnel, including image scaling, normalization, and channel order conversion, to adapt to the input requirements of the Alphapose model.

[0038] S412 Multi-person Pose Estimation: Alphapose is used to perform multi-person pose estimation on cropped pedestrian image segments; the Alphapose model detects the positions of key points on the human body (such as head, shoulders, elbows, knees, etc.) and estimates the pose information of the target person;

[0039] S413 Result Visualization: The results of multi-person pose estimation are combined with the original image or video frame to draw connection lines of key points and pose angle information, obtaining the target person's skeleton map for subsequent analysis and display. The method combining Alphapose and graph convolution effectively avoids dependence on the video environment. Comparative experiments were conducted with multiple models on public datasets, demonstrating that this model has high detection accuracy and low scene dependence.

[0040] Preferably, the spatiotemporal adaptive graph convolutional network (STA-GCN) in step S42 consists of spatial adaptive graph convolution SA-GC and temporal adaptive graph convolution TA-GC, and both spatial adaptive graph convolution SA-GC and temporal adaptive graph convolution TA-GC have embedded graph convolution topology adaptive encoders TAE.

[0041] Preferably, the specific steps in step S42 of using a spatiotemporal adaptive graph convolutional network to extract features from the pedestrian skeleton map through graph convolutional layers are as follows:

[0042] S421: A feature extraction operator will be introduced onto the spatiotemporal graph, first targeting the spatiotemporal graph. In graph convolution, where ε is the set of all nodes spanning T frames in the skeleton sequence. st For the set of spacetime boundaries; v nt In the spacetime diagram A node in the middle; v nt Let v represent a node in the skeleton sequence, where n is the node index and t is the time step index; therefore, v nt It is a node at time step t; v nt This may represent a specific moment, i.e., a position or feature in the skeleton sequence at time step t; n is the node index, used to uniquely identify different nodes in the spatiotemporal graph; in v ntIn particular, n denotes the number of nodes, which is an integer, usually from 1 to N, where N is the total number of nodes; t usually denotes the time step or time frame; it is used to represent each time or time point in the dataset; T denotes the total number of time steps or time frames;

[0043] S422: decompose a spatio-temporal graph into T spatial graphs across time and N temporal graphs across nodes; the spatial graph is represented as where ε s is the spatial edge set, denoted as the spatial adjacency matrix A s ∈R T×N×N ; where R represents the real field of the matrix, R T×N×N represents a three-dimensional matrix, whose elements belong to R, i.e. elements in the real field; when all spatial graphs have the same spatial correlation, the spatial adjacency matrix is degraded to A∈R N×N form; similarly, the temporal graph is represented as denotes the temporal edge set in the temporal graph ; the temporal edge set is used to define the temporal relationship or temporal connection between nodes; specifically, ε t contains the temporal edges connecting the nodes, used to represent the association or dependency relationship between nodes at different time steps; the temporal adjacency matrix is A t ∈R N×T×T ; after the decomposition of the spatio-temporal graph, the spatial graph convolution S-GC and the temporal graph convolution T-GC are developed;

[0044] The spatial graph convolution S-GC is represented as:

[0045] where, is an element of A s , v pt represents a node in the spatio-temporal graph, where p is the index of the node and t is the index of the time step, which is used to represent the node in the spatial graph in the spatial graph convolution S-GC; W represents the weight matrix or weight parameter, which is usually used for linear transformation or weighted aggregation of features; i.e. w(v pt ) represents the weight applied to the node v pt , which is used for weighted aggregation of input features to generate output features on the node v nt ;

[0046] The temporal graph convolution T-GC is represented as:

[0047] where, is an element of A t , v nqalso represents a node in the spatial graph, where n is the index of the node, q is the index of the time step, in the temporal graph convolution T-GC, it is used to represent a node in the temporal graph; W represents the weight matrix (or weight parameter), which is usually used for linear transformation or weighted aggregation of features; that is, w(v nq ) represents the weight applied to the node v nq , which is used for weighted aggregation of input features to generate output features on the node v nt ;

[0048] Again, the spatial graph convolution S-GC and the temporal graph convolution T-GC are used to extract features on the spatio-temporal graph, and the feature vector is obtained, where the extracted features are represented as:

[0049] where, is an element of A t ; represents the weight or connection coefficient used to connect the node v pq and the node v nq in the spatial graph convolution S-GC, which is an element of the spatial adjacency matrix A s in the spatio-temporal graph, used to measure the spatial relationship or connection strength between nodes; specifically, represents the spatial connection strength between the node v pq and the node v nq , which is used to determine how to propagate feature information in the spatial graph; N is used to represent the index range of the nodes, representing the node index of the node v pq and the node v nq ; q represents the index of the time step or time frame, which is used to represent different time steps or time frames in the temporal graph convolution T-GC; p is the index of the node of the spatial graph; v pq represents the node in the spatial graph convolution S-GC, where p is the index of the node, q is the index of the time step; W represents the weight matrix or weight parameter, which is used for linear transformation or weighted aggregation of features, that is, w1(v pq ) and w2(v nq ) are both used for weighted aggregation of input features, w1(v pq ) w1(v pq ) represents the first weight or connection coefficient applied to the node v pq , w2(v nq ) represents the second weight or connection coefficient applied to the node v nq ; These weight matrices are usually learnable parameters in neural network models, which are used to adjust and control the propagation and aggregation of features between different nodes to capture complex feature relationships in the spatio-temporal graph; In the training process of the model, these weight matrices will be adjusted according to the loss function to optimize the performance of the model.

[0050] The method uses a YOLO v5 network model to obtain human target detection box information, confidence information and classification information; the target tracking part of the model uses a method combining YOLO v5 and Deepsort, wherein each tracking instance needs to be matched with the target detection result obtained by YOLO v5, and can be divided into three states according to different situations, is set to a temporary state at initialization, is deleted if no detection result is matched; if a certain number of consecutive matches are matched, it is set to a tracked state; if the number of unmatched times exceeds the set maximum number in the tracked state, it is deleted; in the pose estimation part, the target detection and tracking part detects the target detection box of each frame, and in the pose estimation stage, the model identifies and detects the human skeleton key points of each frame of image through Alphapose, and then normalizes the obtained skeleton key point data; after pose estimation, the information of the human skeleton key points is obtained, the obtained skeleton points are transmitted to the improved space-time adaptive graph convolutional neural network (STA-GCN) in the form of graph information for graph convolution, and the tensor after graph convolution is transmitted to the Linear layer for classification, so as to realize human fall detection; the space-time adaptive graph convolution is introduced to model the space-time relationship of the pedestrian target, the motion trajectory information and key point position information of the pedestrian are used to realize adaptive learning and extraction of features; the space-time adaptive graph convolution can better capture the motion features and posture changes of the pedestrian, and enhances the representation ability of the features.

[0051] Preferably, the specific steps of the step S43 are:

[0052] S431: first, pre-process the feature vector to obtain a pre-processed feature vector;

[0053] S432: then, select a classifier model to train the classifier using the labeled training data;

[0054] S433: then, evaluate the trained classifier using unlabeled test data;

[0055] S434: use the trained classifier to classify and identify the pre-processed feature vector, and determine whether the input data belongs to the fall category according to the feature vector.

[0056] Preferably, the pre-processing method in the step S431 includes feature normalization and dimension reduction; the classifier model in the step S432 selects a support vector machine (SVM); and the performance of the classifier is evaluated by calculating the accuracy, recall rate and F1 score in the step S433.

[0057] Compared with the prior art, the present application has the beneficial effects that:

[0058] (1) High accuracy: YOLOv5 as a target detector has high accuracy, which can quickly and accurately detect pedestrian targets in images or videos. Deepsort can realize continuous tracking of pedestrian targets, reducing false positives and false negatives, and improving the accuracy of fall detection;

[0059] (2) Real-time: YOLOv5 and Deepsort are both target detection and tracking algorithms designed for real-time applications, so the entire system can run efficiently in real-time scenarios, meeting the needs of real-time fall detection.

[0060] (3) Continuous pose estimation: Combined with the continuous tracking of pedestrian targets by Deepsort, fall detection can continuously estimate the pose of pedestrians, not only accurately determining whether a pedestrian has fallen, but also obtaining the pose change information of the pedestrian in the entire video sequence;

[0061] (4) Spatio-temporal adaptive graph convolution: The introduction of spatio-temporal adaptive graph convolution can capture the motion characteristics and pose changes of pedestrian targets, enhancing the representation ability of features. By learning the relationship of pedestrian targets in space and time, the feature representation of fall detection is more comprehensive and meaningful;

[0062] (5) Multi-task learning: Integrating YOLOv5, Deepsort and pose estimation modules forms a multi-task learning framework, with each module cooperating and sharing features, improving the performance of the entire system;

[0063] (6) Universality: The fall detection method based on YOLOv5 and Deepsort can be applied to various types of cameras and scenes, with strong universality and adaptability. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0065] Figure 1 is a flowchart of the skeleton detection and fall detection method based on the improved spatio-temporal adaptive graph convolution of the present application;

[0066] Figure 2 is a total flowchart of pedestrian detection and tracking in the skeleton detection and fall detection method based on the improved spatio-temporal adaptive graph convolution of the present application;

[0067] Figure 3 is a cascade matching flowchart in the Deepsort algorithm in the skeleton detection and fall detection method based on the improved spatio-temporal adaptive graph convolution of the present application;

[0068] Fig. 4 is a space-time adaptive graph convolution network junction in the improved space-time adaptive graph convolution-based skeleton detection and fall detection method of the application;

[0069] Fig. 5 is a skeleton detection schematic diagram in the improved space-time adaptive graph convolution-based skeleton detection and fall detection method of the application. DETAILED DESCRIPTION

[0070] The embodiments of the application will be described in detail below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the application, and cannot be used to limit the protection scope of the application.

[0071] The well-known knowledge in the technical solution includes:

[0072] (1) YOLOv5 target detection algorithm:

[0073] YOLOv5 technical features: YOLOv5 is a single-stage target detection algorithm. This algorithm adds some new improvement ideas on the basis of YOLOv4, so that its speed and accuracy have been greatly improved. Compared with the YOLOv4 algorithm, the YOLOv5 target detection algorithm has the following improvement ideas:

[0074] Input: In the model training stage, some improvement ideas are proposed, mainly including Mosaic data enhancement, adaptive anchor box calculation, etc.

[0075] Mosaic data enhancement randomly selects four different training images and splices them into a large image to form a "Mosaic" image. In the splicing process, the size and position of each image will be randomly adjusted and flipped. In this way, the model can be exposed to more background and target samples during training, increasing the richness and diversity of the data set. Mosaic data enhancement can improve the generalization ability and robustness of the model, and reduce the problem of overfitting.

[0076] Adaptive anchor box calculation is a method of dynamically adjusting the anchor box according to the distribution and size of the target in the image. In traditional target detection algorithms, fixed anchor boxes are usually used to predict the bounding box of the target. Adaptive anchor box calculation dynamically generates anchor boxes according to the distribution of targets in the training set, making them more suitable for the size and aspect ratio of the target. This can improve the detection accuracy of the target detection algorithm for targets of different sizes and shapes. Adaptive anchor box calculation usually combines clustering algorithms or statistical analysis methods to determine the optimal anchor box size and aspect ratio.

[0077] These techniques are widely used in target detection algorithms, aiming to increase data diversity, improve model adaptability and accuracy. Their use can make the model better adapt to various scenarios and targets, and achieve better performance in practical applications.

[0078] Benchmark network: some new ideas from other detection algorithms are integrated, mainly including Focus structure and CSP structure.

[0079] Focus structure is a special structure used in YOLOv5 to replace traditional convolution operations. Traditional convolution operations usually reduce the scale of input feature maps by a large margin, resulting in information loss and increased computational complexity. Focus structure adopts a similar idea to group convolution, dividing input feature maps into smaller sub-maps and performing convolution operations within sub-maps. This can reduce information loss and improve computational efficiency. Focus structure helps to reduce computational complexity while maintaining the richness of feature maps.

[0080] CSP structure is a special network structure used to improve the transmission of feature maps and the extraction of features. CSP structure divides input feature maps into two branches, one branch performs convolution operations, and the other branch directly transmits the original feature map. This branch structure can promote the propagation and fusion of features, improving the model's ability to perceive targets of different scales. CSP structure is applied to different network levels in YOLOv5 to enhance the extraction and expression ability of features.

[0081] The application of Focus structure and CSP structure in YOLOv5 enables the model to better handle targets of different scales, improving the accuracy and robustness of target detection. By integrating new ideas from other detection algorithms, the benchmark network of YOLOv5 achieves good performance in target detection tasks.

[0082] Neck network: target detection networks often insert some layers between the BackBone and the final Head output layer. YOLOv5 adds FPN+PAN structure, which is commonly used in target detection tasks to extract and fuse feature information of different scales, so as to better detect multi-scale targets.

[0083] FPN (Feature Pyramid Network) is a network structure used for processing multi-scale features. It generates a series of feature maps with different scales by constructing a feature pyramid on different levels of the backbone network. These feature maps contain rich semantic and spatial information, which can capture the details and context of targets at different scales. FPN structure uses upsampling and downsampling operations to fuse these feature maps to obtain a series of feature pyramids with different resolutions to meet the detection needs of targets at different scales.

[0084] PAN (Path Aggregation Network) structure is a further improvement based on FPN. It introduces lateral connections and top-down pathways to better fuse and aggregate feature information. Lateral connections are used to combine high-resolution shallow features with low-resolution deep features to enrich and enhance feature representation. Top-down pathways combine lower resolution feature maps with high-resolution feature maps through upsampling operations to obtain more rich context information.

[0085] Head output layer: The anchor mechanism of the output layer is the same as YOLOv4, and the main improvement is the loss function GIOU_Loss during training and the DIOU_nms of the predicted box screening.

[0086] In the Head output layer of YOLOv5, a set of predefined anchor boxes are used to predict the bounding boxes of targets. Each anchor box is usually associated with multiple grid cells, and the position and size of the target are predicted by regression. The size and aspect ratio of these anchor boxes are usually designed and adjusted according to the distribution of targets in the training set to adapt to targets of different scales and shapes.

[0087] During training, YOLOv5 improves the calculation method of the loss function and adopts GIOU_Loss (Generalized Intersection over Union Loss). GIOU_Loss is an improved version of the IoU loss function, which is used to measure the overlap between the predicted bounding box and the true bounding box. It considers the intersection area and the union area between the bounding boxes, and calculates the generalized IoU value to optimize the model. Compared with the traditional IoU loss function, GIOU_Loss can better handle the overlap of bounding boxes in the optimization of target detection tasks, improving the accuracy and robustness of detection.

[0088] In addition, YOLOv5 also improves the method of bounding box screening, introducing DIOU_nms (Distance-IoU Non-Maximum Suppression). DIOU_nms considers the distance and IoU value between bounding boxes during the non-maximum suppression (NMS) process, which is used to select the most representative bounding box. It considers the position and shape information of the bounding box, and retains the most relevant target box while suppressing redundant prediction boxes. This screening method can further improve the accuracy and recall rate of target detection.

[0089] (2) Deepsort target tracking algorithm: The predecessor of Deepsort is the SORT algorithm, which is composed of a target detector and a tracker. The core of the tracker is the Kalman filter algorithm and the Hungarian algorithm. The Kalman filter algorithm is used to predict the state of the detection box in the next frame, and the Hungarian algorithm is used to match the state with the detection results of the next frame to achieve tracking. Once the object is blocked or not detected for other reasons, the state information predicted by the Kalman filter will not match the detection results, and the tracking segment will end early. Deepsort introduces a re-identification algorithm in deep learning to extract the appearance features (low-dimensional vector representation) of the detected object (in the detection box). After each detection and tracking, the appearance features of the detected object are extracted and saved. Every time a step is executed, a similarity calculation between the current frame's detected object appearance features and the previously stored appearance features is performed to avoid missing detection and losing identity ID. Deepsort not only uses the speed and direction trend of the object to track the target, but also uses the appearance features of the object to consolidate the judgment of whether it is the same object.

[0090] Deepsort main features:

[0091] 1. Multi-target tracking: Deepsort can track multiple targets simultaneously by associating and managing the trajectories of each target, achieving accurate tracking of multiple targets in a video.

[0092] 2. Combination of deep learning and traditional methods: Deepsort combines deep learning and traditional Kalman filters. Deep learning is used for target detection and feature extraction, providing rich target feature representation; Kalman filter is used for target state estimation and prediction, providing target position and motion information.

[0093] 3. Strong feature representation: The feature representation extracted by the deep learning network has high discriminability, which can accurately measure the similarity between targets. This helps to accurately associate targets and handle problems such as appearance changes and occlusions.

[0094] 4. Robustness: Deepsort has a certain degree of robustness, and can handle challenges such as the appearance, disappearance and occlusion of targets in common scenarios. By using Kalman filtering for state prediction and trajectory management, it can compensate for the errors or instability of target detection to a certain extent.

[0095] 5. Real-time performance: Deepsort performs well in real-time target tracking, with fast speed and low computational resource requirements. This makes it suitable for real-time target tracking on devices with limited computing resources, such as mobile devices, embedded systems, etc.

[0096] Embodiment: The method for skeleton detection and fall detection based on improved spatiotemporal adaptive graph convolution, as shown in FIG. 1, specifically includes the following steps:

[0097] S1: Collect image data, obtain each frame of image data through video or rtsp stream.

[0098] S2: Use a pre-trained yolov5 target personnel detection model to detect whether there is a target personnel in each frame of image data, if there is a target personnel, go to step S3, if there is no target personnel, end.

[0099] The specific steps for training the yolov5 target personnel detection model in step S2 are as follows:

[0100] S21: Use video frame extraction to process the image data collected in step S1 to generate a fall dataset;

[0101] S22: Enhance the fall dataset to obtain an enhanced dataset, and divide it into a training set and a test set;

[0102] S23: Build a YOLOV5 algorithm model, input data and train to obtain algorithm model weights, and obtain a yolov5 target personnel detection model.

[0103] S3: For each detected target personnel, use the Deepsort target tracking algorithm for target tracking to obtain tracking results, calculate the similarity to obtain the results of target association, and update the trajectory information of each target personnel.

[0104] As shown in FIG. 2, the specific steps of step S3 are as follows:

[0105] S31: Use Kalman filtering to predict the position of the next frame of image of the target personnel, and then perform matching; including cascade matching and IOU matching.

[0106] As shown in FIG. 3, first, the position of the next frame image of the target person is predicted by using Kalman filtering, and cascade matching is performed on the tracked target person, and IOU matching is performed on the track that is not tracked; after cascade matching, the tracking instances that are not matched, the detection instances that are not matched, and the tracking instances that are matched are obtained, and IOU matching is performed again on the detection instances that are not matched; after IOU matching, the tracking instances that are not matched, the detection instances that are not matched, and the tracking instances that are matched are obtained, wherein, for the tracking instances that are matched, step S33 is performed, for the tracking instances that are not matched, if there is no track of the target person in the tracking instances that are not matched, the tracking instances that are not matched are deleted, if there is a track of the target person in the tracking instances that are not matched, it is judged whether the number of times of not matching exceeds the maximum number of times set, if yes, the tracking instances that are not matched are deleted, and if no, the tracking instances that are not matched are transferred to the prediction of the target person by using Kalman filtering.

[0107] S32: data association is performed by using the Hungarian algorithm, that is, the position information of the target person is obtained by inputting the video to the detection network, and then transmitted to the tracking network of the Deepsort target tracking algorithm for data association and matching of the target person in the front and back frames of the target person, so as to obtain the tracking target.

[0108] In the step S32, the similarity is calculated by using the motion information and the appearance information of the target person, the Mahalanobis distance is used to judge the correlation degree of the predicted target person and the detected target person for the motion information, and the formula of the Mahalanobis distance is:

[0109] dd j is the position of the detection frame j; y i is the position predicted by the tracker i; d j -y i represents that the feature vector of the jth data point is subtracted from the feature vector of the ith data point to obtain a difference vector, and the difference vector represents the difference or distance between the two data points in each feature dimension; j -y i T represents that the difference vector is transposed, that is, from a row vector to a column vector, so as to facilitate matrix operation; S i is the covariance matrix of the detection and prediction positions; the Mahalanobis distance considers the uncertainty of the state measurement by calculating the standard deviation between the detection position and the average tracking position;

[0110] When the target is long-term occluded or the angle of view is dithered, the appearance information is introduced, and the problem of identity switching caused by occlusion is solved by using the cosine distance; the expression of the cosine distance is:

[0111] dd j is the detection frame d​j eigenvectors; It is the set of N nearest eigenvectors corresponding to tracker i; in this embodiment, N is set to 100; R i This is a feature vector library for appearance; to fully utilize motion and appearance information, a linear weighted summation method is used, with the formula: c i，j =λd (1) (i,j)+(1-λ)d (2) (i, j);

[0112] Where λ is the weighting parameter, this formula combines Mahalanobis distance and cosine distance, balancing their contributions by adjusting the weighting coefficient λ; its value ranges between [0, 1]; the function c i，j Used to represent the combined distance or similarity between the i-th sample and the j-th sample; by adjusting the weight parameter λ, the importance of the two distance metrics can be balanced when combining distance metrics; if and only if the metric value c i，j Existing in d (1) (i, j) and d (2) Only when there are (i, j) are the targets considered related; where " "The closest to tracker i" refers to the N frames with the highest similarity to the target features in the current frame. This requires calculating the cosine similarity between the target features in the current frame and the target features in each previous frame, and selecting the N frames with the highest similarity. In other words, the trained model is used to extract features from the j-th detection box and the i-th tracking box to obtain the apparent feature vector r. j and apparent eigenvector r i The k-th apparent feature vector of the i-th tracking box Stored in appearance feature library R i middle, Let be the cosine similarity of the appearance features between the j-th detection box and the ith tracking box, and let d be the minimum cosine distance between the appearance features. (2) The sum of (i, j) is 1; where d (2) The smaller (i, j) is, the higher the similarity of the apparent features of the tracking box and the detection box, and the greater the correlation and matching degree between the two.

[0113] S33: update the trajectory to determine the tracking result and determine the ID using Kalman filter; finally, use the Hungarian algorithm for data association to improve the tracking effect and reduce the identity switching problem; input the video to the detection network to obtain the position information of the target person, and then transmit it to the tracking network for data association and matching of the front and back frames to obtain the tracking result; Deepsort calculates the similarity using the motion information of the target; according to the result of target association, update the trajectory information of each existing pedestrian target. The Kalman filter can be used to predict and update the state of the target, and the covariance matrix between the prediction and the observation is used to estimate the position and speed of the pedestrian target. For unassociated new pedestrian targets, create a new target trajectory and assign a unique identifier to it. The trajectory of these new targets will be updated and tracked in subsequent frames. Through the above steps, Deepsort can track pedestrian targets in real time and maintain their trajectory information. In the fall detection, the results of pedestrian tracking can be combined for subsequent fall event detection, for example, by analyzing the posture and motion trajectory of the pedestrian to determine whether a fall event has occurred.

[0114] S4: posture recognition for each target person combined with trajectory information, and a spatio-temporal adaptive graph convolution network is used to extract the feature vector of the posture, and a classifier is used for human behavior classification and recognition to determine whether a fall event has occurred for the target person.

[0115] The specific steps of step S4 are:

[0116] S41 Alphapose posture recognition: use the top-down method of the Alphapose model for skeleton detection to obtain continuous skeleton frames, and then use the SPPE (single person pose estimation) algorithm for posture estimation of the detected target person to obtain the target person skeleton graph. The effect after target detection and posture estimation is shown in Figure 5, which shows that the target detection frame of the target person is obtained, and the human skeleton graph after posture estimation is obtained; Figure 5 first uses target detection and then performs posture estimation, which is a general method for human behavior recognition. The posture estimation method can provide more human information, such as joint position, posture angle, etc. In the posture estimation graph in Figure 5, these information can be used to more comprehensively analyze the human state, thereby improving the accuracy of fall event detection; in addition, the posture estimation method can capture the dynamic changes of the human body, including motion trajectory, posture conversion, etc.; this is very important for distinguishing falls from other normal behaviors, because falls are usually accompanied by unusual movements; by tracking the movement trajectory of human key points over time, the mutual relationship and continuous changes between human body parts can be obtained; during the fall, the trajectory of the key points may show an abnormal pattern, which helps to identify the fall event.

[0117] The step S41 adds a symmetric space transformation network (SSTN), a pose-guided proposal generator (PGPG) and a parametric pose non-maximum suppression (PPNMS) in the Alphapose model to realize skeleton detection; the existing skeleton model mainly has two problems: positioning error and generating redundant detection results. In view of these problems, the Alphapose model adds three modules: a symmetric space transformation network (SSTN), a pose-guided proposal generator (PGPG) and a parametric pose non-maximum suppression (PPNMS). The function of SSTN is to extract the region of interest of the human body detection box, so as to automatically adjust the detection box. Through the addition of this module, the target detection result in the above figure is more accurate. The function of PGPG is to guide the data expansion of the existing data, increase the training samples, and achieve the purpose of data enhancement; it acts on the training of target detection and SPPE; PPNMS is a kind of parametric pose non-maximum suppression method, which eliminates redundant detection boxes by defining pose distance to calculate pose similarity, and deletes redundant boxes when the similarity is lower than a certain threshold. By combining these three modules, Alphapose realizes more accurate skeleton detection; the specific steps are as follows:

[0118] S411 data preprocessing: the cropped target personnel image segment is preprocessed, including image scaling, normalization and channel order conversion and other operations, to adapt to the input requirements of the Alphapose model;

[0119] S412 multi-person pose estimation: using Alphapose to estimate the multi-person pose of the cropped pedestrian image segment; the Alphapose model detects the position of the human key points (such as head, shoulder, elbow, knee, etc.) and estimates the pose information of the target personnel;

[0120] S413 result visualization: combine the results of multi-person pose estimation with the original image or video frame, draw the connection line of key points and the information of pose angle, and get the target personnel skeleton diagram for subsequent analysis and display.

[0121] S42 fall behavior recognition: a spatio-temporal adaptive graph convolution network is used to extract features in the spatial and temporal dimensions to obtain a feature vector of the target person.

[0122] In real videos, action behaviors not only contain single frames, but also contain continuous time. Therefore, after using the Alphapose algorithm to extract the skeleton, the obtained is a continuous skeleton frame. The human skeleton frame sequence is composed of joint coordinates, and the early work of extracting skeleton features mainly gives the spatio-temporal convolutional neural network, which connects the coordinate vectors of the whole frame joints and then forms a feature vector. Although the convolutional neural network can extract spatio-temporal features, this method does not consider the natural connection between the skeleton edges of the skeleton joints and the time edge of the same joint connected between frames. In 2018, researchers proposed the ST-GCN model using the graph convolutional neural network (GCN) to process the skeleton graph. However, there are still two shortcomings in the feature extraction operation on the spatio-temporal graph: (1) In the spatial feature learning stage, the fixed spatial topology is shared among all poses, which may not be optimal for actions with large pose changes. Using a fixed spatial topology may incorrectly enhance irrelevant connections or weaken key connections, and cannot accurately represent the spatial dependency. This fact shows that the spatial topology structure should adapt to each pose in the skeleton sequence. (2) In the learning of time features, the existing method applies a time convolution with a fixed small kernel to extract short-range time features. It leads to a weak ability to model the time long-range joint dependency, which is crucial for action recognition. In order to learn a robust feature representation in the spatial and temporal dimensions, the present scheme proposes an improved ST-GCN network, a spatio-temporal adaptive graph convolutional network (STA-GCN); the spatio-temporal adaptive graph convolutional network (STA-GCN) in the step S42 is composed of a spatial adaptive graph convolution SA-GC and a temporal adaptive graph convolution TA-GC, and both the spatial adaptive graph convolution SA-GC and the temporal adaptive graph convolution TA-GC have a topological adaptive encoder TAE of embedded graph convolution; the SA-GC module extracts spatial features by modeling the spatial adaptive joint dependency. The TA-GC module aims to learn temporal features by capturing direct long-range joint dependencies in the temporal dimension. This model combined with the SA-GC and TA-GC modules can learn discriminative features in both spatial and temporal dimensions; the TAE component is used to learn the spatial adaptive topology and the temporal adaptive topology. The existing GCN method uses a fixed spatio-temporal topology. This fixed spatial topology forces each skeleton frame to adopt the same spatial topology, and the fixed temporal topology forces all trajectories to use the same temporal topology. This fixed topology structure is not sufficient to represent the joint dependency of each pose or each trajectory. Therefore, we propose TAE to solve this problem by learning spatial adaptive topology and temporal adaptive topology. The spatial adaptive topology can generate pose-specific dependencies for each frame in the skeleton sequence to learn discriminative spatial features. The temporal adaptive topology can model the direct long-range dependency between any two nodes in the trajectory graph to extract robust temporal features.

[0123] As shown in Figure 4, the specific steps of using a spatiotemporal adaptive graph convolutional network to extract features from the pedestrian skeleton map through graph convolutional layers in step S42 are as follows:

[0124] S421: Most existing studies treat human skeletal sequences as a spatiotemporal graph, extracting features through spatial graph convolution and temporal convolution. However, due to the use of fixed, small temporal convolution kernels, temporal convolution has limited ability to learn temporal features. Therefore, a more robust feature extraction operator will be introduced onto the spatiotemporal graph, first targeting the spatiotemporal graph... Graph convolution in, where ε is the set of all nodes spanning frames in the skeleton sequence. st For the set of spacetime boundaries; v nt In the spacetime diagram A node in the middle; v nt Let v represent a node in the skeleton sequence, where n is the node index and t is the time step index; therefore, v nt It is a node at time step t; v nt This may represent a position or feature in the skeleton sequence at a specific moment (time step t); n is the node index, used to uniquely identify different nodes in the spatiotemporal graph; in v nt In this context, n represents the node number, which is an integer, usually from 1 to N, where N is the total number of nodes; t usually represents a time step or time frame; it is used to represent each moment or point in time in the dataset; T represents the total number of time steps or time frames.

[0125] S422: Decompose a spatiotemporal graph into T spatial graphs spanning time and N temporal graphs spanning nodes; the spatial graphs are represented as follows: Where ε s It is a spatial edge set, denoted as a spatial adjacency matrix A. s ∈R T×N×N Where R represents the real field of the matrix, R T×N×N Represents a three-dimensional matrix whose elements belong to R, i.e., elements in the real number field; when all spatial graphs have the same spatial correlation, the spatial adjacency matrix is ​​reduced to A∈R. N×N Form; similarly, time diagrams are represented as ε t Representing a timeline The temporal edge set in ε; the temporal edge set is used to define the temporal relationship or temporal connection between nodes; specifically, ε t It includes time edges connecting nodes, used to represent the association or dependency relationship between nodes at different time steps; the time adjacency matrix is ​​A. t ∈R N×T×TAfter the decomposition of the spatio-temporal graph, the spatial graph convolution S-GC and the temporal graph convolution T-GC are developed;

[0126] The spatial graph convolution S-GC is represented as:

[0127] wherein, is an element of A s , v pt represents a node in the spatio-temporal graph, wherein p is the index of the node, t is the index of the time step, in the spatial graph convolution S-GC, it is used to represent the node in the spatial graph; W represents the weight matrix or the weight parameter, which is usually used for linear transformation or weighted aggregation of features; w(v pt ) represents the weight applied to the node v pt , which is used for weighted aggregation of input features to generate output features on the node v nt ;

[0128] The temporal graph convolution T-GC is represented as:

[0129] wherein, is an element of A t , v nq represents a node in the spatio-temporal graph, wherein n is the index of the node, q is the index of the time step, in the temporal graph convolution T-GC, it is used to represent the node in the temporal graph; W represents the weight matrix (or weight parameter), which is usually used for linear transformation or weighted aggregation of features; that is, w(v nq ) represents the weight applied to the node v nq , which is used for weighted aggregation of input features to generate output features on the node v nt ;

[0130] Further, the spatial graph convolution S-GC and the temporal graph convolution T-GC are used to extract features on the spatio-temporal graph to obtain the feature vector, wherein the extracted features are represented as:

[0131] wherein, is an element of A t ; represents the weight or connection coefficient used to connect the node v pq and the node v nq in the spatial graph convolution S-GC, which is an element of the spatial adjacency matrix A s in the spatio-temporal graph, used to measure the spatial relationship or connection strength between nodes; specifically, represents the node v pq and the node v nqSpatial connection strength between nodes, used to determine how to propagate feature information in the spatial graph; N denotes the index range of nodes, representing node v pq and node index in node v nq ; q represents the index of time step or time frame, used to represent different time steps or time frames in the time graph convolution T-GC; Specifically, q represents a time point in the time dimension of the spatio-temporal graph; v pq represents a node in the spatial graph convolution S-GC, where p is the index of the node and q is the index of the time step; W represents the weight matrix or weight parameter, used for linear transformation or weighted aggregation of features, i.e. w1(v pq ) and w2(v nq ) are both used for weighted aggregation of input features, w1(v pq ) represents the first weight or connection coefficient applied to node v pq , and w2(v nq ) represents the second weight or connection coefficient applied to node v nq ; These weight matrices are usually learnable parameters in neural network models, which are used to adjust and control the way features propagate and aggregate between different nodes to capture complex feature relationships in spatio-temporal graphs; During the training process of the model, these weight matrices will be adjusted according to the loss function to optimize the performance of the model.

[0132] S43 Human behavior classification and recognition: using the trained classifier to classify and recognize the feature vector of the target person, which will determine whether the input data belongs to the fall category according to the feature vector; The specific steps of the step S43 are:

[0133] S431: First, pre-process the feature vector to obtain a pre-processed feature vector;

[0134] The pre-processing method in step S431 includes feature normalization and dimensionality reduction; These preprocessing steps help optimize the performance of the classifier;

[0135] S432: Then select a classifier model to train the classifier using labeled training data; The classifier model selected in step S432 is support vector machine SVM; And use the labeled training data to train the classifier. The training data is a set of feature vectors with labels, where the label represents the fall behavior category corresponding to each feature vector;

[0136] S433: Then evaluate the trained classifier using unlabeled test data; The performance of the classifier is evaluated by calculating the accuracy, recall and F1 score in step S433;

[0137] S434: classify and identify the pre-processed feature vector using the trained classifier, and determine whether the input data belongs to the fall category according to the feature vector.

[0138] For those skilled in the art, the specific embodiments are only exemplary description of the present application, it is obvious that the specific implementation of the present application is not limited to the above manner, as long as the method concept and technical solution of the present application are adopted for various non-essential improvements, or the concept and technical solution of the present application are directly applied to other occasions without improvement, all within the protection scope of the present application.

Claims

1. A skeleton detection and fall detection method based on improved spatio-temporal adaptive graph convolution, characterized in that, Specifically comprising the following steps: S1: collecting image data, obtaining each frame of image data; S2: using a pre-trained yolov5 target personnel detection model to detect whether there is a target personnel in each frame of image data, if there is a target personnel, go to step S3, if there is no target personnel, end; S3: for each detected target personnel, using a Deepsort target tracking algorithm for target tracking, obtaining tracking results, and calculating similarity to obtain target association results, updating the trajectory information of each target personnel; S4: combining the trajectory information of each target personnel for posture recognition, and using a spatiotemporal adaptive graph convolution network to extract the feature vector of the posture, using a classifier to classify and identify the human behavior, and determining whether a fall event has occurred for the target personnel.

2. The improved spatio-temporal adaptive graph convolution based skeleton detection and fall detection method of claim 1, wherein, The specific steps for training the yolov5 target personnel detection model in step S2 are: S21: frame the image data collected in step S1 in a video frame extraction manner to generate a fall data set; S22: perform enhancement processing on the fall data set to obtain an enhanced data set, and divide it into a training set and a test set; S23: build a YOLOV5 algorithm model, input data and train to obtain algorithm model weights, and obtain the yolov5 target personnel detection model.

3. The improved spatio-temporal adaptive graph convolution based skeleton detection and fall detection method of claim 2, wherein, The specific steps of step S3 are: S31: use Kalman filtering to predict the position of the next frame of image of the target personnel, and then perform matching; S32: use the Hungarian algorithm for data association, that is, input the video to the detection network to obtain the position information of the target personnel, and then transmit it to the tracking network of the Deepsort target tracking algorithm for data association and matching of the target personnel in the front and back frames of the target personnel, so as to obtain the tracking target; S33: update the trajectory to determine the tracking result and determine the ID.

4. The improved spatio-temporal adaptive graph convolution based skeleton detection and fall detection method of claim 3, wherein, The similarity calculated in the step S32 utilizes the motion information and the appearance information of the target person. The Mahalanobis distance is used to determine the correlation degree between the predicted target person and the detected target person. The formula of the Mahalanobis distance is as follows: where d j is the position of the detection frame j; y i is the position predicted by tracker i; d j -y i represents subtracting the feature vector of the jth data point from the feature vector of the ith data point to obtain a difference vector, which represents the difference or distance between the two data points in each feature dimension;(d j -y i ) T represents transposing the difference vector, i.e., changing from a row vector to a column vector, to facilitate matrix operations; S i is the covariance matrix of the detection and predicted positions; when the target is long-term occluded or the viewing angle is jittered, appearance information is introduced to solve the identity switching problem caused by occlusion through a cosine distance. The expression of the cosine distance is: where r j is an eigenvector of the detection frame d j ; that is, r j represents the feature vector of the jth data point; is r j the transpose of r, where the jth data point's feature value information in each feature dimension is contained; is the set of the nearest N frames of eigenvectors corresponding to the tracker i; R i is the appearance feature vector library; using motion information and appearance information, linear weighting is used to sum up, and the formula is: c i，j = λd (1) (i,j) + (1 - λ)d (2) (i,j); where λ is a weight parameter, and its value is in the range of [0, 1]; function c i，j is used to represent the integrated distance or similarity between the ith sample and the jth sample; by adjusting the weight parameter λ, the importance of the two distance metrics can be balanced when combining the distance metrics; the target association is considered only when and if the metric value c i，j exists in d (1) (i, j) and d (2) (i, j).

5. The improved spatio-temporal adaptive graph convolution based skeleton detection and fall detection method of claim 3, wherein, The specific steps of step S4 are: S41 Alphapose posture recognition: using the top-down method of the Alphapose model to detect the skeleton, obtaining continuous skeleton frames, and then using the SPPE algorithm to estimate the posture of the detected target personnel to obtain the target personnel skeleton map; S42 fall behavior recognition: using a spatiotemporal adaptive graph convolution network to extract features in space and time dimensions to obtain the feature vector of the target personnel; S43 human behavior classification and identification: using a trained classifier to classify and identify the feature vector of the target personnel, and the classifier will determine whether the input data belongs to the fall category according to the feature vector.

6. The improved spatio-temporal adaptive graph convolution based skeleton detection and fall detection method of claim 5, wherein, In step S41, a symmetric spatial transformation network, a pose-guided sample generator, and a pose non-maximum suppressor are added to the Alphapose model to realize skeleton detection; the specific steps are: S411 data preprocessing: preprocessing the image segments of the target personnel cut out; S412 multi-person pose estimation: using Alphapose to estimate the multi-person pose of the cropped pedestrian image segment; the Alphapose model detects the position of the human key points and estimates the pose information of the target personnel; S413: The results of multi-person pose estimation are combined with the original image or video frame, the connection lines of key points and the information of pose angles are drawn, and a target person skeleton diagram is obtained for subsequent analysis and display.

7. The improved spatio-temporal adaptive graph convolution based skeleton detection and fall detection method of claim 6, wherein, The spatio-temporal adaptive graph convolution network in the step S42 is composed of a spatial adaptive graph convolution SA-GC and a temporal adaptive graph convolution TA-GC, and both the spatial adaptive graph convolution SA-GC and the temporal adaptive graph convolution TA-GC have a topological adaptive encoder TAE of an embedded graph convolution.

8. The improved spatio-temporal adaptive graph convolution based skeleton detection and fall detection method of claim 7, wherein, The specific steps of the step S42 of adopting the spatio-temporal adaptive graph convolution network to perform feature extraction on the pedestrian skeleton graph through a graph convolution layer are as follows: S421 : A feature extraction operator is introduced on the spatio-temporal graph, which firstly aims at the spatio-temporal graph graph convolution in which ε is the set of all nodes in the skeleton sequence that span a T-frame st v is the set of spatiotemporal edges; v nt is the spatiotemporal graph represents a node in the skeleton sequence; v nt represents a node in the skeleton sequence, where n is the index of the node and t is the index of the time step; thus, v nt represents a node at time step t; v nt represents a position or feature in the skeleton sequence at a certain time instant, i.e., time step t; n is the index of the node, which is used to uniquely identify different nodes in the spatio-temporal graph; in v nt , n represents the number of the node, which is an integer, usually from 1 to N, where N is the total number of nodes; t represents the time step or time frame, which is used to represent each T represents the total number of time steps or time frames; S422: decompose a spatio-temporal graph into T spatial graphs across time and N temporal graphs across nodes; the spatial graphs are denoted as where ε s is the spatial edge set, denoted as spatial adjacency matrix A s ∈R T×N×N ; where R denotes the field of real numbers, R T×N×N denotes a three-dimensional matrix whose elements belong to R, the field of real numbers; when all the spatial maps have the same spatial correlation, the spatial adjacency matrix is degraded to A e R N×N form; likewise, the temporal map is represented as ε t representation of a time graph temporal edge set; the temporal edge set is used to define a temporal relationship or a temporal connection between nodes; the temporal adjacency matrix is A t ∈R N×T×T ; after decomposition of the space-time graph, a spatial graph convolution S-GC and a temporal graph convolution T-GC are developed; the spatial graph convolution S-GC is represented as: wherein is A s the element v pt represents a node in the spatio-temporal graph, where p is the index of the node, t is the index of the time step, in the spatial graph convolution S-GC, it is used to represent the node in the spatial graph; W represents the weight matrix or the weight parameter, which is used for linear transformation or weighted aggregation of features; that is, w(v pt ) represents the weight applied to the node v pt , which is used for weighted aggregation of input features to generate output features on the node v nt ; The temporal graph convolution T-GC is represented as: wherein is A t the element v nq represents a node in the spatio-temporal graph, where n is the index of the node, and q is the index of the time step, in the temporal graph convolution T-GC, it is used to represent the node in the temporal graph; W represents the weight matrix or the weight parameter, which is used for linear transformation or weighted aggregation of features; that is, w(v nq ) represents the weight applied to the node v nq , which is used for weighted aggregation of input features to generate output features on the node v nt ; Features on the spatio-temporal graph are extracted using spatial graph convolution S-GC and temporal graph convolution T-GC to obtain a feature vector, where the extracted features are denoted as: wherein, denotes the weight or connection coefficient between nodes v pq and v nq in the spatial graph convolution S-GC, which is an element of the spatial adjacency matrix A s in the spatio-temporal graph, for measuring the spatial relationship or connection strength between nodes; in particular, represents a node v pq and a node v nq between spatial connection strength for determining how to propagate feature information in the spatial graph; N denotes the index range of nodes, representing node v pq and node index in node v nq ; q represents the index of time step or time frame, used to represent different time steps or time frames in the temporal graph convolution T-GC; p is the index of the node of the spatial graph; W represents the weight matrix or weight parameter, used for linear transformation or weighted aggregation of features, i.e. w1(v pq ) and w2(v nq ) are both used for weighted aggregation of input features; the weight matrix is a learnable parameter in the neural network model, which is used to adjust and control the way of propagation and aggregation of features between different nodes, in order to capture the complex feature relationships in the spatio-temporal graph; in the training process of the model, the weight matrix will be adjusted according to the loss function to optimize the performance of the model.

9. The improved spatio-temporal adaptive graph convolution based skeleton detection and fall detection method of claim 7, wherein, The specific steps of the step S43 are as follows: S431: First, the feature vector is preprocessed to obtain a preprocessed feature vector; S432: Then, a classifier model is selected to train the classifier using labeled training data; S433: Then, the trained classifier is evaluated using unlabeled test data; S434: The preprocessed feature vector is classified and recognized using the trained classifier, and whether the input data belongs to the falling category is determined according to the feature vector.

10. The improved spatio-temporal adaptive graph convolution based skeleton detection and fall detection method of claim 9, wherein, The preprocessing mode in the step S431 includes feature normalization and dimension reduction; the classifier model in the step S432 selects a support vector machine SVM; and the performance of the classifier is evaluated by calculating the accuracy, recall rate and F1 score in the step S433.

Images