Fall behavior detection method based on YOLOv5s and ResNet-50 improvement

Abstract

The invention discloses a tumbling behavior detection method based on YOLOv5s and ResNet-50 improvement, and the method comprises the steps: S1, collecting a human body data set, forming a tumbling data set after recognition processing, dividing the tumbling data set, and marking tumbling attributes; s2, building a ResNet-50 model, training a fall data set, extracting features, and outputting a fall behavior detection model; and S3, adopting a multi-frame logic judgment strategy to perform falling behavior detection on the test set through the trained ResNet-50 falling behavior detection model, judging whether falling occurs or not according to feature changes, outputting falling attribute results of test set pictures, and performing classification according to different falling attributes. The method has the advantages of high accuracy, strong anti-interference capability and strong environmental adaptability, and can accurately identify the falling behavior under the conditions of different illumination, backgrounds and posture changes.

Description

Technical Field

[0001] The present invention belongs to the field of novel artificial intelligence technology, and in particular relates to a fall behavior detection method based on YOLOv5s and ResNet-50 improvement. Background Art

[0002] Existing fall detection methods mostly rely on sensors or video surveillance systems, which have problems such as long response time, low accuracy, and poor environmental adaptability. In addition, existing deep learning algorithms cannot effectively handle various interference factors in complex environments, such as lighting changes, obstructions, and different backgrounds. These methods are insufficient in real-time and accuracy, and are difficult to meet the needs of practical applications.

[0003] Although existing detection methods based on convolutional neural networks (CNNs) have made significant progress in the field of image recognition, they are still insufficient in terms of real-time performance and accuracy. Especially in complex environments, traditional models are not able to identify falls, which may be confused with other behaviors (such as bending, sitting, squatting, etc.), and are prone to false detection or missed detection, resulting in the inability to respond to falls in a timely and effective manner.

[0004] With the rapid development of computer vision technology, more and more people are beginning to pay attention to the research of human behavior recognition. Human behavior recognition can be widely used in security monitoring, human-computer interaction and virtual reality. Deep convolutional networks have unique advantages in image classification tasks. They integrate multi-level features and classifications in a multi-layer end-to-end manner and increase the number of features through convolution stacking. Therefore, most existing technologies use deep learning methods to detect falling behaviors. This method only requires the installation of a camera and can identify whether someone has fallen through the video.

[0005] At present, the commonly used deep learning methods include transfer learning and convolutional neural networks. Transfer learning is to apply the existing training sample features to similar models, and through the continuous increase of training samples, the captured features are rich and have specific patterns. However, most of the fall detection algorithms based on traditional computer vision methods do not use transfer learning. Some researchers proposed a parameter posture for refining posture suggestions to detect human fall behavior by combining Faster R-CNN and hourglass network models. Some researchers proposed a cascade pyramid network to detect human fall behavior using multi-scale feature maps at different levels. Chen W used OpenPose to extract human skeleton information and judged the fall state by the falling speed of the hip joint center, the angle between the center line of the human body and the ground, and the aspect ratio of the external rectangle of the human body. Miao Tai combined the YOLOv3 algorithm and the Deepsort algorithm to achieve tracking of the detection target, and applied the combination of detection and tracking targets to the field of fall detection. Some researchers applied the YOLOv5 model to fall detection and made improvements, using the K-means clustering algorithm to improve the accuracy of fall detection, and replacing the Clo U loss function with the Elo U loss function to speed up convergence. However, as the number of convolutional layers of the deep learning network increases, the ability to express image features is theoretically stronger. However, in the test, the accuracy of the training set is saturated and the network training is degraded. In addition, the feature extraction ability of fall behavior is insufficient. Fall behavior may be confused with other behaviors (such as bending, sitting, squatting, etc.), which is easy to cause false detection or missed detection, resulting in the inability to respond to fall events in a timely and effective manner.

[0006] Chinese patent document CN 110457984A discloses a pedestrian attribute recognition method based on ResNet-50 in a monitoring scenario, including the following steps: 1) selecting a public data set and a monitoring video to construct a pedestrian data set; 2) selecting a ResNet-50 convolutional neural network model as an experimental model; 3) improving the loss loss function, and proposing an N+1 class prediction method to predict N / A labels; 4) inputting the training set into the ResNet-50 convolutional neural network after the improved loss loss function to obtain a pedestrian attribute recognition model; 5) inputting the test set into the pedestrian attribute recognition model for recognition, and evaluating the experimental results of the pedestrian attribute recognition model; using the ResNet-50 network structure, deeper pedestrian attributes can be analyzed, and the relationship between attributes can be better mined. When mining the relationship between attributes, a unified multi-attribute joint learning framework is proposed to simultaneously identify multiple attributes.

[0007] Therefore, based on the above shortcomings, the convolutional neural residual network was introduced, and the convolutional neural ResNet-50 residual network was selected as the basic model. The convolutional neural residual network can solve the problem of network depth convolution fatigue, can improve the accuracy by increasing the network depth, and solve the gradient vanishing problem caused by increasing the network depth in the deep neural network. However, the ResNet-50 residual network also has shortcomings. For small and similar data sets, ResNet-50 is prone to memorizing training data rather than learning generalization capabilities due to its large number of parameters, resulting in overfitting. In the target detection task, ResNet-50 has limited detection capabilities for small targets in complex backgrounds. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide an improved fall behavior detection method based on YOLOv5s and ResNet-50, which combines the real-time detection capability of YOLOv5s and the feature extraction capability of ResNet-50 to have higher accuracy and robustness.

[0009] In order to solve the above technical problems, the technical solution adopted by the present invention is: the improved fall behavior detection method based on YOLOv5s and ResNet-50 specifically includes the following steps:

[0010] S1: Collect human data sets, form a fall data set after recognition processing, divide the fall data set, and then annotate the fall attributes;

[0011] S2: Build a ResNet-50 model, train the fall data set, extract features, and output a fall behavior detection model;

[0012] S3: Use a multi-frame logic judgment strategy to detect fall behavior on the test set through the trained ResNet-50 fall behavior detection model, determine whether it is a fall based on feature changes, output the fall attribute results of the test set images and classify them according to different fall attributes.

[0013] The above technical solution is adopted to effectively combine the real-time detection capability of YOLOv5s with the feature extraction capability of ResNet-50 to improve the detection accuracy and real-time performance of fall behavior. YOLOv5s is used to identify human objects in the data set and capture and save pictures, providing ResNet-50 with a large number of human behavior data sets with rich postures and simplifying the complex scenes of the data set. ResNet-50 is then used to solve the problem of deep convolution fatigue of the network, and extract the characteristics of human standing, bending, sitting, squatting and other behaviors and distinguish them from the characteristics of fall behavior. YOLOv5s can specify the identification objects in the input data and crop them, thereby constructing a high-quality and diverse fall behavior data set. ResNet-50 can extract and train weight files according to the different features of the input data, improving the performance of the model in image classification tasks. The real-time detection capability of YOLOv5s is combined with the feature extraction capability of ResNet-50. YOLOv5s is used to trim high-quality data sets to enhance ResNet-50's feature extraction capability for fall behavior. The feature extraction capability of ResNet-50 is used to enhance the detection performance of YOLOv5s, strengthen environmental adaptability, and detect fall behavior in complex backgrounds. Therefore, this method has strong environmental adaptability and can accurately identify fall behavior under different lighting, background and posture changes.

[0014] Preferably, the specific steps of step S1 are:

[0015] S11 collects human data sets: collects human body pictures and data of various falling behaviors and saves them in the data set;

[0016] S12YOLOv5s model detects human body: Use the YOLOv5s model to detect human attributes, obtain detection results, and then crop and save the images in the detection results to form a fall data set;

[0017] S13 divides the data set and labels it: First, clean the data. The cleaning process includes removing pictures with low resolution or size less than 50×50, and ensuring that the pictures clearly show the information of human fall behavior and do not contain images of non-human fall behavior; then calculate the number of pictures in the fall data set, and use TensorFlow's image_dataset_from_directory method to randomly divide the fall data set into training set, validation set, and test set; at the same time, shuffle the order of the pictures, and then annotate the fall attributes of the fall data set and export the label file.

[0018] The ratio of the training set, validation set, and test set is 8:1:1. Shuffling the order of the images can improve the generalization ability of the model, balance the impact of samples, and improve training efficiency. The labelImg tool is used to manually annotate the fall dataset to ensure that the target objects in each image are accurately labeled for subsequent model training and evaluation. The fall attribute labeling rules are 0-falling, 1-standing, 2-bending, 3-sitting, and 4-squatting.

[0019] In the above technical solution, in step S11, a suitable data set is searched in PaddlePaddle AI Studio to collect the download address of human image data. A script is written to batch download these images and save them in a specified local folder. At the same time, videos of multiple people performing different falling behaviors are shot, and a script is written to extract the video frame by frame. The extracted image frames will be saved as independent image files and put into the data set together. In step S12, the detect.py detection code of YOLOv5 is designed, the pre-trained YOLOv5s model weight file on the official website is loaded, and the conf-crop parameter in the detect.py detection code is adjusted to change the size of the prediction box, so that the subsequent cropping and saving of the image will not only include the human body but also lose other features except the human body. In step S13, data cleaning is to remove blurred or low-resolution images, ensure that the human fall behavior information is clearly displayed in the image, and do not contain images of non-human fall behaviors, so that the data set includes a variety of fall attributes in different scenarios that we need to detect (such as falling, standing, bending, sitting, squatting, etc.), prevent dependence during model training, improve the generalization ability of the model, balance the influence of samples, and improve training efficiency.

[0020] Preferably, the specific steps of step S12 are:

[0021] S121: First use Keras to build the YOLOv5 algorithm model, use the ResNet-50 pre-trained weights to design the training code, enhance the feature extraction capability, and improve the model's performance in image classification; that is, load the pre-trained YOLOv5s.pt weight file in detect.py and set save_crop to true, so that the images with human attributes can be cropped and saved;

[0022] S122: Input the collected human data set for detection. When the YOLOv5 algorithm model analyzes the human data set, first adjust the image size, adjust the input image to a specified size (such as 640×640) to meet the input requirements of the model; then normalize the image, normalize the image pixel value to the range of [0,1] to improve the processing efficiency of the model; then perform data enhancement; the data enhancement techniques used include flipping, rotation, brightness and contrast adjustment, etc., to improve the robustness of detection;

[0023] S123: Modify the conf-crop parameter in detect.py to increase the prediction box size, enhance the density of feature extraction when using ResNet-50 for subsequent training, and then perform image cropping. Finally, the cropped and saved images in the detection results are summarized to form a fall data set. This data set covers various categories in different scenarios, including different types of human fall behavior images, to ensure that the model can be effectively detected in a variety of environments. When cropping and saving the image, since the cropped image just contains the human body and lacks other features, it is difficult to extract the features of the fall behavior using the ResNet-50 model in the subsequent use, reducing the accuracy. Therefore, by modifying the conf-crop parameter in detect.py, the prediction box size is increased to enhance the density of feature extraction when using ResNet-50 for subsequent training. Modify the prediction box size in the detect.py code of YOLOv5 so that when the human data set is detected and cropped, the cropped image with the human body as the object not only contains human features but also has other features that can be extracted by ResNet-50, enhancing the feature extraction density of ResNet-50.

[0024] Preferably, the specific steps of step S2 are:

[0025] S21: Use Keras to build the ResNet-50 algorithm model, use the ResNet-50 pre-trained weight file, design training code, enhance feature extraction capabilities, and improve the model's performance in image classification;

[0026] S22: Design the training code for ResNet-50 and use the cosine annealing strategy for learning rate scheduling;

[0027] S23: Then input the training set image file and train it, use the ResNet-50 model to extract the features of the fall behavior image, and perform multiple iterations of training to obtain the weight file of the fall behavior recognition model; and continuously iterate and update the feature weights, and use the validation set to evaluate the model.

[0028] Preferably, the convolutional neural network of the ResNet-50 algorithm model constructed in the step S21 has two basic blocks, namely, a convolutional block and an identity block. The structures of the convolutional block and the identity block are composed of three convolutional layers, wherein the first and third convolutional layers are 1×1 convolutional layers, and the middle convolutional layer is a 3×3 convolutional layer; the convolutional block is mainly used to learn image features, and its first convolutional layer is used to reduce the number of channels and reduce computational complexity; the second convolutional layer is used to learn features; the third convolutional layer is used to restore the number of channels and keep the dimension of the feature map consistent; the convolutional block is usually used at the beginning of the ResNet-50 network to extract low-level features of the image; the first convolutional layer of the identity block is used to reduce or restore the number of channels; the second convolutional layer is used to learn features; the third convolutional layer is used to restore the number of channels; the input and output of the identity block have the same dimension, and the input is directly added to the output through a jump connection, retaining the information of the original input; ResNet-50 constructs a rectification network by stacking convolutional blocks and identity blocks. The design of these blocks makes ResNet-50 deeper and easier to train, and achieves better performance on tasks such as image classification.

[0029] Preferably, the convolutional neural network of the ResNet-50 algorithm model in step S21 adopts a batch normalization layer, and the operation of the batch normalization layer includes: first, for each channel, calculating the mean and variance of the small batch input; second, using the calculated mean and variance to normalize the small batch input; then, scaling and translating the normalized input, and adjusting it through learnable parameters; finally, performing a nonlinear transformation on the adjusted input through an activation function.

[0030] Batch normalization is a commonly used regularization technique to accelerate the training process of neural networks and improve the performance of the model. In ResNet-50, the batch normalization layer is usually applied after the convolution layer and before the activation function. The role of batch normalization is to normalize the input of each small batch so that the mean of the input is close to 0 and the variance is close to 1. This helps to alleviate the gradient vanishing and gradient exploding problems and improve the stability and convergence speed of the network.

[0031] Preferably, the training process formula of the deep learning convolutional neural ResNet-50 residual network model in step S2 is:

[0032] A (n+1) =δ n (ω n A n +b n );

[0033] Among them, A nis the output result of the current level node; A (n+1) Output result for the next node of the current level, δ n is the activation function, ω n is the network weight value of the nth layer, b n is the bias offset.

[0034] Preferably, in the step S2, the Dropout layer randomly kills neurons during the training of the convolutional neural network of the ResNet-50 algorithm model, and the BN layer calculates the mean and variance in the batch dimension; the Dropout layer does not work during verification, and the BN layer calculates the mean and variance obtained by the entire training set; the training model is composed of an RPN network and a classification network, and the RPN network loss function is used to perform training on the corresponding fall behavior type to obtain the loss value of the source image features and the extracted features; the classification network is composed of a loss function, and the classification vector dimension is changed according to the required fall behavior image features; the loss function is:

[0035]

[0036]

[0037] Among them, p i is the target probability of the ith anchor; if the iou between the ith anchor point and a real target is greater than 0.7, then If the iou between the i-th anchor point and a real target is less than 0.3, then Other cases Not participating in training; i is the offset between the predicted box and the anchor box, is the true value, N cls is the size of the training sample data, N reg is the number of anchor positions, and iou is the standard for measuring the accuracy of detecting targets in a specific dataset.

[0038] Preferably, in the step S22, when designing the training code of ResNet-50, a cosine annealing strategy is used for learning rate scheduling, wherein cosine annealing first slowly increases the learning rate and then slowly decreases the learning rate during the training process, thereby achieving more efficient training; the principle of cosine annealing is:

[0039]

[0040] Among them, i is the index of the number of times; and Respectively represent the maximum and minimum values ​​of the learning rate, defining the range of the learning rate; T cur Indicates how many epochs are currently executed; T iIndicates the total number of epochs in the i-th index. When building the ResNet-50 training model, the cosine annealing strategy is used for learning rate scheduling. Cosine annealing can slowly increase the learning rate during training, and then slowly decrease the learning rate during training, thereby achieving more efficient training.

[0041] Preferably, the specific steps of step S3 are:

[0042] S31: Design the detection code of ResNet-50, introduce multi-frame logical judgment, use the trained fall behavior detection model as the feature extractor and introduce multi-frame logical judgment, and modify the last fully connected layer of ResNet-50 to adapt to specific classification tasks;

[0043] S32: Input the test set image data and preprocess the input image so that the input image frame is reshaped into an input format suitable for ResNet-50, and then extract the fall attribute features. Through multi-frame logical judgment, the extracted fall attribute features are integrated, and the final feature representation is obtained through a simple average operation;

[0044] S33: outputting the falling behavior of each image and the detection result of classifying the images according to the falling behavior;

[0045] S34: Based on the ACC curve, LOSS curve, confusion matrix and detection results obtained from the training of the ResNet-50 fall detection model, the fall behavior detection model is evaluated.

[0046] In the feature extraction stage, the ResNet-50 convolutional neural network preprocesses the input training set human image data with label attributes, including normalization and standardization, to convert it into feature data that is easy to extract and learn before training. These preprocessing steps help optimize the performance of the fall behavior model obtained after ResNet-50 training. Introducing multi-frame logical judgment in the detection code of ResNet-50 allows ResNet-50 to integrate the features of multiple frames of images to capture temporal changes and dynamic information, so that the trained fall behavior model can judge whether the fall behavior occurs based on the analysis of feature changes in continuous frames (such as characteristic behaviors such as people squatting on the ground, sitting on the ground, bending over, etc.). Through multi-frame logical judgment, the fall behavior model can automatically detect the pictures of several adjacent frames before and after when detecting a picture, extract and analyze the feature changes of these pictures to judge whether there is a fall, and improve the accuracy of fall behavior detection. Multi-frame logical judgment can improve the robustness of the fall behavior model to noise and interference, and reduce the errors that may be caused by single-frame judgment.

[0047] The known technical contents underlying the present invention include: YOLOv5s target detection algorithm and ResNet-50 algorithm.

[0048] (1) YOLOv5 object detection algorithm

[0049] YOLOv5 is a single-stage detection algorithm improved on the basis of YOLOv4, with the characteristics of short training time, fast reasoning speed and small weight file. According to the difference in network depth and width, YOLOv5 can be divided into five models: YOLOv5n, YOLOv5s, YOLOv5m, YOLOv5l and YOLOv5x. The YOLOv5s model is relatively small, and can take into account both detection speed and detection accuracy. Therefore, the present invention performs fall behavior detection based on the YOLOv5s model. The network model of YOLOv5s consists of four parts, which are divided into input end, Backbone, Neck and output end.

[0050] The input end of YOLOv5s mainly includes three modules, namely Mosaic image enhancement, adaptive anchor frame calculation and adaptive image scaling. The Mosaic image enhancement method extracts four pictures, randomly crops and scales them, and then randomly arranges and splices them into one picture, which enhances the model's ability to identify objects of different scales, shapes and relative positions. Adaptive anchor frame calculation can automatically learn and adjust the shape and scale of the anchor frame. Adaptive image scaling can uniformly scale the collected original pictures of different lengths and widths to a standard size and then send them to the network model, solving the problem of scale change of the detection target in the detection task.

[0051] The Backbone of YOLOv5s is mainly composed of Conv module, C3 module and SPPF module. The Conv module mainly converts and extracts input features; the C3 module increases the depth of the network; and the SPPF module maintains the spatial invariance and position invariance of the input data.

[0052] The Neck of YOLOv5s is mainly composed of the FPN module and the PANet module. The FPN module is mainly used to process the features extracted by the backbone network and capture contextual information of different scales. The PANet module is mainly used to solve the fusion and information transmission problems of multi-scale features. The FPN module conveys strong semantic features from top to bottom, and the PANet module conveys strong positioning features from bottom to top. The combination of the two fuses strong semantic features and strong positioning features, enhancing the learning ability of the network.

[0053] The output of YOLOv5s is mainly composed of the Bounding box loss function and NMS non-maximum suppression. The Bounding box loss function is mainly used to measure the difference between the predicted bounding box and the true bounding box. NMS is used to process the results of target detection, eliminate redundant detection boxes, and avoid multiple predicted boxes for one target.

[0054] (2) ResNet-50 algorithm

[0055] ResNet-50 is an image classification algorithm based on deep convolutional neural network (CNN). Compared with the traditional CNN model, ResNet-50 has a deeper network structure. By introducing residual connection, it solves the gradient vanishing problem in the deep network training process and effectively improves the performance of the model.

[0056] CNN is a neural network structure specially used for image processing, which has hierarchical feature extraction capabilities. It extracts image features layer by layer by alternating between convolutional layers, pooling layers, and activation function layers, thereby achieving tasks such as image classification and detection. However, when the network structure becomes very deep, the CNN model is prone to the problem of gradient disappearance and model degradation.

[0057] Residual connection is one of the core ideas of ResNet-50. In the traditional CNN model, the information flow between network layers is sequentially passed from the previous layer to the next layer, and the output of each layer needs to be processed by the activation function. This sequential way of transmitting information is prone to the problem of gradient disappearance, especially in deep networks. ResNet-50 solves the problem of gradient disappearance by introducing residual connections in the network, allowing information to jump directly between network layers.

[0058] The basic building block in ResNet-50 is the residual block, each of which consists of two convolutional layers, which are called the main path and the shortcut connection. The convolutional layer in the main path is used to extract features, while the skip connection directly passes the input information to the output of the main path. By adding the input to the output of the main path, residual learning of the information is achieved. In addition, batch normalization and activation functions (such as ReLU) are used in each residual block to further improve the performance of the model.

[0059] The ResNet-50 network consists of multiple residual blocks, which include some additional layers such as pooling layers and fully connected layers. The structure of the entire network is very deep and has strong feature extraction capabilities. In addition, ResNet-50 uses 50 convolutional layers, which extract features from images at different sizes and depths, allowing the model to capture features at different levels.

[0060] In the residual network, the output of each layer of the network is obtained by calculating H(x) = F(x) + x, where F(x) is the residual map of this layer, x is the input of this layer, and H(x) is the output of this layer. The role of the residual map F(x) is to take the output of the previous layer as input and calculate new features through a series of convolutions and nonlinear activation functions. If an identity map layer is added after the residual map, the output of the network is equal to the input, which makes the network depth of the residual network very large without increasing the error, thereby improving the network's expressive power.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] (1) High accuracy: As a lightweight target detection model, YOLOv5s has a fast inference speed and high accuracy, and can quickly and accurately detect human targets in images or videos. In addition, combined with the deep feature extraction capability of ResNet-50, it can better capture the subtle features of falling behavior, thereby improving the accuracy of detection and reducing false detection or missed detection.

[0063] (2) Multi-scale detection: YOLOv5s supports multi-scale feature fusion, which can detect falling behavior at different scales and adapt to observations at different distances and angles.

[0064] (3) Anti-interference ability: The residual learning mechanism of ResNet-50 enhances the robustness of the model to noise and background interference, so that falling behavior can still be effectively identified in complex environments.

[0065] (4) Transfer learning capability: The pre-trained ResNet-50 model can be used to perform transfer learning using existing datasets, thus reducing training time and improving the generalization capability of the model.

[0066] (5) Scalability: This method can be combined with other sensors to enhance the accuracy and reliability of fall detection and is suitable for a variety of application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a flow chart of the improved fall behavior detection method based on YOLOv5s and ResNet-50 of the present invention;

[0068] Figure 2 It is a network model of a fall behavior detection algorithm in the improved fall behavior detection method based on YOLOv5s and ResNet-50 of the present invention;

[0069] Figure 3 The network structure diagram of ResNet-50 in the improved fall behavior detection method based on YOLOv5s and ResNet-50 of the present invention;

[0070] Figure 4 It is the YOLOv5 detection result in the improved fall behavior detection method based on YOLOv5s and ResNet-50 of the present invention;

[0071] Figure 5 It is a bar chart of the binary and five-classification detection results in the improved fall behavior detection method based on YOLOv5s and ResNet-50 of the present invention;

[0072] Figure 6 It is the binary classification detection result in the improved fall behavior detection method based on YOLOv5s and ResNet-50 of the present invention;

[0073] Figure 7 The ACC curve graph and the LOSS curve graph are obtained by ResNet-50 training in the improved fall behavior detection method based on YOLOv5s and ResNet-50 of the present invention. DETAILED DESCRIPTION

[0074] The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the protection scope of the present invention.

[0075] Example: Figure 1 As shown, the improved fall behavior detection method based on YOLOv5s and ResNet-50 specifically includes the following steps:

[0076] S1: Collect human data sets, form a fall data set after recognition processing, divide the fall data set, and then annotate the fall attributes;

[0077] The specific steps of step S1 are:

[0078] S11 collects human body data sets: collects human body pictures and data on various falling behaviors, and saves them in the data set; in step S11, finds a suitable data set in PaddlePaddle AI Studio and collects the download address of the human body picture data. Write a script to batch download these pictures and save them in a specified local folder. At the same time, shoot videos of multiple people performing different falling behaviors, and write scripts to extract the videos frame by frame. The extracted picture frames will be saved as independent picture files and put into the data set together;

[0079] S12 YOLOv5s model detects human body: Use YOLOv5s model to detect human attributes, obtain detection results, and then crop and save the pictures in the detection results to form a fall data set; in step S12, design the detect.py detection code of YOLOv5, load the pre-trained YOLOv5s model weight file on the official website, adjust the conf-crop parameter in the detect.py detection code to change the size of the prediction box, so that the subsequent cropping and saving of pictures will not only contain the human body but also lose other features except the human body;

[0080] The specific steps of step S12 are:

[0081] S121: First use Keras to build the YOLOv5 algorithm model, use the ResNet-50 pre-trained weights to design the training code, enhance the feature extraction capability, and improve the model's performance in image classification; that is, load the pre-trained YOLOv5s.pt weight file in detect.py and set save_crop to true, so that the images with human attributes can be cropped and saved;

[0082] S122: Input the collected human data set for detection. When the YOLOv5 algorithm model analyzes the human data set, first adjust the image size, adjust the input image to a specified size (such as 640×640) to meet the input requirements of the model; then normalize the image, normalize the image pixel value to the range of [0,1] to improve the processing efficiency of the model; then perform data enhancement; the data enhancement techniques used include flipping, rotation, brightness and contrast adjustment, etc., to improve the robustness of detection;

[0083] S123: Modify the conf-crop parameter in detect.py to increase the prediction box size, enhance the density of feature extraction when using ResNet-50 for subsequent training, and then perform image cropping. Finally, summarize the cropped and saved images in the detection results to form a fall data set. This data set covers various categories in different scenarios, including different types of human fall behavior images, to ensure that the model can perform effective detection in a variety of environments. When cropping and saving the image, since the cropped image just contains the human body and lacks other features, it is difficult to extract the features of the fall behavior using the ResNet-50 model in the subsequent use, reducing the accuracy. Therefore, by modifying the conf-crop parameter in detect.py, the prediction box size is increased to enhance the density of feature extraction when using ResNet-50 for subsequent training. Modify the prediction box size in the detect.py code of YOLOv5 so that when the human data set is detected and cropped, the cropped image with the human body as the object not only contains human features but also has other features that can be extracted by ResNet-50, enhancing the feature extraction density of ResNet-50;

[0084] S13 divides the data set and labels it: first clean the data. The cleaning process includes removing pictures with low resolution or size less than 50×50, and ensuring that the pictures clearly show the information of human falling behavior, and do not contain images of non-human falling behavior; then calculate the number of pictures in the fall data set, and use TensorFlow's image_dataset_from_directory method to randomly divide the fall data set into training set, validation set, and test set; at the same time, shuffle the order of the pictures, and then annotate the fall data set with fall attributes and export the label file. The division ratio of the training set, validation set, and test set is 8:1:1. Shuffling the order of pictures can improve the generalization ability of the model, balance the sample influence, and improve training efficiency. Use the labelImg tool to manually label the fall data set to ensure that the target objects in each image are accurately labeled for subsequent model training and evaluation. The fall attribute labeling rules are 0-falling, 1-standing, 2-bending, 3-sitting, 4-squatting;

[0085] The data cleaning in step S13 is to remove blurred or low-resolution images, ensure that the images clearly show the human fall behavior information, and do not contain images of non-human fall behavior, so that the data set includes a variety of fall attributes in different scenarios that we need to detect (such as falling, standing, bending, sitting, squatting, etc.), prevent dependence during model training, improve the generalization ability of the model, balance the sample impact, and improve training efficiency;

[0086] S2: Build a ResNet-50 model, train the fall data set, extract features, and output a fall behavior detection model;

[0087] The specific steps of step S2 are:

[0088] S21: Use Keras to build the ResNet-50 algorithm model, use the ResNet-50 pre-trained weight file, design training code, enhance feature extraction capabilities, and improve the model's performance in image classification;

[0089] The convolutional neural network of the ResNet-50 algorithm model constructed in the step S21 has two basic blocks, namely, a convolutional block and an identity block. The structures of the convolutional block and the identity block are composed of three convolutional layers, wherein the first and third convolutional layers are 1×1 convolutional layers, and the middle convolutional layer is a 3×3 convolutional layer; the convolutional block is mainly used to learn image features, and its first convolutional layer is used to reduce the number of channels and reduce computational complexity; the second convolutional layer is used to learn features; the third convolutional layer is used to restore the number of channels and keep the dimension of the feature map consistent; the convolutional block is usually used at the beginning of the ResNet-50 network to extract low-level features of the image; the first convolutional layer of the identity block is used to reduce or restore the number of channels; the second convolutional layer is used to learn features; the third convolutional layer is used to restore the number of channels; the input and output of the identity block have the same dimension, and the input is directly added to the output through a jump connection, retaining the information of the original input; ResNet-50 constructs a rectification network by stacking convolutional blocks and identity blocks. The design of these blocks makes ResNet-50 deeper and easier to train, and achieves better performance in tasks such as image classification; the convolutional neural network of the ResNet-50 algorithm model in step S21 adopts a batch normalization layer, and the operation of the batch normalization layer includes: first, for each channel, the mean and variance of the small batch input are calculated; second, the small batch input is normalized using the calculated mean and variance; then, the normalized input is scaled and translated, and adjusted through learnable parameters; finally, the adjusted input is nonlinearly transformed through the activation function; batch normalization is a commonly used regularization technique used to accelerate the training process of neural networks and improve the performance of the model. In ResNet-50, the batch normalization layer is usually applied after the convolution layer and before the activation function. The role of batch normalization is to normalize the input of each small batch so that the mean of the input is close to 0 and the variance is close to 1; this helps to alleviate the problems of gradient vanishing and gradient exploding, and improve the stability and convergence speed of the network; the network structure of ResNet-50 is as follows Figure 3 As shown;

[0090] S22: Design the training code for ResNet-50 and use the cosine annealing strategy for learning rate scheduling;

[0091] The training process formula of the deep learning convolutional neural ResNet-50 residual network model in step S2 is:

[0092] A (n+1) =δ n (ω n A n +b n );

[0093] Among them, A n is the output result of the current level node; A (n+1) Output result for the next node of the current level, δ n is the activation function, ω n is the network weight value of the nth layer, b n is the bias offset;

[0094] In the step S2, during the training of the convolutional neural network of the ResNet-50 algorithm model, the Dropout layer randomly kills neurons, and the BN layer calculates the mean and variance in the batch dimension; during verification, the Dropout layer does not work, and the BN layer calculates the mean and variance obtained by the entire training set; the training model is composed of an RPN network and a classification network, and the RPN network loss function is used to perform training on the corresponding fall behavior type to obtain the loss value of the source image feature and the extracted feature; the classification network is composed of a loss function, and the classification vector dimension is changed according to the required fall behavior image feature; the loss function is:

[0095]

[0096]

[0097]

[0098] Among them, p i is the target probability of the ith anchor; if the iou between the ith anchor point and a real target is greater than 0.7, then If the iou between the i-th anchor point and a real target is less than 0.3, then Other cases Not participating in training; i is the offset between the predicted box and the anchor box, is the true value, N cls is the size of the training sample data, N reg is the number of anchor positions, and iou is the standard for measuring the accuracy of detecting targets in a specific dataset;

[0099] When designing the training code of ResNet-50 in step S22, the cosine annealing strategy is used to schedule the learning rate, wherein the cosine annealing first slowly increases the learning rate and then slowly decreases the learning rate during the training process, thereby achieving more efficient training; the principle of cosine annealing is:

[0100]

[0101] Among them, i is the index of the number of times; and Respectively represent the maximum and minimum values ​​of the learning rate, defining the range of the learning rate; T cur Indicates how many epochs are currently executed; T i Indicates the total number of epochs in the i-th index. When building the ResNet-50 training model, the cosine annealing strategy is used for learning rate scheduling. Cosine annealing can slowly increase the learning rate during training, and then slowly decrease the learning rate during training, thereby achieving more efficient training;

[0102] S23: Then input the training set image file and train it, use the ResNet-50 model to extract the features of the fall behavior image, and perform multiple iterations of training to obtain the weight file of the fall behavior recognition model; and continuously iterate and update the feature weights, and use the validation set to evaluate the model; Figure 2 A network model for the fall behavior detection algorithm;

[0103] S3: Use the multi-frame logic judgment strategy to perform fall behavior detection on the test set through the trained ResNet-50 fall behavior detection model, determine whether it is a fall based on feature changes, output the fall attribute results of the test set images and classify them according to different fall attributes;

[0104] The specific steps of step S3 are:

[0105] S31: Design the detection code of ResNet-50, introduce multi-frame logical judgment, use the trained fall behavior detection model as the feature extractor and introduce multi-frame logical judgment, and modify the last fully connected layer of ResNet-50 to adapt to specific classification tasks;

[0106] S32: Input the test set image data and preprocess the input image so that the input image frame is reshaped into an input format suitable for ResNet-50, and then extract the fall attribute features. Through multi-frame logical judgment, the extracted fall attribute features are integrated, and the final feature representation is obtained through a simple average operation;

[0107] S33: outputting the falling behavior of each image and the detection result of classifying the images according to the falling behavior;

[0108] S34: Evaluation of the fall behavior detection model based on the ACC curve, LOSS curve, confusion matrix and detection results obtained from the training of the ResNet-50 fall detection model;

[0109] In the feature extraction stage, the ResNet-50 convolutional neural network preprocesses the input training set human image data with label attributes, including normalization and standardization, to convert it into feature data that is easy to extract and learn before training. These preprocessing steps help optimize the performance of the fall behavior model obtained after ResNet-50 training. Introducing multi-frame logical judgment in the detection code of ResNet-50 allows ResNet-50 to integrate the features of multiple frames of images to capture temporal changes and dynamic information, so that the trained fall behavior model can judge whether the fall behavior occurs based on the analysis of feature changes in continuous frames (such as characteristic behaviors such as people squatting on the ground, sitting on the ground, bending over, etc.). Through multi-frame logical judgment, the fall behavior model can automatically detect the pictures of several adjacent frames before and after when detecting a picture, extract and analyze the feature changes of these pictures to judge whether there is a fall, and improve the accuracy of fall behavior detection. Multi-frame logical judgment can improve the robustness of the fall behavior model to noise and interference, and reduce the errors that may be caused by single-frame judgment. The trained fall behavior model is loaded into the ResNet-50 detection code and the test set image data is input for detection. The fall behavior model will determine which type of fall behavior the person has based on the features of the person in the image (falling, standing, bending, sitting, squatting). The evaluation results of the fall behavior model can be obtained through the ACC curve and LOSS curve obtained by ResNet-50 training (such as Figure 7 As shown in the figure, as well as the detection and recognition results are used to measure the performance of the model.

[0110] Most existing studies use YOLOv5s to directly train people's falling behaviors. This training method often has deficiencies in real-time and accuracy. It cannot effectively handle various interference factors in complex environments, such as lighting changes, occlusions, different backgrounds, etc. It has insufficient recognition ability for falling behaviors, which easily leads to false detection or missed detection, resulting in failure to respond to falling events in a timely and effective manner, and it is difficult to meet the needs of practical applications. As shown in Table 1, the fall data set is trained with YOLOv5 and ResNet-50 respectively, and the trained model is used to detect 100 fall test pictures. The fall test pictures and the detection results are compared to check whether the results are correct. It is found that the detection accuracy of the model trained with YOLOv5 is 82%, while the detection accuracy of the model trained with ResNet-50 is 91%. Among them, the target detection model of YOLOv5 only divides the falling behavior into two categories: falling and normal for training and detection. It is easy to misdetect the behavior one second before the fall (not yet falling) as falling, resulting in low detection accuracy. Figure 4 As shown; the target detection model of ResNet-50 divides the falling behavior into five categories: falling, standing, bending, sitting, and squatting for training detection, and refines the characteristics of each different falling behavior to improve the detection accuracy. Therefore, the present invention adopts the ResNet-50 residual network model for training.

[0111] Table 1 Comparison of YOLOv5 and ResNet-50 detection results

[0112] Model YOLOv5 ResNet-50 Positive inspection 82 91 False detection 18 9 Accuracy 82％ 91％

[0113] However, most of the existing research on using the ResNet-50 model for training only divides feature attributes into two categories, namely, features with extraction and features without extraction. Figure 5 The following is a bar chart of the detection results of using the trained model to test 100 images after training the fall dataset using two-category and five-category classification. The two-category classification only divides the fall behavior into normal and fall, while the five-category classification divides the fall behavior into fall, standing, bending over, sitting, and squatting. The fall test images and the detection results were compared to check whether the results were correct. It was found that the detection accuracy of the model trained using two-category classification was 85%, while the detection accuracy of the model trained using five-category classification was 95%. Among them, the ResNet-50 using two-category classification is easy to misdetect the behavior of sitting on the ground as fall because the behavioral characteristics of sitting on the ground are similar to those of falling, resulting in low detection accuracy. Figure 6 Therefore, the fall attribute is changed from two categories to five categories, which enriches the features that can be extracted by the subsequent training of the ResNet-50 residual network model and improves the detection accuracy of the fall behavior model.

[0114] Therefore, according to the five-class fall attributes, multi-frame logic judgment is introduced when designing the ResNet-50 detection code. When multi-frame logic judgment is introduced, ResNet-50 will integrate the features of multi-frame images to capture the changes in time and dynamic information, so that the trained fall behavior model can judge whether the fall behavior occurs by analyzing the feature changes in continuous frames (such as characteristic behaviors such as people squatting on the ground, sitting on the ground, bending over, etc.). If the fall behavior is divided into only two categories, no fall and fall, the fall behavior model will have the problem of false detection or missed detection when detecting the fall behavior. So the present invention divides the fall behavior into five categories, namely fall, stand, bend over, sit down, squat. Through multi-frame logic judgment, the fall behavior model can automatically detect the pictures of several adjacent frames before and after when detecting a picture, extract and analyze the feature changes of these pictures to judge whether to fall, and improve the accuracy of fall behavior detection. Multi-frame logic judgment can improve the robustness of the fall behavior model to noise and interference, and reduce the error that may be caused by single-frame judgment.

[0115] For ordinary technicians in this field, the specific embodiments are only illustrative descriptions of the present invention. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned methods. As long as various non-substantial improvements are made using the method concepts and technical solutions of the present invention, or the concepts and technical solutions of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. An improved fall behavior detection method based on YOLOv5s and ResNet-50, characterized in that: The specific steps include: S1: Collect human data sets, form a fall data set after recognition processing, divide the fall data set, and then annotate the fall attributes; S2: Build a ResNet-50 model, train the fall data set, extract features, and output a fall behavior detection model; S3: Use a multi-frame logic judgment strategy to detect fall behavior on the test set through the trained ResNet-50 fall behavior detection model, determine whether it is a fall based on feature changes, output the fall attribute results of the test set images and classify them according to different fall attributes.

2. The improved fall behavior detection method based on YOLOv5s and ResNet-50 according to claim 1, characterized in that: The specific steps of step S1 are: S11 collects human data sets: collects human body images and data of various falling behaviors and saves them in the data set; S12YOLOv5s model detects human body: Use the YOLOv5s model to detect human attributes, obtain detection results, and then crop and save the images in the detection results to form a fall data set; S13 divides the data set and labels it: first clean the data, the cleaning process includes removing pictures with low resolution or size less than 50×50, then calculate the number of pictures in the fall data set, and divide the fall data set into a training set, a validation set, and a test set; then annotate the fall attributes of the fall data set and export the label file.

3. The improved fall behavior detection method based on YOLOv5s and ResNet-50 according to claim 2, characterized in that: The specific steps of step S12 are: S121: First build the YOLOv5 algorithm model, load the pre-trained YOLOv5s.pt weight file in detect.py and set save_crop to true; S122: input the collected human body data set for detection. When the YOLOv5 algorithm model analyzes the human body data set, the image size is first adjusted, the image is normalized, and then data enhancement is performed; S123: Modify the parameters to increase the prediction box size, enhance the density of feature extraction during subsequent ResNet-50 training, and then perform image cropping. Finally, summarize the cropped and saved images in the detection results to form a fall data set.

4. The improved fall behavior detection method based on YOLOv5s and ResNet-50 according to claim 2, characterized in that: The specific steps of step S2 are: S21: Use Keras to build the ResNet-50 algorithm model and use the ResNet-50 pre-trained weight file; S22: Design the training code for ResNet-50 and use the cosine annealing strategy for learning rate scheduling; S23: Then input the training set image file and train it, use the ResNet-50 model to extract the features of the fall behavior image, and perform multiple iterations of training to obtain the weight file of the fall behavior recognition model; and continuously iterate and update the feature weights, and use the validation set to evaluate the model.

5. The improved fall behavior detection method based on YOLOv5s and ResNet-50 according to claim 4, characterized in that: The convolutional neural network of the ResNet-50 algorithm model constructed in the step S21 has two basic blocks, which are divided into a convolutional block and an identity block. The structures of the convolutional block and the identity block are composed of three convolutional layers, wherein the first and third convolutional layers are 1×1 convolutional layers, and the middle convolutional layer is a 3×3 convolutional layer; the convolutional block is used to learn image features, and its first convolutional layer is used to reduce the number of channels and reduce the computational complexity; the second convolutional layer is used to learn features; the third convolutional layer is used to restore the number of channels and keep the dimension of the feature map consistent; the convolutional block is used at the beginning of the ResNet-50 network to extract low-level features of the image; the first convolutional layer of the identity block is used to reduce or restore the number of channels; the second convolutional layer is used to learn features; The third convolutional layer is used to restore the number of channels; the input and output of the identity block have the same dimension, and the input is directly added to the output through a jump connection, retaining the information of the original input; ResNet-50 constructs a rectification network by stacking convolutional blocks and identity blocks.

6. The improved fall behavior detection method based on YOLOv5s and ResNet-50 according to claim 4, characterized in that: The convolutional neural network of the ResNet-50 algorithm model in step S21 adopts a batch normalization layer, and the operation of the batch normalization layer includes: first, for each channel, calculating the mean and variance of the small batch input; second, using the calculated mean and variance to normalize the small batch input; then, scaling and translating the normalized input, and adjusting it through learnable parameters; finally, performing a nonlinear transformation on the adjusted input through an activation function.

7. The improved fall behavior detection method based on YOLOv5s and ResNet-50 according to claim 7, characterized in that: The training process formula of the deep learning convolutional neural ResNet-50 residual network model in step S2 is: A (n+1) =d n (oh n A n +b n ); Among them, A n is the output result of the current level node; A (n+1) Output result for the next node of the current level, δ n is the activation function, ω n is the network weight value of the nth layer, b n A is the bias offset. n It is the output result of the current level node.

8. The improved fall behavior detection method based on YOLOv5s and ResNet-50 according to claim 4, characterized in that: In the step S2, during the training of the convolutional neural network of the ResNet-50 algorithm model, the Dropout layer kills neurons, and the BN layer calculates the mean and variance in the batch dimension; during verification, the Dropout layer does not work, and the BN layer calculates the mean and variance obtained by the entire training set; the training model is composed of an RPN network and a classification network, and the RPN network loss function is used to perform training on the corresponding fall behavior type to obtain the loss value of the source image feature and the extracted feature; the classification network is composed of a loss function, and the classification vector dimension is changed according to the required fall behavior image feature; the loss function is: Where pi is the target probability of the ith anchor; if the iou between the ith anchor and a real target is greater than 0.7, then If the iou between the i-th anchor point and a real target is less than 0.3, then Other cases Not participating in training; i is the offset between the predicted box and the anchor box, is the true value, N cls is the size of the training sample data, N reg is the number of anchor positions, and iou is the standard for measuring the accuracy of detecting targets in the dataset.

9. The improved fall behavior detection method based on YOLOv5s and ResNet-50 according to claim 8, characterized in that: When designing the training code of ResNet-50 in step S22, the cosine annealing strategy is used to schedule the learning rate, wherein the cosine annealing first increases the learning rate and then decreases the learning rate during the training process. The principle of cosine annealing is: Among them, i is the index of the number of times; and Respectively represent the maximum and minimum values ​​of the learning rate, defining the range of the learning rate; T cur Indicates how many epochs are currently executed; T i Indicates the total number of epochs in the i-th index.

10. The improved fall behavior detection method based on YOLOv5s and ResNet-50 according to claim 8, characterized in that: The specific steps of step S3 are: S31: Design the detection code of ResNet-50, introduce multi-frame logical judgment, use the trained fall behavior detection model as the feature extractor and introduce multi-frame logical judgment, and modify the last fully connected layer of ResNet-50 to adapt to specific classification tasks; S32: Input the test set image data and preprocess the input image so that the input image frame is reshaped into an input format suitable for ResNet-50, and then extract the fall attribute features. Through multi-frame logical judgment, the extracted fall attribute features are integrated, and the final feature representation is obtained through an average operation; S33: outputting the falling behavior of each image and the detection result of classifying the images according to the falling behavior; S34: Based on the ACC curve, LOSS curve, confusion matrix and detection results obtained from the training of the ResNet-50 fall detection model, the fall behavior detection model is evaluated.

Images