Fall detection method based on feedback fusion and label distribution

Abstract

The invention discloses a fall detection method based on feedback fusion and label distribution, and belongs to the technical field of target detection. According to the invention, the problem of low fall detection precision caused by unreasonable reference frame setting, small feature information loss and fuzzy label distribution is solved. According to the invention, by improving the label box clustering method, the reference box is set more reasonably, and the obtained reference box better fits the fall detection data set; a feedback channel is added after convolution operation to ensure that small feature information is not lost; and dynamic label optimal allocation is realized from a global perspective by using a label allocation thought of minimizing total cost, so that the tumble detection precision is improved. The method can be applied to the field of fall detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular relates to a fall detection method based on feedback fusion and label allocation. Background Art

[0002] According to statistics, by the end of 2023, the number of elderly people aged 65 and above in my country will account for 13.5% of the total population, of which more than half are empty-nest left-behind elderly people. According to data from the Centers for Disease Control and Prevention, falls are one of the main causes of injury or even death for people aged 65 and above in my country. On average, 4 out of every 10 elderly people have fallen, and more than half of the falls occur at home. It is urgent to detect falls in time to avoid secondary injuries caused by the elderly being unable to call for help due to no one around or pain. At present, research on fall detection is mainly divided into two directions: sensor perception and computer vision. Sensor-based devices are mainly divided into wearable devices and environmental deployment devices. Wearable sensors give false alarms when the human body moves with a large amplitude, and wearable sensors cause inconvenience to daily actions. Environmental deployment devices extract features through background difference method, calculate the width and height ratio of the human anchor frame to determine whether a fall has occurred, and have strong environmental limitations. In the field of computer vision, target detection algorithms are mainly divided into three categories: traditional target detection algorithms, candidate frames combined with deep learning classification, and regression methods based on deep learning. The YOLO-based object detection algorithm can recognize the entire image during training and testing, and its overall performance on the COCO dataset with an input image size of 640*640 is very ideal. However, the image size of the fall detection dataset is not fixed. In the YOLO object detection network based on the reference frame (anchor), firstly, the rationality of the reference frame setting greatly affects the detection performance of the final model. In order to ensure the accuracy of object detection, it is important to re-cluster the appropriate reference frame for the dataset; secondly, the convolution layer and the pooling layer can reduce the image features to achieve data dimensionality reduction, but it may cause the loss of small feature information, thereby affecting the detection accuracy; thirdly, label allocation is a key step in the object detection training process. When the target frames overlap, it will cause a reference frame to be a positive sample of multiple label frames (ground-truth, gt). At this time, the label allocation will be ambiguous and cause detection errors. Therefore, it is very necessary to solve the problem of low fall detection accuracy caused by unreasonable reference frame setting, small feature information loss, and ambiguous label allocation. Summary of the invention

[0003] The purpose of the present invention is to solve the problem of low fall detection accuracy caused by unreasonable reference frame setting, loss of small feature information and fuzzy label assignment, and propose a fall detection method based on feedback fusion and label assignment.

[0004] The technical solution adopted by the present invention to solve the above technical problems is: a fall detection method based on feedback fusion and label allocation, the method specifically comprising the following steps:

[0005] Step 1: Obtain a fall image training dataset;

[0006] Step 2: Add a label frame to each fall image in the training data set, cluster the added label frames, and use the label frames corresponding to each cluster center as a reference frame;

[0007] Step 3: Use the fall images with labeled frames and the obtained reference frames to train the improved YOLOv5 network, and the trained improved YOLOv5 network is used to perform fall detection on the image to be detected;

[0008] The improved YOLOv5 network is a network obtained by replacing the C3 convolution module in the original YOLOv5 network with a C3-F module, wherein the C3-F module includes a first Conv layer, a first BN layer, a first Silu activation function layer, a first branch, a second branch, a second Conv layer, a second BN layer, and a second Silu activation function layer;

[0009] The working process of C3-F module is:

[0010] The input of the C3-F module is recorded as x. In the C3-F module, the input x is used as the input of the first Conv layer;

[0011] The output of the first Conv layer is then used as the input of the first BN layer;

[0012] The output of the first BN layer is then used as the input of the first Silu activation function layer;

[0013] Perform Split on the output of the first Silu activation function layer, dividing the output of the first Silu activation function layer into two parts of features;

[0014] The first part of the features obtained by segmentation is used as the input of the first branch, the second part of the features obtained by segmentation is used as the input of the second branch, and the output of the first branch is fused with the output of the second branch;

[0015] The fusion result is used as the input of the second Conv layer;

[0016] The output of the second Conv layer is used as the input of the second BN layer;

[0017] The output of the second BN layer is used as the input of the second Silu activation function layer;

[0018] The output of the second Silu activation function layer is used as the output of the C3-F module.

[0019] Furthermore, the fall images in the training data set include fall images acquired from a known data set and fall images obtained through simulation.

[0020] Furthermore, the first branch includes n Bottleneck units;

[0021] In the first branch, the input of the first branch is used as the input of the first Bottleneck unit;

[0022] Then use the output of the first Bottleneck unit as the input of the second Bottleneck unit;

[0023] Then use the output of the second Bottleneck unit as the input of the third Bottleneck unit;

[0024] And so on, the output of the n-1th Bottleneck unit is used as the input of the nth Bottleneck unit;

[0025] The outputs of the 1st Bottleneck unit to the nth Bottleneck unit are fused to obtain a fusion result P1, and the fusion result P1 is used as the output of the first branch.

[0026] Furthermore, the second branch includes a Conv layer, a BN layer and a Silu activation function layer;

[0027] In the second branch, the input of the second branch is used as the input of the Conv layer;

[0028] Then use the output of the Conv layer as the input of the BN layer;

[0029] Then use the output of the BN layer as the input of the Silu activation function layer;

[0030] The output of the Silu activation function layer is the output of the second branch.

[0031] Furthermore, clustering the added label frames is specifically performed as follows:

[0032] Step 21: Randomly select a label box from all the added label boxes as the first initial clustering center;

[0033] Step 22: Set k=2;

[0034] Step 2: Determine the kth initial cluster center based on the first k-1 initial cluster centers;

[0035] Step 24: Determine whether k=K is satisfied;

[0036] If satisfied, all K initial cluster centers are obtained, and then the final cluster center is determined based on the clustering algorithm and the initial cluster centers;

[0037] If not satisfied, set k=k+1 and return to step 2 and 3.

[0038] Furthermore, the kth initial cluster center is determined according to the first k-1 initial cluster centers, and the specific process is as follows:

[0039] (1) When k = 2, the distance between each label box of the non-initial cluster center and the first initial cluster center is calculated respectively:

[0040]

[0041] Among them, a′1 is the first initial cluster center, x1 is the width of the first initial cluster center, y1 is the height of the first initial cluster center, and x i is the width of the label box of the i-th non-initial cluster center, y i is the height of the label box of the i-th non-initial cluster center, D(a′1,a i ) is the distance between the label box of the i-th non-initial cluster center and the first initial cluster center;

[0042] Then the probability that the label box of the i-th non-initial cluster center is selected as the k-th initial cluster center is:

[0043]

[0044] Among them, P(a i ) is the probability that the label box of the i-th non-initial cluster center is selected as the k-th initial cluster center, A is the set of all label boxes of non-initial cluster centers, a i is the label box of the i-th non-initial cluster center in set A;

[0045] The label box with the highest probability of being selected is taken as the kth initial cluster center;

[0046] (2) When k>2, the distance between each label box of the non-initial cluster center and each initial cluster center is calculated respectively:

[0047]

[0048] Among them, a′ j is the jth initial cluster center, j = 1, 2, ..., k-1, x j is the width of the jth initial cluster center, y j is the height of the jth initial cluster center, xi is the width of the label box of the i-th non-initial cluster center, y i is the height of the label box of the i-th non-initial cluster center, D(a′ j ,a i ) is the distance between the label box of the i-th non-initial cluster center and the j-th initial cluster center;

[0049] Then the probability that the label box of the i-th non-initial cluster center is selected as the k-th initial cluster center is:

[0050]

[0051] Among them, P(a i ) is the probability that the label box of the i-th non-initial cluster center is selected as the k-th initial cluster center, D′(a′ j ,a i ) is the sum of the distances between the label frame of the i-th non-initial cluster center and each initial cluster center, A is the set of all label frames of the non-initial cluster center, a i is the label box of the i-th non-initial cluster center in set A;

[0052] The label box with the highest probability will be selected as the kth initial cluster center.

[0053] Furthermore, in step 3, the improved YOLOv5 network is trained using the fall images with label frames and the obtained reference frames. The specific process of label allocation during the training process is as follows:

[0054] Define the label box as the positive supplier, the background as the negative supplier, the prediction box as the demand side, record the number of positive suppliers as m, and the number of demand sides as n;

[0055] Each positive supplier provides k′ positive labels s l =k′,l=1,2,...,m,s l is the number of positive labels provided by the lth positive supplier, and each demander needs a label d p =1,p=1,2,...,n,d p is the number of labels required by the pth demander, then the number of negative labels provided by the negative supplier is nm×k′;

[0056] The cost of a positive supplier passing a label to a demander is recorded as The cost of a negative supplier passing a label to the demander is recorded as according to and The total cost C incurred by the positive and negative suppliers in delivering labels to the demander is calculated, and the label delivery scheme that minimizes the total cost C is obtained to achieve initial label allocation.

[0057] Furthermore, the cost of the positive supplier passing a label to the demander for:

[0058]

[0059] Among them, θ is the model parameter, and is the classification and bounding box corresponding to the p-th prediction box generated by the model, and is the classification and bounding box corresponding to the lth positive supplier, L cls and L reg Represent the cross entropy loss and IoU loss respectively, α is the balance coefficient, It is the cost of the lth positive supplier delivering the label to the pth demander.

[0060] Furthermore, the cost incurred by the negative supplier in delivering a label to the demander for:

[0061]

[0062] in, represents the category of negative suppliers, is the classification corresponding to the p′th prediction box generated by the model, It is the cost incurred by the negative supplier for delivering the label to the demander.

[0063] Furthermore, the initial label allocation result is processed to obtain a final label allocation result, and the specific process of the processing is:

[0064] For the label boxes g1, g2, ..., g q In the case of sharing a prediction box, the label boxes g1, g2, ..., g are calculated respectively. q The total cost in the process of transferring labels is used to select the label box with the smallest total cost. The label of the shared prediction box is the same as the label of the selected label box.

[0065] The beneficial effects of the present invention are:

[0066] The present invention improves the label frame clustering method to make the reference frame setting more reasonable, and the obtained reference frame is more in line with the fall detection data set. A feedback channel is added after the convolution operation to ensure that small feature information is not lost. The label allocation idea of ​​minimizing the total cost is used to realize dynamic label allocation from a global perspective, thereby improving the fall detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a structural diagram of the YOLOv5 network in the present invention;

[0068] In the figure, the CBS module includes Conv layer, BN layer and Silu activation function layer;

[0069] Figure 2 is a flow chart of the clustering algorithm in the present invention;

[0070] Figure 3 This is the structure diagram of the C3 module in the original YOLOv5 network;

[0071] Figure 4 It is a structural diagram of the C3-F module in the improved YOLOv5 network;

[0072] Figure 5 It is a hardware design structure diagram of the present invention;

[0073] Figure 6 It is a physical connection diagram;

[0074] Figure 7 This is a comparison chart of the fall detection Precision before and after the improvement in the present invention;

[0075] Figure 8 This is a comparison diagram of fall detection recall before and after improvement in the present invention;

[0076] Fig. 9 This is a comparison chart of mAP@0.5 of fall detection before and after the improvement in the present invention;

[0077] Fig.10 This is a schematic diagram of the confusion matrix before improvement;

[0078] Fig.11 It is a schematic diagram of the improved confusion matrix;

[0079] Fig.12 This is a schematic diagram of the test results Figure 1 ;

[0080] Fig.13 This is a schematic diagram of the test results Figure 2 . DETAILED DESCRIPTION

[0081] Specific implementation method 1: A fall detection method based on feedback fusion and label allocation described in this implementation method specifically includes the following steps:

[0082] Step 1: Obtain a fall image training dataset;

[0083] Step 2: Add a label frame to each fall image in the training data set, cluster the added label frames, and use the label frames corresponding to each cluster center as a reference frame (anchor);

[0084] Step 3: Use the fall images with labeled frames and the obtained reference frames to train the improved YOLOv5 network, and the trained improved YOLOv5 network is used to perform fall detection on the image to be detected;

[0085] The improved YOLOv5 network is a network obtained by replacing the C3 convolution module in the original YOLOv5 network with a C3-F module, wherein the C3-F module includes a first Conv layer, a first BN layer, a first Silu activation function layer, a first branch, a second branch, a second Conv layer, a second BN layer, and a second Silu activation function layer;

[0086] The working process of C3-F module is:

[0087] The input of the C3-F module is recorded as x. In the C3-F module, the input x is used as the input of the first Conv layer;

[0088] The output of the first Conv layer is then used as the input of the first BN layer;

[0089] The output of the first BN layer is then used as the input of the first Silu activation function layer;

[0090] Perform Split on the output of the first Silu activation function layer, and divide the output of the first Silu activation function layer into two parts of features (the two parts of features are the same);

[0091] The first part of the features obtained by segmentation is used as the input of the first branch, the second part of the features obtained by segmentation is used as the input of the second branch, and the output of the first branch is fused with the output of the second branch;

[0092] Use the fusion result as the input of the second Conv layer;

[0093] The output of the second Conv layer is used as the input of the second BN layer;

[0094] The output of the second BN layer is used as the input of the second Silu activation function layer;

[0095] The output of the second Silu activation function layer is used as the output of the C3-F module.

[0096] The structure diagram of the original YOLOv5 network is as follows Figure 1 As shown, the present invention utilizes C3-F module Figure 3The structure of the C3 convolution module in the improved C3-F module is shown in the figure below. Figure 4 shown.

[0097] Since the image size and ratio contained in the fall detection dataset are not fixed, in order to ensure the accuracy of target detection, it is extremely important to cluster different reference frame sizes and ratios from the training data to better adapt to the distribution of targets in the dataset. At the same time, when detecting small targets, in order to prevent the loss of feature information, it is necessary to timely feedback the results of each convolution operation for concat fusion. When the target frames overlap, the loss function value is minimized according to the minimum cost transportation strategy to achieve the global optimal dynamic allocation of labels. For the requirements of using YOLO to detect fall behavior, it is very important to reasonably set the size and ratio of the reference frame, feedback and fuse small feature information, and complete the global optimal dynamic allocation of labels to improve the detection accuracy of fall behavior.

[0098] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that the fall images in the training data set include fall images obtained from a known data set and fall images obtained through simulation.

[0099] The other steps and parameters are the same as those in the first embodiment.

[0100] Specific implementation method three: This implementation method is different from specific implementation methods one or two in that the first branch includes n Bottleneck units;

[0101] In the first branch, the input of the first branch is used as the input of the first Bottleneck unit;

[0102] Then use the output of the first Bottleneck unit as the input of the second Bottleneck unit;

[0103] Then use the output of the second Bottleneck unit as the input of the third Bottleneck unit;

[0104] And so on, the output of the n-1th Bottleneck unit is used as the input of the nth Bottleneck unit;

[0105] The outputs of the 1st Bottleneck unit to the nth Bottleneck unit are fused to obtain a fusion result P1, and the fusion result P1 is used as the output of the first branch.

[0106] The other steps and parameters are the same as those in the first or second embodiment.

[0107] In the first branch of the present invention, each Bottleneck unit contains two channels, one channel passes the extracted features to the next Bottleneck unit for further extraction, and the other channel is fused with the low-level features through the feedback channel to ensure that the feature information after each passing through the Bottleneck unit will not be lost, and the output of the first branch is concat-fused with the output of the second branch to ensure that the feature information processed by the first branch is not ignored, and the deep features and shallow features are completely fused from top to bottom, ensuring that small feature information is not lost after the convolution operation, thereby improving the accuracy of fall detection. Moreover, after being processed by several Bottleneck units, data dimensionality reduction can be achieved, the amount of calculation can be reduced, and the operation speed can be improved.

[0108] Specific implementation method 4: This implementation method is different from any one of specific implementation methods 1 to 3 in that the second branch includes a Conv layer, a BN layer and a Silu activation function layer;

[0109] In the second branch, the input of the second branch is used as the input of the Conv layer;

[0110] Then use the output of the Conv layer as the input of the BN layer;

[0111] Then use the output of the BN layer as the input of the Silu activation function layer;

[0112] The output of the Silu activation function layer is the output of the second branch.

[0113] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0114] Specific implementation method five: Combination Figure 2 This embodiment is different from the first to fourth embodiments in that the added label frames are clustered as follows:

[0115] Step 21: Randomly select a label box from all the added label boxes as the first initial clustering center;

[0116] Step 22: Set k=2;

[0117] Step 2: Determine the kth initial cluster center based on the first k-1 initial cluster centers;

[0118] Step 24: Determine whether k=K is satisfied;

[0119] If satisfied, all K initial cluster centers are obtained, and then the final cluster center is determined based on the clustering algorithm and the initial cluster centers (the clustering adopts the K-Means algorithm);

[0120] If not satisfied, set k=k+1 and return to step 2 and 3.

[0121] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0122] Specific implementation method 6: This implementation method is different from any one of the specific implementation methods 1 to 5 in that the kth initial clustering center is determined based on the first k-1 initial clustering centers. The specific process is as follows:

[0123] (1) When k = 2, the distance between each label box of the non-initial cluster center and the first initial cluster center is calculated respectively:

[0124]

[0125] Among them, a′1 is the first initial cluster center, x1 is the width of the first initial cluster center, y1 is the height of the first initial cluster center, and x i is the width of the label box of the i-th non-initial cluster center, y i is the height of the label box of the i-th non-initial cluster center, D(a′1,a i ) is the distance between the label box of the i-th non-initial cluster center and the first initial cluster center;

[0126] Then the probability that the label box of the i-th non-initial cluster center is selected as the k-th initial cluster center is:

[0127]

[0128] Among them, P(a i ) is the probability that the label box of the i-th non-initial cluster center is selected as the k-th initial cluster center, A is the set of all label boxes of non-initial cluster centers, a i is the label box of the i-th non-initial cluster center in set A;

[0129] The label box with the highest probability of being selected is taken as the kth initial cluster center;

[0130] (2) When k>2, the distance between each label box of the non-initial cluster center and each initial cluster center is calculated respectively:

[0131]

[0132] Among them, a′ j is the jth initial cluster center, j = 1, 2, ..., k-1, x j is the width of the jth initial cluster center, y j is the height of the jth initial cluster center, x i is the width of the label box of the i-th non-initial cluster center, yi is the height of the label box of the i-th non-initial cluster center, D(a′ j ,a i ) is the distance between the label box of the i-th non-initial cluster center and the j-th initial cluster center;

[0133] Then the probability that the label box of the i-th non-initial cluster center is selected as the k-th initial cluster center is:

[0134]

[0135]

[0136] Among them, P(a i ) is the probability that the label box of the i-th non-initial cluster center is selected as the k-th initial cluster center, D′(a′ j ,a i ) is the sum of the distances between the label frame of the i-th non-initial cluster center and each initial cluster center, A is the set of all label frames of the non-initial cluster center, a i is the label box of the i-th non-initial cluster center in set A;

[0137] The label box with the highest probability will be selected as the kth initial cluster center.

[0138] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0139] The selection of cluster centers by the method of the present invention is not random, and the cluster center selection method of the present invention can improve the success rate of clustering and the convergence speed of the algorithm.

[0140] The present invention clusters the labeled boxes with annotations and clusters the sizes of the labeled boxes. Each cluster in the clustering result represents the size of a reference box (anchor). When an unlabeled picture is input (in the test process after the model training is completed), the model will first generate many prediction boxes (i.e., the reference boxes after size transformation) at a certain position. The redundant prediction boxes can be filtered out through NMS (non-maximum suppression), and then the model algorithm is used to finally select the label and bounding box with the highest probability as the result output.

[0141] Specific implementation method 7: This implementation method is different from any one of specific implementation methods 1 to 6 in that, in step 3, the improved YOLOv5 network is trained using each fall image with a labeled frame and the obtained reference frame. The specific process of label allocation during the training process is as follows:

[0142] Input each fall image with a labeled frame into the improved YOLOv5 network. For any target position in each fall image, a reference frame of each size is generated at the target position. The generated reference frame is then used through the RPN model (translation and scaling operation) to generate a prediction frame of the target position (which can be understood as a reference frame after size transformation).

[0143] Then use NMS (non-maximum suppression) to filter the generated prediction boxes (filter the prediction boxes with too much overlap) to obtain the remaining prediction boxes, and then assign labels based on the label boxes and the remaining prediction boxes.

[0144] Define the label box as the positive supplier, the background as the negative supplier, the prediction box as the demand side, record the number of positive suppliers as m, and the number of demand sides as n;

[0145] Each positive supplier provides k′ positive labels s l =k′,l=1,2,...,m,s l is the number of positive labels provided by the lth positive supplier, and each demander needs a label d p =1,p=1,2,...,n,d p is the number of labels required by the pth demander. In the standard optimal allocation problem, the number of suppliers and demanders is the same, then the number of negative labels provided by the negative supplier is nm×k′;

[0146] The cost of a positive supplier passing a label to a demander is recorded as The cost of a negative supplier passing a label to the demander is recorded as according to and The total cost C incurred by the positive and negative suppliers in delivering labels to the demander is calculated, and the label delivery scheme that minimizes the total cost C is obtained to achieve initial label allocation.

[0147] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0148] The dynamic label allocation method of the present invention enables the network to learn the optimal label allocation method by itself by minimizing the cost, eliminating the need for manual parameter selection to achieve label allocation, and at the same time improves the detection accuracy of the model.

[0149] Specific implementation eight: This implementation differs from one of specific implementations one to seven in that the positive supplier delivers a tag to the demander at a cost of for:

[0150]

[0151] Among them, θ is the model parameter, and is the classification and bounding box corresponding to the p-th prediction box generated by the model, and is the classification and bounding box corresponding to the lth positive supplier, L cls and L reg Represent the cross entropy loss and IoU loss respectively, α is the balance coefficient, It is the cost of the lth positive supplier delivering the label to the pth demander.

[0152] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0153] Specific implementation method 9: This implementation method is different from the specific implementation methods 1 to 8 in that the negative supplier transfers a label to the demander. for:

[0154]

[0155] in, represents the category of negative suppliers, is the classification corresponding to the p′th prediction box generated by the model, It is the cost incurred by the negative supplier for delivering the label to the demander.

[0156] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0157] Specific implementation method 10: This implementation method is different from the specific implementation methods 1 to 9 in that the initial label allocation result is processed to obtain the final label allocation result, and the specific process of the processing is:

[0158] For the label boxes g1, g2, ..., g q Sharing a prediction box (the value of q is greater than or equal to 2, g1, g2, ..., g q Represent the first, second, ... qth label frames sharing a prediction frame respectively), and calculate the label frames g1, g2, ..., g q The total cost in the process of transferring labels is used to select the label box with the smallest total cost. The label of the shared prediction box is the same as the label of the selected label box.

[0159] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.

[0160] Experimental part

[0161] The falling process is divided into three postures: up (standing), bending (bending, squatting), and down (lying down, falling). The fall detection dataset contains about 4,000 images, of which 3,400 are indoor images. In order to enhance the universality of the model, about 550 images with complex backgrounds and portraits are captured from the Internet and used as fall detection data for various scenes.

[0162] The experimental environment configuration is given in Table 1, and the dataset is trained and tested according to the environment shown in Table 1.

[0163] Table 1 Experimental environment configuration

[0164]

[0165] First, update the NVIDIA driver and install Python 3.8 in the virtual environment created in Anaconda. Second, select the corresponding CUDA version. The experimental training environment consists of the Windows 11 operating system and Python 3.8. NVIDIA discrete graphics cards are used for computing.

[0166] Precision, Recall and mAP@0.5 were selected as evaluation indicators to obtain the training curve, the curve was analyzed and verified, and the YOLOv5 network was improved. The improved YOLOv5 network was compared with the original YOLOv5 network. Table 2 shows the comparison of fall detection indicators before and after the improvement of the YOLOv5 network.

[0167] Table 2 Comparison of fall detection indicators before and after improvement

[0168]

[0169] Precision comparison before and after improvement Figure 7 As shown in Figure 2, the Recall comparison before and after the improvement is as follows: Figure 8 As shown in Figure 2. The mAP@0.5 comparison before and after the improvement is as follows: Fig. 9 As shown in the figure. Before the improvement, Precision and Recall converged to 90.60% and 91.77%, respectively. After the improvement, Precision and Recall converged to 93.48% and 94.15%, respectively, which were improved by 2.88% and 2.38%, respectively. mAP@0.5 converged to 94.77% before the improvement, and converged to 96.19% after the improvement, which was improved by 1.42%. The confusion matrix before and after the improvement is shown in the figure. Fig.10 and Fig.11As shown in the confusion matrix, it can be seen that the improved method improves the detection accuracy of the "down" label from 79.0% to 83.0%, a total increase of 5.0%, thereby improving the overall accuracy of fall detection. The schematic diagrams of the detection results are shown in Fig.12 and Fig.13 shown.

[0170] Table 3 shows the comparison of the indicators of the method of the present invention and other fall detection methods.

[0171] Table 3 Comparison of different fall detection methods

[0172]

[0173]

[0174] It can be seen that the method of the present invention outperforms other methods in performance, with Precision increased by 5.83%, Recall increased by 5.64%, and mAP@0.5 increased by 3.63%. This shows that the fall detection method based on feedback fusion and optimal allocation model of the present invention has good target detection performance and provides an effective guarantee for accurately detecting fall behavior.

[0175] Build a hardware platform for fall detection testing. The hardware design structure diagram and physical connection diagram are as follows: Figure 5 and Figure 6 The webcam captures the subject's behavior in real time, and the collected video or image data is input into the improved model for processing and analysis. When the elderly person falls, the corresponding string is recognized and communicated with the microcontroller, so that the buzzer sounds a loud alarm to remind people nearby that the elderly person may fall.

[0176] In the image test, a total of 273 images were randomly selected as input, 191 images were identified as "up", 49 images were identified as "bending", 33 images were identified as "down", 227 images were accurately identified, and the detection accuracy reached 83.15%.

[0177] In the real-time camera test, the camera was fixed in the corridor and the person being tested simulated three postures: up, bending, and down. During the test, efforts were made to ensure that the entire body of the person being tested was not blocked. The average confidence level of the detection exceeded 85%, and the effect was ideal.

[0178] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A fall detection method based on feedback fusion and label assignment, characterized in that: The method specifically comprises the following steps: Step 1: Obtain a fall image training dataset; Step 2: Add a label frame to each fall image in the training data set, cluster the added label frames, and use the label frames corresponding to each cluster center as a reference frame; Step 3: Use the fall images with labeled frames and the obtained reference frames to train the improved YOLOv5 network, and the trained improved YOLOv5 network is used to perform fall detection on the image to be detected; The improved YOLOv5 network is a network obtained by replacing the C3 convolution module in the original YOLOv5 network with a C3-F module, wherein the C3-F module includes a first Conv layer, a first BN layer, a first Silu activation function layer, a first branch, a second branch, a second Conv layer, a second BN layer, and a second Silu activation function layer; The working process of C3-F module is: The input of the C3-F module is recorded as x. In the C3-F module, the input x is used as the input of the first Conv layer; The output of the first Conv layer is then used as the input of the first BN layer; The output of the first BN layer is then used as the input of the first Silu activation function layer; Perform Split on the output of the first Silu activation function layer, dividing the output of the first Silu activation function layer into two parts of features; The first part of the features obtained by segmentation is used as the input of the first branch, the second part of the features obtained by segmentation is used as the input of the second branch, and the output of the first branch is fused with the output of the second branch; The fusion result is used as the input of the second Conv layer; The output of the second Conv layer is used as the input of the second BN layer; The output of the second BN layer is used as the input of the second Silu activation function layer; The output of the second Silu activation function layer is used as the output of the C3-F module.

2. A fall detection method based on feedback fusion and label allocation according to claim 1, characterized in that: The fall images in the training data set include fall images acquired from a known data set and fall images obtained through simulation.

3. A fall detection method based on feedback fusion and label allocation according to claim 1, characterized in that: The first branch includes n Bottleneck units; In the first branch, the input of the first branch is used as the input of the first Bottleneck unit; Then use the output of the first Bottleneck unit as the input of the second Bottleneck unit; Then use the output of the second Bottleneck unit as the input of the third Bottleneck unit; And so on, the output of the n-1th Bottleneck unit is used as the input of the nth Bottleneck unit; The outputs of the 1st Bottleneck unit to the nth Bottleneck unit are fused to obtain a fusion result P1, and the fusion result P1 is used as the output of the first branch.

4. A fall detection method based on feedback fusion and label allocation according to claim 1, characterized in that: The second branch includes a Conv layer, a BN layer and a Silu activation function layer; In the second branch, the input of the second branch is used as the input of the Conv layer; Then use the output of the Conv layer as the input of the BN layer; Then use the output of the BN layer as the input of the Silu activation function layer; The output of the Silu activation function layer is the output of the second branch.

5. A fall detection method based on feedback fusion and label allocation according to claim 1, characterized in that: The clustering of the added label frames is specifically as follows: Step 21: Randomly select a label box from all the added label boxes as the first initial clustering center; Step 22: Set k=2; Step 2: Determine the kth initial cluster center based on the first k-1 initial cluster centers; Step 24: Determine whether k=K is satisfied; If satisfied, all K initial cluster centers are obtained, and then the final cluster center is determined based on the clustering algorithm and the initial cluster centers; If not satisfied, set k=k+1 and return to step 2 and 3.

6. A fall detection method based on feedback fusion and label allocation according to claim 5, characterized in that: The kth initial cluster center is determined according to the first k-1 initial cluster centers, and the specific process is as follows: (1) When k = 2, the distance between each label box of the non-initial cluster center and the first initial cluster center is calculated respectively: Among them, a′1 is the first initial cluster center, x1 is the width of the first initial cluster center, y1 is the height of the first initial cluster center, and x i is the width of the label box of the i-th non-initial cluster center, y i is the height of the label box of the i-th non-initial cluster center, D(a′1,a i ) is the distance between the label box of the i-th non-initial cluster center and the first initial cluster center; Then the probability that the label box of the i-th non-initial cluster center is selected as the k-th initial cluster center is: Among them, P(a i ) is the probability that the label box of the i-th non-initial cluster center is selected as the k-th initial cluster center, A is the set of all label boxes of non-initial cluster centers, a i is the label box of the i-th non-initial cluster center in set A; The label box with the highest probability of being selected is taken as the kth initial cluster center; (2) When k>2, the distance between each label box of the non-initial cluster center and each initial cluster center is calculated respectively: Among them, a′ j is the jth initial cluster center, j = 1, 2, ..., k-1, x j is the width of the jth initial cluster center, y j is the height of the jth initial cluster center, x i is the width of the label box of the i-th non-initial cluster center, y i is the height of the label box of the i-th non-initial cluster center, D(a′ j ,a i 0 is the distance between the label box of the i-th non-initial cluster center and the j-th initial cluster center; Then the probability that the label box of the i-th non-initial cluster center is selected as the k-th initial cluster center is: Among them, P(a i ) is the probability that the label box of the i-th non-initial cluster center is selected as the k-th initial cluster center, D′(a′ j ,a i ) is the sum of the distances between the label frame of the i-th non-initial cluster center and each initial cluster center, A is the set of all label frames of the non-initial cluster center, a i is the label box of the i-th non-initial cluster center in set A; The label box with the highest probability will be selected as the kth initial cluster center.

7. A fall detection method based on feedback fusion and label allocation according to claim 6, characterized in that: In the step 3, the improved YOLOv5 network is trained using the fall images with label frames and the obtained reference frames. The specific process of label allocation during the training process is: Define the label box as the positive supplier, the background as the negative supplier, the prediction box as the demand side, record the number of positive suppliers as m, and the number of demand sides as n; Each positive supplier provides k′ positive labels s l =k′,l=1,2,...,m,s l is the number of positive labels provided by the lth positive supplier, and each demander needs a label d p =1,p=1,2,...,n,d p is the number of labels required by the pth demander, then the number of negative labels provided by the negative supplier is nm×k′; The cost of a positive supplier passing a label to a demander is recorded as The cost of a negative supplier passing a label to the demander is recorded as according to and The total cost C incurred by the positive and negative suppliers in delivering labels to the demander is calculated, and the label delivery scheme that minimizes the total cost C is obtained to achieve initial label allocation.

8. A fall detection method based on feedback fusion and label allocation according to claim 7, characterized in that: The cost of the positive supplier passing a label to the demander for: Among them, θ is the model parameter, and is the classification and bounding box corresponding to the p-th prediction box generated by the model, and is the classification and bounding box corresponding to the lth positive supplier, L cls and L reg Represent the cross entropy loss and IoU loss respectively, α is the balance coefficient, It is the cost of the lth positive supplier delivering the label to the pth demander.

9. A fall detection method based on feedback fusion and label allocation according to claim 8, characterized in that: The negative supplier passes a label to the demand side of the cost for: in, represents the classification of negative suppliers, is the classification corresponding to the p′th prediction box generated by the model, It is the cost incurred by the negative supplier for delivering the label to the demander.

10. A fall detection method based on feedback fusion and label allocation according to claim 7, characterized in that: The initial label allocation result is processed to obtain the final label allocation result. The specific process of the processing is as follows: For the label boxes g1, g2, ..., g q In the case of sharing a prediction box, the label boxes g1, g2, ..., g are calculated respectively. q The total cost in the process of transferring labels is used to select the label box with the smallest total cost. The label of the shared prediction box is the same as the label of the selected label box.

Images