Fall detection method, electronic device, and medium

Abstract

This application provides a fall detection method, electronic device, and medium, belonging to the field of visual detection technology. The method includes: acquiring a video segment to be detected; and obtaining a detection result based on the video segment to be detected and a trained deep learning model. The detection result is used to characterize whether a fall has occurred in the video segment to be detected, or whether a fall has not occurred in the video segment to be detected. In the technical solution provided in this application, after acquiring the video segment to be detected, the detection result is obtained through a trained deep learning model. The deep learning model is trained using a first training video with a first label and a second training video with a second label. Therefore, coarse-grained video-level labels are used during the training phase, rather than fine-grained video segment labels or frame-level labels, thereby significantly reducing the workload of video data annotation.

Description

Technical Field

[0001] This invention relates to the field of visual inspection technology, and more specifically, to a fall detection method, electronic device, and medium. Background Technology

[0002] In video-based detection of specific anomalous behaviors, deep learning methods have generally outperformed traditional hand-designed feature methods. However, most of these current methods rely on fully supervised learning to distinguish specific anomalous behaviors from other behaviors. The quality of the network model's learning is heavily influenced by the accuracy of video segment or frame labels. Fine-grained video labeling is extremely time-consuming and labor-intensive, and may not always be completely accurate, leading to difficulties in model training and hindering its transfer to other applications. Furthermore, current methods for detecting specific anomalous behaviors generally exhibit low robustness. Summary of the Invention

[0003] In view of this, one of the objectives of this application is to provide a fall detection method to solve the following problems existing in the prior art: the fine-grained annotation process of video is time-consuming and labor-intensive and the accuracy is difficult to guarantee, resulting in low model training efficiency, insufficient generalization ability, and difficulty in adapting to the needs of different application scenarios.

[0004] In a first aspect, embodiments of this application provide a fall detection method, the method comprising:

[0005] Obtain the video segment to be detected;

[0006] Based on the video segment to be detected, a detection result is obtained using a trained deep learning model. The detection result is used to characterize whether the video segment to be detected has experienced a fall or has not experienced a fall.

[0007] The deep learning model is obtained by training a deep learning model using the RFDS dataset. The RFDS dataset includes a first training video and a second training video. The first training video and the second training video are respectively labeled with a first label to represent the occurrence of a fall and a second label to represent the absence of a fall.

[0008] Furthermore, the number of the first training videos and the second training videos are equal;

[0009] The first training video includes a number of consecutive first training video segments in time, and the second training video includes a number of consecutive second training video segments in time, wherein the number of the first training video segments and the number of the second training video segments are equal.

[0010] The loss function of the deep learning model is:

[0011]

[0012] in, and These represent the first hyperparameter and the second hyperparameter, respectively.

[0013] Indicates the first The first training video The score for the fall in the first training video segment;

[0014] It indicates the first The second training video is the first The fall score in the second training video segment;

[0015]

[0016]

[0017] in, Indicates the first The first training video The feature vector of the first training video segment;

[0018] Indicates the first The second training video is the first The feature vector of the second training video segment;

[0019] This represents the weight matrix of the deep learning model;

[0020] b represents the bias term;

[0021] and They represent the first The first training video and the first The second training video.

[0022] Furthermore, the objective function of the deep learning model is:

[0023]

[0024] in, This represents the third hyperparameter.

[0025] Furthermore, the deep learning model includes a first branch and a second branch, the first branch including a first feature extraction module and a first classification detection module, and the second branch including a second feature extraction module and a second classification detection module;

[0026] The step of obtaining detection results based on the video segment to be detected and the trained deep learning model includes:

[0027] The video segment to be detected is processed into a first specified format file and a second specified format file;

[0028] The first specified format file is input into the first feature extraction module to obtain a first feature vector; the second specified format file is input into the second feature extraction module to obtain a second feature vector.

[0029] The first feature vector is input into the first classification and detection module to obtain a first fall detection score; the second feature vector is input into the second classification and detection module to obtain a second fall detection score.

[0030] The detection result is obtained based on the first feature vector, the second feature vector, the first detected fall score, and the second detected fall score.

[0031] Furthermore, obtaining the detection result based on the first feature vector, the second feature vector, the first detected fall score, and the second detected fall score includes:

[0032] Calculate the similarity between the first feature vector and the second feature vector;

[0033] The similarity is input into the weight calculation model to obtain the first weight of the first detected fall score and the second weight of the second detected fall score;

[0034] Perform a Hadamard product operation on the first weight, the second weight, the first detected fall score, and the second detected fall score to obtain the fused detected fall score;

[0035] Get the average pixel value of all frame images in the first specified format file, and the average amplitude value of the second specified format file;

[0036] A fall score threshold is determined based on the pixel mean and amplitude mean. When the fused fall score is greater than the fall score threshold, the detection result indicates that the video segment to be detected has experienced a fall. When the fused fall score is less than or equal to the fall score threshold, the detection result indicates that the video segment to be detected has not experienced a fall.

[0037] The weight calculation model is as follows:

[0038]

[0039] Indicates the first weight; Indicates the second weight; This represents the normalized similarity. Furthermore, the first specified format file and the second specified format file are in the formats of RGB data and optical flow data, respectively.

[0040] Furthermore, the RFDS dataset includes multiple training video groups, each training video group including one first training video and one corresponding second training video, wherein the scene in which the first training video is filmed is the same as the scene in which the corresponding second training video is filmed.

[0041] Furthermore, before acquiring the video segment to be detected, the method further includes:

[0042] Obtain the video to be tested;

[0043] The video to be detected is divided into multiple time-continuous video segments, wherein each video segment includes a specified number of video frames.

[0044] Secondly, embodiments of this application provide an electronic device, which includes a processor and a memory coupled to each other. The memory stores a computer program, and when the computer program is executed by the processor, the electronic device performs the above-described method.

[0045] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described above.

[0046] The invention employing the above technical solution has the following advantages:

[0047] In the technical solution provided in this application, after acquiring the video segment to be detected, the detection result is obtained through a trained deep learning model. The deep learning model is trained using a first training video with a first label and a second training video with a second label. Therefore, coarse-grained video-level labels are used in the training stage, rather than fine-grained video segment labels or frame-level labels, which significantly reduces the workload of video data annotation.

[0048] In the technical solution provided in this application, by setting a loss function, the highest fall score of the first training video with fall behavior is made as large as possible during the training process, and the highest fall score of the second training video without fall behavior is made as small as possible, thereby improving the accuracy of the deep learning model in recognizing fall behavior after training.

[0049] In the technical solution provided in this application, the video segment to be detected is processed into a first specified format file and a second specified format file, respectively. Then, a first feature vector and a second feature vector are extracted, and these are input into a first classification detection module and a second classification detection module, respectively, to obtain a first fall detection score and a second fall detection score. Finally, the final detection result is determined based on the fusion result of the first and second fall detection scores. This application integrates the detection results of two branches based on two paths, achieving the technical objective of robustly detecting fall behavior even in complex scenes. Attached Figure Description

[0050] This application can be further illustrated by the non-limiting embodiments given in the accompanying drawings. It should be understood that the following drawings only illustrate some embodiments of this application and should not be considered as limiting the scope. For those skilled in the art, other related drawings can be obtained from these drawings without any inventive effort.

[0051] Figure 1 A flowchart provided for an embodiment of this application.

[0052] Figure 2 The percentage of the total video duration during which falls occur in all the first training videos provided for embodiments of this application.

[0053] Figure 3 This is a sub-flowchart of S120 provided in an embodiment of this application.

[0054] Figure 4 The flowchart is an exemplary one provided for embodiments of this application.

[0055] Figure 5 The following are ROC curves of the method tested on different datasets, where (a) is the ROC curve from the URFD dataset, (b) is the ROC curve from the Le2i dataset, and (c) is the ROC curve from the RFDS dataset.

[0056] Figure 6 The following are the experimental results of this method on different datasets, where (a) is the experimental result from the URFD dataset, (b) is the experimental result from the Le2i dataset, and (c) is the experimental result from the RFDS dataset.

[0057] Figure 7 Here are the visual feature maps from the UR dataset, where (a) represents the RGB mode and (b) represents the optical flow mode.

[0058] Figure 8Examples of fall score curves obtained using the method of this embodiment and representative video frames are provided, where (a) represents a fall while running on a sports field, (b) represents a fall while walking on the street, (c) represents a fall while walking in a room, (d) represents a fall while riding an elevator, (e) represents a fall during indoor activities, (f) represents a fall due to not sitting properly, (g) represents a fall in a coffee shop, (h) represents a fall in a dimly lit home, and (i) represents normal activities. Detailed Implementation

[0059] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that similar or identical parts are referred to by the same reference numerals in the drawings or description. Implementations not shown or described in the drawings are forms known to those skilled in the art. In the description of this application, terms such as "first" and "second" are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0060] This application provides an electronic device that may include a processing module and a storage module. The storage module stores a computer program, which, when executed by the processing module, enables the electronic device to perform the corresponding steps in the fall detection method described below.

[0061] In this embodiment, the processing module can be an integrated circuit chip with signal processing capabilities. The processing module can be a general-purpose processor. For example, the processor can be a Central Processing Unit (CPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0062] The storage module can be, but is not limited to, random access memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, etc.

[0063] Please refer to Figure 1 This application also provides a fall detection method. The fall detection method may include the following steps:

[0064] S110, Obtain the video segment to be detected;

[0065] S120: Based on the video segment to be detected, a detection result is obtained using a trained deep learning model. The detection result is used to characterize whether a fall occurred in the video segment to be detected, or whether a fall did not occur in the video segment to be detected. The trained deep learning model is obtained by training the deep learning model based on the RFDS dataset. The RFDS dataset includes a first training video and a second training video. The first training video and the second training video are respectively labeled with a first label to characterize a fall and a second label to characterize a fall.

[0066] In the above implementation, the deep learning model is trained using a weakly supervised learning mode. The deep learning model includes a first branch and a second branch. The first branch includes a first feature extraction module and a first classification and detection module, and the second branch includes a second feature extraction module and a second classification and detection module. Both the first and second classification and detection modules can be fully connected neural networks, hereinafter referred to as FC-NN.

[0067] FC-NN is a 3-layer fully connected network. The first layer has 512 neurons, the second layer has 32 neurons, and the last layer has 1 neuron as the output layer. FC-NN uses 60% Dropout regularization between layers, the first layer uses the ReLU activation function, and the last layer (the output layer) uses the Sigmoid activation function. The output layer generates a fall regression score for each instance, ranging from 0 to 1; a higher score indicates a greater probability of a fall.

[0068] In this embodiment, the deep learning model is trained independently using the RGB data features and optical flow data features of the video. The training adopts the multi-instance learning method in weakly supervised learning, which uses weak labels to learn how to fall, that is, it uses coarse-grained video-level labels to train the model.

[0069] In this embodiment, the videos used for training in the RFDS dataset are processed into RGB data and optical flow data respectively. Specifically, the first training video is processed into a first training video of RGB data and a first training video of optical flow data, and the second training video is processed into a second training video of RGB data and a second training video of optical flow data.

[0070] RGB data is the raw video data, which is the video read in the form of RGB video frames, with the size of each video frame adjusted to... Pixels; Optical flow data is obtained by calculating the optical flow between adjacent video frames, thus converting the original video data into optical flow data. Optical flow data is an effective representation of object motion information.

[0071] RGB data is used to encode appearance information, such as static visual cues like human posture, ground environment, and furniture position. Optical flow data is used to encode motion information, such as dynamic change patterns like sudden falls or violent limb movements.

[0072] RGB data and optical flow data can be used together to capture the completeness of a movement. For example, "falling" requires the following conditions to be met simultaneously: the person's posture is tilted, the movement is rapid, and the movement is unsupported.

[0073] Videos are considered bags, and video segments within videos are considered instances. The training process provides bag labels. When labeling videos, a video is classified as either a first or second training video solely based on whether it contains a falling action. If a video contains a falling action, it is labeled as a first training video (falling video); otherwise, it is labeled as a second training video (normal video). A first training video must contain at least one positive instance (falling video segment), while all instances in a second training video are negative instances (all non-falling video segments).

[0074] In this embodiment, the first training video is a first training video, and the second training video is a second training video. The number of first training videos and second training videos is the same. The first training videos and second training videos are respectively labeled with a first annotation and a second annotation, which respectively represent the occurrence of a fall and the absence of a fall. The first training video includes several first training video segments that are consecutive in time, and the second training video includes several second training video segments. The number of first training video segments and the number of second training video segments are the same.

[0075] In this embodiment, existing video processing software can be used to segment the first training video and the second training video to obtain all the first training video segments and the second training video segments.

[0076] The deep learning model was trained using the RFDS dataset, which included the first and second training videos. The deep learning model was given a fall score between 0 and 1 for each video segment to be detected. The expectation was that the model would output a high fall score for each video segment with a fall and a low fall score for each video segment without a fall.

[0077] Therefore, the objective function for model training is designed as follows:

[0078] (1)

[0079] It is the loss function, specifically defined as follows:

[0080] (2)

[0081] in, and These represent the first hyperparameter and the second hyperparameter, respectively.

[0082] Indicates the first The first training video The score for the fall in the first training video segment;

[0083] It indicates the first The second training video is the first The fall score in the second training video segment;

[0084]

[0085]

[0086] in, Indicates the first The first training video The feature vector of the first training video segment;

[0087] Indicates the first The second training video is the first The feature vector of the second training video segment;

[0088] This represents the weight matrix of the deep learning model;

[0089] b represents the bias term;

[0090] and They represent the first The first training video and the first The second training video.

[0091] This represents the third hyperparameter.

[0092] In the public notice (2), the first item It is a squared loss, which minimizes the maximum fall score in the first training video segment (i.e., close to 1) and the maximum fall score in the second training video segment (i.e., close to 0). This design is because the first training video contains fall events, while the second training video does not.

[0093] Second item This is a sparsity constraint term. Minimizing this term minimizes the sum of fall scores for all first training video segments in the first training video and the sum of fall scores for all second training video segments in the second training video. Since all second training video segments represent normal activities (non-falling behaviors), each second training video segment should be assigned a small fall score, and the sum of fall scores for all second training video segments should be as small as possible. Furthermore, although the first training video contains fall events, these events typically occur only for a short period, and most instances in the first training video are still non-falling behaviors, thus they should be assigned small fall scores, and the sum of fall scores for all instances should also be small.

[0094] Third item This is a temporal smoothing constraint for the second training video. Minimizing this constraint minimizes the difference in fall scores between adjacent video segments (second training video segments). Because the video segments are continuous and there are no falls in the second training video, the fall scores between adjacent video segments in the second training video should exhibit smoothness. However, for the first training video, a temporal smoothing constraint is not required because falls are generally sudden.

[0095] Currently, publicly available vision-based datasets mainly include the UR Fall Detection Dataset, the Multicam Fall Detection Dataset, the FDD Fall Detection Dataset, and the Le2i Fall Detection Dataset. The UR Fall Detection Dataset is a simulated dataset containing 30 fall videos and 40 normal videos. The Multicam dataset contains 192 videos from 24 home scenes, with 8 videos of the same content but different shooting angles for each scene. The videos in the first 22 scenes all contain fall behavior, while the videos in the last 2 scenes do not. The FDD Fall Detection Dataset consists of video frames, containing a total of 22,636 images. The Le2i dataset contains 191 videos, including 143 fall videos and 48 normal videos. The videos are from four scenes: home, coffee shop, office, and lecture hall. The above dataset has the following main characteristics: 1) The total number of samples is small; 2) The number of the first training video and the second training video is unbalanced; 3) The scenes usually only include indoor living alone situations, and all falling behaviors are simulated falls; 4) The ages of the people in all video samples are not significantly different, and there are no falling samples of different age groups.

[0096] To address the limitations of the aforementioned datasets, this embodiment constructs the RFDS dataset—the Realistic RFDS dataset (RFDS). This dataset is characterized by its rich scene types, large scale, equal number of positive second training videos, diversity, and realistic fall representations. The scenes in the RFDS dataset are not limited to indoor situations but include both indoor and outdoor scenarios, such as homes, streets, hospitals, shopping malls, parks, stairwells, and offices. Furthermore, RFDS covers both single-person and complex multi-person scenarios, and includes fall cases from different age groups (including children, young adults, middle-aged adults, and the elderly).

[0097] The RFDS dataset contains multiple training video sets. Each training video set includes one first training video and one corresponding second training video. The first training video is filmed in the same scene as the corresponding second training video. Using first and second training videos with the same scene but different fall outcomes helps improve the accuracy of deep learning models in recognizing fall behavior.

[0098] In this embodiment, the RFDS dataset contains 240 videos, including 120 video samples containing falling behavior (first training videos) and 120 normal video samples without falling behavior (second training videos). All video samples in the RFDS dataset were collected from various video and news websites worldwide, and all falling behaviors are real-world falls, not simulated falls. During data collection, the first and second training videos were collected in pairs, ensuring an equal number of first and second training videos, with each first training video having a second training video collected in the same or similar scenario. The equal number of first and second training videos in the dataset, forming a corresponding comparison, helps the model learn falling behavior better. Furthermore, the video sample durations in the constructed RFDS dataset vary, ranging from a few seconds to over a minute; additionally, the percentage of falling behavior time in the first training videos also varies, with each first training video having a different percentage of falling behavior time. Figure 2 As shown.

[0099] The steps of the fall detection method will be explained in detail below:

[0100] Before S110, there is also S100, which specifically includes the following steps:

[0101] S100: Obtain the video to be detected, and divide the video to be detected into multiple time-continuous video segments, wherein each video segment includes a specified number of video frames.

[0102] In this embodiment, each video to be detected It's just a bag The video to be detected is divided into Each video segment to be detected is a video segment to be detected. It's a bag One example, here Regarding the division of the video segments to be detected, the method proposed in this embodiment is to divide the video to be detected into 32 video segments, namely... The specific division principle is as follows: Let Indicates video Total number of video frames Representing the There are 16 consecutive videos. Since a pre-trained I3D feature extractor is used (I3D extracts features based on 16 video frames as the basic unit), 16 consecutive frames from each video are taken in a non-overlapping and continuous manner as a video clip. The number of clips is denoted as _____. It is equal to:

[0103] (3)

[0104] in, for( The integer obtained. When If the number is not divisible by 16, there is a remainder. The last clip is by and The last frame in the middle ( ( ) copies are combined to obtain. Then, compare... and The size relationship between them determines whether a clip corresponds exactly to a video segment (when...). ), or multiple clips merged into one video segment (when ), or one clip corresponds to multiple video segments (when In short, it is to... Each clip is transformed into A video clip.

[0105] In this embodiment, the number of video frames in the video segment to be detected cannot be too large. This is because after step S120, the video segment in which the fall occurred can be identified, and the specific time of the fall can be determined based on the timing sequence. If the number of video frames in the video segment to be detected is too large, it will affect the accuracy of locating the time of the fall. For example, if the video segment to be detected contains N video frames, making the duration of the video segment 8 seconds, then when a fall occurs in the video segment, regardless of which video frame the fall occurs in, it can be clearly identified as occurring within these 8 seconds, thus the accuracy of determining the time of the fall is high. If the video segment to be detected contains M video frames, making the duration of the video segment 8 minutes, then when a fall occurs in the video segment, it is necessary to pinpoint the video frame in which the fall occurred to accurately determine the time of the fall. However, the deep learning model in this method is based on weakly supervised training and cannot determine the specific video frame in which the fall occurred. Therefore, the specified quantity can be determined based on the accuracy requirements for the timing of the fall.

[0106] In S120, such as Figure 3 As shown, it may include the following steps:

[0107] S121: Process the video segment to be detected into a first specified format file and a second specified format file;

[0108] S122: Input the first specified format file into the first feature extraction module to obtain the first feature vector; input the second specified format file into the second feature extraction module to obtain the second feature vector;

[0109] S123: Input the first feature vector into the first classification detection module to obtain the first fall detection score; input the second feature vector into the second classification detection module to obtain the second fall detection score;

[0110] S124: Based on the first feature vector, the second feature vector, the first detected fall score, and the second detected fall score, the detection result is obtained.

[0111] In S121, after acquiring all the video segments to be detected, the video segments to be detected are processed into a first specified format file and a second specified format file based on a preset processing model. The first specified format file and the second specified format file are in the formats of RGB data and optical flow data, respectively.

[0112] In S122, both the first and second eigenvectors can be obtained through the following methods, specifically:

[0113] The first specified format file and the second specified format file are respectively input into the first feature extraction module and the second feature extraction module to obtain the first feature vector and the second feature vector. In this embodiment, the first feature extraction module and the second feature extraction module are both dilated 3D convolutional networks.

[0114] In this embodiment, the Dilated 3D Convolutional Network (I3D model) is a publicly available, pre-trained model. In this embodiment, it is used as a feature extractor to extract 3D features from the video data. The features selected are the output of the last Inception module of the I3D network model, after average pooling. Specifically, the I3D network model extracts features using 16 consecutive frame clips as basic units. After extracting features from all video clips, the features of each video segment are obtained by calculating the average of the features from all clips contained in that video segment.

[0115] In S124, the detection result is determined based on the following method, specifically:

[0116] Calculate the similarity between the first feature vector and the second feature vector; input the similarity into the weight calculation model to obtain the first weight of the first detected fall score and the second weight of the second detected fall score; perform a Hadamard product operation on the first weight, the second weight, the first detected fall score, and the second detected fall score to obtain the fused detected fall score; obtain the pixel mean of all frame images in the first specified format file and the amplitude mean of the second specified format file; determine the fall score threshold based on the pixel mean and the amplitude mean, wherein when the fused detected fall score is greater than the fall score threshold, the detection result indicates that the video segment to be detected has experienced a fall, and when the fused detected fall score is less than or equal to the fall score threshold, the detection result indicates that the video segment to be detected has not experienced a fall.

[0117] The weight calculation model is as follows:

[0118]

[0119] Indicates the first weight; Indicates the second weight; This represents the similarity after normalization.

[0120] The similarity in this embodiment is determined based on the following formula:

[0121]

[0122]

[0123] Represents the first eigenvector. This represents the second eigenvector.

[0124] The first and second weights are calculated using a weight calculation model. According to the model, the higher the similarity, the more consistent the bi-branch features, and the equal the first and second weights. ≈ (≈0.5), the lower the similarity, the more reliable the single-branch feature, and the weights are tilted towards the high-quality branch.

[0125] In this embodiment, after obtaining the pixel mean and amplitude mean, the scene factor is calculated according to the following formula. The scene factor is used to characterize whether the corresponding image is a "complex scene that is easy to misjudge" or a "simple scene with clear features".

[0126]

[0127] I represents the scene brightness factor, which is obtained by normalizing the pixel mean.

[0128] M represents the motion intensity factor, which is obtained by normalizing the mean amplitude.

[0129] This represents the balance factor (typically 0.5, with brightness and motion each accounting for half).

[0130] Then, the fall score threshold T is calculated based on the following formula.

[0131]

[0132] This represents the adjustment coefficient, which is typically 0.2.

[0133] In this embodiment, after performing a Hadamard product operation on the first weight, the second weight, the first detected fall score, and the second detected fall score, the result is then subjected to one-dimensional Gaussian smoothing. This increases the difference in fall scores between normal video segments and fall video segments, making it easier to correctly classify and distinguish them subsequently. One-dimensional Gaussian smoothing ensures that the scores of normal video segments with low fall scores remain low or are even suppressed to be even lower, while further increasing the scores of fall video segments with high fall scores, making them significantly different from the scores of normal video segments.

[0134] Specifically, set and Representing the first The first and second fall detection scores were obtained from the videos. and Perform the Hadamard product operation to obtain the initial fall score after merging. :

[0135] (4)

[0136] Then Perform one-dimensional Gaussian smoothing to obtain the final fall score:

[0137] (5)

[0138] in This is a hyperparameter.

[0139] In this embodiment, the deep learning model includes a first branch and a second branch. The first specified format file is processed by the first branch to obtain a first fall detection score. The second specified format file is processed by the second branch to obtain a second fall detection score. Then, through post-fusion, the detection result used to characterize whether a fall has occurred is directly obtained.

[0140] In this embodiment, as Figure 4 As shown, the exemplary method of S100-S120 can be represented as follows:

[0141] First, the video to be tested is segmented into... Several consecutive video segments to be detected are processed into a first specified format file and a second specified format file, i.e., converted into RGB data and optical flow data. The RGB data and optical flow data are respectively fed into the upper branch (RGB flow) and lower branch (optical flow flow) of the bimodal network. In the upper and lower branches, the corresponding feature extractors are used to obtain the first feature vector and the second feature vector. The first feature vector is input into the corresponding trained fully connected neural network to obtain the first fall detection score. The second feature vector is input into the corresponding trained fully connected neural network to obtain the second fall detection score. The first fall detection score and the second fall detection score are fused by formula (4) and formula (5) to obtain the final fused fall detection score. When the fused fall detection score is greater than the fall score threshold, the detection result indicates that the video segment to be detected has fallen. When the fused fall detection score is less than or equal to the fall score threshold, the detection result indicates that the video segment to be detected has not fallen. In this way, the time point of the fall can be determined.

[0142] In this embodiment, the dilated 3D convolutional network performs sliding operations on the spatial and temporal dimensions of the video using 3D convolutional kernels to extract a first feature vector and a second feature vector from the input first specified format file and second specified format file. Both the first and second feature vectors are dynamic spatiotemporal features, rather than static image features. Both the first and second feature vectors are 3D feature vectors, and their specific dimensions depend on the network structure of the dilated 3D convolutional network.

[0143] In this embodiment, the aforementioned deep learning model can be Figure 4 In the bimodal network, the first branch can be Figure 4 The upper branch, the second branch can be Figure 4 In the lower branch, the first feature extraction module can be the feature extractor I3D in the upper branch, the second feature extraction module can be the feature extractor I3D in the lower branch, the first classification and detection module can be the fully connected neural network in the upper branch, and the second classification and detection module can be the fully connected neural network in the lower branch.

[0144] This embodiment evaluates the proposed method on two publicly available benchmark datasets (UR and Le2i) and a self-built dataset (RFDS). Relevant information for these three datasets is shown in Table 1. For the training and test set partitioning of each dataset, this embodiment partitions all first training videos and all second training videos in a 4:1 ratio between the training and test sets. Extensive experiments are conducted to evaluate the performance of the proposed method. The superiority of this method is verified through comparison with existing fall detection methods. Furthermore, the superiority of the proposed fusion strategy is verified by comparing various information fusion strategies. Finally, ablation experiments demonstrate the effectiveness of the proposed objective function, and the effectiveness of each term in the objective function is analyzed.

[0145] Table 1 Experimental Dataset Information

[0146]

[0147] The method proposed in this embodiment first outputs the fall score for each video segment from a bimodal network model. Then, it obtains the fall score for each video frame based on the correspondence between video segments and video frames. The fall score of a video frame is compared with a preset threshold to determine whether the video frame is a fall frame. If the fall score of a video frame is greater than the threshold, it is determined to be a fall frame; otherwise, it is a normal frame. The threshold is set based on the ROC curve to find the optimal threshold. Specifically, the ROC curves of the method obtained from experiments on the UR, Le2i, and RFDS datasets are plotted here, as shown below. Figure 5 As shown. Each point on the ROC curve corresponds to a threshold, and the point closest to the top left corner is the optimal point (i.e., Figure 5 (Points marked with small red boxes in the middle). By trying the thresholds corresponding to the optimal point and nearby points, the threshold with the best overall performance is selected as the optimal threshold in the specific experiment.

[0148] The method proposed in this embodiment is implemented using PyTorch, and the training and testing of the model are run on a single TITAN XP GPU.

[0149] To evaluate the performance of the method proposed in this embodiment, based on existing research, five evaluation metrics were used in the experiment: sensitivity, specificity, precision, accuracy, and F1 score.

[0150] (1) Sensitivity

[0151] Sensitivity, also known as recall, represents the model's ability to identify the first training video. It refers to the proportion of samples that are actually the first training video, correctly identified by the model. Its calculation formula is:

[0152] (6)

[0153] (2) Specificity

[0154] Specificity represents the model's ability to identify the second training video. It refers to the proportion of samples that are actually the second training video, which the model correctly identifies as such. The formula is as follows:

[0155] (7)

[0156] (3) Precision

[0157] Precision indicates the accuracy with which the model predicts the first training video. It represents the proportion of all videos identified as the first training video that are actually the first training video. The formula is:

[0158] (8)

[0159] (4) Accuracy

[0160] Accuracy represents the overall predictive accuracy of the model. It refers to the proportion of correct predictions out of all predictions, including both the first and second training videos. The formula for calculation is:

[0161] (9)

[0162] (5) F1 score

[0163] The F1 score strikes a balance between precision and recall, and is suitable for imbalanced class distributions. It is the harmonic mean of precision and recall, providing a comprehensive evaluation of both. Its formula is:

[0164] (10)

[0165] In the above formula, TP (True Positive) represents the number of correctly predicted first training videos, i.e., the number of correctly detected fall frames; FP (False Positive) represents the number of incorrectly predicted first training videos, i.e., the number of normal frames that were mistakenly detected as fall frames; FN (False Negative) represents the number of incorrectly predicted second training videos, i.e., the number of fall frames that were mistakenly detected as normal frames; and TN (True Negative) represents the number of correctly predicted second training videos, i.e., the number of correctly detected normal frames.

[0166] This embodiment conducted experiments on the RFDS, UR, and Le2i datasets to evaluate the performance of the method. In the RFDS dataset, the parameters in equations (4) and (5) are... They were set to 1.0 and 1×10 respectively. -3 and 9×10 -3 When integrating the fall score, the parameters in formula (3) These were set to 1.0, 1.0, and 0.7 respectively. In the UR dataset, the above-mentioned parameters were set as follows: In the Le2i dataset, the parameters in equations (4) and (5) They were set to 1.0 and 8×10 respectively. -5 and 8×10 -5 The parameters in formula (3) The values ​​were set to 1.0, 1.0, and 0.7, respectively. The optimal thresholds for the method on the RFDS, UR, and Le2i datasets were set to 0.450, 0.402, and 0.479, respectively.

[0167] The experimental results of the method proposed in this embodiment on different datasets are as follows: Figure 6 As shown in the figure, this method achieves good detection results on all datasets. As previously mentioned, compared to the UR and Le2i datasets, the self-built RFDS dataset is more complex and diverse, covering real-life fall behaviors in various life scenarios. However, even with such a challenging dataset, this method still achieves good detection results.

[0168] To further verify the effectiveness and demonstrate the superiority of the proposed method, this section compares the proposed method with current state-of-the-art methods and evaluates its performance using only RGB or optical streams, as shown in Tables 2, 3, and 4. In each table, the best results for each evaluation metric are highlighted in bold.

[0169] Table 2. Performance comparison of different methods for fall detection on the UR dataset.

[0170]

[0171] Tables 2 and 3 show the comparison results of the proposed method and existing methods on the UR and Le2i datasets, respectively. It is worth noting that the proposed method is the first one based on weakly supervised learning, while the other compared methods are based on supervised learning. Generally, supervised learning methods outperform weakly supervised learning methods. However, experimental results show that the overall performance of the proposed method is comparable to, or even better than, supervised learning methods. Furthermore, Table 4 presents the comparison results of the proposed method and existing methods on the self-built RFDS dataset. The results show that the proposed method achieves good detection results on such a complex dataset containing indoor and outdoor scenes, as well as single and multi-person scenes, outperforming other compared methods and demonstrating good robustness.

[0172] Table 3. Performance comparison of different methods for fall detection on the Le2i dataset.

[0173]

[0174] Table 4. Performance comparison of different methods for fall detection on the RFDS dataset

[0175]

[0176] In addition, it is worth emphasizing that the method proposed in this embodiment is easy to deploy and apply in practice. Because this method is a weakly supervised learning framework, compared with supervised learning, it greatly reduces the workload of data annotation in video data annotation, saves a lot of time, facilitates the use of large-scale datasets and new datasets for model training, and makes it easier to transfer the model to different application scenarios.

[0177] As shown in the experimental data in the last three rows of Tables 2 to 4, the method's performance is significantly inferior to the method in dual-stream mode when using only RGB or optical flow. This embodiment employs a dual-stream mode, utilizing dual-modal features to improve detection performance. Features are extracted from the RGB and optical flow data of the video using the I3D network model. Visual examples of these two modal features are shown in the figure below. Figure 7 As shown, they have different characteristic distributions.

[0178] For information fusion, there are generally two strategies: early fusion and late fusion. The fusion method designed in this paper belongs to late fusion. Early fusion refers to fusion at the feature level, and then the fused features are used for model training or testing. Late fusion, on the other hand, is fusion at the detection score level. That is, features from different modalities are processed by their respective models to obtain detection scores, and then the detection scores output by multiple models are fused. Early fusion usually includes feature concatenation fusion and feature superposition fusion, while late fusion usually includes averaging and maximizing.

[0179] The fusion strategy in this method was changed to different fusion methods. Experiments were conducted to compare the early-concatenate, early-addition, late-average, and late-maximum fusion strategies with the fusion strategy designed in this method. The experimental results are shown in Tables 5, 6, and 7. As can be seen from the tables, the fusion strategy designed in this method has the best detection performance among all strategies.

[0180] Table 5. Comparison results of different fusion strategies on the UR dataset.

[0181]

[0182] Table 6. Comparison results of different fusion strategies on the Le2i dataset.

[0183]

[0184] Table 7. Comparison results of different fusion strategies on the RFDS dataset.

[0185]

[0186] The objective function designed in this method underwent ablation experiments to verify the effectiveness of its various design components. Here, the first term (squared loss) in formula (5) is denoted as SL, and the second term (sparse constraint term) is denoted as S. P+NLet the third term (time series smoothing constraint term) be denoted as T. N Table 8 shows the results of ablation experiments conducted on the self-built dataset RFDS in this embodiment. The results show that on SL+S... P+N +T N Under these conditions, the method achieves the best detection results. This experiment was conducted on the self-built RFDS dataset because this dataset is more complex and can better reflect the impact of adding or removing components from the objective function.

[0187] Table 8 Ablation experimental results on the RFDS dataset

[0188]

[0189] exist Figure 8 The image shows an example of a fall score curve obtained by detecting a test video using the method of this embodiment, along with representative video frames of normal / falling conditions. The light green rectangles in the image represent the locations where falls occur, and the green curves are the fall score curves. Figure 8 As can be visually observed, the method proposed in this embodiment can indeed respond correctly to normal behavior and falls; that is, the method gives a higher fall score to the fall portion of the video and a lower fall score to the normal portion of the video.

[0190] The experimental results above show that the fall detection method proposed in this embodiment has achieved excellent detection results, and its overall performance is better than the currently compared supervised learning fall detection methods, while significantly reducing the workload of data annotation. The superior performance described above can be attributed to the following factors: (1) A target function combining the characteristics of fall detection is designed, and the loss function in the target function can widen the fall score gap between fall instances and normal instances; (2) The use of bimodal data is beneficial to improving the detection capability of the method, which is better than the method using only a single modality; (3) An effective bimodal information fusion strategy is designed in this embodiment. This strategy has been experimentally verified to be superior to other fusion strategies and has made a strong contribution to improving the detection performance of the method; (4) The method uses I3D to extract high-quality spatiotemporal features of the data. Usually, the quality of features has a great impact on the performance of the detection method; (5) The method adopts a weakly supervised learning mode based on a multi-instance learning framework, which makes the training of the model only require coarse-grained video-level labels. This greatly reduces the workload of data labeling. Therefore, the method is easier to deploy and apply in practice than the supervised fall detection method. It is convenient to use large-scale datasets or new datasets to train the model.

[0191] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the electronic device described above can be referred to the corresponding steps in the aforementioned method, and will not be elaborated further here.

[0192] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when run on a computer, causes the computer to perform the fall detection method as described in the above embodiments.

[0193] Based on the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by hardware or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, electronic device, or network device, etc.) to execute the methods of various implementation scenarios of this application.

[0194] In the embodiments provided in this application, it should be understood that the disclosed methods can also be implemented in other ways. The method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or part of code, which includes one or more executable instructions for implementing a specified logical function. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0195] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A fall detection method, characterized in that, The method includes: Obtain the video segment to be detected; Based on the video segment to be detected, a detection result is obtained using a trained deep learning model. The detection result is used to characterize whether the video segment to be detected has experienced a fall or has not experienced a fall. The deep learning model is obtained by training a deep learning model using the RFDS dataset. The RFDS dataset includes a first training video and a second training video. The first training video and the second training video are respectively labeled with a first label to represent the occurrence of a fall and a second label to represent the absence of a fall.

2. The method according to claim 1, characterized in that, The number of the first training video and the second training video are equal; The first training video includes a number of consecutive first training video segments in time, and the second training video includes a number of consecutive second training video segments in time, wherein the number of the first training video segments and the number of the second training video segments are equal. The loss function of the deep learning model is: ； in, and These represent the first hyperparameter and the second hyperparameter, respectively. Indicates the first The first training video The score for the fall in the first training video segment; It indicates the first The second training video is the first The fall score in the second training video segment; ； ； in, Indicates the first The first training video The feature vector of the first training video segment; Indicates the first The second training video is the first The feature vector of the second training video segment; This represents the weight matrix of the deep learning model; b represents the bias term; and They represent the first The first training video and the first The second training video.

3. The method according to claim 2, characterized in that, The objective function of the deep learning model is: ； in, This represents the third hyperparameter.

4. The method according to claim 1, characterized in that, The deep learning model includes a first branch and a second branch. The first branch includes a first feature extraction module and a first classification and detection module, and the second branch includes a second feature extraction module and a second classification and detection module. The step of obtaining detection results based on the video segment to be detected and the trained deep learning model includes: The video segment to be detected is processed into a first specified format file and a second specified format file; The first specified format file is input into the first feature extraction module to obtain a first feature vector; the second specified format file is input into the second feature extraction module to obtain a second feature vector. The first feature vector is input into the first classification and detection module to obtain a first fall detection score; the second feature vector is input into the second classification and detection module to obtain a second fall detection score. The detection result is obtained based on the first feature vector, the second feature vector, the first detected fall score, and the second detected fall score.

5. The method according to claim 4, characterized in that, The detection result is obtained based on the first feature vector, the second feature vector, the first detected fall score, and the second detected fall score, including: Calculate the similarity between the first feature vector and the second feature vector; The similarity is input into the weight calculation model to obtain the first weight of the first detected fall score and the second weight of the second detected fall score; Perform a Hadamard product operation on the first weight, the second weight, the first detected fall score, and the second detected fall score to obtain the fused detected fall score; Get the average pixel value of all frame images in the first specified format file, and the average amplitude value of the second specified format file; A fall score threshold is determined based on the pixel mean and amplitude mean. When the fused fall score is greater than the fall score threshold, the detection result indicates that the video segment to be detected has experienced a fall. When the fused fall score is less than or equal to the fall score threshold, the detection result indicates that the video segment to be detected has not experienced a fall. The weight calculation model is as follows: ； Indicates the first weight; Indicates the second weight; This represents the similarity after normalization.

6. The method according to claim 4, characterized in that, The first specified format file and the second specified format file are in the formats of RGB data and optical flow data, respectively.

7. The method according to claim 1, characterized in that, The RFDS dataset includes multiple training video groups, each training video group including one first training video and one corresponding second training video, wherein the scene in which the first training video is filmed is the same as the scene in which the corresponding second training video is filmed.

8. The method according to claim 1, characterized in that, Before acquiring the video segment to be detected, the method further includes: Obtain the video to be tested; The video to be detected is divided into multiple time-continuous video segments, wherein each video segment includes a specified number of video frames.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory coupled together, the memory storing a computer program that, when executed by the processor, causes the electronic device to perform the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-8.

Images