A fall detection method and system in a complex scene, an electronic device, and a medium

Abstract

The application discloses a fall detection method and system in a complex scene, electronic equipment and a medium. The method comprises the following steps: S1, obtaining a detection frame region containing a human body in a video frame; S2, establishing a VGG-BiNLSTM fall behavior detection model; the VGG-BiNLSTM fall behavior detection model comprises a VGG network and a BiNLSTM network; the BiNLSTM network is constructed by improving an LSTM network, embedding a preset storage unit of the LSTM network into an original LSTM network storage unit, and constructing an NLSTM network storage unit; S3, using the VGG network to extract features of the detection frame region containing the human body in the video frame, so as to capture a feature vector of a human behavior; and sending the feature vector into the VGG-BiNLSTM fall behavior detection model, using time sequence information between continuous frames, and realizing detection and identification of a fall behavior. The method can solve the technical problem of low generalization of a traditional deep learning fall detection mode, and realizes accurate detection of a pedestrian fall behavior.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, in particular to a fall detection method and system in a complex scene, an electronic device and a medium. BACKGROUND

[0002] With the continuous acceleration of the aging process of society, the fall of the elderly has become a common phenomenon. Therefore, a fall detection technology, which can quickly and accurately detect the fall behavior of the elderly, is of great significance to reduce the risk and harm of the fall of the elderly. At present, the fall detection technology is mainly realized based on video monitoring system and sensor equipment. The video monitoring system usually uses computer vision algorithm to extract pedestrian features and detect fall behavior. The sensor equipment detects human posture and action by measuring acceleration and angular velocity and other data, and then identifies the fall event. These technologies still have some problems such as false detection and missed detection to some extent.

[0003] In recent years, with the development of deep learning technology, the fall detection technology based on deep network has gradually attracted attention. The deep network has strong adaptive learning ability and representation ability, which can learn and extract features from a large amount of data, and realize efficient detection of fall behavior. In the existing deep learning framework, the model learns at the training end. Once trained, the model is fixed and does not change in the test end or network inference process. In this way, the model has no automatic adaptation ability in the test field, which leads to the problem of generalization. SUMMARY

[0004] In view of the above-mentioned shortcomings of the prior art, the present application provides a fall detection method and system in a complex scene, an electronic device and a medium to solve the technical problem of low generalization of the traditional deep learning fall detection method, and realize accurate detection of pedestrian fall behavior.

[0005] To achieve the above effects, the technical scheme of the present application is as follows:

[0006] In a first aspect, the present application provides a fall detection method in a complex scene, comprising the following steps:

[0007] S1: obtaining a detection frame region containing a human body in a video frame;

[0008] S2: establishing a VGG-BiNLSTM fall behavior detection model: the VGG-BiNLSTM fall behavior detection model comprises a VGG network and a BiNLSTM network; the BiNLSTM network is constructed by: improving the LSTM network, embedding the preset storage unit of the LSTM network into the original LSTM network storage unit, and constructing a NLSTM network storage unit;

[0009] S3: using VGG network to extract features of the detection frame region containing human body in the video frame to capture feature vectors of human behavior; sending the feature vectors into the VGG-BiNLSTM fall behavior detection model to realize detection and identification of the fall behavior by using the time sequence information between continuous frames.

[0010] Further, step 1 further comprises: performing rectangular frame labeling on the detection frame region of the human body by using a Labelimg labeling tool to obtain an XML format file corresponding to the target category and the rectangular frame coordinates.

[0011] Further, the VGG network is a VGG16 network.

[0012] Further, the LSTM network storage unit is as formula (1), and the state update of the LSTM network storage unit is as formula (2):

[0013] (1)

[0014] In the formula, x t represents an input vector at t moment, x t x t represents an input vector at t moment, h t-1 h t-1 represents a state vector of a hidden layer at t-1 moment, W xi and W hi respectively, x t and h t-1 corresponding weight matrix, b i is a bias vector; and sigma represents a sigmod function, i t y t represents an output vector of an update gate in each storage unit at t moment, f t y f represents an output vector of a forget gate at t moment, o t y o represents an output vector of an output gate at t moment. g t and c t is an intermediate calculation variable at t moment.

[0015] (2)

[0016] In the formula, x t represents an input vector at t moment, x t represents an input vector at t moment, h t-1 represents a state vector of a hidden layer at t-1 moment.

[0017] Further, the improved LSTM network nests the preset storage unit of the LSTM network into the original LSTM network storage unit to construct an NLSTM network storage unit; specifically:

[0018] The LSTM network storage unit is used to replace the calculation equation in the original LSTM model to obtain the NLSTM network storage unit; the outer NLSTM network storage unit is referred to as an external storage unit, and the internal NLSTM network storage unit is referred to as an internal storage unit; in the original LSTM network storage unit, the update of the state of the storage unit is shown in formula (3):

[0019] (3)

[0020] In the NLSTM network storage unit, the update of the state of the storage unit is replaced by the internal storage unit, wherein and are used as short-term memory and long-term memory inputs respectively; the structure of the internal storage unit is shown in formula (4):

[0021] (4)

[0022] In the formula, and are weight matrices corresponding to and respectively, is a bias vector; represents a sigmod function, represents an output vector of the update gate in each storage unit at time t, represents an output vector of the forgetting gate at time t, represents an output vector of the output gate at time t; and are intermediate calculation variables at time t;

[0023] Due to the existence of the internal storage, the update mode of the state of the storage unit is changed to formula (5):

[0024] (5). Further, the BiNLSTM network comprises an Input layer, a Forword layer, a backword layer and an Output layer connected in sequence;

[0025]

[0026] ​​​The Forward layer includes 6 shared weights; forward calculation is performed in the Forward layer from the starting time to time t, the output of each time of the Forward layer is obtained and saved; backward calculation is performed in the Backward layer from time t to the starting time, the output of each time of the Backward layer is obtained and saved; the corresponding output results of each time of the Forward layer and the Backward layer are fused to obtain the output, as formula (6) to formula (10);

[0027] (6)

[0028] (7)

[0029] (8)

[0030] (9)

[0031] (10)

[0032] In the formula, W i (i=1,…,6) is a weight when one unit layer is transmitted to another unit layer, x t is a feature vector obtained by extracting behavior features from a video frame through a VGG layer, h is a feature sequence (…, x t-1 , x t , x t+1 ,…) input into an NLSTM network storage unit, h ’ is a feature sequence unit (…, x t+1 , x t , x t-1 ,…) input into an NLSTM, o t is output data of the feature vector after passing through a BiNLSTM network; , , , represents a bias of the BiNLSTM network structure, and respectively represent the results of the external storage unit and the internal storage unit of the NLSTM network on the processing of the feature vector output by the VGG layer; formula (10) obtains the feature vector O t as the corresponding time and The average value is output.

[0033] Furthermore, the attention calculation process is as follows: calculate the input vector k t Score S t The input vector k is obtained. t The degree of influence on the output value; the score k is evaluated using the Softmax function. t Normalization is performed to obtain the weight coefficients. a t Through weighting coefficients a t and input vector k t Calculate the weighted vector o t The calculation formulas are shown in formulas (11) to (13):

[0034] (11)

[0035] (12)

[0036] (13)

[0037] In the formula, W h This represents the weight matrix of the BiNLSTM network. b h This represents the bias vector of the BiNLSTM network.

[0038] Secondly, the present invention provides a fall detection system for complex scenes, comprising:

[0039] The detection box region acquisition module is used to acquire the detection box region containing a human body in a video frame;

[0040] The VGG-BiNLSTM fall behavior detection model construction module is used to build a VGG-BiNLSTM fall behavior detection model. The VGG-BiNLSTM fall behavior detection model includes a VGG network and a BiNLSTM network. The BiNLSTM network construction process is as follows: improve the LSTM network by nesting the preset storage units of the LSTM network into the original LSTM network storage units to construct the NLSTM network storage units.

[0041] The recognition module is used to extract features from the detection box regions containing human bodies in video frames using the VGG network to capture the feature vectors of human behavior; the feature vectors are then fed into the VGG-BiNLSTM fall behavior detection model, which uses the temporal information between consecutive frames to detect and recognize fall behavior.

[0042] In a third aspect, the present application provides an electronic device comprising a memory and a processor, the memory storing a computer program, and the processor being configured to execute the fall detection method in complex scenes by the computer program.

[0043] In a fourth aspect, the present application provides a computer readable storage medium comprising a stored computer program, wherein the computer program performs the fall detection method in complex scenes when running.

[0044] Compared with the prior art, the technical scheme of the present application has the beneficial effects that:

[0045] The present application proposes a VGG-BiNLSTM fall behavior detection model to detect the elderly who fall, uses a VGG convolution network to extract features from the input pedestrian monitoring video, improves the generalization and robustness of the fall detection network, and reduces the occurrence of false positives and false negatives.

[0046] The VGG convolution network and the improved BiNLSTM network are combined, and the attention mechanism is introduced, and BiNLSTM is used to judge the time series data. The fall behavior of pedestrians is accurately detected, and the accuracy, robustness and real-time performance are superior, and has good application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 The fall detection method in complex scenes provided by the embodiments of the present application is shown in the figure;

[0048] Figure 2a The VGG16 network structure provided by the embodiments of the present application is shown in the figure;

[0049] Figure 2b The internal structure of the NLSTM storage unit provided by the embodiments of the present application is shown in the figure;

[0050] Figure 3 The BiNLSTM network structure provided by the embodiments of the present application is shown in the figure;

[0051] Figure 4 The VGG-BiNLSTM fall behavior detection model provided by the embodiments of the present application is shown in the figure;

[0052] Figure 5 The data set identification provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0053] Other advantages and novel features of the present application will become apparent from the following detailed description of the application when considered in conjunction with the drawings. The application can be put into practice by different embodiments, and the details of the application can be modified in different ways without departing from the spirit and scope of the application. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application, as claimed.

[0054] It should be noted that the drawings provided in the following embodiments are only schematic and are intended to provide the basic understanding of the application. In the drawings, the shape, the number and the size of components are not drawn according to the actual implementation, and the shape, the number and the size of components in the actual implementation can be changed arbitrarily, and the layout of components can be more complex.

[0055] Embodiment

[0056] The present embodiment proposes a fall detection method in a complex scene. Please refer to Figure 1 , comprising the following steps:

[0057] S1: obtaining a detection frame region containing a human body in a video frame;

[0058] S2: establishing a VGG-BiNLSTM fall behavior detection model: the VGG-BiNLSTM fall behavior detection model comprises a VGG network and a BiNLSTM network; the BiNLSTM network is constructed by: improving the LSTM network, embedding the preset storage unit of the LSTM network into the original LSTM network storage unit, and constructing a NLSTM network storage unit;

[0059] S3: using the VGG network to extract features of the detection frame region containing the human body in the video frame to capture the feature vector of the human behavior; sending the feature vector into the VGG-BiNLSTM fall behavior detection model, and using the time sequence information between the continuous frames to realize the detection and recognition of the fall behavior.

[0060] It should be noted that BiNLSTM is a bidirectional long short-term memory network. A recurrent neural network (RNN) is a network mainly used to process sequence data, which can remember the output of the previous period and apply it to the current output. The nodes between the hidden layers are connected, and the input of the hidden layer not only has the output of the input layer, but also the output of the hidden layer at the last time, so a better result related to the association can be obtained. However, RNN has a long-term dependence problem, and can only save the dependence of a short time span, and cannot learn the dependence of a long time span. To solve the existing problems, Hochreiter S created a long short-term memory (LSTM). LSTM is composed of a forgetting gate, an input gate and an output gate. The forgetting gate is used to selectively forget the unit state. The input gate is used to selectively save new information to the unit state. The output gate is used to output the final value according to the cell state. NLSTM is an improvement of LSTM, which improves the prediction performance by memorizing additional information in historical data. The NLSTM network storage unit is to nest an LSTM network storage unit into the original LSTM network storage unit, and the external storage unit can freely and selectively read and write the relevant long-term information of the internal storage unit. This structure improves the robustness of the original LSTM network structure and can remember and process longer historical information. In the LSTM network, the output gate follows a principle that information unrelated to the current time step will still be remembered. Therefore, NLSTM can retain more features of the input data, improving the learning ability of NLSTM. The internal structure of the NLSTM network storage unit is shown in FIG. 2.

[0061] The attention mechanism is used to capture the influence of the feature condition of unequal time on fall detection. The effectiveness of the improved NLSTM network structure is verified through comparative experiments, which can be effectively applied in real scenes. NLSTM is an improvement of long short-term memory (LSTM), which makes the output value range of the forgetting gate more accurate, thereby retaining more data features.

[0062] As a preferred technical solution, in the embodiment, after step 1, the method further includes: performing rectangular frame labeling on the detected frame region of the human body by using a Labelimg labeling tool to obtain an XML format file corresponding to the target category and the rectangular frame coordinates.

[0063] As a preferred technical solution, in the embodiment, the VGG network is a VGG16 network.

[0064] The VGG network for extracting image features adopts a VGG16 network model (Visual Geometry Group 16, VGG16 network), and the VGG16 network structure is deep, including a convolutional layer, a pooling layer, a fully connected layer and a Softmax layer. The VGG16 network uses a 3*3 convolution kernel, so that the network model can obtain more image feature information, the network parameters are less, and the operation ability and accuracy of the network model are effectively improved; the VGG16 network has 13 convolutional layers, each convolutional layer maintains the length and width size of the feature map of the previous convolutional layer, and the convolution realizes the increase of the channel number, the VGG16 network includes 5 maximum pooling layers, which are used to reduce the length and width size of the output feature map, and then improve the anti-interference ability of the network; the VGG16 network structure is as follows Figure 1 .

[0065] The basic principle of the VGG16 network is that the convolutional layer first extracts image features by convolution on the image, and the input image of the algorithm is a human body rectangular frame image detected from the video, instead of the entire video frame; the convolutional layer uses the method of weight sharing and local receptive field, which can reduce the input parameters and effectively improve the network efficiency and speed; then batch normalization operation is performed on the input data, and the data is processed by the ReLU activation function, and the data is nonlinearly processed. The purpose of the pooling layer is to compress the image, and the convolution kernel obtains the average value or maximum value of the corresponding position, which can effectively retain the main features and reduce the parameters; after four times of operation, the output feature map is output through the last three fully connected layers, the role of the fully connected layer is to map the learned distributed features to the label space of the sample, so as to perform classification task, convert the convolution output into a one-dimensional vector, obtain the specific compressed image features, and finally use the SoftMax classifier to classify the features, and pass to the subsequent algorithm network structure.

[0066] In order to improve the performance index of the algorithm for pedestrian fall detection, the LSTM network is improved: the preset LSTM network storage unit is nested in the original LSTM network storage unit, the external storage unit freely and selectively reads and writes the related long-term information of the internal storage unit, which improves the robustness of the original LSTM network structure as a whole, and can remember and process historical information for a long time.

[0067] The VGG16 network can find obvious features in data, so the VGG16 network is used to extract features; the basic idea of BiNLSTM is to input a time sequence into two layers of NLSTM in a forward and reverse direction respectively, and finally obtain an output value according to the outputs of the two layers of NLSTM; BiNLSTM is used for time series due to its good sequence-based calculation characteristics; the attention layer captures the importance of the feature conditions of the unstable moments in time series prediction, and is usually applied after RNN; therefore, the attention after BiNLSTM calculation focuses on the features that affect the results, so as to improve the prediction accuracy.

[0068] On the basis of target human body detection and tracking, the application extracts features of the detection frame region containing human bodies in the original video frame by using the VGG16 network, so as to capture key information of human behaviors; then, the feature vectors are sent into a VGG-BiNLSTM fall behavior detection model, so as to realize detection and identification of the fall behavior by using the time sequence information existing between continuous frames; the NLSTM model has two directions of forward and backward, and simultaneously considers past and future information, while the attention mechanism can adaptively select the importance of features, so that the model can better focus on key information, and improve the accuracy and robustness of fall detection.

[0069] Among them, the fall detection method based on the VGG convolution network and the recurrent network (RNN) has achieved good detection effect. The VGG network can extract image features, and the RNN network can process time sequence information, and the combination of the two can better realize the detection of the fall behavior. In addition, the attention mechanism is also widely used in fall detection, which can selectively focus on important feature regions, further improving the detection accuracy.

[0070] As a preferred technical solution, in the embodiment, the LSTM network storage unit is as formula (1), and the state update of the LSTM network storage unit is as formula (2):

[0071] (1)

[0072] In the formula, x t is an input vector at t moment, h t-1 is a state vector of a hidden layer at t-1 moment, W xi and W hi respectively, x t and h t-1 corresponding weight matrix, b i is a bias vector; and σ represents a sigmod function,i t denotes the output vector of the update gate in each memory cell at time t, f t denotes the output vector of the forget gate at time t, o t denotes the output vector of the output gate at time t; g t and c t is an intermediate calculation variable at time t;

[0073] (2)

[0074] wherein, is an input vector at time t, is a state vector of the hidden layer at time t-1.

[0075] As a preferred technical solution, in the embodiment, the improved LSTM network nests the LSTM network preset storage unit into the original LSTM network storage unit to construct an NLSTM network storage unit; specifically,

[0076] The LSTM network storage unit is used to replace the calculation equation in the original LSTM model to form a nested LSTM network storage unit, and an NLSTM network storage unit is obtained; the outer NLSTM network storage unit is called an external storage unit, and the internal NLSTM network storage unit is called an internal storage unit; in the original LSTM network storage unit, the update of the state of the storage unit is shown in equation (3):

[0077] (3)

[0078] In the NLSTM network storage unit, the update of the state of the storage unit is replaced by the internal storage unit, wherein and are used as short-term memory and long-term memory inputs, respectively; the structure of the internal storage unit is shown in equation (4):

[0079] (4)

[0080] wherein, and are weight matrices corresponding to and respectively, is a bias vector; denotes a sigmod function, denotes the output vector of the update gate in each memory cell at time t, denotes an output vector of the forget gate at time t, denotes an output vector of the output gate at time t; and is an intermediate calculation variable at time t;

[0081] Due to the existence of internal memory, the update mode of the state of the storage unit is changed to formula (5) as shown:

[0082] (5)

[0083] The NLSTM network has the structure of external memory and internal memory, compared with the traditional LSTM network, the NLSTM network can remember information for a longer time; therefore, the NLSTM network is applied to target fall behavior prediction in the application, so as to obtain better prediction performance.

[0084] In the process of upgrading from the starting RNN recurrent network to the LSTM long short-term memory network, and then to the improved NLSTM of the application, the reasoning and learning process of these networks is only carried out in one direction, so that the mining of reverse information is ignored, and a solution is found in BiLSTM for this case; for video data, the input of the data at this moment not only comes from the previous video frame, but also comes from the subsequent video frame, so it is necessary to consider more deeply.

[0085] As a preferred technical solution, in the embodiment, the BiNLSTM network comprises an Input (input) layer, a Forword (forward hidden) layer, a backword (backward hidden) layer and an Output (output) layer connected in sequence;

[0086] The BiNLSTM network fully utilizes the information before and after the video frame in the time sequence, and the BiNLSTM network structure is as shown in Figure 3 The Forward layer includes 6 shared weights

W i (i=1,…,6)

[0087] (6)

[0088] (7)

[0089] (8)

[0090] (9)

[0091] (10)

[0092] wherein, W i (i = 1, …, 6) is the weight of a unit layer when passing to another unit layer, x t is the feature vector (1*4096) obtained by extracting the behavior features from the video frame through the VGG layer, h is the feature sequence (…, x t-1 , x t , x t+1 , …), h ’ is the feature sequence unit (…, x t+1 , x t , x t-1 , …), o t is the output data of the feature vector after passing through the BiNLSTM network; , , , represents the bias of the BiNLSTM network structure, and respectively represent the results of the corresponding time feature vector processing of the external storage unit, the internal storage unit of the NLSTM network for the VGG layer output; formula (10) outputs the feature vector O t as the average value of and , and inputs the feature vector O t to the attention to learn the NLSTM network weight.

[0093] As a preferred technical solution, in the embodiment, the attention mainly imitates the human visual and unique brain signal processing mechanism, highlights some important features, and discovers the data features at different time in the time sequence; therefore, the attention selects important information from all information; by calculating the feature vector weight output by the BiNLSTM network at different time steps, the VGG-BiNLSTM falling behavior detection model shows better performance; the calculation process of the attention is as follows: the score S t of the input vector kt , obtain an input vector k t The degree of influence on the output value; the score k t is normalized by the Softmax function to obtain a weight coefficient a t The weighted vector is calculated by the weight coefficient a t and the input vector k t o t The calculation formula is shown in formulas (11) to (13):

[0094] (11)

[0095] (12)

[0096] (13)

[0097] In the formula, W h represents the weight matrix of the BiNLSTM network, b h represents the bias vector of the BiNLSTM network.

[0098] As a preferred technical solution, in the embodiment,

[0099] As a preferred technical solution, in the embodiment,

[0100] As an embodiment of the present application, it should be noted that the present application uses the PASCAL VOC2012 dataset and a large number of video pictures collected by manual collection, and generates a self-made dataset by manual annotation through labelImg. The fall detection dataset uses a total of 5426 pictures, of which 4197 pictures are used as a training set and 1049 pictures are used as a test set. In order to verify the pedestrian detection index of the improved algorithm, the present application uses Precision, F1-Sore index and detection frame rate as evaluation indexes, which can be used to comprehensively reflect the algorithm index. The related formulas are shown in formulas 14 to 16:

[0101] (14)

[0102] (15)

[0103] (16)

[0104] ​TP (1 Positive) represents a situation where a pedestrian falls and is detected as having fallen; FP (0 ​​Positive) represents a situation where a pedestrian does not fall but is detected as having fallen; FN (0 Negative) represents a situation where a pedestrian falls but is not detected as having fallen; TN (1 Negative) represents a situation where a pedestrian does not fall and is not detected as having fallen.

[0105] Frame rate (FPS) represents the number of video frames that a network can process per second. It is generally used to evaluate the detection speed of a network and to determine whether the model can meet the real-time requirements.

[0106] To verify the superiority of the experimental results of the network structure of this invention, a dataset with pedestrian feature boxes pre-processed by YOLOv5 was used for subsequent fall detection. Four algorithms, namely VGG+LSTM attention, VGG+BiLSTM attention, VGG+NLSTM attention, and VGG+BiNLSTM attention, were selected for experiments to compare their effects. A self-made fall dataset was used to compare the fall detection accuracy of the above four algorithms and to conduct comparative analysis.

[0107] Table 1. Fall detection prediction performance of different algorithms on the MCFD dataset.

[0108]

[0109] The results on the self-made fall detection dataset are shown in Table 1. The data clearly shows that BiNLSTM has a significant advantage in recall and precision compared to unidirectional NLSTM, achieving good detection results for simple scene datasets. BiNLSTM also demonstrates good real-time performance in detecting video frame rates. Overall, the VGG+BiNLSTM attention method performs best in the fall detection task, exhibiting high detection accuracy and fast processing speed. The experimental results of the proposed detection method are as follows: Figure 5 As shown:

[0110] In complex scenes, the behavior of pedestrians is a continuous process, and the information in a single image is insufficient to accurately determine the specific behavior category. To address the issues of potential errors in target behavior judgment and poor detection performance in complex scenes, this invention proposes the VGG-BiNLSTM attention network algorithm, which better integrates the information features of the pedestrian before and after in video frames, thus achieving the detection and judgment of fall behavior. Experimental results show that the proposed detection model can effectively improve the detection rate of pedestrian fall behavior in complex scenes.

[0111] In another embodiment of the present invention, a fall detection system for complex scenes is also provided, comprising:

[0112] a detection frame region acquisition module, configured to acquire a detection frame region containing a human body in a video frame;

[0113] a VGG-BiNLSTM fall behavior detection model construction module, configured to establish a VGG-BiNLSTM fall behavior detection model, wherein the VGG-BiNLSTM fall behavior detection model comprises a VGG network and a BiNLSTM network, and the BiNLSTM network is constructed by improving an LSTM network, embedding a storage unit preset in the LSTM network into an original LSTM network storage unit, and constructing a NLSTM network storage unit;

[0114] a recognition module, configured to extract features of the detection frame region containing the human body in the video frame by using the VGG network, so as to capture a feature vector of a human behavior, and send the feature vector into the VGG-BiNLSTM fall behavior detection model, so as to realize detection and recognition of the fall behavior by using time sequence information between continuous frames.

[0115] In another embodiment of the present application, an electronic device is also provided, comprising: one or more processors; a storage device configured to store one or more programs, which, when executed by the one or more processors, cause the present application to implement the fall detection method in a complex scene as in any of the above embodiments.

[0116] In the present embodiment, a computer system of an electronic device suitable for implementing the embodiments of the present application comprises a central processing unit (CPU), which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) or loaded from a storage part into a random access memory (RAM), such as the methods described in the above embodiments. In the RAM, various programs and data required for system operation are also stored. The CPU, the ROM, and the RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0117] The following components are connected to the I / O interface: an input section including a keyboard, a mouse, etc.; an output section including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), etc., and a speaker, etc.; a storage section including a hard disk, etc.; and a communication section including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as necessary. A removable medium such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive as necessary, so that a computer program read out from it is installed in the storage section as necessary.

[0118] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present application. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing a computer program for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), various functions defined in the system of the present application are executed.

[0119] It should be noted that, although several modules or units of the apparatus for action execution are mentioned in the foregoing detailed description, such division into modules or units is not mandatory. Indeed, according to embodiments of the present application, the features and functionalities of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functionalities of one module or unit described above can be further divided into several modules or units.

[0120] In the present embodiment, the present application also provides a computer readable storage medium having stored thereon a computer program, which, when executed by a processor of a computer, causes the computer to perform the method for fall detection in complex scene as provided in any of the foregoing embodiments. The computer readable storage medium can be included in the electronic device described in the foregoing embodiments, or can exist separately and not be assembled into the electronic device.

[0121] Those skilled in the art can clearly understand the example embodiments described herein through the above description of the example embodiments that the example embodiments described herein can be implemented by software or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or on a network, and includes a plurality of instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0122] Obviously, the above embodiments of the present application are merely examples for clearly illustrating the present application, and are not intended to limit the embodiments of the present application. Based on the above description, other different forms of changes or variations can be made by those of ordinary skill in the art. Here, all the embodiments need not and cannot be exhausted. Any modification, equivalent replacement, and improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. A fall detection method in a complex scene, characterized in that, The method comprises the following steps: S1: obtaining a detection frame region containing a human body in a video frame; S2: establishing a VGG-BiNLSTM fall behavior detection model: the VGG-BiNLSTM fall behavior detection model comprises a VGG network and a BiNLSTM network; The BiNLSTM network construction process is: improving the LSTM network, nesting the LSTM network preset storage unit into the original LSTM network storage unit to construct an NLSTM network storage unit; The LSTM network storage unit is as shown in formula (1), and the state update of the LSTM network storage unit is as shown in formula (2): In the formula, x t Let h be the input vector at time t. t-1 Let W be the state vector of the hidden layer at time t-1. xi and W hi x t with h t-1 The corresponding weight matrix, b i For the biased vector; σ represents the sigmoid function, i t f represents the output vector of the gate updated at time t in each memory cell. t O represents the output vector of the forget gate at time t. t g represents the output vector of the output gate at time t; t and c t These are intermediate calculation variables at time t; wherein is the input vector at time t, is the state vector of the hidden layer at time t-1; The improved LSTM network nests the LSTM network preset storage unit into the original LSTM network storage unit to construct an NLSTM network storage unit; specifically: The calculation equation in the original LSTM network storage unit is replaced by an NLSTM network storage unit to obtain an NLSTM network storage unit; the outer NLSTM network storage unit is referred to as an external storage unit, and the inner NLSTM network storage unit is referred to as an internal storage unit; in the original LSTM network storage unit, the update of the state c t of the storage unit is shown in formula (3): In the NLSTM network storage unit, the update of the state c t of the storage unit is replaced by an internal storage unit, where and are used as short-term memory and long-term memory inputs, respectively; the structure of the internal storage unit is shown in equation (4): wherein, and are respectively and corresponding weight matrix, is a bias vector; denotes a sigmod function, denotes an output vector of the update gate in each memory cell at time t, denotes an output vector of the forget gate at time t, denotes an output vector of the output gate at time t; and are intermediate calculation variables at time t; Due to the presence of the internal memory, the updating manner of the state c t of the storage unit becomes as shown in equation (5): S3: using the VGG network to perform feature extraction on the detection frame region containing the human body in the video frame to capture a feature vector of the human body behavior; The feature vector is sent into the VGG-BiNLSTM fall behavior detection model, and the time sequence information between continuous frames is utilized to realize detection and identification of the fall behavior.

2. The method of claim 1, wherein, After step 1, the method further comprises: performing rectangular frame labeling on the detection frame region of the human body by using a Labelimg labeling tool to obtain an XML format file containing a target category and rectangular frame coordinates.

3. The method of claim 1, wherein, The VGG network is a VGG16 network.

4. The method of claim 1, wherein, The BiNLSTM network comprises an Input layer, a Forword layer, a backword layer and an Output layer connected in sequence; The Forward layer comprises six shared weights; forward calculation is performed on the Forward layer from the starting time to the time t, and the output of each time of the Forward layer is obtained and saved; backward calculation is performed on the Backward layer from the time t to the starting time, and the output of each time of the backword layer is obtained and saved; the corresponding output results of each time of the Forward layer and the Backward layer are fused to obtain the output, as shown in formula (6) to formula (10); wherein W i (i = 1, …, 6) is the weight of a unit layer when it is transferred to another unit layer, x t is the feature vector obtained by extracting the behavior features from the video frame through the VGG layer, h is the feature sequence (…, x t-1 , x t , x t+1 , …) input into the NLSTM network storage unit, h' is the feature sequence unit (…, x t+1 , x t , x t-1 , …) input into the NLSTM, o t is the output data of the feature vector after passing through the BiNLSTM network; represents the bias of the BiNLSTM network structure, o t ' and o t " respectively represent the results of the feature vector processed by the external storage unit and the internal storage unit of the NLSTM network for the VGG layer output; formula (10) outputs the feature vector O t as the average value of o t ' and o t " at the corresponding moment.

5. A fall detection system in a complex scene, characterized by, It comprises: a detection frame region acquisition module, configured to acquire a detection frame region containing a human body in a video frame; a VGG-BiNLSTM fall behavior detection model construction module, configured to establish a VGG-BiNLSTM fall behavior detection model: the VGG-BiNLSTM fall behavior detection model comprises a VGG network and a BiNLSTM network; The BiNLSTM network construction process is: improving the LSTM network, nesting the LSTM network preset storage unit into the original LSTM network storage unit to construct an NLSTM network storage unit; The LSTM network storage unit is as shown in formula (1), and the state update of the LSTM network storage unit is as shown in formula (2): In the formula, x t Let h be the input vector at time t. t-1 Let W be the state vector of the hidden layer at time t-1. xi and W hi x t with h t-1 The corresponding weight matrix, b i For the biased vector; σ represents the sigmoid function, i t f represents the output vector of the gate updated at time t in each memory cell. t O represents the output vector of the forget gate at time t. t g represents the output vector of the output gate at time t; t and c t These are intermediate calculation variables at time t; wherein is the input vector at time t, is the state vector of the hidden layer at time t-1; The improved LSTM network nests the LSTM network preset storage unit into the original LSTM network storage unit to construct an NLSTM network storage unit; specifically: The calculation equation in the original LSTM network storage unit is replaced by an NLSTM network storage unit to obtain an NLSTM network storage unit; the outer NLSTM network storage unit is referred to as an external storage unit, and the inner NLSTM network storage unit is referred to as an internal storage unit; in the original LSTM network storage unit, the update of the state c t of the storage unit is shown in formula (3): In the NLSTM network storage unit, the update of the state c t of the storage unit is replaced by an internal storage unit, where and are used as short-term memory and long-term memory inputs, respectively; the structure of the internal storage unit is shown in equation (4): wherein, and are respectively and corresponding weight matrix, is a bias vector; denotes a sigmod function, denotes an output vector of the update gate in each memory cell at time t, denotes an output vector of the forget gate at time t, denotes an output vector of the output gate at time t; and are intermediate calculation variables at time t; Due to the presence of the internal memory, the updating manner of the state c t of the storage unit becomes as shown in equation (5): an identification module, configured to use the VGG network to perform feature extraction on the detection frame region containing the human body in the video frame to capture a feature vector of the human body behavior; The feature vector is sent into a VGG-BiNLSTM fall behavior detection model, and the time sequence information between continuous frames is utilized to realize detection and identification of the fall behavior. The memory stores a computer program, and the processor is configured to execute the fall detection method in a complex scene according to any one of claims 1 to 4 by the computer program. 6.An electronic device comprising a memory and a processor, the electronic device characterized by, The computer readable storage medium comprises a stored computer program, wherein the computer program executes the fall detection method in a complex scene according to any one of claims 1 to 4 when running.

7. A computer-readable storage medium, characterized in that, ​

Images