A method, device and system for identifying falling action
By combining the convolutional neural network of TRN timing network and Backbone network, the problems of low efficiency and large calculation volume in the existing technology are solved, and more efficient fall action recognition is achieved, and the user has been improved by sending prompt information. Security.
Patent Information
- Application Number
- CN202210666059.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-14
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-06-14
AI Technical Summary
The prior art has problems such as low efficiency, large amount of calculation and difficulty in extracting long-term timing relationships in fall action recognition. In particular, the dual-stream method and 3D CNN method based on deep learning require pre-calculation of optical flow information and large amount of 3D convolution calculation.
The convolutional neural network that combines the TRN timing network and the Backbone network is used to train the preset training data set, extract the timing characteristics and action features of the action images, and enhance the timing processing capabilities of the network through time composite functions to improve the recognition efficiency .
It improves the efficiency of falling action recognition and sends prompt information to the user when falling action is detected, improving safety.
Smart Images

Figure CN115116131B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of recognition of falling actions, and in particular to a method, device and system for recognition of falling actions. Background Art
[0002] With the growth of the aging population in China, the health monitoring of elderly people living alone is becoming more and more important. When an elderly person falls, timely treatment can prevent the elderly from being injured again. Therefore, it is crucial for the monitoring device to identify the falling action.
[0003] In the prior art, the recognition of falling actions is mainly based on wearable devices, environmental sensors and computer vision. Among them, wearable devices are not conducive to the daily activities of the elderly, and cannot be detected in real time when they forget to wear them; environmental sensors are easily disturbed by the surrounding environment, and the equipment is expensive and has low accuracy. Therefore, the prior art generally uses computer vision to identify falling actions. In the field of deep learning, the methods used for fall behavior detection are mainly dual-stream method and 3D CNN. The dual-stream method fuses optical flow information with RGB information to improve detection accuracy; 3DCNN uses 3D convolution kernels to extract temporal features from continuous RGB video sequences.
[0004] However, the existing technology still has the following defects: (1) The optical flow information needs to be calculated in advance, which reduces the efficiency of the algorithm; (2) When consecutive frames are input into the network, there is redundancy in the information between consecutive frames, and the 3D convolution calculation is large; (3) Since the frame sequence of the network is usually 20-30 frames, it is difficult for the network to extract long-term temporal relationships.
[0005] Therefore, there is a need for a method, device and system for identifying a falling action, so as to overcome the above-mentioned defects in the prior art. Summary of the invention
[0006] In view of the above existing technical problems, the object of the present invention is to provide a method, device and system for identifying a fall action, so as to improve the efficiency of identifying a fall action.
[0007] The present invention provides a method for identifying a fall action, the method comprising: training a preset first fall recognition network according to a preset training data set to obtain a fall recognition network; the training data set is a URFall data set; the first fall recognition network comprises a TRN timing network and a Backbone network; obtaining a group of action images to be identified; analyzing the timing features and action features of the action image group according to the fall recognition network, identifying and classifying the action image group, thereby obtaining a recognition result.
[0008] In one embodiment, according to a preset training data set, a preset first fall recognition network is trained to obtain a fall recognition network, specifically comprising: inputting the preset training data set into the Backbone network in the first fall recognition network, thereby extracting a feature data group of the training data set; the feature data group includes multiple feature data; performing time relationship reasoning on the multiple feature data according to the TRN timing network in the first fall recognition network to obtain the time relationship between the multiple feature data, and splicing the output results according to the time relationship, thereby obtaining a training recognition result; correcting the first fall recognition network according to the training recognition result, and outputting the corrected first recognition network as a fall recognition network.
[0009] In one embodiment, performing time relationship reasoning on the multiple feature data according to the TRN timing network in the first fall recognition network to obtain the time relationship between the multiple feature data specifically includes: inputting the multiple feature data into the TRN timing network in the first fall recognition network, so that the TRN timing network obtains the first time composite functions of the multiple feature data on multiple time scales as the corresponding time relationship according to a preset time composite function; wherein the time composite function is: Where V is the input video, which has n ordered frames, where f i represents the i-th frame, where and g θ They are the corresponding functions in MLP respectively.
[0010] In one embodiment, a preset training data set is input into a Backbone network in a first fall recognition network, thereby extracting a feature data group of the training data set, specifically comprising: scaling the preset training data set according to preset input requirements to obtain an input data set; the input data set corresponds to the training data set; and extracting image features in the input data set through the Backbone network in the first fall recognition network to obtain a feature data group.
[0011] In one embodiment, the first fall recognition network is corrected according to the training recognition result, and the corrected first recognition network is output as a fall recognition network, specifically including: calculating a loss value according to the training recognition result, and judging whether the first fall recognition network converges according to the loss value; if not, reversely transmitting the loss value to adjust the parameters of the first fall recognition network according to the loss value, and repeating the above steps; if converged, outputting the first fall recognition network as a fall recognition network.
[0012] In one embodiment, the output results are spliced according to the time relationship to obtain the training recognition result, which specifically includes: accumulating the first time composite functions corresponding to each time relationship to obtain the training recognition result.
[0013] In one embodiment, after analyzing the temporal features and the motion features of the action image group according to the fall recognition network and identifying and classifying the action image group to obtain the recognition result, it also includes: when the recognition result is that a fall behavior is detected, sending a prompt message to the user.
[0014] The present invention also provides a device for identifying a fall action, the identification device comprising a model training unit, an image acquisition unit and a fall recognition unit, wherein the model training unit is used to train a preset first fall recognition network according to a preset training data set to obtain a fall recognition network; the training data set is a UR Fall data set; the first fall recognition network includes a TRN timing network and a Backbone network; the image acquisition unit is used to obtain a group of action images to be identified; the fall recognition unit is used to analyze the timing characteristics and action characteristics of the action image group according to the fall recognition network, identify and classify the action image group, and thus obtain a recognition result.
[0015] In one embodiment, the model training unit is also used to: input a preset training data set into the Backbone network in the first fall recognition network, so as to extract a feature data group of the training data set; the feature data group includes multiple feature data; perform time relationship reasoning on the multiple feature data according to the TRN timing network in the first fall recognition network to obtain the time relationship between the multiple feature data, and splice the output results according to the time relationship to obtain a training recognition result; modify the first fall recognition network according to the training recognition result, and output the modified first recognition network as a fall recognition network.
[0016] In one embodiment, the recognition device further includes a fall prompt unit, and the fall prompt unit is used to send a prompt message to the user when the recognition result is that a fall behavior is detected.
[0017] The present invention also provides a system for identifying falling actions, which includes a behavior recognition module, a data storage module and one or more cameras. The behavior recognition module is communicated with the data storage module and the camera respectively. The data storage module is used to store all data. The camera is used to collect a group of action images to be identified and send the group of action images to the behavior recognition module. The behavior recognition module is used to execute the method for identifying falling actions as described above.
[0018] In one embodiment, the recognition system further includes a user interaction module, which is communicatively connected to the behavior recognition module, and is configured to send a prompt message to the user when the recognition result is that a fall behavior is detected.
[0019] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:
[0020] The present invention provides a method, device and system for identifying falling actions. The time series processing capability of a convolutional neural network is enhanced by a TRN time series network, and the convolutional neural network is used to identify and classify the acquired action image group to be identified. The recognition method, device and system improve the recognition efficiency of falling actions.
[0021] Furthermore, the method, device and system for identifying a fall action provided by the present invention also improve safety by sending a prompt message to the user when the identification result is that a fall behavior is detected. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The present invention will be further described below in conjunction with the accompanying drawings, wherein:
[0023] Figure 1 A flow chart of an embodiment of a method for identifying a falling action according to the present invention is shown;
[0024] Figure 2 A structural diagram showing an embodiment of a device for identifying a falling action according to the present invention;
[0025] Figure 3 A structural diagram of an embodiment of a system for identifying a falling action according to the present invention is shown. DETAILED DESCRIPTION
[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Specific embodiment 1
[0028] The embodiment of the present invention first describes a method for identifying a falling action. Figure 1 A flow chart of an embodiment of a method for identifying a falling action according to the present invention is shown.
[0029] like Figure 1 As shown, the method comprises the following steps:
[0030] S1: According to a preset training data set, a preset first fall recognition network is trained to obtain a fall recognition network.
[0031] The training data set is a UR Fall data set. The first fall recognition network includes a TRN temporal network and a Backbone network.
[0032] In one embodiment, according to a preset training data set, a preset first fall recognition network is trained to obtain a fall recognition network, specifically comprising: inputting the preset training data set into the Backbone network in the first fall recognition network, thereby extracting a feature data group of the training data set; the feature data group includes multiple feature data; performing time relationship reasoning on the multiple feature data according to the TRN timing network in the first fall recognition network to obtain the time relationship between the multiple feature data, and splicing the output results according to the time relationship, thereby obtaining a training recognition result; correcting the first fall recognition network according to the training recognition result, and outputting the corrected first recognition network as a fall recognition network.
[0033] Before using the fall recognition network to identify and classify the action image group to be identified, it is first necessary to train the initial convolutional neural network (described herein as the "first fall recognition network") according to the preset training data set to obtain the fall recognition network. Therefore, the embodiment of the present invention first inputs the preset training data set into the Backbone network in the first fall recognition network to preliminarily extract the spatial features of the training data set, thereby obtaining the corresponding feature data set. In one embodiment, the feature data set includes multiple feature data.
[0034] In one embodiment, a preset training data set is input into a Backbone network in a first fall recognition network, thereby extracting a feature data group of the training data set, specifically comprising: scaling the preset training data set according to preset input requirements to obtain an input data set; the input data set corresponds to the training data set; and extracting image features in the input data set through the Backbone network in the first fall recognition network to obtain a feature data group.
[0035] Since the recognition of falling actions requires the analysis and judgment of a series of images, and the time and space relationship between images is close, there are often computing load problems caused by a large number of time series feature analyses when recognizing falling actions. However, the network models used for recognition in the prior art often have weak time series processing capabilities and cannot effectively or efficiently complete recognition and classification. In this regard, when designing the network model for fall recognition, the embodiment of the present invention deliberately adds a TRN time series network on the basis of the Backbone network, thereby improving the time series processing capabilities of the first fall recognition network.
[0036] Since the TRN timing network will first group multiple images, and then analyze and process each group to obtain the results of each group and the time relationship between the results, after obtaining the results and the time relationship between the results, it is necessary to splice the output results according to the time relationship to obtain the training recognition results.
[0037] In one embodiment, time relationship reasoning is performed on the multiple feature data according to the TRN timing network in the first fall recognition network to obtain the time relationship between the multiple feature data, specifically including: inputting the multiple feature data into the TRN timing network in the first fall recognition network, so that the TRN timing network obtains the first time compound functions of the multiple feature data on multiple time scales as the corresponding time relationship according to a preset time compound function.
[0038] Wherein, the time composite function is:
[0039]
[0040] Where V is the input video, which has n ordered frames, where f i represents the i-th frame, where and They are the corresponding functions in MLP respectively.
[0041] After obtaining the training recognition result, the training recognition result needs to be verified to calculate the loss value of the current first fall recognition network, so as to determine whether the current first fall recognition network converges according to the loss value and the preset threshold, and then determine whether to continue training.
[0042] When it is determined that the current first fall recognition network has not converged, it is necessary to continue network model training, and the loss value calculated in the previous step is transmitted in reverse, so as to correct the model parameters of the current first fall recognition network according to the loss value, and enter the next round of training, and repeat this process until it is determined that the first fall recognition network has converged.
[0043] When it is determined that the current first fall recognition network converges, it means that the current first fall recognition network can be directly used for recognition and classification of fall actions. At this time, the first fall recognition network can be output as a preset fall recognition network for subsequent recognition applications.
[0044] In one embodiment, the first fall recognition network is corrected according to the training recognition result, and the corrected first recognition network is output as a fall recognition network, specifically including: calculating a loss value according to the training recognition result, and judging whether the first fall recognition network converges according to the loss value; if not, reversely transmitting the loss value to adjust the parameters of the first fall recognition network according to the loss value, and repeating the above steps; if converged, outputting the first fall recognition network as a fall recognition network.
[0045] In one embodiment, the output results are spliced according to the time relationship to obtain the training recognition result, which specifically includes: accumulating the first time composite functions corresponding to each time relationship to obtain the training recognition result.
[0046] S2: Obtain a group of action images to be identified.
[0047] In order to identify the falling action, it is first necessary to set up a camera to collect and obtain a group of action images to be identified in real time. The action image group includes several action images.
[0048] S3: Analyze the temporal features and motion features of the action image group according to the fall recognition network, identify and classify the action image group, and thus obtain a recognition result.
[0049] In the field of deep learning, the methods used for fall behavior detection are mainly two-stream method and 3D CNN. The two-stream method fuses optical flow information with RGB information to improve detection accuracy; 3D CNN uses 3D convolution kernels to extract temporal features from continuous RGB video sequences. However, the above two methods still have the following defects: optical flow information needs to be calculated in advance, which reduces the efficiency of the algorithm; when continuous frames are input into the network, there is redundancy in information between continuous frames, and the 3D convolution calculation is large; since the network's frame sequence is usually 20-30 frames, it is difficult for the network to extract long-term temporal relationships.
[0050] In this regard, an embodiment of the present invention adopts a preset fall recognition network for identification and classification, wherein the fall recognition network is a convolutional neural network composed of a TRN timing network and a Backbone network training. The TRN timing network can learn and discover possible time relationships on multiple time scales, thereby enhancing the network's timing processing capabilities.
[0051] In one embodiment, after analyzing the temporal features and the motion features of the action image group according to the fall recognition network and identifying and classifying the action image group to obtain the recognition result, it also includes: when the recognition result is that a fall behavior is detected, sending a prompt message to the user.
[0052] An embodiment of the present invention describes a method for identifying a falling action. The time processing capability of a convolutional neural network is enhanced by a TRN time network, and the convolutional neural network is used to identify and classify the acquired action image group to be identified. The recognition method improves the recognition efficiency of a falling action. Furthermore, a method for identifying a falling action described in an embodiment of the present invention also improves safety by sending a prompt message to the user when the recognition result is that a falling behavior is detected. Specific embodiment 2
[0054] In addition to the above method, the embodiment of the present invention also describes a device for identifying a falling action. Figure 2 A structural diagram of an embodiment of a device for identifying a falling action according to the present invention is shown.
[0055] As shown in the figure, the recognition device includes a model training unit 11, an image acquisition unit 12 and a fall recognition unit 13.
[0056] The model training unit 11 is used to train the preset first fall recognition network according to the preset training data set to obtain the fall recognition network. The training data set is the UR Fall data set; the first fall recognition network includes a TRN time series network and a Backbone network.
[0057] The image acquisition unit 12 is used to acquire a group of action images to be identified.
[0058] The fall recognition unit 13 is used to analyze the temporal features and the motion features of the action image group according to the fall recognition network, and to recognize and classify the action image group, so as to obtain a recognition result.
[0059] In one embodiment, the model training unit is also used to: input a preset training data set into the Backbone network in the first fall recognition network, so as to extract a feature data group of the training data set; the feature data group includes multiple feature data; perform time relationship reasoning on the multiple feature data according to the TRN timing network in the first fall recognition network to obtain the time relationship between the multiple feature data, and splice the output results according to the time relationship to obtain a training recognition result; modify the first fall recognition network according to the training recognition result, and output the modified first recognition network as a fall recognition network.
[0060] In one embodiment, the recognition device further includes a fall prompt unit, and the fall prompt unit is used to send a prompt message to the user when the recognition result is that a fall behavior is detected.
[0061] An embodiment of the present invention describes a device for identifying falling actions. The time processing capability of a convolutional neural network is enhanced through a TRN time network, and the convolutional neural network is used to identify and classify the acquired action image group to be identified. The identification device improves the recognition efficiency of falling actions. Furthermore, the device for identifying falling actions described in an embodiment of the present invention also improves safety by sending a prompt message to the user when the recognition result is that a falling behavior is detected. Specific embodiment three
[0063] In addition to the above method and device, the present invention also describes a system for identifying falling actions. Figure 3 A structural diagram of an embodiment of a system for identifying a falling action according to the present invention is shown.
[0064] As shown in the figure, the recognition system includes a behavior recognition module 1, a data storage module 2 and one or more cameras 3. The behavior recognition module 1 is communicated with the data storage module 2 and the camera 3 respectively. The data storage module 2 is used to store all data. The camera 3 is used to collect the action image group to be recognized and send the action image group to the behavior recognition module 1. The behavior recognition module 1 is used to execute the recognition method of the falling action as described above.
[0065] In one embodiment, the recognition system further includes a user interaction module, which is communicatively connected to the behavior recognition module 1 , and is configured to send a prompt message to the user when the recognition result is that a fall behavior is detected.
[0066] An embodiment of the present invention describes a system for identifying falling actions. The time processing capability of a convolutional neural network is enhanced through a TRN time network, and the convolutional neural network is used to identify and classify the acquired action image group to be identified. The recognition system improves the recognition efficiency of falling actions. Furthermore, a system for identifying falling actions described in an embodiment of the present invention also improves safety by sending a prompt message to the user when the recognition result is that a falling behavior is detected.
[0067] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for identifying a falling action, characterized in that: The identification method comprises: According to a preset training data set, a preset first fall recognition network is trained to obtain a fall recognition network; the training data set is a UR Fall data set; the first fall recognition network includes a TRN timing network and a Backbone network; Obtaining a group of action images to be identified; Analyzing the temporal features and the motion features of the action image group according to the fall recognition network, identifying and classifying the action image group, thereby obtaining a recognition result; The step of training a preset first fall recognition network according to a preset training data set to obtain a fall recognition network specifically includes: Inputting a preset training data set into the Backbone network in the first fall recognition network, thereby extracting a feature data group of the training data set; the feature data group includes a plurality of feature data; Performing time relationship reasoning on the multiple feature data according to the TRN timing network in the first fall recognition network to obtain the time relationship between the multiple feature data, and splicing the output results according to the time relationship to obtain a training recognition result; Modifying the first fall recognition network according to the training recognition result, and outputting the modified first recognition network as a fall recognition network; The performing time relationship reasoning on the plurality of feature data according to the TRN timing network in the first fall recognition network to obtain the time relationship between the plurality of feature data specifically includes: The plurality of characteristic data are input into the TRN timing network in the first fall recognition network, so that the TRN timing network respectively obtains the first time compound functions of the plurality of characteristic data on a plurality of time scales as corresponding time relations according to a preset time compound function; wherein the time compound function is: ; In the formula, is the input video, with n ordered frames, where represents the i-th frame, here and They are the corresponding functions in MLP respectively.
2. The method for identifying a falling action according to claim 1, characterized in that: Inputting a preset training data set into the Backbone network in the first fall recognition network, thereby extracting a feature data group of the training data set, specifically including: According to a preset input requirement, scaling a preset training data set to obtain an input data set; the input data set corresponds to the training data set; The image features in the input data set are extracted through the Backbone network in the first fall recognition network to obtain a feature data set.
3. The method for identifying a falling action according to claim 1, characterized in that: The first fall recognition network is modified according to the training recognition result, and the modified first recognition network is output as a fall recognition network, specifically including: Calculating a loss value according to the training recognition result, and judging whether the first fall recognition network converges according to the loss value; If it does not converge, the loss value is transmitted in reverse order to adjust the parameters of the first fall recognition network according to the loss value, and the above steps are repeated; If converged, the first fall recognition network is output as a fall recognition network.
4. The method for identifying a falling action according to claim 1, characterized in that: The output results are spliced according to the time relationship to obtain the training recognition result, which specifically includes: The first time composite functions corresponding to each time relationship are accumulated to obtain the training recognition result.
5. The method for identifying a falling action according to any one of claims 1 to 4, characterized in that: After analyzing the temporal features and the action features of the action image group according to the fall recognition network and identifying and classifying the action image group to obtain the recognition result, the method further includes: When the recognition result is that a fall behavior is detected, a prompt message is sent to the user.
6. A device for identifying a falling action, characterized in that: The recognition device includes a model training unit, an image acquisition unit and a fall recognition unit, wherein: The model training unit is used to train a preset first fall recognition network according to a preset training data set to obtain a fall recognition network; the training data set is a UR Fall data set; the first fall recognition network includes a TRN timing network and a Backbone network; The image acquisition unit is used to acquire a group of action images to be identified; The fall recognition unit is used to analyze the temporal features and the action features of the action image group according to the fall recognition network, and to recognize and classify the action image group, so as to obtain a recognition result; The model training unit trains a preset first fall recognition network according to a preset training data set to obtain a fall recognition network, specifically including: The model training unit inputs a preset training data set into the Backbone network in the first fall recognition network, thereby extracting a feature data group of the training data set; the feature data group includes a plurality of feature data; Performing time relationship reasoning on the multiple feature data according to the TRN timing network in the first fall recognition network to obtain the time relationship between the multiple feature data, and splicing the output results according to the time relationship to obtain a training recognition result; Modifying the first fall recognition network according to the training recognition result, and outputting the modified first recognition network as a fall recognition network; The model training unit performs time relationship reasoning on the multiple feature data according to the TRN timing network in the first fall recognition network to obtain the time relationship between the multiple feature data, specifically including: The model training unit inputs the multiple feature data into the TRN time series network in the first fall recognition network, so that the TRN time series network obtains the first time composite functions of the multiple feature data on multiple time scales as corresponding time relations according to a preset time composite function; wherein the time composite function is: ; In the formula, is the input video, with n ordered frames, where represents the i-th frame, here and They are the corresponding functions in MLP respectively.
7. A system for identifying a falling action, characterized in that: The recognition system includes a behavior recognition module, a data storage module and one or more cameras. The behavior recognition module is communicated with the data storage module and the camera respectively. The data storage module is used to store all data. The camera is used to collect a group of action images to be recognized and send the group of action images to the behavior recognition module. The behavior recognition module is used to execute the method for recognizing a falling action as described in any one of claims 1 to 5.
8. The system for identifying a fall according to claim 7, characterized in that: The recognition system further includes a user interaction module, which is communicatively connected to the behavior recognition module, and is configured to send a prompt message to the user when a fall behavior is detected as a recognition result.
