Digital rural elderly living alone safety assistance method, system, device and storage medium

Abstract

The present invention discloses a digital rural elderly living alone safety help method, system, device and storage medium. It first extracts multiple behavior state monitoring key frames from the behavior state monitoring video of the monitored elderly living alone collected by a camera, then extracts a sequence of optical flow images from the multiple behavior state monitoring key frames based on an optical flow feature extraction network, then passes the multiple behavior state monitoring key frames through a spatial flow feature extractor based on a three-dimensional convolutional neural network model to obtain a first behavior monitoring feature map, then passes the sequence of optical flow images through a temporal flow feature extractor based on a three-dimensional convolutional neural network model to obtain a second behavior monitoring feature map, then fuses the first behavior monitoring feature map and the second behavior monitoring feature map to obtain a classification feature map, and finally passes the classification feature map through a classifier to obtain a classification result for indicating whether a safety help request is generated. In this way, a safety help request can be generated in a timely and effective manner.

Description

Technical Field

[0001] The present application relates to the field of intelligent monitoring, and more specifically, to a method, system, device and storage medium for providing safety assistance to elderly people living alone in digital villages. Background Art

[0002] As the aging of the population in my country continues to deepen, the construction of digital villages has gradually become a hot issue of social concern. In digital villages, the life safety of elderly people living alone is particularly prominent. Once an accident occurs, the elderly living alone may not be able to get rescue in time for a long time, or even endanger their lives.

[0003] Accordingly, considering that in the actual process of safety monitoring of elderly people living alone, the elderly may not be able to ask for help on their own in unexpected situations, in order to enable the elderly to get rescue help in a timely and effective manner in unexpected situations, it is possible to determine whether to automatically generate a safety help request based on the elderly's behavior status monitoring video. However, since there is a large amount of information in the behavior status monitoring video, the characteristic information about the elderly's behavior status is an implicit behavior semantic feature in the video. In other words, since each elderly person has different behavioral habits, their behavior in the monitoring video is different. Therefore, there will be great difficulties in actually extracting the semantic features of the elderly's behavior status, making it impossible for the elderly to get timely rescue help.

[0004] Therefore, there is a need for a digital rural elderly person living alone method, system, device and storage medium for safe assistance. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The present application provides a method, system, device and storage medium for safe help-seeking of elderly people living alone in digital villages. It first extracts multiple behavior state monitoring key frames from the behavior state monitoring video of the monitored elderly people living alone collected by the camera, then extracts a sequence of optical flow images from the multiple behavior state monitoring key frames based on the optical flow feature extraction network, then passes the multiple behavior state monitoring key frames through a spatial flow feature extractor based on a three-dimensional convolutional neural network model to obtain a first behavior monitoring feature map, then passes the sequence of optical flow images through a temporal flow feature extractor based on a three-dimensional convolutional neural network model to obtain a second behavior monitoring feature map, then fuses the first behavior monitoring feature map and the second behavior monitoring feature map to obtain a classification feature map, and finally passes the classification feature map through a classifier to obtain a classification result for indicating whether a safety help request is generated. In this way, a safety help request can be generated in a timely and effective manner.

[0006] According to one aspect of the present application, a digital rural elderly living alone safety assistance system is provided, which includes:

[0007] The elderly behavior status video acquisition module is used to obtain the behavior status monitoring video of the monitored elderly living alone collected by the camera;

[0008] A key frame extraction module, used to extract multiple behavior state monitoring key frames from the behavior state monitoring video;

[0009] An optical flow image extraction module, configured to extract a sequence of optical flow images from the plurality of behavior state monitoring key frames based on an optical flow feature extraction network;

[0010] A spatial stream image feature extraction module, used for passing the plurality of behavior state monitoring key frames through a spatial stream feature extractor based on a three-dimensional convolutional neural network model to obtain a first behavior monitoring feature map;

[0011] A time stream image feature extraction module, used for passing the sequence of optical flow images through a time stream feature extractor based on a three-dimensional convolutional neural network model to obtain a second behavior monitoring feature map;

[0012] A behavior feature fusion module, used for fusing the first behavior monitoring feature graph and the second behavior monitoring feature graph to obtain a classification feature graph;

[0013] The safety help detection module is used to pass the classification feature map through a classifier to obtain a classification result, and the classification result is used to indicate whether a safety help request is generated.

[0014] In the above-mentioned digital rural elderly living alone safety assistance system, the spatial stream image feature extraction module is used to:

[0015] Use the spatial stream feature extractor to perform the following operations on the input data in the forward pass of the layer:

[0016] Performing three-dimensional convolution processing on the input data based on a three-dimensional convolution kernel to obtain a first convolution feature map;

[0017] Performing mean pooling processing on the first convolutional feature map to obtain a first pooled feature map;

[0018] Performing nonlinear activation on the first pooled feature map to obtain a first activated feature map;

[0019] The output of the last layer of the spatial stream feature extractor is the first behavior monitoring feature map, and the input of the first layer of the spatial stream feature extractor is the multiple behavior state monitoring key frames.

[0020] In the above-mentioned digital rural elderly living alone safety assistance system, the time stream image feature extraction module is used to:

[0021] Use the temporal stream feature extractor to perform the following operations on the input data in the forward pass of the layer:

[0022] Performing three-dimensional convolution processing on the input data based on a three-dimensional convolution kernel to obtain a second convolution feature map;

[0023] Performing mean pooling processing on the second convolutional feature map to obtain a second pooled feature map;

[0024] Performing nonlinear activation on the second pooled feature map to obtain a second activated feature map;

[0025] The output of the last layer of the temporal flow feature extractor is the second behavior monitoring feature map, and the input of the first layer of the temporal flow feature extractor is the sequence of optical flow images.

[0026] In the above-mentioned digital rural elderly living alone safety assistance system, the behavior feature fusion module includes:

[0027] An optimization factor calculation unit, used to calculate the Gaussian regression uncertainty factors of the first behavior monitoring feature graph and the second behavior monitoring feature graph respectively to obtain a first Gaussian regression uncertainty factor and a second Gaussian regression uncertainty factor;

[0028] A weighted optimization unit, configured to perform weighted optimization on the first behavior monitoring feature graph and the second behavior monitoring feature graph using the first Gaussian regression uncertainty factor and the second Gaussian regression uncertainty factor as weighted coefficients to obtain an optimized first behavior monitoring feature graph and an optimized second behavior monitoring feature graph;

[0029] The optimized feature fusion unit is used to fuse the optimized first behavior monitoring feature map and the optimized second behavior monitoring feature map to obtain the classification feature map.

[0030] In the above-mentioned digital rural elderly living alone safety assistance system, the optimization factor calculation unit is used to:

[0031] Calculating the Gaussian regression uncertainty factors of the first behavior monitoring feature graph and the second behavior monitoring feature graph respectively using the following optimization formula to obtain the first Gaussian regression uncertainty factor and the second Gaussian regression uncertainty factor;

[0032] Wherein, the optimization formula is:

[0033]

[0034]

[0035] Among them, f 1i is the feature value of the ith position in the first behavior monitoring feature map, f 2iis the feature value of the ith position in the second behavior monitoring feature map, S is the scale of the feature map, μ1 and σ1 2 are the mean and variance of the feature value sets of each position in the first behavior monitoring feature map, and μ2 and σ2 2 are respectively the mean and variance of the feature value sets at each position in the second behavior monitoring feature graph, log is a logarithmic function with base 2, w1 and w2 are respectively the first Gaussian regression uncertainty factor and the second Gaussian regression uncertainty factor.

[0036] In the above-mentioned digital rural elderly living alone safety assistance system, the optimized feature fusion unit is used to:

[0037] The optimized first behavior monitoring feature graph and the optimized second behavior monitoring feature graph are fused with the following fusion formula to obtain the classification feature graph;

[0038] Among them, the fusion formula is:

[0039] F c =Concat[F'1,F'2]

[0040] Wherein, F'1 represents the first behavior monitoring characteristic graph after optimization, F'2 represents the second behavior monitoring characteristic graph after optimization, Concat[·] represents the cascade function, and F c Represents the classification feature map.

[0041] In the above-mentioned digital rural elderly living alone safety help system, the safety help detection module includes:

[0042] A feature map expansion unit, used to expand the classification feature map into a classification feature vector according to a row vector or a column vector;

[0043] A uniform activation unit, used for performing uniform activation on the semantic information expressed by the feature rank of the classification feature vector to obtain an optimized classification feature vector;

[0044] A fully connected encoding unit, configured to perform fully connected encoding on the optimized classification feature vector using a plurality of fully connected layers of the classifier to obtain an encoded classification feature vector;

[0045] The classification unit is used to pass the encoded classification feature vector through the Softmax classification function of the classifier to obtain the classification result.

[0046] In the above-mentioned digital rural elderly living alone safety assistance system, the homogenization activation unit is used to: perform homogenization activation of the semantic information expressed by the feature rank of the classification feature vector using the following activation formula to obtain the optimized classification feature vector;

[0047] Wherein, the activation formula is:

[0048]

[0049] Where V is the classification feature vector, v i is the ith eigenvalue of the classification feature vector V, ||V||2 represents the bi-norm of the classification feature vector, log is the logarithm with base 2, and α is the weight hyperparameter, v' i The i-th eigenvalue of the optimized classification feature vector.

[0050] According to one aspect of the present invention, a method for helping elderly people living alone in a digital village safely is also provided, comprising:

[0051] Obtaining the behavior status monitoring video of the monitored elderly living alone collected by the camera;

[0052] Extracting a plurality of behavior state monitoring key frames from the behavior state monitoring video;

[0053] Extracting a sequence of optical flow images from the plurality of behavior state monitoring key frames based on an optical flow feature extraction network;

[0054] The plurality of behavior state monitoring key frames are passed through a spatial stream feature extractor based on a three-dimensional convolutional neural network model to obtain a first behavior monitoring feature graph;

[0055] Passing the sequence of optical flow images through a time flow feature extractor based on a three-dimensional convolutional neural network model to obtain a second behavior monitoring feature map;

[0056] fusing the first behavior monitoring feature graph and the second behavior monitoring feature graph to obtain a classification feature graph;

[0057] The classification feature graph is passed through a classifier to obtain a classification result, and the classification result is used to indicate whether a safety help request is generated.

[0058] According to one aspect of the present application, an electronic device is also provided, comprising: a processor; and a memory, wherein computer program instructions are stored in the memory, and when the computer program instructions are executed by the processor, the processor executes the above method.

[0059] According to one aspect of the present application, a computer-readable storage medium is further provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor is caused to execute the above method.

[0060] Compared with the prior art, the digital rural elderly living alone safety help method, system, device and storage medium provided by the present application first extract multiple behavior state monitoring key frames from the behavior state monitoring video of the monitored elderly living alone collected by the camera, then extract a sequence of optical flow images from the multiple behavior state monitoring key frames based on the optical flow feature extraction network, then pass the multiple behavior state monitoring key frames through a spatial flow feature extractor based on a three-dimensional convolutional neural network model to obtain a first behavior monitoring feature map, then pass the sequence of optical flow images through a temporal flow feature extractor based on a three-dimensional convolutional neural network model to obtain a second behavior monitoring feature map, then fuse the first behavior monitoring feature map and the second behavior monitoring feature map to obtain a classification feature map, and finally pass the classification feature map through a classifier to obtain a classification result for indicating whether a safety help request is generated. In this way, a safety help request can be generated in a timely and effective manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. The following drawings are not deliberately scaled according to the actual size, and the focus is on illustrating the main purpose of the present application.

[0062] Figure 1 This is an application scenario diagram of a digital rural elderly living alone safety assistance system according to an embodiment of the present application;

[0063] Figure 2 It is a block diagram of a digital rural elderly living alone safety assistance system according to an embodiment of the present application;

[0064] Figure 3 It is a block diagram of the behavior feature fusion module in the digital rural elderly living alone safety assistance system according to an embodiment of the present application;

[0065] Figure 4 It is a block diagram of the safety help detection module in the safety help system for elderly people living alone in a digital village according to an embodiment of the present application;

[0066] Figure 5 A flowchart of a method for helping elderly people living alone in a digital village safely according to an embodiment of the present application;

[0067] Figure 6 A schematic diagram of a system architecture of a method for providing safety assistance to elderly people living alone in a digital village according to an embodiment of the present application;

[0068] Figure 7is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0069] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work also fall within the scope of protection of the present application.

[0070] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0071] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0072] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0073] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0074] As mentioned above, in digital villages, the life safety of elderly people living alone is particularly prominent. Once an accident occurs, the elderly living alone may not be rescued in time for a long time, or even endanger their lives. Therefore, a method, system, device and storage medium for safe help for elderly people living alone in digital villages are needed.

[0075] Accordingly, considering that in the actual process of safety monitoring of elderly people living alone, the elderly may not be able to ask for help on their own in unexpected situations, in order to enable the elderly to get rescue help in a timely and effective manner in unexpected situations, it is possible to determine whether to automatically generate a safety help request based on the elderly's behavior status monitoring video. However, since there is a large amount of information in the behavior status monitoring video, the characteristic information about the elderly's behavior status is an implicit behavior semantic feature in the video. In other words, since each elderly person has different behavioral habits, their behavior in the monitoring video is different. Therefore, there will be great difficulties in actually extracting the semantic features of the elderly's behavior status, making it impossible for the elderly to get timely rescue help.

[0076] Figure 1 This is an application scenario diagram of the digital rural elderly living alone safety assistance system according to the embodiment of the present application. Figure 1 As shown, in this application scenario, first, obtain the image captured by the camera (for example, Figure 1 The behavior status monitoring video of the monitored elderly living alone (e.g., Figure 1 D), then, the behavior status monitoring video is input into a server deployed with a digital rural elderly living alone safety assistance algorithm (for example, Figure 1 S) as shown in the figure, wherein the server can use the digital rural elderly living alone safety help algorithm to process the behavior status monitoring video to obtain a classification result indicating whether a safety help request is generated.

[0077] Figure 2 Schematic diagram of a digital rural elderly living alone safety assistance system according to an embodiment of the present application. Figure 2As shown, according to the embodiment of the present application, the digital rural elderly living alone safety help system 100 includes: an elderly behavior status video acquisition module 110, which is used to obtain the behavior status monitoring video of the monitored elderly living alone collected by a camera; a key frame extraction module 120, which is used to extract multiple behavior status monitoring key frames from the behavior status monitoring video; an optical flow image extraction module 130, which is used to extract a sequence of optical flow images from the multiple behavior status monitoring key frames based on an optical flow feature extraction network; a spatial flow image feature extraction module 140, which is used to pass the multiple behavior status monitoring key frames through a spatial flow feature extractor based on a three-dimensional convolutional neural network model to obtain a first behavior monitoring feature map; a temporal flow image feature extraction module 150, which is used to pass the sequence of optical flow images through a temporal flow feature extractor based on a three-dimensional convolutional neural network model to obtain a second behavior monitoring feature map; a behavior feature fusion module 160, which is used to fuse the first behavior monitoring feature map and the second behavior monitoring feature map to obtain a classification feature map; and a safety help detection module 170, which is used to pass the classification feature map through a classifier to obtain a classification result, and the classification result is used to indicate whether a safety help request is generated.

[0078] More specifically, in the embodiment of the present application, the elderly behavior status video acquisition module 110 is used to obtain the behavior status monitoring video of the monitored elderly living alone collected by the camera. In order to enable the elderly to obtain rescue help in a timely and effective manner in an unexpected situation, it can be determined whether to automatically generate a safety request for help based on the elderly behavior status monitoring video.

[0079] More specifically, in an embodiment of the present application, the key frame extraction module 120 is used to extract multiple behavior state monitoring key frames from the behavior state monitoring video. In the behavior state monitoring video, the behavior state change characteristics of the elderly living alone can be represented by the difference between adjacent monitoring frames in the behavior state monitoring video, that is, the time series change of the behavior state of the elderly living alone is represented by the image representation of adjacent image frames. However, considering that the difference between adjacent frames in the behavior state monitoring video is small and there is a large amount of data redundancy, therefore, in order to reduce the amount of calculation and avoid the adverse effects of data redundancy on detection, the behavior state monitoring video is sampled by key frames at a predetermined sampling frequency to extract multiple behavior state monitoring key frames from the behavior state monitoring video.

[0080] More specifically, in an embodiment of the present application, the optical flow image extraction module 130 is used to extract a sequence of optical flow images from the multiple behavior state monitoring key frames based on an optical flow feature extraction network. In this way, continuous information on changes in the behavior state of elderly people living alone can be obtained. It should be understood that optical flow refers to the displacement of pixels in an image in the direction of the time axis, which can be used to describe the movement trajectory and speed of the elderly in the image sequence. Therefore, in the technical solution of the present application, after extracting multiple behavior state monitoring key frames from the behavior state monitoring video, the time flow characteristics of the behavior state of the elderly living alone can be obtained by analyzing the optical flow image sequence between these key frames, which is conducive to a more accurate understanding of the behavioral semantics of the elderly in the subsequent process.

[0081] More specifically, in an embodiment of the present application, the spatial stream image feature extraction module 140 is used to pass the multiple behavior state monitoring key frames through a spatial stream feature extractor based on a three-dimensional convolutional neural network model to obtain a first behavior monitoring feature map. In this way, the changes in the behavior state of the elderly with spatial position can be better described. Specifically, in the technical solution of the present application, the multiple behavior state monitoring key frames are passed through a spatial stream feature extractor based on a three-dimensional convolutional neural network model. The three-dimensional convolutional neural network model can automatically learn and extract the characteristics of the elderly's behavior state that changes with spatial position, and generate a spatial stream feature map, that is, the first behavior monitoring feature map. In this way, the changes in the elderly's behavior state can be described more comprehensively, and the accuracy of the elderly's behavior semantic understanding can be improved.

[0082] It should be understood that Convolutional Neural Network (CNN) is an artificial neural network that is widely used in image recognition and other fields. Convolutional neural network may include input layer, hidden layer and output layer, wherein the hidden layer may include convolution layer, pooling layer, activation layer and fully connected layer, etc. The previous layer performs corresponding operations according to the input data and outputs the operation results to the next layer. The input initial data is processed through multiple layers of operations to obtain a final result.

[0083] Accordingly, in a specific example, the spatial stream image feature extraction module 140 is used to: use the spatial stream feature extractor to perform the following on the input data in the forward pass of the layer: perform three-dimensional convolution processing on the input data based on the three-dimensional convolution kernel to obtain a first convolution feature map; perform mean pooling processing on the first convolution feature map to obtain a first pooling feature map; and perform nonlinear activation on the first pooling feature map to obtain a first activation feature map; wherein the output of the last layer of the spatial stream feature extractor is the first behavior monitoring feature map, and the input of the first layer of the spatial stream feature extractor is the multiple behavior state monitoring key frames.

[0084] More specifically, in the embodiment of the present application, the time stream image feature extraction module 150 is used to pass the sequence of optical flow images through a time stream feature extractor based on a three-dimensional convolutional neural network model to obtain a second behavior monitoring feature map. The sequence of optical flow images is further subjected to feature mining in a time stream feature extractor based on a three-dimensional convolutional neural network model to extract dynamic change feature information of the state features of the elderly's behavior in the time dimension to generate a second behavior monitoring feature map.

[0085] In particular, here, the convolution kernel of the time flow feature extractor based on the three-dimensional convolutional neural network model is a three-dimensional convolution kernel, which has W (width), H (height) and C (channel dimension). In the technical solution of the present application, the channel dimension of the three-dimensional convolution kernel corresponds to the time dimension of the sequence of the optical flow images arranged as a three-dimensional input tensor. Therefore, when performing three-dimensional convolution encoding, the dynamic change characteristics of the distribution characteristics of the behavioral state of the elderly with the time dimension can be extracted, which is conducive to the understanding of the behavioral semantics of the elderly.

[0086] Accordingly, in a specific example, the time stream image feature extraction module 150 is used to: use the time stream feature extractor to perform the following on the input data in the forward pass of the layer: perform three-dimensional convolution processing on the input data based on the three-dimensional convolution kernel to obtain a second convolution feature map; perform mean pooling processing on the second convolution feature map to obtain a second pooling feature map; and perform nonlinear activation on the second pooling feature map to obtain a second activation feature map; wherein the output of the last layer of the time stream feature extractor is the second behavior monitoring feature map, and the input of the first layer of the time stream feature extractor is the sequence of optical flow images.

[0087] More specifically, in an embodiment of the present application, the behavior feature fusion module 160 is used to fuse the first behavior monitoring feature graph and the second behavior monitoring feature graph to obtain a classification feature graph. The change of the elderly's behavior state not only changes with time, but also changes with the spatial position. Therefore, it is necessary to fuse the time flow characteristics and spatial flow characteristics of the elderly's behavior state in order to comprehensively characterize the changes in the elderly's behavior state, so as to more accurately understand the semantics of the elderly's behavior. Based on this, in the technical solution of the present application, the first behavior monitoring feature graph and the second behavior monitoring feature graph are further fused to obtain a classification feature graph. In this way, the temporal dynamic change feature information of the elderly's behavior state changing with time and the change feature information of the elderly's behavior state changing with spatial position can be fused, so as to more comprehensively describe the changes in the elderly's behavior state, which is conducive to a more comprehensive semantic understanding of the elderly's behavior, thereby generating a safety request for help in a timely and effective manner.

[0088] Accordingly, in a specific example, Figure 3 As shown, the behavior feature fusion module 160 includes: an optimization factor calculation unit 161, used to calculate the Gaussian regression uncertainty factors of the first behavior monitoring feature graph and the second behavior monitoring feature graph respectively to obtain a first Gaussian regression uncertainty factor and a second Gaussian regression uncertainty factor; a weighted optimization unit 162, used to perform weighted optimization on the first behavior monitoring feature graph and the second behavior monitoring feature graph using the first Gaussian regression uncertainty factor and the second Gaussian regression uncertainty factor as weighting coefficients to obtain an optimized first behavior monitoring feature graph and an optimized second behavior monitoring feature graph; and an optimization feature fusion unit 163, used to fuse the optimized first behavior monitoring feature graph and the optimized second behavior monitoring feature graph to obtain the classification feature graph.

[0089] In particular, in the technical solution of the present application, here, considering the semantic error of the source image of the behavior status monitoring video during the data acquisition process, the sequence of the optical flow images will be further introduced when the optical flow feature extraction network is used for optical flow feature extraction, and the spatial flow feature extractor based on the three-dimensional convolutional neural network model and the temporal flow feature extractor based on the three-dimensional convolutional neural network model perform feature extraction independently, which makes the first behavior monitoring feature map and the second behavior monitoring feature map have their own Gaussian distribution error uncertainty of feature distribution, which leads to the classification regression error of the classification feature map when the first behavior monitoring feature map and the second behavior monitoring feature map are fused to obtain the classification feature map, affecting the accuracy of the classification result obtained by the classification feature map through the classifier. Based on this, in the technical solution of the present application, the Gaussian regression uncertainty factors of the first behavior monitoring feature map and the second behavior monitoring feature map are calculated respectively.

[0090] Accordingly, in a specific example, the optimization factor calculation unit 161 is used to calculate the Gaussian regression uncertainty factors of the first behavior monitoring feature graph and the second behavior monitoring feature graph respectively using the following optimization formula to obtain the first Gaussian regression uncertainty factor and the second Gaussian regression uncertainty factor; wherein the optimization formula is:

[0091]

[0092]

[0093] Among them, f 1i is the feature value of the ith position in the first behavior monitoring feature map, f 2i is the feature value of the ith position in the second behavior monitoring feature map, S is the scale of the feature map, μ1 and σ1 2 are the mean and variance of the feature value sets of each position in the first behavior monitoring feature map, and μ2 and σ2 2 are respectively the mean and variance of the feature value sets at each position in the second behavior monitoring feature graph, log is a logarithmic function with base 2, w1 and w2 are respectively the first Gaussian regression uncertainty factor and the second Gaussian regression uncertainty factor.

[0094] Here, for the unknown regression of the classification feature graph that may be caused by the distribution uncertainty information of the integrated feature set of each of the first behavior monitoring feature graph and the second behavior monitoring feature graph, the mean and variance as statistical quantization parameters are used to perform scalar measurement of the statistical characteristics of the feature set, so as to extend the normal distribution cognitive model represented by the feature error to the unknown distribution regression model, and realize the transfer learning based on the natural distribution transfer on the feature set scale. In this way, the weighted optimization unit 162 can realize the uncertainty correction based on self-calibration when the first behavior monitoring feature graph and the second behavior monitoring feature graph are fused by weighting the first behavior monitoring feature graph and the second behavior monitoring feature graph respectively with the above-mentioned Gaussian regression uncertainty factor, so as to correct the classification regression error existing in the classification feature graph and improve the accuracy of the classification result obtained by the classification feature graph through the classifier. In this way, the semantics of the behavior state of the elderly can be accurately identified and fully understood, so as to automatically generate a safety request for help for the elderly who need help, so as to ensure the life safety of the elderly living alone.

[0095] Accordingly, in a specific example, the optimized feature fusion unit 163 is used to fuse the optimized first behavior monitoring feature graph and the optimized second behavior monitoring feature graph using the following fusion formula to obtain the classification feature graph; wherein the fusion formula is:

[0096] F c =Concat[F'1,F'2]

[0097] Wherein, F'1 represents the first behavior monitoring characteristic graph after optimization, F'2 represents the second behavior monitoring characteristic graph after optimization, Concat[·] represents the cascade function, and F c Represents the classification feature map.

[0098] More specifically, in the embodiment of the present application, the safety help detection module 170 is used to pass the classification feature graph through a classifier to obtain a classification result, and the classification result is used to indicate whether a safety help request is generated. Based on the classification result, a safety help request is automatically generated for the elderly who need help to ensure the life safety of the elderly living alone.

[0099] It should be understood that the role of the classifier is to use the given categories and known training data to learn classification rules, and then classify (or predict) unknown data. Logistic regression and SVM are often used to solve binary classification problems. For multi-class classification problems, logistic regression or SVM can also be used, but multiple binary classifications are required to form a multi-classification. However, this is prone to errors and is not efficient. Commonly used multi-classification methods include the Softmax classification function.

[0100] Accordingly, in a specific example, Figure 4 As shown, the security help detection module 170 includes: a feature map expansion unit 171, used to expand the classification feature map into a classification feature vector according to a row vector or a column vector; a uniform activation unit 172, used to perform uniform activation on the semantic information of the feature rank expression of the classification feature vector to obtain an optimized classification feature vector; a fully connected encoding unit 173, used to use multiple fully connected layers of the classifier to fully connect encode the optimized classification feature vector to obtain an encoded classification feature vector; and a classification unit 174, used to pass the encoded classification feature vector through the Softmax classification function of the classifier to obtain the classification result.

[0101] Here, in the technical solution of the present application, the first behavior monitoring feature map and the second behavior monitoring feature map respectively represent the temporal flow features based on the three-dimensional convolution kernel and the spatial flow features based on the three-dimensional convolution kernel of the behavior state of the elderly object, that is, considering the semantic error of the source image of the behavior state monitoring video during the data acquisition process, the sequence of optical flow images will be further introduced when the optical flow feature extraction network is used for optical flow feature extraction, and the spatial flow feature extractor based on the three-dimensional convolutional neural network model and the temporal flow feature extractor based on the three-dimensional convolutional neural network model perform feature extraction independently, which makes the first behavior monitoring feature map and the second behavior monitoring feature map have Gaussian distribution error uncertainty of their own feature distribution, resulting in the scale-heuristic class probability mapping of the local features of the classification feature map obtained by fusing the first behavior monitoring feature map and the second behavior monitoring feature map for the image semantic feature domain transfer expression scale, considering the local feature semantic mismatch of the local feature matrices of the first behavior monitoring feature map and the second behavior monitoring feature map, resulting in the training efficiency of the classifier being reduced. Based on this, when the classification feature graph is classified by a classifier, the present application performs homogenization activation on the semantic information expressed by the feature rank of the classification feature vector obtained after the classification feature graph is expanded.

[0102] In a specific example, the homogenization activation unit 172 is used to: perform homogenization activation of the semantic information expressed by the feature rank of the classification feature vector using the following activation formula to obtain the optimized classification feature vector;

[0103] Wherein, the activation formula is:

[0104]

[0105] Where V is the classification feature vector, v i is the i-th eigenvalue of the classification feature vector V, ||v||2 represents the bi-norm of the classification feature vector, log is the logarithm with base 2, and α is the weight hyperparameter, v' i The i-th eigenvalue of the optimized classification feature vector

[0106] Here, considering the feature distribution mapping of the feature distribution of the classification feature vector V from the high-dimensional feature space to the class regression space, different mapping modes will be presented at different feature distribution levels based on diversified features, resulting in the inability of the scale-heuristic mapping strategy to obtain the optimal efficiency. Therefore, the semantic information of the rank expression based on the feature vector norm is homogenized instead of the scale for feature matching. Similar feature rank expressions can be activated in a similar manner and the correlation between feature rank expressions with large differences can be reduced, thereby solving the problem of low efficiency of probability expression mapping of the feature distribution of the classification feature vector V under different spatial rank expressions and improving the training efficiency of the classification feature map when classified by the classifier.

[0107] In summary, the digital rural elderly living alone safety help system 100 based on the embodiment of the present application is explained, which first extracts multiple behavior status monitoring key frames from the behavior status monitoring video of the monitored elderly living alone collected by the camera, then extracts a sequence of optical flow images from the multiple behavior status monitoring key frames based on the optical flow feature extraction network, then passes the multiple behavior status monitoring key frames through a spatial flow feature extractor based on a three-dimensional convolutional neural network model to obtain a first behavior monitoring feature map, then passes the sequence of optical flow images through a temporal flow feature extractor based on a three-dimensional convolutional neural network model to obtain a second behavior monitoring feature map, then, fuses the first behavior monitoring feature map and the second behavior monitoring feature map to obtain a classification feature map, and finally, passes the classification feature map through a classifier to obtain a classification result for indicating whether a safety help request is generated. In this way, a safety help request can be generated in a timely and effective manner.

[0108] As described above, the digital rural elderly living alone safety assistance system 100 according to the embodiment of the present application can be implemented in various terminal devices, such as a server having a digital rural elderly living alone safety assistance algorithm based on the embodiment of the present application. In one example, the digital rural elderly living alone safety assistance system 100 based on the embodiment of the present application can be integrated into the terminal device as a software module and / or a hardware module. For example, the digital rural elderly living alone safety assistance system 100 based on the embodiment of the present application can be a software module in the operating system of the terminal device, or it can be an application developed for the terminal device; of course, the digital rural elderly living alone safety assistance system 100 based on the embodiment of the present application can also be one of the many hardware modules of the terminal device.

[0109] Alternatively, in another example, the digital rural elderly living alone safety assistance system 100 based on the embodiment of the present application and the terminal device may also be separate devices, and the digital rural elderly living alone safety assistance system 100 may be connected to the terminal device via a wired and / or wireless network, and transmit interactive information in accordance with an agreed data format.

[0110] Figure 5 This is a flow chart of a method for helping elderly people living alone in a digital village safely according to an embodiment of the present application. Figure 5 As shown, according to the embodiment of the present application, the method for safe help-seeking for elderly people living alone in a digital village includes: S110, obtaining a behavior status monitoring video of the monitored elderly people living alone collected by a camera; S120, extracting multiple behavior status monitoring key frames from the behavior status monitoring video; S130, extracting a sequence of optical flow images from the multiple behavior status monitoring key frames based on an optical flow feature extraction network; S140, passing the multiple behavior status monitoring key frames through a spatial flow feature extractor based on a three-dimensional convolutional neural network model to obtain a first behavior monitoring feature map; S150, passing the sequence of optical flow images through a temporal flow feature extractor based on a three-dimensional convolutional neural network model to obtain a second behavior monitoring feature map; S160, fusing the first behavior monitoring feature map and the second behavior monitoring feature map to obtain a classification feature map; and, S170, passing the classification feature map through a classifier to obtain a classification result, and the classification result is used to indicate whether a safety help request is generated.

[0111] Figure 6 Schematic diagram of the system architecture of the method for helping the elderly living alone in a digital village safely according to an embodiment of the present application. Figure 6 As shown, in the system architecture of the digital rural elderly living alone safety help-seeking method, first, a behavior status monitoring video of the monitored elderly living alone collected by a camera is obtained; then, a plurality of behavior status monitoring key frames are extracted from the behavior status monitoring video; then, a sequence of optical flow images is extracted from the plurality of behavior status monitoring key frames based on an optical flow feature extraction network; then, the plurality of behavior status monitoring key frames are passed through a spatial flow feature extractor based on a three-dimensional convolutional neural network model to obtain a first behavior monitoring feature map; then, the sequence of optical flow images is passed through a temporal flow feature extractor based on a three-dimensional convolutional neural network model to obtain a second behavior monitoring feature map; then, the first behavior monitoring feature map and the second behavior monitoring feature map are fused to obtain a classification feature map; finally, the classification feature map is passed through a classifier to obtain a classification result, and the classification result is used to indicate whether a safety help request is generated.

[0112] In a specific example, in the above-mentioned digital village elderly people living alone safety help-seeking method, the multiple behavior state monitoring key frames are passed through a spatial stream feature extractor based on a three-dimensional convolutional neural network model to obtain a first behavior monitoring feature map, including: using the spatial stream feature extractor to perform the following on the input data in the forward pass of the layer: performing three-dimensional convolution processing on the input data based on the three-dimensional convolution kernel to obtain a first convolution feature map; performing mean pooling processing on the first convolution feature map to obtain a first pooling feature map; and, performing nonlinear activation on the first pooling feature map to obtain a first activation feature map; wherein the output of the last layer of the spatial stream feature extractor is the first behavior monitoring feature map, and the input of the first layer of the spatial stream feature extractor is the multiple behavior state monitoring key frames.

[0113] In a specific example, in the above-mentioned digital rural elderly people living alone safety help-seeking method, the sequence of optical flow images is passed through a time stream feature extractor based on a three-dimensional convolutional neural network model to obtain a second behavior monitoring feature map, including: using the time stream feature extractor to perform the following on the input data in the forward pass of the layer: performing three-dimensional convolution processing on the input data based on the three-dimensional convolution kernel to obtain a second convolution feature map; performing mean pooling processing on the second convolution feature map to obtain a second pooling feature map; and, performing nonlinear activation on the second pooling feature map to obtain a second activation feature map; wherein the output of the last layer of the time stream feature extractor is the second behavior monitoring feature map, and the input of the first layer of the time stream feature extractor is the sequence of optical flow images.

[0114] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned digital rural elderly living alone safety assistance method have been referred to above. Figures 1 to 4 The digital rural elderly living alone safety assistance system 100 has been introduced in detail, and therefore, its repeated description will be omitted.

[0115] Below, reference Figure 7 To describe an electronic device according to an embodiment of the present application.

[0116] Figure 7 A block diagram of an electronic device according to an embodiment of the present application is illustrated.

[0117] like Figure 7 As shown, the electronic device 10 includes one or more processors 11 and a memory 12 .

[0118] The processor 11 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0119] The memory 12 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the functions of the methods of the various embodiments of the present application described above and / or other desired functions. Various contents such as vectors may also be stored in the computer-readable storage medium.

[0120] In one example, the electronic device 10 may further include: an input device 13 and an output device 14, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0121] The input device 13 may include, for example, a keyboard, a mouse, etc.

[0122] The output device 14 can output various information to the outside, including classification results, etc. The output device 14 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0123] Of course, to simplify, Figure 7 Only some of the components related to the present application in the electronic device 10 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 10 may also include any other appropriate components.

[0124] According to another aspect of the present application, a non-volatile computer-readable storage medium is provided, on which computer-readable instructions are stored. When the instructions are executed by a computer, the above-mentioned method can be executed.

[0125] The program part of the technology can be considered as a "product" or "manufactured product" in the form of executable code and / or related data, which is participated in or realized by computer-readable media. Tangible and permanent storage media can include any memory or storage used by any computer, processor, or similar device or related module. For example, various semiconductor memories, tape drives, disk drives or any similar devices that can provide storage functions for software.

[0126] All or part of the software may sometimes communicate over a network, such as the Internet or other communications network. Such communications can load software from one computer device or processor to another. For example: from a server or host computer of a video target detection device to a hardware platform of a computer environment, or other computer environment that implements the system, or a system with similar functions related to providing information required for target detection. Therefore, another medium capable of transmitting software elements can also be used as a physical connection between local devices, such as light waves, radio waves, electromagnetic waves, etc., transmitted through cables, optical cables or air. Physical media used to carry carriers, such as cables, wireless connections or optical cables and the like, can also be considered as media that carry software. As used herein, unless limited to tangible "storage" media, other terms referring to computer or machine "readable media" refer to media involved in the process of the processor executing any instructions.

[0127] The present application uses specific words to describe the embodiments of the present application. For example, "first / second embodiment", "one embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of the present application. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or multiple times in different positions in this specification does not necessarily refer to the same embodiment. In addition, some features, structures or characteristics in one or more embodiments of the present application can be appropriately combined.

[0128] In addition, it will be appreciated by those skilled in the art that various aspects of the present application may be illustrated and described by a number of patentable categories or situations, including any new and useful process, machine, product or combination of substances, or any new and useful improvements thereto. Accordingly, various aspects of the present application may be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may all be referred to as "data blocks", "modules", "engines", "units", "components" or "systems". In addition, various aspects of the present application may be represented as a computer product located in one or more computer-readable media, which includes computer-readable program code.

[0129] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology and should not be interpreted in an idealized or extremely formal sense, unless explicitly defined as such herein.

[0130] The above is an explanation of the present invention and should not be considered as a limitation thereof. Although several exemplary embodiments of the present invention have been described, it will be readily appreciated by those skilled in the art that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present invention. Therefore, all such modifications are intended to be included within the scope of the present invention as defined in the claims. It should be understood that the above is an explanation of the present invention and should not be considered as being limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present invention is defined by the claims and their equivalents.

Claims

1. A digital rural elderly person living alone safety assistance system, characterized by: include: The elderly behavior status video acquisition module is used to obtain the behavior status monitoring video of the monitored elderly living alone collected by the camera; A key frame extraction module, used to extract multiple behavior state monitoring key frames from the behavior state monitoring video; An optical flow image extraction module, configured to extract a sequence of optical flow images from the plurality of behavior state monitoring key frames based on an optical flow feature extraction network; A spatial stream image feature extraction module, used for passing the plurality of behavior state monitoring key frames through a spatial stream feature extractor based on a three-dimensional convolutional neural network model to obtain a first behavior monitoring feature map; A time stream image feature extraction module, used for passing the sequence of optical flow images through a time stream feature extractor based on a three-dimensional convolutional neural network model to obtain a second behavior monitoring feature map; A behavior feature fusion module, used for fusing the first behavior monitoring feature graph and the second behavior monitoring feature graph to obtain a classification feature graph; A safety help detection module, used for passing the classification feature graph through a classifier to obtain a classification result, wherein the classification result is used to indicate whether a safety help request is generated; The behavior feature fusion module includes: An optimization factor calculation unit, used to calculate the Gaussian regression uncertainty factors of the first behavior monitoring feature graph and the second behavior monitoring feature graph respectively to obtain a first Gaussian regression uncertainty factor and a second Gaussian regression uncertainty factor; A weighted optimization unit, configured to perform weighted optimization on the first behavior monitoring feature graph and the second behavior monitoring feature graph using the first Gaussian regression uncertainty factor and the second Gaussian regression uncertainty factor as weighted coefficients to obtain an optimized first behavior monitoring feature graph and an optimized second behavior monitoring feature graph; An optimization feature fusion unit, used for fusing the optimized first behavior monitoring feature graph and the optimized second behavior monitoring feature graph to obtain the classification feature graph; The optimization factor calculation unit is used for: Calculating the Gaussian regression uncertainty factors of the first behavior monitoring feature graph and the second behavior monitoring feature graph respectively using the following optimization formula to obtain the first Gaussian regression uncertainty factor and the second Gaussian regression uncertainty factor; Wherein, the optimization formula is: Among them, f 1i is the feature value of the ith position in the first behavior monitoring feature map, f 2i is the feature value of the ith position in the second behavior monitoring feature map, S is the scale of the feature map, μ1 and σ1 2 are the mean and variance of the feature value sets of each position in the first behavior monitoring feature map, and μ2 and σ2 2 are the mean and variance of the feature value sets at each position in the second behavior monitoring feature graph, log is a logarithmic function with base 2, w1 and w2 are the first Gaussian regression uncertainty factor and the second Gaussian regression uncertainty factor, respectively; The safety help detection module includes: A feature map expansion unit, used to expand the classification feature map into a classification feature vector according to a row vector or a column vector; A uniform activation unit, used for performing uniform activation on the semantic information expressed by the feature rank of the classification feature vector to obtain an optimized classification feature vector; A fully connected encoding unit, configured to perform fully connected encoding on the optimized classification feature vector using a plurality of fully connected layers of the classifier to obtain an encoded classification feature vector; A classification unit, used for passing the encoded classification feature vector through the Softmax classification function of the classifier to obtain the classification result; The homogenization activation unit is used to: perform homogenization activation on the semantic information expressed by the feature rank of the classification feature vector using the following activation formula to obtain the optimized classification feature vector; Wherein, the activation formula is: Where V is the classification feature vector, v i is the ith eigenvalue of the classification feature vector V, ||V||2 represents the bi-norm of the classification feature vector, log is the logarithm with base 2, and α is the weight hyperparameter, v' i the i-th eigenvalue of the optimized classification feature vector; The semantic information of the rank expression based on the norm of the feature vector is homogenized instead of the scale for feature matching, similar feature rank expressions are activated in a similar way, and the correlation between feature rank expressions with large differences is reduced to solve the problem of low efficiency of probability expression mapping of the feature distribution of the classification feature vector under different spatial rank expressions.

2. The digital rural elderly living alone safety assistance system according to claim 1 is characterized in that: The spatial stream image feature extraction module is used to: Use the spatial stream feature extractor to perform the following operations on the input data in the forward pass of the layer: Performing three-dimensional convolution processing on the input data based on a three-dimensional convolution kernel to obtain a first convolution feature map; Performing mean pooling processing on the first convolutional feature map to obtain a first pooled feature map; Performing nonlinear activation on the first pooled feature map to obtain a first activated feature map; The output of the last layer of the spatial stream feature extractor is the first behavior monitoring feature map, and the input of the first layer of the spatial stream feature extractor is the multiple behavior state monitoring key frames.

3. The digital rural elderly living alone safety assistance system according to claim 2 is characterized in that: The time stream image feature extraction module is used to: Use the temporal stream feature extractor to perform the following operations on the input data in the forward pass of the layer: Performing three-dimensional convolution processing on the input data based on a three-dimensional convolution kernel to obtain a second convolution feature map; Performing mean pooling processing on the second convolutional feature map to obtain a second pooled feature map; Performing nonlinear activation on the second pooled feature map to obtain a second activated feature map; The output of the last layer of the temporal flow feature extractor is the second behavior monitoring feature map, and the input of the first layer of the temporal flow feature extractor is the sequence of optical flow images.

4. The digital rural elderly living alone safety assistance system according to claim 3 is characterized in that: The optimized feature fusion unit is used to: The optimized first behavior monitoring feature graph and the optimized second behavior monitoring feature graph are fused with the following fusion formula to obtain the classification feature graph; Among them, the fusion formula is: F c =Concat[F′1,F′2] Wherein, F'1 represents the first behavior monitoring characteristic graph after optimization, F'2 represents the second behavior monitoring characteristic graph after optimization, Concat[·] represents the cascade function, and F c Represents the classification feature map.

5. A digital rural elderly person living alone safety assistance method used in the digital rural elderly person living alone safety assistance system as claimed in claim 1, characterized in that: include: Obtaining the behavior status monitoring video of the monitored elderly living alone collected by the camera; Extracting a plurality of behavior state monitoring key frames from the behavior state monitoring video; Extracting a sequence of optical flow images from the plurality of behavior state monitoring key frames based on an optical flow feature extraction network; The plurality of behavior state monitoring key frames are passed through a spatial stream feature extractor based on a three-dimensional convolutional neural network model to obtain a first behavior monitoring feature graph; Passing the sequence of optical flow images through a time flow feature extractor based on a three-dimensional convolutional neural network model to obtain a second behavior monitoring feature map; fusing the first behavior monitoring feature graph and the second behavior monitoring feature graph to obtain a classification feature graph; The classification feature graph is passed through a classifier to obtain a classification result, and the classification result is used to indicate whether a safety help request is generated.

6. The method for helping elderly people living alone in a digital village safely according to claim 5 is characterized in that: The fusing the first behavior monitoring feature graph and the second behavior monitoring feature graph to obtain a classification feature graph includes: Calculating Gaussian regression uncertainty factors of the first behavior monitoring feature graph and the second behavior monitoring feature graph respectively to obtain a first Gaussian regression uncertainty factor and a second Gaussian regression uncertainty factor; Using the first Gaussian regression uncertainty factor and the second Gaussian regression uncertainty factor as weighting coefficients, weighted optimization is performed on the first behavior monitoring feature graph and the second behavior monitoring feature graph to obtain an optimized first behavior monitoring feature graph and an optimized second behavior monitoring feature graph; The optimized first behavior monitoring feature graph and the optimized second behavior monitoring feature graph are fused to obtain the classification feature graph.

7. An electronic device, characterized in that: include: processor; and a memory, wherein computer program instructions are stored in the memory, and when the computer program instructions are executed by the processor, the processor is caused to perform the method according to any one of claims 5 to 6.

8. A computer-readable storage medium, characterized in that: Computer program instructions are stored thereon, and when the computer program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 5 to 6.

Images