Multi-target real-time fall detection method and device in human-machine-object fusion scene

Abstract

The application discloses a multi-target real-time fall detection method and device in a human-machine-object fusion scene, comprising generating query pictures according to the current states of the targets; respectively mapping the query pictures and a plurality of reference pictures to a same mapping space; and obtaining real-time fall detection results based on query embedding vectors and reference embedding vectors in the mapping space. The application realizes a multi-target fall detection monitoring system under different privacy protection settings, saves computing power, improves efficiency, realizes real-time detection effect, and still has high accuracy and low misjudgment rate in the case of insufficient fall data, avoiding human and financial losses.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly to a multi-target real-time fall detection method and device in a human-machine-object fusion scene. BACKGROUND

[0002] With the continuous aggravation of population aging and the increase of the number of the elderly in China, according to the records of CDC (2017), the threat of falls to the elderly is increasing, and gradually becoming a problem to be solved. The increasing number of falls not only causes a serious burden to the family and society, but also in the case of injury caused by falling, if there is a lack of immediate care and support, it is extremely likely to threaten life. At the same time, with the rapid development and acceleration of the fusion of Internet of Things, big data and artificial intelligence technology, the continuous deepening of the fusion of human-machine-object through various sensors networking to collect various types of data in real time, and then through data mining and machine learning methods to process and understand intelligently in terminal devices, edge devices or cloud, how to promote the cooperation of human-machine-object in the human-machine-object fusion scene, and solve the social and safety problems caused by falling is particularly important. Therefore, it has very important practical significance to apply computer vision to fall detection and realize fall detection in the human-machine-object scene.

[0003] The fall detection method based on computer vision mainly includes 1) fall detection based on body posture analysis: by using the static features of human posture, the posture of human body (lying, standing, sitting, etc.) is recognized, and then the lying posture is considered as falling. The problem of this method is that it cannot distinguish the activities such as sleeping in the lying posture, has a large misrecognition rate, and is greatly affected by the environment. 2) fall detection based on static detection: by analyzing whether the human body is in a lying posture after a period of time to determine whether a fall has occurred. This method still has the disadvantages of body posture analysis, and still has the same problems. 3) fall detection based on head movement analysis: by using methods such as depth camera and thermal camera to track head movement, and by analyzing the movement of the head to determine whether a fall has occurred. This method has a very high movement of the head when falling, which is easy to lose the tracking of the head, resulting in low detection accuracy, and has very obvious requirements for the device, which is difficult to implement. 4) fall detection method based on deep learning / machine learning: by learning the posture features of the human body, using neural network or traditional machine learning method, the precision is high, and the generalization is good.

[0004] In the disclosed machine learning / deep learning-based fall detection method, the fall detection process is generally divided into two steps: motion target detection and fall determination, one network is used for human body detection, and another network is used for fall determination. When performing human body detection, all human bodies are detected without screening and human body feature extraction is performed, and many human bodies that do not need to be extracted are also extracted, which obviously wastes a lot of computing power and computing time, and it is difficult to meet the real-time requirement. In addition, the disclosed network model does not train for easy misjudgment, resulting in a high misjudgment rate. Moreover, the number of fall samples in the disclosed human behavior data set is extremely small, and the potential model pays more attention to other actions rather than fall actions, resulting in a low fall detection accuracy. SUMMARY

[0005] In order to overcome the deficiencies of the existing fall detection algorithm such as poor real-time performance, low accuracy and high false positive rate, and better cope with the situation of insufficient data, the present application provides a multi-target real-time fall detection method and device in a human-machine-thing fusion scene. By using the metric learning method, the fall recognition in the small sample situation can be well solved.

[0006] The technical solution adopted by the present application to solve the technical problems is:

[0007] A multi-target real-time fall detection method in a human-machine-thing fusion scene, comprising the following steps:

[0008] Generating a query image I j , wherein j is the target serial number;

[0009] Mapping each query image I j to the same mapping space with a plurality of reference images, wherein each reference image is a representative action image;

[0010] Based on the query embedding vector and the reference embedding vector in the mapping space, the real-time fall detection result is obtained.

[0011] Further, the method for generating the query image I j includes one or more of the following: based on the skeleton node data generation, based on the inertia data generation and based on the acceleration data generation, wherein the skeleton node generation is realized by the following steps:

[0012] 1) using a Yolo model to obtain a target;

[0013] 2) performing skeleton node extraction on the target with fall signs to obtain a queue of a plurality of skeleton node sequences;

[0014] 3) generating a matrix S based on each skeleton node sequencej each row of the matrix S j represents a bone node, and each column represents the coordinate position of a certain skeleton node when sampled at a specific time interval;

[0015] 4) converting the matrix S j into a matrix S' j by a regularization method;

[0016] 5) encoding each matrix S' j in the RGB channel respectively to generate a query picture I j .

[0017] Further, the target with falling signs is obtained by the following steps:

[0018] 1) obtaining the aspect ratio of each target detection frame;

[0019] 2) obtaining the target with falling signs according to the set threshold and the aspect ratio.

[0020] Further, the method for extracting the bone node includes AlphaPose algorithm, OpenPose algorithm, CPN algorithm, MSPN algorithm, RMPE algorithm or improved AlphaPose algorithm using a lighter network.

[0021] Further, the bone node includes a nose node, a left eye node, a right eye node, a left ear node, a right ear node, a left shoulder node, a right shoulder node, a left elbow node, a right elbow node, a left wrist node, a right wrist node, a left knee node, a right knee node, a left ankle node, a right ankle node, a left hip node and a right hip node.

[0022] Further, the query picture I j is mapped to the mapping space by the following steps:

[0023] 1) extracting the features of the query picture I j ;

[0024] 2) generating a query embedding vector using a two-layer perception machine based on the features;

[0025] 3) mapping the query embedding vector to the mapping space.

[0026] Further, the features of the query picture I j are extracted by the following steps:

[0027] 1) using Resnet18 network as the skeleton of the feature extraction network and initializing using the pre-trained feature extraction network parameters;

[0028] 2) based on the training set of the bone node sequence, a training query picture and a label of the training query picture are generated;

[0029] 3) the training query picture is input into a feature extraction network, and the feature extraction network is trained based on the label of the training query picture and a triple loss / multi-similarity loss, to obtain a trained feature extraction network;

[0030] 4) the query picture I j is input into the trained feature extraction network to obtain the feature of the query picture.

[0031] Further, the detection result is obtained by the following steps:

[0032] 1) in the embedding space, a query point of each query embedding vector and a reference point of each reference embedding vector are obtained;

[0033] 2) the nearest neighbor algorithm is used to find the reference point closest to each query point, and the label of the reference point is taken as the predicted label of the query data point, to obtain the detection result.

[0034] Further, if the detection result contains a target predicted to fall, the corresponding contact person is notified by email or short message.

[0035] A storage medium, the storage medium stores a computer program, wherein the computer program is set to execute the above method when running.

[0036] An electronic device comprising a memory and a processor, wherein the memory stores a program for executing the above method.

[0037] The beneficial effects of the present application are:

[0038] 1. By using the deep metric learning method on a limited small sample of fall data set, high accuracy and low misjudgment rate can be achieved even in the case of insufficient fall data;

[0039] 2. Because of the characteristics of metric learning, various falling postures and easily misidentified data can be used as reference sets, thereby reducing the error recognition rate and the loss of manpower and financial resources caused by false alarms;

[0040] 3. The method of combining deep learning and metric learning improves the learning ability of metric learning and provides more accurate fall recognition accuracy;

[0041] 4. By adding a screening and filtering step after human body detection based on Yolo, the computing power and time of subsequent feature extraction are reduced, and real-time detection in a multi-person scene is realized;

[0042] 5. Through the optimization of the data processing flow, unnecessary data processing is saved, the computing power is saved, the efficiency is improved, the effect of real-time detection is realized, and the effect of 15fps can be achieved;

[0043] 6. According to different privacy protection requirements, a fall detection monitoring system under different privacy protection settings is realized.

[0044] 7. 100% detection accuracy is realized on the public dataset SisFall dataset, and 96.75% accuracy is realized on the complex scene dataset ImVia Fall Detection dataset, and the false detection rate is less than 1%. BRIEF DESCRIPTION OF DRAWINGS

[0045] Fig. 1 is the fall detection method architecture of the application.

[0046] Fig. 2 is the fall detection method flowchart of the application.

[0047] Fig. 3 is the data feature extraction representation method flowchart of the application. DETAILED DESCRIPTION

[0048] In the following, exemplary embodiments of the application will be described with reference to the accompanying drawings. In the specification, all features of the actual implementation are not described for the sake of clarity and brevity. However, it should be appreciated that many implementation-specific decisions must be made in the process of developing any such actual implementation, in order to achieve the specific goals of the developer, such as compliance with system- and business-related constraints, which can vary from one implementation to another.

[0049] In addition, it should be noted that, in order to avoid unnecessary details obscuring the application, only the parts closely related to the scheme according to the application are shown in the drawings, and other details not closely related to the application are omitted.

[0050] The fall detection method of the application first detects the human body in the video, and screens out the human body that may fall through screening. The human body state is detected for the second time through deep metric learning and nearest neighbor algorithm, and then it is determined whether a fall event occurs. Through the use of the method combining deep learning and metric learning, training and fall detection recognition are realized in the case of less fall data. At the same time, for the easily confused data, the distance between the easily confused data and the fall data is continuously increased through the deep metric learning method, so as to improve the detection accuracy and reduce the misjudgment rate, and good results are achieved.

[0051] The application is realized through the following steps:

[0052] (1) Preprocess the data of the public data set, eliminate data with low quality, and integrate several existing data sets as a fusion data set for fall detection.

[0053] (2) Feature extraction is performed on the data set, the sequence of human key nodes is extracted, and the human key node sequence is encoded in a special coding manner, and the encoding result is represented by an image.

[0054] (3) Build a deep metric learning network model for training and verification.

[0055] (4) When testing, select representative action pictures as the reference set of metric learning.

[0056] (5) Use Yolo algorithm for human body detection on real-time video

[0057] (6) Filter out normal human bodies that are unlikely to fall by screening the detected human bodies.

[0058] (7) Extract human key nodes from the screened human bodies

[0059] (8) Encode the human key nodes of the human body within a period of time into an image

[0060] (9) Map the image and the image in the reference set to the mapping space through the deep metric learning network

[0061] (10) Use the nearest neighbor algorithm to determine the label of the data point in the reference set closest to the to-be-detected data point.

[0062] (11) In the monitoring background, according to different privacy settings, such as displaying complete video pictures in public places, and in private places such as private homes, only the recognized human skeleton key points can be displayed, so that the background picture and the figure picture are hidden, thereby protecting privacy.

[0063] (12) If it is determined that a fall has occurred, send a fall alarm through email / sms, etc.

[0064] The specific implementation is as follows:

[0065] The present embodiment proposes a multi-person real-time fall detection method based on computer vision and deep metric learning, as shown in Figs. 1-2 The flowchart of the multi-person real-time fall detection method based on computer vision and deep metric learning of the present embodiment.

[0066] Firstly, the key skeleton points in the embodiment include a nose node, a left eye node, a right eye node, a left ear node, a right ear node, a left shoulder node, a right shoulder node, a left elbow node, a right elbow node, a left wrist node, a right wrist node, a left knee node, a right knee node, a left ankle node, a right ankle node, a left hip node, and a right hip node, totaling 17 nodes.

[0067] Since the data set used is the human key node information collected by using the Microsoft Kinect, the detection and extraction of the human skeleton nodes are not required in the training process. However, in the test, since the input data is video data, the human detection and the human skeleton node extraction need to be performed by the application.

[0068] In the human detection, the open-source Yolo model is used to perform frame-by-frame human detection on the test video, and the human detected is selected as the one with a confidence greater than 90%.

[0069] In the process of processing the human skeleton nodes, the application can divide the human skeleton node coordinates of each person in a period of time according to different nodes and different times to obtain a matrix S corresponding to each human skeleton node coordinate, wherein each row of the matrix represents a human skeleton node, and each column represents the coordinate position of a skeleton node sampled according to a specific time interval.

[0070] The application converts the length M of the matrix S j into W and converts the width N of the matrix into H, where M represents the sampling time length, N represents the number of skeleton nodes, W is the width of the picture after conversion into a picture representation, and H is the height of the picture after conversion into a picture representation. By encoding each regularized matrix in the RGB channel, the application can obtain a data set containing N training pictures I1,…,I N and C labels y i ∈1,…,C. Each picture represents a human skeleton key node sequence.

[0071] In addition, the application can also obtain the acceleration, inertia, and other data of the human body at different times through other devices or technical means, and generate a picture according to one or more of the human skeleton node data, acceleration data, and inertia data. For example, a model is trained for the acceleration data and the inertia data respectively, when one of the inertia data or the acceleration data is used, the corresponding model is used to generate a picture; when both the inertia data and the acceleration data are available, the two models are used simultaneously to generate a data set D 1 and a data set D 2 , and then the corresponding picture and picture Simple stitching is performed to generate a picture I i .

[0072] In the feature extraction network, the present application uses Resnet18 as the network skeleton of the present application, and uses the pre-trained model parameters for initialization. After the last feature extraction layer, the present application uses two layers of perception to convert the features into embedding size. The embedding feature vector is improved by the method of deep metric learning.

[0073] The present application combines the triplet loss with the multi-similarity excavator to excavate good triplet candidates during training.

[0074] During the training process, the triplet loss defined by the present application is: given a triplet: an anchor picture I o , a positive example picture Ip representing the same action category, and a negative example picture I n representing a different action category, then the triplet loss is: L t (I o ,I p ,I n )=max(||f(I o )-f(I p )||2-||f(I o )-f(I n )||2+δ,0)。

[0075] Where f represents the mapping function learned using metric learning, f(I o ) represents mapping a picture I o to an embedding vector in the embedding space; δ represents a boundary value, which is a constant set.

[0076] In order to unify the triplet loss and the classification loss, the present application uses the following loss function as the overall training loss function: L=αL t +βL c , where α represents the weight of the triplet loss, β represents the weight of the classifier loss, and L c represents the loss function of the classifier.

[0077] Through the training of the feature extraction network, an Encoder encoder is finally obtained, which can extract the features of the pictures coded by the present application and map them into the embedding space.

[0078] When testing, the present application selects 10 data different in features for each category as ReferenceSample.

[0079] In the test, the real-time video data is first detected for human body, and the human body detection algorithm used here is Yolo.

[0080] The detected human body target is preliminarily screened, and the length-width ratio of the human body target detection frame is used as a screening value here, and the threshold is set to 0.6. All human body target data with a threshold less than 0.6 is input to the next process for detection, and the human body target greater than 0.6 is considered to have no falling sign at present.

[0081] Then the human body target to be detected is extracted by the AlphaPose algorithm. In order to improve the human body skeleton key node detection speed of AlphaPose, a lighter model is used in the present application, which improves the detection efficiency of human body key nodes on the basis of very small loss of accuracy compared with the original model.

[0082] It is easy to understand that the present application can also use other methods for extracting skeleton nodes when extracting skeleton nodes: for example, OpenPose, CPN, MSPN, RMPE, etc.

[0083] For each different human body target, the present application maintains a queue storing human body skeleton node sequences of a fixed time interval, and the size of each queue is set to 30. When the number in the queue is greater than 30, the earliest skeleton node will be deleted from the queue.

[0084] All the skeleton node sequences in the queue are encoded, and then the skeleton sequences are encoded into a picture in the same way as when training data. In this way, for each queue, there is a corresponding picture after encoding. If there are 10 queues (i.e. 10 tracking targets) at this time, there are 10 pictures in the query data set at this time.

[0085] Then, for each picture in the query data set, the feature network trained by the present application is used for processing to obtain an embedding vector. The embedding vector is the coordinate of the corresponding point in the embedding space.

[0086] Similarly, the present application processes each picture in the reference data set selected in advance using the same feature network to obtain a series of reference embedding vectors, which represent some points with known labels in the embedding space.

[0087] The present application finds the nearest reference point in the embedding space for each point in the query data set by the nearest neighbor algorithm, and the label of the reference point is used as the predicted label of the query data point.

[0088] In the monitoring background, according to different privacy settings, complete video pictures can be displayed in public places, while in private places such as private families, only the identified human skeleton key points can be displayed, so as to hide the background picture and the figure picture, and privacy is well protected.

[0089] If there is a target human body predicted to fall, the corresponding medical rescue support organization is informed through an email or a short message, timely medical support is provided for the faller, and the life safety is ensured.

[0090] Obviously, the above embodiments of the present application are only considered as distance made for the clear illustration of the present application, and are not the limitation of the embodiments of the present application. On the basis of the above description, other different forms of changes or variations can be made by the ordinary skilled in the art, and all the embodiments need not and cannot be exhausted. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the claims of the present application.​​

Claims

1. A method for multi-target real-time fall detection in a human-machine convergence scene, comprising the steps of: According to the current state of each target, a query picture I is generated j , wherein j is a target serial number, and the method for generating the query picture I j includes one or more of: generating based on skeleton node data, generating based on inertial data, and generating based on acceleration data, and the process for generating the query picture I j based on skeleton node data includes: obtaining targets using a Yolo model; extracting bone nodes of targets with fall signs to obtain a queue of bone node sequences; wherein the targets with fall signs are obtained by the following steps: obtaining the length-width ratio of each target detection frame; obtaining targets with fall signs according to a set threshold and the length-width ratio; Based on each bone node sequence, a matrix S is generated j Each row of the matrix S j represents a bone node, and each column represents the coordinate position of a certain skeleton node when sampled at a specific time interval. By a regularization method, the matrix S j is converted into the matrix S' j ; The matrices S' are generated by encoding each matrix S j in the RGB channels, respectively, to generate the query image I j ; Each query image I j is mapped to the same mapping space as a number of reference images, each of which is a representative action image; obtaining real-time fall detection results based on query embedding vectors and reference embedding vectors in a mapping space; wherein the detection results are obtained by the following steps: in the embedding space, obtaining query points of each query embedding vector and reference points of each reference embedding vector; finding the nearest reference point to each query point by a nearest neighbor algorithm, and taking the label of the reference point as the predicted label of the query data point to obtain the detection results.

2. The method of claim 1, wherein, The method for extracting bone nodes includes AlphaPose algorithm, OpenPose algorithm, CPN algorithm, MSPN algorithm, RMPE algorithm or improved AlphaPose algorithm using a lighter network.

3. The method of claim 1, wherein, The bone nodes include nose node, left eye node, right eye node, left ear node, right ear node, left shoulder node, right shoulder node, left elbow node, right elbow node, left wrist node, right wrist node, left knee node, right knee node, left ankle node, right ankle node, left hip node and right hip node.

4. The method of claim 1, wherein, The query image I is mapped to the mapping space by the following steps: j mapping to the mapping space: 1) Extracting features of the query image I j ; 2) Based on the features, a two-layer perception machine is used to generate query embedding vectors; 3) The query embedding vectors are mapped to a mapping space.

5. The method of claim 4, wherein, The query image I is extracted by the following steps j features of the query image I: 1) Resnet18 network is used as the skeleton of the feature extraction network, and the pre-trained feature extraction network parameters are used for initialization; 2) Based on the training set of bone node sequences, training query pictures and labels of the training query pictures are generated; 3) The training query pictures are input into the feature extraction network, and the feature extraction network is trained based on the labels of the training query pictures and the triplet loss / multi-similarity loss to obtain the trained feature extraction network; 4) query picture I j Input the trained feature extraction network to obtain the feature of the query picture.

6. The method of claim 1, wherein, If the detection results contain targets predicted to fall, the corresponding contacts are notified by email or SMS. 7.An electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method according to any one of claims 1-6.

Images