Personnel tumble detection method based on human body segmentation network and tumble classification model

Abstract

The invention relates to a human body segmentation network and tumble classification model training-based personnel tumble detection method. The method comprises the following steps of S100, data preparation; step S200, performing video frame cutting; step S300, data annotation preprocessing: annotating the intercepted video frame picture by using an annotation tool; step S400: data division: dividing each folder after the frame cutting processing into training data and verification data; step S500: model training: performing segmentation training on a human body in the video frame picture to obtain a human body segmentation network model Mseg, and performing classification training on the feature vectors by using an SVM model to obtain a continuous multi-frame tumble judgment model Mfall; step S600, carrying out model adaptation; step S700, acquiring data by a camera; step S800: model reasoning: reasoning the video frame data acquired in the step S700 by using trained models Mseg and Mfall, and returning a result of whether personnel fall and a personnel mask coverage result; and step S900, reporting the alarm information.

Description

Technical Field

[0001] The present invention relates to the field of intelligent monitoring, and in particular to a method for detecting and protecting the privacy of a person falling by using an intelligent camera and based on a human body segmentation network and a fall classification model training. Background Art

[0002] As an important application in monitoring systems, fall detection has broad application prospects and social value, especially in the fields of elderly care and smart home. With the increasing awareness of privacy protection, especially in the home environment, users need to be able to detect and notify family members in time when they fall (for example, sound prompts or sound, light, picture or video warnings or transmission to other family members or other designated personnel or institutions), and on the other hand, they have increasingly stringent requirements for monitoring equipment and privacy protection.

[0003] In current monitoring equipment such as camera monitoring applications, the monitoring system cannot automatically capture, discover and accurately determine the fall of a person in real time, and cannot automatically cover the person (e.g., face) to protect the privacy of the person. Such equipment and technology are urgently needed in the field to solve or improve these problems and other technical problems.

[0004] The information included in this background section of the present specification, including any references cited herein and any description or discussion thereof, is included for technical reference purposes only and is not to be construed as subject matter that will limit the scope of the present invention. Summary of the invention

[0005] The present invention is proposed in view of the above and other more concepts.

[0006] Therefore, the present invention aims to provide an effective personnel fall monitoring solution, which can detect the fall event in real time and perform privacy processing on the human body part in the fall image to protect personal privacy.

[0007] In addition, the present invention may also include the function of sending fall alarm information to ensure that relevant personnel can be notified in time when a fall occurs, thereby achieving rapid response and assistance. By combining advanced image processing and deep learning technology, the present invention can fully consider the user's privacy protection needs while meeting security monitoring needs.

[0008] According to the present invention, intelligent monitoring of human falls can be performed through intelligent cameras. When a fall event is intelligently monitored, the human body part in the fall image can be privacy processed and protected in the intelligent camera, and an alarm message of the fall can be sent to notify others, thereby ensuring that the privacy of the fallen person is protected while obtaining timely response and assistance.

[0009] According to one aspect of the present invention, a method for detecting a person falling based on a human body segmentation network model and a human body fall classification model training is provided, comprising the following steps:

[0010] Step S100: data preparation, collecting a large number of video clips containing people falling and non-falling, and naming each of the video clips, wherein the name of each video clip has a fall label or a non-fall label, the number of people appearing in each video clip is always one, and the duration of each video clip is 4-10 seconds;

[0011] Step S200: Video frame capture, using a frame capture tool or code to capture 4 frames of images per second for each file of the video clip at equal intervals, creating a folder named after the name of the video clip, and saving the captured video frame images to the corresponding folder;

[0012] Step S300: data annotation preprocessing, using an annotation tool to annotate the captured video frame images, annotate the human body contours of all video frames in each of the folders, and generate a JSON annotation file;

[0013] Step S400: data division, dividing each of the folders after the frame cutting process into training data and verification data according to a ratio of 7:3;

[0014] Step S500: Model training, using the modified UNet network structure to perform segmentation training on the human body in the video frame image, and training to obtain the human body segmentation network model M seg , and use the SVM model to classify the feature vectors and train a continuous multi-frame fall judgment model M fall ;

[0015] Step S600: Model adaptation, the trained human body segmentation network model M seg and the fall classification model M fall Deploy and transplant to the smart camera, use the corresponding SDK for adaptation, and transform the human body segmentation network model M seg and the fall classification model M fall The file is converted into the specified format and the human body segmentation network model M seg and the fall classification model M fall Quantify;

[0016] Step S700: Camera data acquisition, collecting data through the smart camera, decoding video frames at equal intervals at a frequency of 4 frames per second, and sending the decoded video frame data to the model in step S800 for reasoning;

[0017] Step S800: Model reasoning, using the trained human body segmentation network model M seg and the fall classification model M fall Inferring the video frame data collected in step S700, and returning a result of whether a person falls in the video frame data, and a result of mask coverage of the person in the video frame; and

[0018] Step S900: reporting alarm information. If a fallen person is detected in step S800, the last frame of the image is masked with the human body, and the processed image is sent to the alarm receiver as an alarm image.

[0019] According to one embodiment, the naming format of the video clip is "number_state.mp4", wherein the falling state is named "fall" and the non-falling state is named "nofall".

[0020] According to an embodiment, in the step S300 of data labeling preprocessing, the labeling tool used is Labelme.

[0021] According to one embodiment, in the model training step, the deep learning framework used is PyTorch.

[0022] According to one embodiment, the model used in the model adaptation step S600 includes a human body segmentation network model M seg and fall judgment model M fall .

[0023] According to an embodiment, in the model inference step S800, the returned data format includes the horizontal coordinates and the vertical coordinates of the human body contours of the fallen person and the non-fallen person.

[0024] According to one embodiment, in the step S800 of model inference, the returned data format is:

[0025] {"fall":[[[x 1 ,y 1 ],[x 2 ,y 2 ],....,[x n ,y n ]],...], "nofall":[[[x′ 1 , x′ 1 ],[x′ 2 , x′ 2 ],...,[x′ m , x′ m ]], ...]}

[0026] Among them, [[x 1 ,y1 ],[x 2 ,y 2 ], ..., [x n ,y n ]] is the horizontal and vertical coordinates of the human body contour of the falling person, [[x′ 1 , x′ 1 ],[x′ 2 , x′ 2 ],...,[x′ m , x′ m ]] is the horizontal coordinate of the human body contour of the non-falling person.

[0027] According to one embodiment, in the alarm information reporting step, the color of the mask covering the fallen person is red, and the color of the mask covering the non-fallen person is green.

[0028] According to one embodiment, the alarm information includes a picture covered with a mask of the fallen person, a timestamp and a name of the alarm event.

[0029] According to an embodiment, in the step S900, the mask covering includes performing mask covering processing on at least the head of the human body to provide privacy protection.

[0030] The person fall detection and privacy protection method of the present invention can effectively identify human fall events while protecting the privacy of the monitored person, and has broad application prospects, especially in nursing homes, hospitals, special monitoring places, elderly care and smart homes.

[0031] More embodiments of the present invention can also achieve other advantageous technical effects that are not listed one by one. These other technical effects may be partially described below and are predictable and understandable to those skilled in the art after reading the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above features and advantages of these embodiments and other features and advantages and the manner in which they are achieved will become more apparent, and embodiments of the present invention may be better understood, by referring to the following description taken in conjunction with the accompanying drawings, in which:

[0033] Figure 1 It is a schematic processing flow chart of a method for human fall detection and privacy protection based on a smart camera according to an embodiment of the present invention.

[0034] Figure 2 The human body segmentation network model M is obtained by segmenting the human body in the image based on the modified UNet network structure according to an example of the present invention. seg Schematic diagram of the network structure. DETAILED DESCRIPTION

[0035] In the following description of the drawings and specific embodiments, the details of one or more embodiments of the present invention will be described. Other features, objects and advantages of the present invention can be clearly seen from these descriptions, drawings and claims.

[0036] It should be understood that the illustrated and described embodiments are not limited in application to the details of the construction and arrangement of the components set forth in the following description or illustrated in the accompanying drawings. The illustrated embodiments may be other embodiments and may be implemented or executed in various ways. Each example is provided by explaining the disclosed embodiments rather than limiting them. In fact, it will be apparent to those skilled in the art that various modifications and variations may be made to the embodiments of the present invention without departing from the scope or essence of the present invention. For example, a feature illustrated or described as part of one embodiment may be used together with another embodiment to still produce another embodiment. Therefore, the present invention discloses such modifications and variations that fall within the scope of the appended claims and their equivalent elements.

[0037] Likewise, it is understood that the phrases and terms used herein are for descriptive purposes and should not be considered restrictive. The use of "include," "comprise," or "have" and variations thereof herein is intended to be open-ended to include the items listed thereafter and their equivalents and additional items.

[0038] The present invention will be described in more detail below with reference to several specific embodiments of the present invention.

[0039] like Figure 1 As shown schematically, Figure 1 The processing steps and flow of a method for human fall detection and privacy protection based on a smart camera according to an embodiment of the present invention are schematically illustrated.

[0040] like Figure 1 As shown, according to this embodiment of the present invention, a method for detecting a person falling and protecting privacy based on a smart camera is disclosed, which includes the following steps:

[0041] S100: Data Preparation

[0042] Through the Internet, various public databases and various resources, a large number of video clips containing people falling and not falling are collected. It is required that there is always one person in the video clip, and each video clip is required to be about 4-10 seconds long. In addition, for each video clip, add fall and non-fall tags to the video name, such as 00001_fall.mp4 (indicates fall tag), 00002_nofall.mp4 (indicates non-fall tag), ..., and so on. Each video clip is added with a numbered tag.

[0043] S200: Video frame capture

[0044] For each video file (i.e., the above video clip), use a frame capture tool or code to capture video frames at equal intervals, for example, capture 4 frames per second, and create a folder named after the video file name, and save the captured video frame pictures (images) to the corresponding folder. For example, the video frames captured from the 00001_fall.mp4 video are named according to the sequential numbering, as follows: 00001_fall_001.jpg, 00001_fall_002.jpg, 00001_fall_003.jpg, ..., and so on.

[0045] S300: Data annotation preprocessing

[0046] Use the labelme annotation tool to annotate the images captured in the S200-video frame capture step, annotate the human body contours of all video frames in folders named after the video file names, and finally generate a json annotation file.

[0047] S400: Data Division

[0048] According to a certain ratio, for example, 7:3, the video folders of the captured frames are divided into training data and verification data. For example, 70% of the data is used as a training data set, which is used to train the model; and, for example, 30% of the data is used as a verification data set, which is used to evaluate the model effect during the model training process.

[0049] S500: Model Training

[0050] In this embodiment of the present invention, the human body segmentation training in the image is implemented by using the modified UNet network structure. The human body segmentation network model M is obtained through segmentation training. seg ,like Figure 2 shown. Figure 2 The human body segmentation network model M is obtained by segmenting the human body in the video image file based on the modified UNet network structure according to an example of the present invention. seg Schematic diagram of the network structure.

[0051] Therefore, all the picture frames in a certain folder in step S200-video frame capture step are 1 ,frame 2 ,...,frame m As input, frame 1 It is the first picture saved in the folder after the video is captured.2 It is the second picture saved in the folder after the video is captured, ..., and so on; use M seg The model infers the image, obtains multiple layers of decode network layers, reshapes them and splices them to form a new feature layer f n , where n is the nth video frame in the folder n , all the feature vectors in this obtained folder are [f 1 , f 2 ,...,f m ], the label corresponding to the feature vector is the category of the video "fall" or "non-fall". The SVM model is used to classify the video according to the category, and a continuous multi-frame fall judgment model M can be trained. fall Among them, SVM (SVM model) is a classic machine learning algorithm, which is often used in classification scenarios. In the process of using the SVM model to analyze multiple consecutive frames of images in this patent application, each image is processed by the human body segmentation model M seg During calculation, the data of multiple decoder layers are concatenated to obtain an array. Multiple consecutive images will obtain multiple data. These data are concatenated as the input parameters of SVM to train the fall judgment model M. fall . Among them, encoder means encoder, and decoder means decoder.

[0052] S600: Model Adaptation

[0053] In step S500-model training, the hardware resources of NVIDIA GPU are used, and the deep learning framework is pytorch. After obtaining the final trained fall judgment model M fall And the human segmentation network model M seg After that, the model is deployed and transplanted to the smart camera for personnel fall detection (monitoring). In the deployment and transplantation process, the corresponding SDK is used to adapt according to the AI ​​chip model in the smart camera. seg and M fall The file can be converted to the specified format. seg and M fall Quantization is performed to reduce the amount of computation required for model inference at the terminal. "Deployment and migration" means that the model trained on a GPU, for example, needs to be deployed on an edge device. At this time, the algorithm model needs to be adapted according to the specific model of the edge device so that the trained model can be run and analyzed on the edge device. This process is called deployment and migration.

[0054] S700: Camera Data Acquisition

[0055] Data is collected through a smart camera, and video frames are decoded at equal intervals, for example, at a frequency of 4 frames per second. The decoded image data is sent to the model of the following step S800 - model reasoning step for reasoning.

[0056] S800: Model Reasoning

[0057] Continuing from the above, the video frame data of, for example, 16 frames (a group) acquired in time sequence in the step of S700-camera data acquisition are used to train the model M seg and M fall Perform a combined analysis of the inference results and return the result of whether there is a person falling in the 16 video frames, as well as the mask coverage (mask) result of the person in the video frame. The returned data format is:

[0058] {"fall":[[x 1 ,y 1 ][x 2 ,y 2 ], ..., [x n ,y n ]],...], "nofall":[[[x′ 1 , x′ 1 ],[x′ 2 ,x′ 2 ],...,[x′ m , x′ m ]], ...]}

[0059] Among them, [[x 1 ,y 1 ],[x 2 ,y 2 ], ..., [x n ,y n ]] is the horizontal and vertical coordinates of the human body contour of the falling person, [[x′ 1 , x′ 1 ],[x′ 2 , x′ 2 ],...,[x′ m , x′ m ]] are the body contour coordinates of the non-falling person.

[0060] S900: alarm information reporting

[0061] In the case where a person falls down in step S800 - the model reasoning step, a mask is applied to the human body part of the last frame of the picture, for example, based on the returned data result. A red mask can be used for a person who falls down, and a green mask can be used for a person who does not fall down, so as to protect the privacy of the human body part and use the masked picture of a person who falls down as an alarm picture, and can be combined with the event time and the alarm event name, etc., as an alarm message, and sent to the alarm message receiver. In this way, the alarm message receiver, such as a remote person, can view the alarm message, especially the alarm picture, to determine whether it is a real person fall event and whether further response is required. Since one alarm picture needs to be sent when a person falls down, only one frame of the picture needs to be masked. In the implementation example of this step, the last frame of the picture is selected.

[0062] The personnel fall detection and privacy protection method of the present invention can effectively identify fall events while protecting user privacy, and has broad application prospects, especially in application scenarios such as elderly care, hospitals, nursing homes, mental hospitals, detention centers, specific monitoring places, and the smart home field, etc.

[0063] The foregoing description of several embodiments of the present invention is presented for illustrative purposes. The foregoing description is not intended to be exhaustive, nor is it intended to limit the present invention to the precise steps and / or forms disclosed, and it is apparent that many modifications and variations may be made in light of the above teachings. The scope of the present invention and all equivalents are intended to be defined by the appended claims.

Claims

1. A method for detecting a person falling based on a human body segmentation network model and a human body fall classification model training, characterized in that: The following steps are involved: Step S100: data preparation, collecting a large number of video clips containing people falling and non-falling, and naming each of the video clips, wherein the name of each video clip has a fall label or a non-fall label, the number of people appearing in each video clip is always one, and the duration of each video clip is 4-10 seconds; Step S200: Video frame capture, using a frame capture tool or code to capture 4 frames of images per second for each file of the video clip at equal intervals, creating a folder named after the name of the video clip, and saving the captured video frame images to the corresponding folder; Step S300: data annotation preprocessing, using an annotation tool to annotate the captured video frame images, annotate the human body contours of all video frames in each of the folders, and generate a JSON annotation file; Step S400: data division, dividing each of the folders after the frame cutting process into training data and verification data according to a ratio of 7:3; Step S500: Model training, using the modified UNet network structure to perform segmentation training on the human body in the video frame image, and training to obtain the human body segmentation network model M seg , and use the SVM model to classify the feature vectors and train a continuous multi-frame fall judgment model M fall ; Step S600: Model adaptation, the trained human body segmentation network model M seg and the fall classification model M fall Deploy and transplant to the smart camera, use the corresponding SDK for adaptation, and transform the human body segmentation network model M seg and the fall classification model M fall The file is converted into the specified format and the human body segmentation network model M seg and the fall classification model M fall Quantify; Step S700: Camera data acquisition, collecting data through the smart camera, decoding video frames at equal intervals at a frequency of 4 frames per second, and sending the decoded video frame data to the model in step S800 for reasoning; Step S800: Model reasoning, using the trained human body segmentation network model M seg and the fall classification model M fall Inferring the video frame data collected in step S700, and returning a result of whether a person falls in the video frame data, and a result of mask coverage of the person in the video frame; and Step S900: reporting alarm information. If a fallen person is detected in step S800, the last frame of the image is masked with the human body, and the processed image is sent to the alarm receiver as an alarm image.

2. The method according to claim 1, characterized in that The naming format of the video clips is "numbered state.mp4", where the falling state is named "fall" and the non-falling state is named "nofall".

3. The method according to claim 1, characterized in that In the step S300 of data labeling preprocessing, the labeling tool used is Labelme.

4. The method according to claim 1, characterized in that: In the model training step, the deep learning framework used is PyTorch.

5. The method according to claim 1, characterized in that In the model reasoning step S800, the returned data format includes the horizontal coordinates and vertical coordinates of the human body contours of the fallen person and the non-fallen person.

6. The method according to claim 5, characterized in that In the step S800 of model inference, the returned data format is: {″fall″:[[[x1,y1],[x2,y2],...,[x n ,y n ]],...],″nofall″:[[[x′1,x′1],[x′2,x′2],...,[x′m,x′ m ]],...]} Among them, [[x1, y1], [x2, y2], ..., [x n ,y n ]] is the horizontal and vertical coordinates of the human body contour of the falling person, [[x′1, x′1], [x′2, x′2], ..., [x′ m , x′ m ]] is the horizontal coordinate of the human body contour of the non-falling person.

7. The method according to claim 1, characterized in that In the alarm information reporting step, the color of the mask covering the fallen person is red, and the color of the mask covering the non-fallen person is green.

8. The method according to claim 1, characterized in that The alarm information includes a picture covered with a mask of the fallen person, a timestamp and a name of the alarm event.

9. The method according to claim 1, characterized in that: In the step S900, the mask covering includes performing mask covering processing on at least the head of the human body to provide privacy protection.

Images