Real-time detection and early warning system and method for falling of patient

Abstract

The invention provides a real-time detection and early warning system and method for falling of a patient, and relates to the technical field of artificial intelligence, and the system comprises a monitoring device which is used for collecting a real-time video in a monitoring area, and transmitting the real-time video to a middle station server; the middle station server comprises a storage unit, a calculation unit and an early warning unit; the storage unit is used for storing a real-time video uploaded by monitoring equipment; the calculation unit is used for processing the real-time video through a preset patient tumble real-time detection model to judge whether a tumble event occurs or not; the early warning unit is used for sending early warning information to notify the early warning terminal when the calculation unit detects that the falling event occurs; and the early warning terminal is used for receiving and processing the early warning information of the falling event and giving an early warning prompt. On the basis of referring to an existing model, an attention mechanism of patient clothing features is added, a tracking branch and a posture multi-classification branch are added, and the real-time performance and accuracy of a fall detection model are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a real-time detection and early warning system and method for patient falls. Background Art

[0002] In recent years, with the rapid rise of artificial intelligence technology represented by deep learning, the ability of artificial intelligence technology to recognize videos, images, texts, time series signals, etc. has been developing rapidly. In order to detect in-hospital patient falls in real time, it is first necessary to segment in-hospital patient instances based on video data, and at the same time track and analyze each instance, perform posture classification, and thus identify fall events.

[0003] Instance segmentation is widely used in geographic information systems, autonomous driving, robotics and other fields. Instance segmentation is a combination of target detection and semantic segmentation. The target is detected in the image (target detection), and then each pixel is labeled. The instance segmentation model generally consists of three parts: image input, instance segmentation processing, and segmentation result output. After the image is input, the model generally uses backbone networks such as VGGNet and ResNet to extract image features, and then processes it through the instance segmentation model. In the model, the location and category of the target instance can be determined by target detection, and then segmentation can be performed at the selected area, or the semantic segmentation task can be performed first, and then different instances can be distinguished, and finally the instance segmentation result can be output.

[0004] For instance segmentation models, the Fast R-CNN model proposed in reference [1], the Faster R-CNN model proposed in reference [2], and the Mask R-CNN model proposed in reference [3] have gradually evolved. They first detect the target bounding box and then perform instance segmentation to obtain the segmentation mask. The accuracy is continuously improved, but the real-time performance is not enough. On the other hand, the YOLACT model proposed in reference [4] greatly improves the real-time performance by parallelizing the above tasks and predicting the non-local prototype mask and the mask linear combination coefficients at the same time, but the tracking of video instances is poor.

[0005] Patients in the hospital may suffer accidental falls due to diseases of the bones, muscles, nervous systems, etc., causing injuries of varying degrees. If the most appropriate measures are not taken in time, secondary injuries may occur to the patients.

[0006] For the real-time detection of fall events, many existing technologies are based on various sensor data (acceleration, angular velocity, joint feature points, etc.) worn by patients or point cloud data received by wireless transceiver devices. References [5]-[8] analyze these data to obtain the human body contour, posture and its time series characteristics, and use machine learning models, filtering algorithms, threshold algorithms, etc. to predict the patient's fall event. The disadvantages of this type of technology are: 1) Some sensors are invasive and will cause trouble to the patient's health; 2) Some patients are unwilling to wear sensors, which is time-consuming and labor-intensive, and some are uncomfortable; 3) Sensors are easily displaced, resulting in large errors; 4) This type of technology predicts a high false positive rate of falls.

[0007] Some other technologies use deep learning models to detect key points of the human body for gait recognition and fall detection. References [9]-

[10] first segment the human body contour, detect the key points of the human skeleton, and then classify the gait to identify falls. However, this type of technology is easily affected by partial occlusion of the patient's body and complex background, and the recognition accuracy is not high.

[0008] The background description provided herein is for the purpose of generally presenting the context of the present application. Unless otherwise indicated herein, the materials described in this section are not prior art to the claims of this application and are not admitted to be prior art by inclusion in this section.

[0009] References are as follows:

[0010] [1] R. Girshick. Fast R-CNN. In ICCV, 2015.

[0011] [2] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NIPS, 2015.

[0012] [3] Kaiming He, Georgia Gkioxari, Piotr Dolla r, and Ross Gir-shick. Maskr-cnn. In ICCV, 2017.

[0013] [4] D. Bolya, C. Zhou, F. Xiao and YJ Lee, "YOLACT: Real-Time InstanceSegmentation," 2019 IEEE / CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019, pp.9156-9165, doi:10.1109 / ICCV.2019.00925.

[0014] [5] CN118072472B, fall detection warning method, device, equipment and storage medium.

[0015] [6] CN118021294B, a fall detection method and system based on multiple sensors.

[0016] [7] CN118363021B, a wireless fall detection method based on hybrid drive of data and model.

[0017] [8] CN114814775B, Radar fall detection method and device based on ResNet network.

[0018] [9] CN118135652A, a fall detection method based on skeleton key point association and key feature perception.

[0019]

[10] CN112699706B, fall detection method, system and storage medium. Summary of the invention

[0020] In view of the above technical problems in the related art, the present invention proposes a real-time detection and early warning system and method for patient falls.

[0021] In order to achieve the above-mentioned invention object, the present invention adopts the following technical scheme:

[0022] In the first aspect, the present invention provides a patient fall real-time detection and warning system, including a monitoring device, a middle station server and an early warning terminal:

[0023] The monitoring device is used to collect real-time video in the monitoring area and transmit the real-time video to the middle station server;

[0024] The middle station server includes a storage unit, a computing unit and an early warning unit;

[0025] The storage unit is used to store the real-time video uploaded by the monitoring device;

[0026] The computing unit is used to process the real-time video by using a preset patient fall real-time detection model to determine whether a fall event occurs;

[0027] The preset patient fall real-time detection model is trained by using patient pictures and surveillance videos collected in advance, and is used to complete instance segmentation and posture classification of patients. The real-time video in the storage unit is pre-processed to obtain multiple sub-videos, which are input into the preset patient fall real-time detection model to obtain a posture classification result. When the posture classification result is a fall, it is determined that a fall event has occurred.

[0028] The preset patient fall real-time detection model includes: a feature backbone network, a feature pyramid network, multiple prediction heads, and a prototype frame prediction network; the feature backbone network receives the monitoring video, and sends the patient image to the feature pyramid after convolution; the feature pyramid network output includes a first output and a second output; the first output is sent to the prototype frame prediction network, and the patient image is convolved again after the convolution feature and sent to the prototype frame prediction network to obtain a prototype mask; the second output is sent to the multiple prediction heads and non-maximum suppression is performed to obtain a mask coefficient; the patient in the monitoring video is instance segmented according to the prototype mask and the mask coefficient, and then posture classification is performed;

[0029] The warning unit is used to send warning information to the warning terminal when the computing unit detects a fall event; the warning information includes the fall area and the fall video;

[0030] The early warning terminal is used to receive and process early warning information of a fall event and give an early warning reminder.

[0031] Specifically, the preset patient fall real-time detection model is trained by pre-collected patient pictures and surveillance videos to complete the instance segmentation, tracking and posture classification of the patient, specifically including:

[0032] The patient image is subjected to feature extraction through the first convolutional neural network module to obtain the first feature, and the video image in the surveillance video is subjected to the second convolutional neural network module to obtain the second feature; the first feature and the second feature are dot-multiplied to form attention, and the attention is used as the input of the feature pyramid network; the third feature is extracted through the backbone network of the feature pyramid network as the output;

[0033] The first feature is transformed by the third convolutional neural network module and then dot-multiplied with the third feature to further form attention and input into the prototype box prediction network to predict k non-local prototype masks for each video image. At the same time, the third feature is input into the fully connected layer to predict the bounding box coordinates of the instance in each video image, and the linear combination coefficients of k masks are predicted for each instance, and the bounding boxes with non-maximum scores are suppressed.

[0034] Then, the feature map is cut according to the bounding box of the instance in the predicted video image, and the feature values ​​below the adjustable threshold are set to 0 to obtain an instance cut feature map. Based on the instance cut feature map, each instance category is predicted, and the instance category includes patient or non-patient and the corresponding classification prediction probability score; finally, a mask, a category and its classification prediction probability score are generated for each instance through linear combination;

[0035] The preset patient fall real-time detection model also includes a fully connected tracking branch, which is used to use the features of the instances recognized in the previous video pictures and their segmentation masks to perform feature matching and classification with the latest recognized instances in the current video pictures, thereby realizing continuous tracking of the target instances in the video;

[0036] The preset patient fall real-time detection model also includes a posture classification branch, which uses a fully convolutional neural network module to perform posture classification on the instance based on the previous and next frame features of the same instance.

[0037] Specifically, the method of using the features of the instance identified in the previous video picture and its segmentation mask to perform feature matching and classification with the latest identified target instance in the current video picture, thereby achieving continuous tracking of the target instance in the video is specifically as follows:

[0038] For a newly identified target instance and its segmentation mask, two fully connected layers are used to transform the features of the previous N instances and the latest target instance. The matching calculation of the two sets of features is realized through dot multiplication, and the cross entropy probability is maximized to classify and record the target instance.

[0039] Specifically, the method of classifying the latest target instance by maximizing the cross entropy probability is as follows:

[0040] Classify the target instance i and determine whether it belongs to the existing category j = (1, 2, ..., N) or the new category 0; the cross entropy probability p of the target instance i and the existing category j i (j) is calculated as follows:

[0041]

[0042] Where N is the number of categories, f i is the feature of target instance i; T is the transposed feature vector; f k is the feature of the existing instance k, e is the base of the natural logarithm; the category j of the latest target instance i corresponds to p i (j) Largest category.

[0043] Specifically, the preset patient fall real-time detection model also includes a posture classification branch, based on the previous and next frame features of the same instance, a fully convolutional neural network module is used to perform posture classification on the instance, specifically:

[0044] The features of the previous and next frames of the same instance are spliced ​​in the spatial dimension, and 2 convolutional layers, 1 global average pooling layer, and softmax function are used to obtain 7 posture classification logical values ​​​​logits. The category corresponding to the maximum logical value logits is the posture classification of this instance.

[0045] Specifically, the real-time video is composed of continuous video clips. The video clips collected by the monitoring device are uploaded to the middle station server once every preset time period. The overlap degree of adjacent video clips is a preset overlap degree and is stored in the storage unit of the server.

[0046] Specifically, the preprocessing of the real-time video in the storage unit specifically includes:

[0047] The real-time video segments are organized into 3-second video segments and then downsampled 3 times to obtain 3 sub-videos; if the real-time video segment parameters do not match the calibration parameters, they are converted into calibration parameters through resampling in the spatial and temporal dimensions.

[0048] Specifically, the preprocessing of the real-time video in the storage unit and inputting it into a preset patient fall real-time detection model to obtain a posture classification result specifically includes:

[0049] Each of the three sub-videos is input into the preset patient fall real-time detection model to obtain the seven classification results of the three sub-videos of each instance, and the final posture classification result is obtained by the majority voting method; if the three posture classification results are different, the maximum confidence winning strategy is adopted to obtain the final posture classification result; the posture classification includes: standing / still, walking, sitting / still, sitting / moving, lying / still, lying / moving, and falling.

[0050] Specifically, the monitoring area is divided into a plurality of continuous sub-monitoring areas, and there are a plurality of monitoring devices, the number of which is the same as the number of sub-monitoring areas. A monitoring device is installed in each sub-monitoring area, and both are numbered and a corresponding relationship is established.

[0051] In a second aspect, the present invention provides a method for real-time detection and early warning of patient falls, based on the real-time detection and early warning system for patient falls described in any one of the first aspects, comprising the following steps:

[0052] S1. Collect videos in the monitoring area;

[0053] S2, pre-processing the collected video to obtain three sub-videos;

[0054] S3, inputting the three sub-videos into a preset patient fall real-time detection model to obtain three groups of posture classification results of the instance, and integrating the three groups of posture classification results to determine whether a fall event has occurred;

[0055] The preset patient fall real-time detection model includes: a feature backbone network, a feature pyramid network, multiple prediction heads, and a prototype frame prediction network; the feature backbone network receives the monitoring video, and sends the patient image to the feature pyramid after convolution; the feature pyramid network output includes a first output and a second output; the first output is sent to the prototype frame prediction network, and the patient image is convolved again after the convolution feature and sent to the prototype frame prediction network to obtain a prototype mask; the second output is sent to the multiple prediction heads and non-maximum suppression is performed to obtain a mask coefficient; the patient in the monitoring video is instance segmented according to the prototype mask and the mask coefficient, and then posture classification is performed;

[0056] S4. When a fall event is detected, an early warning message is sent to notify the early warning terminal.

[0057] The present invention proposes a patient fall real-time detection and early warning system and method, which aims to detect fall events in real time and notify relevant personnel for timely rescue. This system uses a monitoring camera network to collect real-time video of the public area of ​​the ward, and detects fall events of in-hospital patients in real time through a deep learning video detection model. The early warning terminal of the early warning system promptly notifies the nearest nurse station of the location of the fall event and provides playback of the fall video. Nursing staff can take the most appropriate rescue measures, thereby reducing the injuries caused by falls of in-hospital patients.

[0058] Based on the existing model, the present invention adds an attention mechanism for the patient's clothing features, a tracking branch and a posture multi-classification branch, and effectively improves the real-time performance and accuracy of the fall detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 2 is a schematic diagram of a patient fall real-time detection and warning system according to an embodiment of the present invention;

[0061] Figure 2 is a schematic diagram of the structure of a preset patient fall real-time detection model provided according to an embodiment of the present invention;

[0062] Figure 3It is a schematic diagram of a method for real-time detection and early warning of patient falls provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0064] In the description of the present invention, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0065] In the description of the present invention, it should be understood that descriptions involving orientation, such as orientation or positional relationship indicated as up, down, etc., are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0066] In the description of the present invention, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention in combination with the specific content of the technical solution.

[0067] Embodiment 1

[0068] refer to Figure 1 This embodiment provides a patient fall real-time detection and warning system, including a monitoring device, a middle station server and a warning terminal:

[0069] The monitoring device is used to collect real-time video in the monitoring area and transmit the real-time video to the middle station server;

[0070] Specifically: the real-time video is composed of continuous video clips. The video clips collected by the monitoring equipment are uploaded to the middle station server once every preset time period. The overlap between adjacent video clips is a preset overlap and is stored in the storage unit of the server for easy video processing.

[0071] In this embodiment, the preset duration is 1 second, and the preset overlap is 5%. The values ​​of the preset duration and the preset overlap can also be changed according to specific needs, which are not limited here.

[0072] The overlapping degree of adjacent video segments is 5%, which means that each video segment has a part of the same content as the video segments before and after it in time, and the duration of this part of the same content accounts for 5% of the total duration of each video segment.

[0073] Specifically, if the duration of each video segment is 1 second, then the overlap between each video segment and the adjacent video segments is 0.05 seconds (i.e. 5% of 1 second). In this way, each 1-second video segment actually has 0.05 seconds of content that is the same as the previous video segment, and also the same as the first 0.05 seconds of content of the next video segment.

[0074] This overlap setting ensures that when processing videos, such as event detection or continuous behavior analysis, no important information will be missed due to the segmentation between video clips. At the same time, it also facilitates the continuous playback of videos and avoids frame skipping during playback.

[0075] The video clips collected by the monitoring equipment are uploaded to the middle-end server through the transmission device. The transmission device calculates the start and end time of each video clip according to the preset overlap degree, and adjusts the start time of the clip to ensure that there is overlapping content with a preset overlap degree between adjacent video clips. This is the existing technology and will not be repeated here.

[0076] The calibration parameters of the video clips collected by the monitoring equipment are: image resolution of 640x480, 3 colors (3 color channels), and frame rate of 30 frames per second;

[0077] If the video parameters of the real-time video clip do not match the calibration parameters, the middle server converts them into calibration parameters by resampling the spatial and temporal dimensions.

[0078] In this embodiment, the monitoring area is divided into multiple continuous sub-monitoring areas. There are multiple monitoring devices, and the number is the same as the number of sub-monitoring areas. Each sub-monitoring area corresponds to a monitoring device, and both are numbered and a corresponding relationship is established. For example, the sub-monitoring areas are numbered: monitoring area 1, monitoring area 2, ..., monitoring area N, and the corresponding monitoring devices are numbered: camera 1, camera 2, ..., camera N, camera 1 corresponds to monitoring area 1, camera 1 corresponds to monitoring area 3, and so on; wherein N is the number of divided sub-monitoring areas, and there are multiple ways of numbering. As long as the above-mentioned corresponding relationship is achieved, they are all protected, which will not be repeated here.

[0079] In another possible implementation, the monitoring area may be divided according to the area that each monitoring device can monitor and cover, and then a corresponding relationship between the monitoring device and the sub-monitoring area is established.

[0080] In this embodiment, the monitoring device is a camera.

[0081] Specifically, use the hospital's WiFi communication network or network cables to build a monitoring device network, assign each monitoring device a unique identification code, set up the sub-monitoring area corresponding to each monitoring device, and record the contact information of the nearest nursing point for the monitoring device.

[0082] The monitoring area of ​​each monitoring device is a rectangular area, ensuring that adjacent monitoring device areas overlap appropriately and the area should not be too large. The specific overlapping area is selected according to actual conditions to avoid reducing the monitoring accuracy of distant patients. When a patient leaves the monitoring area of ​​one monitoring device and arrives at the monitoring area of ​​the next monitoring device, the middle station server automatically identifies it.

[0083] The middle station server includes a storage unit, a computing unit and an early warning unit;

[0084] The storage unit is used to store the real-time video uploaded by the monitoring device;

[0085] When the storage unit stores a video, it also stores the serial number of the video clip, the start and end time of the video clip, the camera to which it belongs, and the corresponding position of the sub-monitoring area to which it belongs. There is a corresponding relationship between them, which is convenient for other sub-units to read. This is a mature existing technology and will not be repeated here.

[0086] The storage unit also stores patient pictures, which are pictures of patients wearing hospital clothes; the patient pictures are 3-channel color pictures with a resolution of 640x480.

[0087] The computing unit is used to process the video by using a preset patient fall real-time detection model to determine whether a fall event has occurred;

[0088] The preset patient fall real-time detection model is trained by using patient pictures and surveillance videos collected in advance, and is used to complete instance segmentation, tracking and posture classification of patients. The real-time video in the storage unit is pre-processed to obtain multiple sub-videos, which are input into the preset patient fall real-time detection model to obtain a posture classification result. When the posture classification result is a fall, it is determined that a fall event has occurred;

[0089] The preset patient fall real-time detection model includes: a feature backbone network, a feature pyramid network, multiple prediction heads, and a prototype frame prediction network; the monitoring video is convolved through the feature backbone network and then sent to the feature pyramid network; the patient image is convolved and then sent to the feature pyramid; the feature pyramid network output includes a first output and a second output; the first output is sent to the prototype frame prediction network, and the patient image is convolved again through the post-convolution features and then sent to the prototype frame prediction network to obtain a prototype mask; the second output is sent to the multiple prediction heads and non-maximum suppression is performed to obtain a mask coefficient; the patient in the monitoring video is instance segmented according to the prototype mask and the mask coefficient, and then posture classification is performed;

[0090] The patient image is subjected to feature extraction through the first convolutional neural network module to obtain a first feature, and the video image in the monitoring video is subjected to convolution through the feature backbone network to obtain a second feature; the first feature and the second feature are dot-multiplied and then input into the feature pyramid network; a third feature is extracted through the backbone network of the feature pyramid network as an output; the third feature is a feature map of the video image; two outputs are established based on the third feature, which are the first output and the second output respectively;

[0091] The first feature is converted by the third convolutional neural network module, and then point-multiplied with the first output and input into the prototype box prediction network, and K prototype masks for the original image are predicted for each video image; at the same time, the second output is input into the multiple prediction heads, and the bounding box of the instance in each video image is predicted by the fully connected layer, and the linear combination coefficients of k masks are predicted for each instance, and the mask coefficients are obtained after the bounding box is filtered by the non-maximum suppression algorithm; then the feature map is cut according to the bounding box of the instance, and the feature values ​​below the adjustable threshold are set to 0 to obtain an instance cut feature map, and each instance category is predicted based on the instance cut feature map, and the instance category includes patient or non-patient and the corresponding classification prediction probability score; finally, a mask, a category and its classification prediction probability score are generated for each instance through linear combination;

[0092] The preset patient fall real-time detection model also includes a fully connected tracking branch, which is used to use the features of the instances recognized in the previous video pictures and their segmentation masks to perform feature matching and classification with the latest recognized instances in the current video pictures, thereby realizing continuous tracking of the target instances in the video;

[0093] The preset patient fall real-time detection model also includes a posture classification branch, which uses a fully convolutional neural network module to perform posture classification on the instance based on the previous and next frame features of the same instance.

[0094] The preset patient fall real-time detection model is a multi-branch neural network model. The model structure is referenced Figure 2:The model includes two inputs: patient images and surveillance videos; the patient images are subjected to feature extraction through the first convolutional neural network module to obtain a first feature, and the video images in the surveillance video are subjected to the second convolutional neural network module to obtain a second feature; the video images are input into the second convolutional neural network module, and the second feature is extracted through its feature backbone network; the second feature includes a plurality of feature backbones of different scales, and the feature backbones include C1-C5;

[0095] There are 5 convolution modules in the ResNe-101 network, from conv1, conv2_x, conv3_x, conv4_x, conv5_x. After the image passes through the convolution module, it corresponds to C1, C2 to C5 of the feature backbone respectively.

[0096] The first convolutional neural network module and the third convolutional neural network module are ResNet-101 networks, and the patient image uses its conv5_x to obtain the first feature;

[0097] The second convolutional neural network module is a ResNet-101 network, and the video image is transformed through conv1-conv5_x to obtain the feature backbone C1-C5;

[0098] Keep C3-C5 of the feature backbone and connect them backward. C5 of the feature backbone has the same size as the first feature, i.e., the convolution result of the patient image. The first feature and C5 of the second feature can be dot-multiplied to form attention. The dot-multiplication result and C3 and C4 are used as inputs of the feature pyramid network. The dot-multiplication result is passed through the backbone network of the feature pyramid network to obtain the third feature as output. The third feature is a feature pyramid, including P3, P4, P5, P6 and P7.

[0099] The third feature is a feature map of the video image; two outputs are established based on the third feature, namely, a first output and a second output; the first output is P3, and the second output is P3-P7;

[0100] Specifically, the dot product result is passed through a convolution layer to obtain P5, the upsampled features of P5 are added to C4 to obtain P4, the upsampled features of P4 are added to C3 to obtain P3; P5 is downsampled to obtain P6, and P6 is downsampled to obtain P7. P3-P7 are feature maps, P3 is the largest feature map, and P7 is the smallest feature map.

[0101] Next, parallel operations are performed; the largest feature map P3 is used for prototype box prediction.

[0102] The first feature is transformed by the third convolutional neural network module and then dot-multiplied with the first output of the third feature to further form attention and input into the prototype box prediction network to predict K non-local prototype masks for each video picture, independent of any instance.

[0103] After the patient's attention, the instances will be automatically screened and the patient instances will appear with a high probability.

[0104] At the same time, the second output of the third feature is input into multiple prediction heads, and the bounding box coordinates of the instance in each video picture are predicted through a fully connected layer (FC), and the linear combination coefficients (Mask Coefficients) of K masks are predicted for each instance, and the output dimension is 4*c+K; where c is the number of instances, 4*c represents the coordinates of the bounding box (each instance has 4 coordinate values), and K represents the number of mask coefficients; and the bounding box with non-maximum score is suppressed by a non-maximum suppression algorithm to obtain the mask coefficient; then the feature map is cut according to the bounding box of the predicted instance, and the feature values ​​below the adjustable threshold are set to 0 to obtain an instance cut feature map, and each instance category is predicted based on the instance cut feature map, and the instance category includes patient or non-patient and the corresponding classification prediction probability score; finally, a mask, category and its classification prediction probability score are generated for each instance through linear combination.

[0105] The adjustable threshold is a configurable parameter, which is set according to the balance between detection rate and false positives.

[0106] In this process, the conv layer (good at generating spatial coherence masks) and the FC layer (good at generating semantic vectors) are used to generate "prototype masks" and "mask coefficients" respectively. This structure abandons the two-stage structure of detection first and then segmentation, which can greatly improve real-time performance.

[0107] The model also includes a fully connected tracking branch, which is used to use the features of the instances and their segmentation masks recognized in the previous video pictures to perform feature matching and classification with the latest recognized instances in the current video pictures, thereby achieving continuous tracking of the target instances in the video.

[0108] The linear combination is to multiply multiple mask coefficients with the corresponding prototype mask and add the results to generate the final instance mask. This process is a key step in the instance segmentation task. This is a prior art and will not be repeated here.

[0109] Specifically: for a newly identified instance and its segmentation mask, two fully connected layers are used to perform feature conversion on the previous N instances and the latest instance. Through the matching calculation (dot product) of the two sets of features, the cross entropy probability is maximized to classify and record the latest instance (instance ID); the previously identified N instances and their segmentation masks have been stored in the storage unit, and there is no limitation on how to obtain them.

[0110] Using cross entropy probability maximization to classify the latest target instance specifically includes:

[0111] Classify the latest instance i and determine whether it belongs to the existing category j = (1, 2, ..., N) or the new category 0; the cross entropy probability p of the target instance i and the existing category j i (j) is calculated as follows:

[0112]

[0113] Where N is the number of categories, f i is the logits (features) of the latest target instance i; T is the transposed feature vector; f k is the logits (features) of the existing instance k, e is the base of the natural logarithm; the category j of the latest target instance i corresponds to p i (j) Largest category.

[0114] The preset patient fall real-time detection model also includes a posture classification branch, which uses a fully convolutional neural network module to perform posture classification on the instance based on the previous and next frame features of the same instance (in the storage unit); the posture classification includes: standing / still, walking, sitting / still, sitting / moving, lying / still, lying / moving, and falling, a total of 7 posture classifications;

[0115] Specifically, the features of the previous and next frames of the same instance are spliced ​​in the spatial dimension, and two convolutional layers (including BN and ReLU), one global average pooling layer, and a softmax function are used to obtain seven posture classification logical values ​​logits. The seven posture classifications are mutually exclusive, and the category corresponding to the maximum logical value logits is the posture classification of this instance;

[0116] The monitoring video is three sub-videos obtained by downsampling the collected real-time video segments into 3-second video segments by 3 times; since the application scenario in this embodiment is relatively fixed and the patient's movements are relatively slow, the video segments collected in advance are firstly assembled into 3-second video segments, and the overlap degree of adjacent 3-second segments is 0.5 seconds; then the 3-second video segments are downsampled by 3 times, that is, one frame is extracted every 3-second time interval, thereby reducing the amount of data and forming three different sub-video sequences, forming 3 sub-videos in total, namely {0, 3, 6..}, {1, 4, 7..}, {2, 5, 8..}, expressed as {N}, {N+1}, {N+2} sub-videos, where N is the sub-video sequence.

[0117] Each of the three sub-videos is input into the preset patient fall real-time detection model to obtain three groups of posture classification results, and the three posture classification results are integrated to give the final detection result.

[0118] The preset patient fall real-time detection model also includes integrating the posture classification results of the three sub-videos, and finally giving whether each patient has fallen, specifically:

[0119] For the 7 classification results of the 3 sub-videos of each instance, the majority voting method is used to obtain the final posture classification result; if the 3 posture classification results are different, the maximum confidence win strategy is used to obtain the final posture classification result;

[0120] For example, three sub-videos are input into the preset patient fall real-time detection model to obtain three groups of posture classification results, namely: standing / still (0.6), walking (0.75), walking (0.72); the brackets are classification logic values, and it can be seen that the posture classification results of the two sub-videos are walking, and the final posture classification result obtained by the majority voting method is walking;

[0121] If three sub-videos are input into the preset patient fall real-time detection model, three groups of posture classification results are obtained, namely: standing / still (0.6), walking (0.75), and falling (0.78); the values ​​in brackets are the classification logic values. It can be seen that the three posture classification results are different, and the maximum confidence winning strategy is adopted to select falling as the final posture classification result.

[0122] Majority voting is a commonly used method to integrate the results of multiple classifiers. The basic idea of ​​this method is to calculate the classification results given by all classifiers for each sample to be classified, and then select the classification with the most occurrences as the final classification result.

[0123] The maximal confidence win strategy is a decision rule that selects the classification with the highest classification confidence as the final classification result. In multi-classification problems, when multiple classifiers give different classification results for the same sample, this strategy can be used to determine which classification is the most reliable.

[0124] When a patient falls in the posture classification results, a fall event is issued;

[0125] Extracting features (P3-P7) from surveillance video images through the backbone network of ResNet-101 and Feature Pyramid Network (FPN) means using ResNet-101 as the basic network to extract multi-scale features of the image, and further constructing a feature pyramid through the FPN structure. These feature pyramids contain feature maps of different scales, usually marked as P3, P4, P5, P6 and P7.

[0126] The preset patient fall real-time detection model of this embodiment is trained and inferred in an end-to-end manner. During the training process, the loss function consists of five parts: 1) Instance bounding box loss function L box (Smooth L1), only the selected bounding box and the reference bounding box are used to calculate the loss function; 2) Instance classification loss function L class , using the binary cross entropy function of whether the patient is a patient; 3) instance segmentation mask loss function L mask =BCE(M,M gt ) / Area(M gt ), which is the pixel-level binary cross entropy function, M is the instance segmentation mask, M gt is the instance reference mask, Area(M gt ) is the area of ​​the instance reference mask (which is beneficial for the detection of small targets); 4) Tracking loss function L track =-∑ i log(p i (j)), using the cross entropy function of instance ID multi-classification; 5) fall classification loss function L fall , using the cross entropy function of posture multi-classification.

[0127] The ResNet-101 network is a deep convolutional neural network consisting of multiple convolutional layers that can extract high-level features from images, which is the existing technology.

[0128] FPN, the full name of which is Feature Pyramid Networks, is a network proposed at the 2017 CVPR that mainly solves the multi-scale problem in target detection. FPN significantly improves the performance of small object detection by simply changing the network connection without increasing the computational complexity of the original model.

[0129] This embodiment uses a multi-branch neural network model to complete the instance segmentation, tracking and posture classification of the patient. The model structure is as follows: Figure 2 As shown. In order to achieve the goal of real-time detection of in-hospital patient falls, the instance segmentation model of YOLACT is used as a reference. Its design idea is to extract features (P3-P7) from the original video image through the backbone network of ResNet-101 and feature pyramid network (FPN), and split the instance segmentation task into two parallel subtasks based on the feature pyramid: 1) Through a fully convolutional Protonet network (prototype box prediction network), k non-local prototype masks for the original image are predicted for each image, which does not depend on any instance; 2) The bounding box coordinates of each instance are predicted through a fully connected layer (fully-connected, FC), and the linear combination coefficients (Mask Coefficients) of k masks are predicted for each instance, and the output dimension is 4*c+k; and the bounding box of non-maximum score is suppressed (non-maximum Suppression); then the feature map is cut according to each predicted bounding box, and the feature values ​​below the adjustable threshold are set to 0. Based on this instance-cut feature map, each instance category (patient / non-patient, predicted score) is predicted; finally, through linear combination, a mask, category and its classification probability score are generated for each instance. In this process, the conv layer (good at generating spatial coherence masks) and the FC layer (good at generating semantic vectors) are used to generate "prototype masks" and "mask coefficients" respectively. This structure abandons the two-stage structure of detection first and segmentation later, which can greatly improve real-time performance.

[0130] The present invention transforms the YOLACT model, and the transformation idea is as follows: 1) extract features from some photos of patients wearing hospital clothes through a convolutional neural network module (Conv), generally a layer of ResNet-101, and multiply these features with the input features of the feature pyramid to form attention, and transform the features of the patient photos through a convolutional neural network module (Conv) and then multiply them with the input of the Protonet network to further form attention; highlight the detection and segmentation of patient instances, and reduce the impact of the environment and other personnel; 2) add a fully connected (FC) tracking branch, assuming that the N instances and their segmentation masks identified in the previous video have been stored in the storage unit, for the latest identified instance and its segmentation mask, use 2 fully connected layers to convert the features of the previous N instances and the latest instance, and through the matching calculation (dot product) of the two sets of features, use the cross entropy probability maximization to classify and record the latest instance (instance ID); 3) Added a posture classification branch, based on the previous and next frame features of the same instance (in the storage unit), a fully convolutional neural network module was used to classify the instance (standing / still, walking, sitting / still, sitting / moving, lying / still, lying / moving, falling); 4) Integrate the classification results of the three sub-videos to finally give a result for each patient whether they have fallen.

[0131] An early warning unit, configured to send early warning information to the early warning terminal when the computing unit detects a fall event; the early warning information includes the fall area and the fall video;

[0132] Specifically, the early warning unit maps the fall event to the spatiotemporal position of the original video, saves the video to a designated storage unit, and comprehensively calibrates the position of the monitoring equipment to locate the fall area of ​​the fallen patient and find the nurse station with the closest physical distance and the second closest nurse station.

[0133] The early warning unit performs graded notifications based on the distance between the nurse station and the location where the fall occurred; specifically:

[0134] The early warning unit sends the fall video to the early warning terminal app at the nearest nurse station and marks it in red in the app, indicating that this is an emergency that requires immediate response; at the same time, the early warning unit also sends the same information to the early warning terminal app at the next nearest nurse station, but marks it in orange in the app, indicating that this is a situation that requires attention but is less urgent than red.

[0135] The early warning terminal is used to receive and process early warning information of fall events and give early warning reminders.

[0136] After receiving a red or orange warning, the early warning terminal app at the nurse station will sound an alarm and display relevant information on the early warning terminal app screen to remind nurses to take action. Nurses can also view the fall video playback through the early warning terminal app.

[0137] The nursing station that receives the warning of a fall incident can take appropriate measures in a timely manner to rescue the patient who has fallen in the hospital, thereby reducing the harm caused by the fall incident to the patient.

[0138] This embodiment considers the fall events that occur when hospital patients wear inpatient clothes and leave the ward and move around in the public area of ​​the ward. The purpose is to detect fall events in real time and notify relevant personnel for timely rescue. A patient fall real-time detection and early warning method based on deep learning is proposed. This system uses a surveillance camera network to collect real-time video of the public area of ​​the ward, and detects fall events of hospital patients in real time through a deep learning video detection model. The early warning system terminal promptly notifies the nearest nurse station of the location of the fall event and provides playback of the fall video. Nursing staff can take the most appropriate rescue measures, thereby reducing the injuries caused by falls of hospital patients.

[0139] Based on the existing model, this embodiment adds an attention mechanism for the patient's clothing features, adds a tracking branch and a posture multi-classification branch, and greatly improves the real-time performance and accuracy of the fall detection model.

[0140] Embodiment 2

[0141] refer to Figure 3 This embodiment provides a patient fall real-time detection and warning method. Based on the first embodiment, this embodiment provides a patient fall real-time detection and warning system, including the following steps:

[0142] S1. Collect videos in the monitoring area;

[0143] Specifically: the video is composed of continuous video clips. The video clips collected by the monitoring device are uploaded to the middle station server once every preset time period. The overlap between adjacent video clips is a preset overlap and is stored in the storage unit of the server for easy video processing.

[0144] Specifically, the video clips of multiple continuous sub-monitoring areas are collected and uploaded to the middle station server once every preset time, and the overlap degree of adjacent video clips is the preset overlap degree;

[0145] S2, pre-processing the collected video to obtain three sub-videos; specifically: the collected video segments are formed into 3-second video segments and then down-sampled by 3 times to obtain three sub-videos;

[0146] S3, inputting the three sub-videos into a preset patient fall real-time detection model to obtain three posture classification results of the instance, and integrating the three posture classification results to determine whether a fall event has occurred;

[0147] The integration of three posture classification results is as follows: for the seven classification results of the three sub-videos, the majority voting method is used to obtain the final posture classification result; if the three classification results are different, the maximal confidence win strategy is used to obtain the final classification result as the final posture classification result;

[0148] The preset patient fall real-time detection model includes: a feature backbone network, a feature pyramid network, multiple prediction heads, and a prototype frame prediction network; the feature backbone network receives the monitoring video, and sends the patient image to the feature pyramid after convolution; the feature pyramid network output includes a first output and a second output; the first output is sent to the prototype frame prediction network, and the patient image is convolved again after the convolution feature and sent to the prototype frame prediction network to obtain a prototype mask; the second output is sent to the multiple prediction heads and non-maximum suppression is performed to obtain a mask coefficient; the patient in the monitoring video is instance segmented according to the prototype mask and the mask coefficient, and then posture classification is performed;

[0149] S4. Send warning information to the warning terminal when a fall event is detected;

[0150] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as a microprocessor, such as software executed by a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0151] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A patient fall real-time detection and warning system, characterized in that: Including monitoring equipment, middle-end server and early warning terminal: The monitoring device is used to collect real-time video in the monitoring area and transmit the real-time video to the middle station server; The middle station server includes a storage unit, a computing unit and an early warning unit; The storage unit is used to store the real-time video uploaded by the monitoring device; The computing unit is used to process the real-time video by using a preset patient fall real-time detection model to determine whether a fall event occurs; The preset patient fall real-time detection model is trained by using patient pictures and surveillance videos collected in advance, and is used to complete instance segmentation and posture classification of patients. The real-time video in the storage unit is pre-processed to obtain multiple sub-videos, which are input into the preset patient fall real-time detection model to obtain a posture classification result. When the posture classification result is a fall, it is determined that a fall event has occurred. The preset patient fall real-time detection model includes: a feature backbone network, a feature pyramid network, multiple prediction heads, and a prototype box prediction network; The feature backbone network receives the surveillance video, and sends the patient image to the feature pyramid after convolution; the feature pyramid network output includes a first output and a second output; the first output is sent to the prototype frame prediction network, and the patient image is convolved again after the convolution feature and sent to the prototype frame prediction network to obtain a prototype mask; the second output is sent to the multiple prediction heads and non-maximum suppression is performed to obtain a mask coefficient; the patient in the surveillance video is instance segmented according to the prototype mask and the mask coefficient, and then posture classification is performed; The warning unit is used to send warning information to the warning terminal when the computing unit detects a fall event; the warning information includes the fall area and the fall video; The early warning terminal is used to receive and process early warning information of a fall event and give an early warning reminder.

2. The patient fall real-time detection and warning system according to claim 1 is characterized in that: The preset patient fall real-time detection model is trained by using patient pictures and surveillance videos collected in advance, and is used to complete the instance segmentation and posture classification of the patient, specifically including: The patient image is subjected to feature extraction through a first convolutional neural network module to obtain a first feature, and the video image in the monitoring video is subjected to a second convolutional neural network module to obtain a second feature; the first feature and the second feature are subjected to point multiplication and then input into a feature pyramid network; a third feature is extracted as an output through a backbone network of the feature pyramid network; the third feature is a feature map of the video image; two outputs are established based on the third feature, namely, a first output and a second output; The first feature is converted by the third convolutional neural network module, and then point-multiplied with the first output and input into the prototype box prediction network, and K prototype masks for the original image are predicted for each video image; at the same time, the second output is input into the multiple prediction heads, and the bounding box of the instance in each video image is predicted by the fully connected layer, and the linear combination coefficients of K masks are predicted for each instance, and the mask coefficients are obtained after the bounding box is filtered by the non-maximum suppression algorithm; then the feature map is cut according to the bounding box of the instance, and the feature values ​​below the adjustable threshold are set to 0 to obtain an instance cut feature map, and each instance category is predicted based on the instance cut feature map, and the instance category includes patient or non-patient and the corresponding classification prediction probability score; finally, a mask, a category and its classification prediction probability score are generated for each instance through linear combination; The preset patient fall real-time detection model also includes a fully connected tracking branch, which is used to use the features of the instances recognized in the previous video pictures and their segmentation masks to perform feature matching and classification with the latest recognized instances in the current video pictures, thereby realizing continuous tracking of the target instances in the video; The preset patient fall real-time detection model also includes a posture classification branch, which uses a fully convolutional neural network module to perform posture classification on the instance based on the previous and next frame features of the same instance.

3. The patient fall real-time detection and warning system according to claim 2, characterized in that: The method of using the features of the instance identified in the previous video picture and its segmentation mask to perform feature matching and classification with the latest identified target instance in the current video picture, thereby achieving continuous tracking of the target instance in the video is specifically as follows: For a newly identified target instance and its segmentation mask, two fully connected layers are used to transform the features of the previous N instances and the latest target instance. The matching calculation of the two sets of features is realized through dot multiplication, and the cross entropy probability is maximized to classify and record the target instance.

4. The patient fall real-time detection and warning system according to claim 3, characterized in that: The method of classifying the latest target instance by maximizing the cross entropy probability is as follows: Classify the target instance i and determine whether it belongs to the existing category j = (1, 2, ..., N) or the new category 0; the cross entropy probability p of the target instance i and the existing category j i( j ) The calculation is as follows: Where N is the number of categories, f i is the feature of target instance i; T is the transposed feature vector; f k is the feature of the existing instance k, e is the base of the natural logarithm; the category j of the latest target instance i corresponds to p i (j) Largest category.

5. The patient fall real-time detection and warning system according to claim 2, characterized in that: The preset patient fall real-time detection model also includes a posture classification branch, which uses a fully convolutional neural network module to perform posture classification on the instance based on the previous and next frame features of the same instance; The features of the previous and next frames of the same instance are spliced ​​in the spatial dimension, and 2 convolutional layers, 1 global average pooling layer, and softmax function are used to obtain 7 posture classification logical values. The category corresponding to the maximum logical value is the posture classification of this instance.

6. The patient fall real-time detection and warning system according to claim 1, characterized in that: The real-time video is composed of continuous video clips. The video clips collected by the monitoring device are uploaded to the middle station server once every preset time period. The overlap degree of adjacent video clips is a preset overlap degree and is stored in the storage unit of the server.

7. The patient fall real-time monitoring and early warning system according to claim 1 is characterized in that: The preprocessing of the real-time video in the storage unit specifically includes: The real-time video segments are organized into 3-second video segments and then downsampled 3 times to obtain 3 sub-videos; if the real-time video segment parameters do not match the calibration parameters, they are converted into calibration parameters through resampling in the spatial and temporal dimensions.

8. The patient fall real-time detection and warning system according to claim 4, characterized in that: The step of pre-processing the real-time video in the storage unit and inputting it into a preset patient fall real-time detection model to obtain a posture classification result specifically includes: Each of the three sub-videos is input into the preset patient fall real-time detection model to obtain the seven classification results of the three sub-videos of each instance, and the final posture classification result is obtained by the majority voting method; if the three posture classification results are different, the maximum confidence winning strategy is adopted to obtain the final posture classification result; the posture classification includes: standing / still, walking, sitting / still, sitting / moving, lying / still, lying / moving, and falling.

9. The patient fall real-time detection and warning system according to claim 1, characterized in that: The monitoring area is divided into a plurality of continuous sub-monitoring areas. There are a plurality of monitoring devices, the number of which is the same as the number of sub-monitoring areas. One monitoring device is installed in each sub-monitoring area, and both are numbered and a corresponding relationship is established.

10. A patient fall real-time detection and warning method, based on the patient fall real-time detection and warning system according to any one of claims 1 to 9, characterized in that: The following steps are involved: S1. Collect videos in the monitoring area; S2, pre-processing the collected video to obtain three sub-videos; S3, inputting the three sub-videos into a preset patient fall real-time detection model to obtain three groups of posture classification results of the instance, and integrating the three groups of posture classification results to determine whether a fall event has occurred; The preset patient fall real-time detection model includes: a feature backbone network, a feature pyramid network, multiple prediction heads, and a prototype frame prediction network; the feature backbone network receives the monitoring video, and sends the patient image to the feature pyramid after convolution; the feature pyramid network output includes a first output and a second output; the first output is sent to the prototype frame prediction network, and the patient image is convolved again after the convolution feature and sent to the prototype frame prediction network to obtain a prototype mask; the second output is sent to the multiple prediction heads and non-maximum suppression is performed to obtain a mask coefficient; the patient in the monitoring video is instance segmented according to the prototype mask and the mask coefficient, and then posture classification is performed; S4. When a fall event is detected, an early warning message is sent to notify the early warning terminal.

Images