A rass sedation score discrimination method based on a depth map neural network

Abstract

The application discloses a RASS sedation score discrimination method based on a depth map neural network, face detection and image correction are carried out based on a Retinaface improved network, and image preprocessing is carried out, feature extraction based on pain detection is carried out through an improved VGG network; a cascade pyramid model (CPN) is used for human posture estimation, the action posture and the limb key point of a patient are obtained, a multilayer 3D convolution and down sampling of a fixed size in a space and time dimension are carried out on an image block in a key point region, key point motion feature extraction is obtained; relationship graph modeling and joint optimization learning are carried out through a depth map neural network, more discriminative high-dimensional feature fusion is obtained, the RASS score of a single image is obtained through calculation of weighted combination and softmax discrimination of the feature fusion; in a continuous image sequence, weighted average is carried out, and more robust RASS score automatic discrimination and prediction are obtained. The application improves the accuracy and efficiency of pain expression detection and abnormal behavior recognition fusion judgment, and reduces the missed detection and false detection rate.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and machine learning, and particularly relates to a RASS sedation score discrimination method based on a deep graph neural network. BACKGROUND

[0002] With the development of artificial intelligence and intelligent information technology and the continuous improvement of computer performance, deep neural network models have made great progress in image and video recognition, and intelligent recognition algorithms have been widely applied in many actual detection scenarios. In the medical and treatment aspects, the intelligent recognition algorithms mainly embody intelligent monitoring of patients, early warning of emergencies, etc.

[0003] In the monitoring process of patients, ICU critically ill patients need to receive invasive diagnosis and treatment, catheter indwelling, mechanical ventilation and other treatment schemes due to the severity of their diseases. Adopting appropriate sedation and analgesia has become a common treatment method. The current widely used light sedation strategy has the risk of unexpected restlessness, delirium and even increases the probability of adverse events such as accidental extubation and falling out of bed. Timely use of Richmond agitation-sedation score (RASS score) to judge the sedation effect of patients plays a very important role in protecting patient safety.

[0004] At present, the monitoring of ICU patients in hospitals mainly relies on the day and night shift of medical staff, and the RASS sedation score is highly dependent on artificial discrimination. The abnormal behaviors of patients are mainly found by observation of medical staff. The existing RASS sedation score discrimination method has the following problems: 1) the judgment of medical staff on the RASS sedation score standard is subjective and easy to lead to inaccurate discrimination; 2) the number of medical staff cannot meet the needs of individual monitoring, and abnormal behaviors of patients are easy to be overlooked, leading to adverse consequences; 3) some abnormal behaviors of patients are usually accompanied by premonitory symptoms, and medical staff are not easy to predict the occurrence of abnormal behaviors through the subtle performance of patients. In order to solve the above problems, it is of great significance to realize an automatic RASS score algorithm suitable for the high complexity and timely response demand of ICU scene. SUMMARY

[0005] In view of the problems existing in the existing RASS sedation score discrimination method, the purpose of the present application is to provide a RASS sedation score discrimination method based on a deep graph neural network, which detects the pain expressions and abnormal behaviors of people in videos or dynamic images by means of a deep convolutional network, and finally learns and fuses multiple features through a deep graph neural network architecture to discriminate the RASS sedation score.

[0006] In order to achieve the above purpose, the following technical scheme is proposed:

[0007] A RASS sedation score discrimination method based on a deep map neural network, comprising the following steps:

[0008] Step one: face detection and image correction based on the improved Retinaface network;

[0009] Step two: for the corrected image face frame and face activity unit position, image preprocessing is carried out, and feature extraction based on pain detection is carried out through the improved VGG network;

[0010] Step three: adopt a cascade pyramid model to estimate the human body posture, and obtain the action posture and limb key points of the patient;

[0011] Step four: the image block in the key point area is subjected to multi-layer 3D convolution and down-sampling in space and time dimensions with fixed size, and key point motion feature extraction is obtained;

[0012] Step five: through the deep map neural network, the face area feature and the limb motion feature are modeled and jointly optimized to learn the more discriminative high-dimensional feature fusion;

[0013] Step six: the RASS score of a single image is obtained by calculating the weighted combination of feature fusion and softmax discrimination;

[0014] Step seven: in the continuous image sequence, the RASS score is weighted and averaged to obtain a more robust RASS score automatic discrimination and prediction.

[0015] A RASS sedation score discrimination method based on a deep map neural network, comprising the following steps:

[0016] Step one: obtain video image data, and perform patient face detection and abnormal behavior recognition in two ways, wherein the face detection part adopts an improved network based on Retinaface, and a face correction module is added, for the detected eye coordinates e0, e1, the included angle α (counterclockwise) of the line e0e1 with the horizontal direction is calculated, for any point P0(x0, y0) in the image, the corresponding point P(x, y) after rotating the image center O counterclockwise by α degrees is P(x, y), then x=x0 cosα+y0 sinα, y=y0 cosα-x0 sinα. The rotated image is input into the face detection network to obtain the coordinates of the patient's face frame;

[0017] Step two: according to the returned face frame left upper coordinate P1(x1, y1) and right lower coordinate P2(x2, y2), the original picture is cut, only the original picture [x1:x2, y1:y2] part is reserved, that is, the face part, and the image size is scaled to 224x224 pixel size, and the image is subjected to grayscale, histogram equalization, and mean filter preprocessing. The processed picture is input into the improved convolutional neural network of the VGG network for feature extraction based on pain detection;

[0018] Step three: behavior detection based on human body posture analysis is performed, and a cascade pyramid (CPN) human body posture estimation method is adopted to obtain the action posture and limb key points of the patient, including shoulder, arm joint, wrist joint, and hand position detection;

[0019] Step four: for continuous input t frames, a feature map with a shoulder, hand, and arm joint size of 60x40 pixels is extracted to generate multi-channel information. Five channels of information are extracted for each frame of the input layer, which are: gray value, x direction gradient, y direction gradient, x direction optical flow, and y direction optical flow. The first three values are calculated for each frame, and the x and y direction optical flows need two consecutive frames to calculate;

[0020] Step five: a 3D convolution kernel of w x h x t (w x h in spatial dimension, t is time dimension) is used to perform convolution on each of the five channels. In order to increase the number of feature maps, two different convolution kernels are used at each position, and a 2x2 window is used for downsampling in the next downsampling layer to obtain the same number of feature map embeddings with lower spatial resolution. After 3 layers of convolution and 2 times of downsampling, each input image block of t consecutive frames is converted into a 128-dimensional feature vector, which captures the motion information of the posture key points of the input frame;

[0021] Step six: the activity unit of the target face is located and feature extracted, the overall feature extracted for face pain classification is taken as context feature, and a relationship graph related to face feature is constructed. The local feature extraction of the joint and hand, and the context of the upper body action feature of the target are used to construct the relationship graph of limb activity, and the graph model related to RASS score is obtained by establishing and fusing the face and posture relationship graphs;

[0022] Step seven: the parameters of the above graph neural network are learned and optimized, and based on the graph model, the high-dimensional feature combination of the face feature and posture feature fusion is obtained by inference, pooling, and dimension reduction. The weighted sum of the high-dimensional features and the softmax layer can predict and estimate the RASS evaluation level and score of each frame;

[0023] Step eight: a group of T frame interval evaluation results P are obtained through a period of RASS evaluation scores t={p1,p2,...,p T}, and return a set of C t ={c1,c2,...,c T} represents the confidence level of the detection result for each frame, expressed by the formula. The final RASS sedation score (-5≤RASS≤4) was obtained.

[0024] The technical concept of this invention is as follows: the pose estimation result is obtained by using the Cascaded Pyramid (CPN) pose estimation algorithm, and the motion features of each key image block are extracted by multi-layer 3D convolutional downsampling; in addition, pain detection-related features are learned by using a convolutional neural network; the RASS estimate of each frame is obtained by using a graph convolutional neural network algorithm, and finally, a multi-frame weighted average is performed to obtain the final RASS sedation score result.

[0025] The beneficial effects of this invention are as follows: by extracting relatively reliable motion features and pain expression features in the spatial and temporal dimensions through convolutional neural networks, a robust, fast, and accurate RASS sedation scoring method is obtained through graph neural network methods, which improves the accuracy and efficiency of pain expression detection and abnormal behavior recognition fusion evaluation, and reduces the false negative and false positive rates. Attached Figure Description

[0026] Fig. 1 This is a schematic diagram illustrating the construction and prediction of a graph neural network based on face activity units and pose estimation.

[0027] Fig. 2 This is a flowchart of the present invention. Detailed Implementation

[0028] The invention will now be further described with reference to the accompanying drawings.

[0029] Reference Figs. 1-2 A method for RASS sedation scoring based on graph deep neural networks includes the following steps:

[0030] Step one: the present application adopts Huawei intelligent camera (model D3250-10-SIU) arranged above the patient's bed to obtain video image data. The data is collected in two parts: (1) face detection and face activity unit human body; (2) arm abnormal behavior recognition. Since the face has a certain angle, the face detection part adopts the network improved based on Retinaface, and adds a face correction module. For the detected eye coordinates e0, e1, the included angle a (counterclockwise) between the line e0e1 and the horizontal direction of the image is calculated. For any point pixel P0(x0, y0) in the image, the corresponding point P(x, y) is obtained by rotating the image center clockwise, then x=x0 cos a+y0 sin a, y=y0 cos a-x0 sin a. The rotated image is input into the face detection network to obtain the coordinates of the patient's face frame;

[0031] Step two: the position and features of the face activity unit are obtained according to face alignment, and the main activity units include left and right eye regions, mouth region movements. In order to obtain the overall facial features representing the pain classification, the face image needs to be cropped according to the returned left upper coordinates P1(x1, y1) and right lower coordinates P2(x2, y2) of the face frame, and the face part image is scaled to 224x224 pixels, and the pretreatment such as gray scale, histogram equalization and mean filtering is carried out, and the processed picture is input into the improved convolutional neural network of VGG network to extract the features based on pain detection;

[0032] Step three: in order to obtain the feature representation of arm abnormal behavior, behavior detection based on human posture analysis is needed. The present method adopts the method of cascade pyramid (CPN) human posture estimation to obtain the action posture and limb key points of the patient, including shoulder, arm joint, wrist joint and hand position detection;

[0033] Step four: for the extraction of joint motion features, considering the continuous input of t frames (t=3), the multi-channel feature map with a size of 60x40 pixels around the shoulder, hand and arm joint is obtained. Five channels of information are extracted for each frame of the input layer, which are: gray value, x direction gradient, y direction gradient, x direction optical flow and y direction optical flow. The first three values are calculated for each frame, and the x and y direction optical flows are calculated using two consecutive frames;

[0034] Step five: calculate the features of these channels, using 3D convolution kernels in space and time dimensions, the size of the convolution kernel is w x h x t (w x h in space, take w = h = 7, t is the time dimension, take t = 3) Convolution is performed in each channel respectively. In order to increase the number of feature maps, two different convolution kernels are used at each location, and a 2x2 window is used for downsampling in the next downsampling layer to obtain the same number of feature map embeddings with lower spatial resolution; After 3 layers of convolution and 2 times of downsampling, a 128-dimensional feature vector can be obtained, which represents the motion information of each pose key point;

[0035] Step six: construct the relationship graph model of these components according to the pain-related pose and expression key points and overall features. First, the positioning and features of the active unit of the target face are combined with the overall features extracted by the face pain classification to construct a relationship graph related to the face features. Second, the local relative connection relationship is obtained by the joint and hand detection, and the context is constructed by combining the target upper body action features to construct the relationship graph of the limb activity. Then, the face and pose relationship graphs are fused to obtain the graph model associated with the RASS score;

[0036] Step seven: the optimization of the above graph model adopts a 6-level graph convolutional neural network algorithm, and the parameter matrix of the graph convolution transformation is learned and optimized through the labeled training samples. In the prediction process, based on this graph model, feature transformation is performed, and pooling and dimension reduction are performed to obtain a high-dimensional feature combination of face features and pose features. The linear weighting and classification layer of this high-dimensional feature can predict and estimate the RASS evaluation level and score of each frame;

[0037] Step eight: in order to obtain more stable RASS evaluation scores and make reliable prediction, the RASS evaluation scores of each time period T (T = 12 in this method) are averaged. The specific method is to save and obtain a group of evaluation results P t ={p1,p2,...,p T} in T frame interval, consider its confidence C t ={c1,c2,...,c T}, through the formula to obtain the final RASS sedation score (-5 ≤ RASS ≤ 4) discriminant result.

[0038] As described above, the specific implementation steps of the patent implementation make the invention clearer. Any modification and change made to the invention within the spirit and protection scope of the claims falls within the protection scope of the invention.

Claims

1. A deep map neural network-based RASS sedation score discrimination method, characterized in that Specifically comprising the following steps: Step one: obtaining video image data, based on the improved network of Retinaface, correcting the image, rotating any pixel in the image to obtain a rotated image, inputting the image into the face detection network to obtain the coordinates of the patient's face frame; Step two: cropping the original image according to the left upper coordinates and right lower coordinates of the patient's face frame obtained in step one, reducing the image size, and performing grayscale, histogram equalization, and mean filtering preprocessing on the image, and inputting the processed image into the improved convolutional neural network of VGG network for feature extraction based on pain detection; Step three: behavior detection based on human body posture analysis, using the method of cascade pyramid CPN human body posture estimation to obtain the action posture and limb key points of the patient, including shoulder, arm joint, wrist joint, and hand position detection; Step four: for continuous input t frames, extract the feature map of the size of 60x40 pixels of the shoulder, hand and arm joint, generate multi-channel information, extract five channels of information for each frame of the input layer, which are: gray value, x direction gradient, y direction gradient, x direction optical flow and y direction optical flow, The first three values are calculated for each frame, and the x and y direction optical flow needs two consecutive frames to calculate; Step five: a 3D convolution kernel of w x h x t is used to convolve each of the five channels, w x h is in the spatial dimension, and t is in the time dimension; After 3 layers of convolution and 2 times of down-sampling, each input image block of t consecutive frames is converted into a 128-dimensional feature vector, which captures the motion information of the input frame key points; Step six: positioning and feature extraction of the target face activity unit, using the overall features extracted by the face pain classification as the context features, constructing the relationship graph related to the face features, using the joint points and hands to extract local features, and using the target upper body action features as the context to construct the relationship graph of the limb activity, and establishing and fusing the face and posture relationship graphs to obtain the graph model associated with the RASS score; Step seven: learning and optimizing the parameters of the above graph neural network, based on the graph model to speculate, pool, and reduce the dimension to obtain the high-dimensional feature combination of the face feature and posture feature fusion, and the weighted sum of the high-dimensional features and the softmax layer predict the RASS evaluation level and score of each frame; Step eight: get a set of T-frame interval results by RASS score of a time period , and return a set of confidence of each frame detection result, get the final determination of sedation score, RASS .​ 2. The RASS sedation score discrimination method based on a deep graph neural network according to claim 1, wherein The specific operation of rotating any pixel in the image in step one is that the face detection network adopts a network improved based on Retinaface, a face correction module is added, and the detected eye coordinates are calculated The included angle α between the connecting line and the horizontal direction is counterclockwise, and for any point pixel in the image, the corresponding point after rotating degrees clockwise around the center O of the image is , so . 3.The RASS sedation score discrimination method based on a deep graph neural network according to claim 1, wherein The cropping process in step two involves using the top-left coordinates of the patient's face bounding box. and bottom right coordinates Crop the original image to retain only the original image. The portion refers to the face, and its image size is scaled down to 224×224 pixels.

4. The RASS sedation score discrimination method based on a deep graph neural network according to claim 1, wherein In step five, the convolution process uses two different convolution kernels at each position to increase the number of feature maps, and a 2x2 window is used for down-sampling in the next down-sampling layer to obtain the same number of feature map embeddings with lower spatial resolution.

Images