ICU patient dysphoria video identification method based on space-time motion detection

Abstract

The invention relates to an ICU patient dysphoria video recognition method based on space-time motion detection. Compared with the prior art, the defect that dysphoria video recognition is difficult to carry out on an ICU patient is overcome. The method comprises the following steps: acquiring and preprocessing a data set; constructing an ICU patient dysphoria identification model; training an ICU patient dysphoria identification model; obtaining video stream data to be identified; iCU patient dysphoria atom action recognition; and performing dysphoria quantitative classification based on an RASS score table. According to the method, the Slowfast basic model is improved, the detection precision and speed of the model are remarkably improved, the dysphoria behavior of the patient is automatically and accurately detected and understood from the perspective of space-time motion detection, the dysphoria condition of the patient is automatically scored according to the RASS score table, and accurate recognition of the dysphoria video of the ICU patient is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of video image recognition, and in particular to a method for recognizing agitated videos of ICU patients based on spatiotemporal motion detection. Background Art

[0002] Patients in the intensive care unit (ICU) often show agitation and struggle due to a variety of reasons, such as the disease itself, environmental factors, pain, and psychological pressure. This not only aggravates the patient's pain, but may also interfere with treatment and nursing work, and even threaten the patient's life safety in severe cases. Agitation is common in critically ill patients, and its incidence is high, which can lead to increased complications and mortality in patients. Therefore, ICU nursing staff need to identify the patient's signs of agitation in a timely manner and alleviate the patient's discomfort as much as possible.

[0003] The most commonly used sedation assessment scale in clinical practice is the RASS (Richmond Agitation-Sedation Scale). According to the RASS scoring scale, the types of agitation in ICU patients are mainly the following: restless, agitated, very agitated, and dangerously agitated. In clinical practice, medical staff mainly observe the type, amplitude, and frequency of the patient's body movements, and assign a score to these movements based on the RASS scoring scale, thereby classifying the patient's agitation.

[0004] The traditional RASS scoring method has the following shortcomings: (1) The manual RASS scoring is mainly based on the direct observation of the patient's condition by medical staff or the patient's self-report. This method is highly subjective and may lead to misjudgment of the patient's condition. (2) In the intensive care unit (ICU), medical staff are faced with high-intensity nursing work. Frequent measurement of critical care indicators for patients increases the workload and may lead to burnout of medical staff. It is reported that about 33% of medical team members experience burnout, which may affect the quality of nursing work and pose a threat to the safety of ICU patients. (3) The patient's agitated movements may be relatively mild and may be obscured by objects such as quilts and are not easy to detect. This increases the risk of patient agitation not being discovered in time, which may cause nursing staff to miss the opportunity to intervene in time. It is crucial to develop accurate agitation assessment tools to detect patient agitation in real time so that patients can receive the best intervention measures.

[0005] Monitoring of patients in the intensive care unit generates a large amount of data every day. Many related artificial intelligence and applications have been developed in the intensive care unit (ICU). For example, using static facial expression images to detect the agitation level of ICU patients, or using wearable sensors and digital imaging technology to identify the agitation state of ICU patients. However, Figure 3As shown in the figure, since ICU patients are often severely occluded and their agitation movements are sometimes mild, these research methods face the problems of insufficient detection of motion categories and insufficient attention to the spatiotemporal information of the target patient's movements.

[0006] Therefore, how to utilize rich spatiotemporal information to accurately detect patient objects and movements when facing partial occlusion and motion changes of ICU patients still needs further research. Summary of the invention

[0007] The purpose of the present invention is to solve the defect that it is difficult to perform agitation video recognition for ICU patients in the prior art, and to provide an agitation video recognition method for ICU patients based on spatiotemporal motion detection to solve the above problem.

[0008] In order to achieve the above object, the technical solution of the present invention is as follows:

[0009] A method for identifying agitated videos of ICU patients based on spatiotemporal motion detection comprises the following steps:

[0010] 11) Dataset acquisition and preprocessing: Obtain ICU patient video dataset, convert it into jpg format and perform preprocessing;

[0011] 12) Construction of ICU patient agitation recognition model: Construction of ICU patient agitation recognition model based on SlowFast;

[0012] 13) Training of ICU patient agitation recognition model: Use the preprocessed ICU patient video dataset to train the ICU patient recognition model;

[0013] 14) Acquisition of video stream data to be identified: Acquisition of video data of ICU patients to be identified and preprocessing;

[0014] 15) Identification of agitated atomic movements of ICU patients: The complex agitated movements of ICU patients are decomposed into five atomic movements: head movement, hand movement, shoulder movement, standing up, and leg bending according to the RASS score table. The agitated recognition model of ICU patients then recognizes the atomic movements according to the changes of the previous and next frames of the key frame.

[0015] 16) Quantitative classification of agitation based on the RASS scoring table: Use the RASS scoring table to give motion scores for different atomic movements, and use the agitation quantification scoring formula to sum up all scores to classify the patient's agitation.

[0016] The acquisition and preprocessing of the data set includes the following steps:

[0017] 21) Obtain ICU patient video data, extract frames from it using FFmpeg technology, and save the extracted frames in jpg format;

[0018] 22) Perform data enhancement processing: First, use the image processing library OpenCV to flip the image horizontally, vertically, and horizontally and vertically respectively, and then use the python script to add Gaussian noise, shift, scale, and randomly crop all the photos in the dataset;

[0019] 23) The data-enhanced video images were divided into training set and validation set in a ratio of 8:2, and the data sets were annotated. Annotation refers to decomposing the agitated movements of ICU patients into five categories: head movement, shoulder movement, manual movement, standing up and leg bending. The VIA3.0 tool was used to annotate the human body and movements in the experimental data set.

[0020] The annotated dataset format is converted to AVA format using a python script; the training set is used to directly participate in the training of the SlowFast network model for feature extraction, and the validation set is used to adjust the hyperparameters of the SlowFast network model.

[0021] The construction of the ICU patient agitation recognition model includes the following steps:

[0022] 31) An ICU patient agitation recognition model is set based on SlowFast, which includes two branches, namely the fast branch Fastpath and the slow branch Slowpath. The fast branch Fastpath and the slow branch Slowpath both have an input terminal Input, a backbone network Backbone and an output terminal Output. The output terminals of the fast branch Fastpath and the slow branch Slowpath are sent as input to the fully connected layer softmax for fusion;

[0023] 32) Set the input end Input to use Mosaic data enhancement, adaptive initial anchor frame calculation, image scaling and cropping to pre-process the image and input it into the backbone network Backbone;

[0024] 33) The backbone network Backbone of the fast branch Fastpath and the slow branch Slowpath is set to extract features from the processed input image, wherein the slow branch Slowpath uses sparsely sampled frames with a large time span τ=16 as input, that is, the number of frames skipped per second, to obtain spatial semantics, and the fast branch Fastpath uses densely sampled frames with a small time span τ / α, α=8 as input to capture temporal semantics;

[0025] The number of channels in the Fastpath backbone network is β times that of the Slowpath network, β = 1 / 8, to keep the Fastpath network lightweight. In the Slowpath network, residual blocks are stacked and convolution kernels are used to extract spatiotemporal features to characterize actions. The temporal information extracted by the Fastpath network is fused with the spatial semantic information obtained by the Slowpath network through lateral connections.

[0026] The fast path and the slow path work at different time rates, and the extracted spatiotemporal feature information is globally averaged and fused through the fully connected softmax layer.

[0027] 34) Set the prediction result after fusion through the fully connected layer softmax. The loss function uses the BCELoss binary cross entropy loss function to calculate the positioning and confidence loss for the supervision of the bounding box. The optimized BCELoss is used to calculate the atomic action loss and supervise the human body position.

[0028] 35) Introduce the attention mechanism ActionNet in the backbone network of the fast path, that is, insert ActionNet before each residual block of the backbone network of the fast path to enhance the attention to the agitation atomic information and timing information of ICU patients;

[0029] 36) The TAdaconv convolution module is introduced into the backbone network of the slow path to replace the ordinary residual block in the backbone network of the slow path to enhance the extraction of spatial features of the agitated atomic movements of ICU patients.

[0030] The training of the ICU patient agitation recognition model includes the following steps:

[0031] 41) Set the training parameters of the SlowFast network model, including the number of iterations, batch size, and initial learning rate. The number of iterations epoch is 200, the batch size batch-size is 64, the initial learning rate lr0 of the model is set to 0.001, and the Adam optimizer is used to reduce the learning rate;

[0032] 42) Set the data enhancement parameters of the SlowFast network model, including hue hsv_h, saturation hsv_s, brightness hsv_v, rotation angle degrees, up and down flip probability flipud and left and right flip probability fliplr, where hsv_h is 0.025, hsv_s is 0.6, hsv_v is 0.3, degrees is 0, flipud is 0, and fliplr is 0.6;

[0033] 43) The training set and validation set of the preprocessed ICU patient dataset are input into the ICU patient agitation recognition model using the data loader DataLoader and batch processed:

[0034] The input module uses Mosaic data enhancement to randomly scale, crop, and arrange images.

[0035] The input modules of the fast branch Fastpath and the slow branch Slowpath both send the processed training images to the improved backbone network Backbone module. The fast branch Fastpath of the ActionNet with the attention mechanism uses high-frequency sampled image frames, that is, densely sampled frames, as input to capture the timing information in the video. The slow branch Slowpath backbone network with the TAdaconv convolution module uses sparsely sampled image frames, that is, sparsely sampled frames, as input to extract the spatial feature information of the training images.

[0036] The fast path network Backbone module extracts temporal feature information of different scales and sends it to the slow path network Slowpath through lateral connections to achieve the fusion of temporal semantic features and spatial semantic features. Then the two branches output different temporal and spatial feature information of the training image respectively.

[0037] The feature vectors of different scales output by the last two branches are concatenated and fused in the fully connected softmax layer, and the category prediction is performed to achieve feature extraction, feature fusion and atomic action detection;

[0038] 45) Calculate the loss value based on the prediction result and the true label, back-propagate the loss value, update the network parameters, and repeat the above 43) steps until the preset number of training rounds is reached or other stopping conditions are met.

[0039] The agitation atomic motion recognition of ICU patients comprises the following steps:

[0040] 51) Obtain the video stream data to be identified;

[0041] 52) The video stream data to be identified is input into the trained ICU patient agitation recognition model to perform spatiotemporal motion detection, feature extraction, feature fusion and atomic motion detection.

[0042] If no human target is detected, no spatiotemporal action detection is performed and None is output;

[0043] If a human target is detected, the agitated atomic motion of the ICU patient is detected;

[0044] 53) After detecting the agitated atomic motion of the ICU patient, output the human target frame data information and the agitated atomic motion information of the ICU patient;

[0045] Take the key frame as the reference frame, use IoU to measure the overlap between each border B and the true label G, and calculate the difference between the key frame and the current frame and the reference distance between the previous and next frames as the motion information prediction threshold d i , perform atomic action recognition, threshold d i The calculation formula is as follows, where f i is the current frame, f r is the reference frame, d(f r ) is the reference distance;

[0046]

[0047] 54) Atomic motion recognition includes: no atomic motion corresponding threshold is within 50%, small-amplitude atomic motion corresponding threshold is within 50% to 70%, medium-amplitude atomic motion corresponding threshold is within 70% to 85%, and large-amplitude atomic motion corresponding threshold is above 85%;

[0048] 55) After the atomic action is identified, agitation identification and classification based on the RASS scoring table is performed.

[0049] The agitation identification classification based on the RASS scoring table includes the following steps:

[0050] 61) Based on the RASS scoring tool, agitation is defined as restless, agitated, very agitated, and dangerously agitated;

[0051] 62) The identified atomic movements were scored for restlessness, with 1 point for head movement, 1 point for shoulder movement, 1 point for manual movement, 1 point for standing up, and 1 point for leg flexion. The total score was less than or equal to 4 points.

[0052] 63) The above atomic action scores are weighted and summed using the agitation quantification calculation formula, and the patient's agitation state is determined based on the RASS score table;

[0053] If the recognition result is less than the set threshold, the system does not output the agitation warning signal;

[0054] If it is greater than the set threshold, the system will output a restlessness warning signal and its judgment basis. The warning signal includes restlessness, agitation, extreme agitation, and dangerous agitation. The judgment basis is the five atomic movement types of head movement, shoulder movement, manual movement, standing up, and leg bending.

[0055] Among them, the quantitative calculation formula of agitation based on judgment is as follows:

[0056] R total =R h (S1)+R a (S2)+R S (S3)+R g (S4)+R l (S5), R total <=4,

[0057] Among them, R total is the total score of restlessness quantification, R h , R a , R s , R g , R l They are head movement, hand movement, shoulder movement, standing up, and leg bending corresponding to different thresholds s i Quantitative score of restlessness, S i The corresponding thresholds are S1, S2, S3, S4, and S5 respectively.

[0058] Beneficial Effects

[0059] Compared with the prior art, the method for identifying agitated videos of ICU patients using spatiotemporal motion detection of the present invention significantly improves the detection accuracy and speed of the model by improving the Slowfast basic model. From the perspective of spatiotemporal motion detection, the method can automatically and accurately detect and understand the agitated behavior of patients, and automatically score the agitated conditions of patients according to the RASS scoring table, thereby realizing accurate identification of agitated videos of ICU patients.

[0060] First, the present invention introduces the ActionNet attention mechanism into the backbone network of the fast branch Fastpath, so that the network's ability to extract motion features and temporal features of characters in the video is further enhanced; secondly, the TAdaconv convolution module is integrated into the slow branch Slowpath, so that the network's ability to characterize the spatial features of video objects is improved. The present invention classifies the complex agitated behaviors of ICU patients in detail according to the RASS score table, including head movement, shoulder movement, manual movement, standing up and leg bending, and maps the weighted sum of the atomic actions into the RASS score table, realizing the specific quantification of the agitated behaviors of ICU patients.

[0061] Experimental results show that the improved model has an average precision mAP of 80.61% in atomic motion detection, which is 13.33% higher than the original Slowfast model. It greatly improves the recognition and classification performance of the model and can basically meet the requirements of ICU patient agitation recognition and classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is a method sequence diagram of the present invention;

[0063] Figure 2 is a logic flow chart of the present invention;

[0064] Figure 3 A schematic diagram of an ICU patient agitation dataset in the prior art;

[0065] Figure 4 This is a diagram of the SlowFast network structure involved in the present invention;

[0066] Figure 5 The overall structure diagram of the ActionNet attention mechanism involved in the present invention;

[0067] Figure 6 This is the internal network structure diagram of the ActionNet attention mechanism involved in the present invention;

[0068] Figure 7 The TAdaconv convolution module network structure diagram involved in the present invention;

[0069] Figure 8 This is the effect diagram of atomic action recognition of the present invention. DETAILED DESCRIPTION

[0070] In order to have a further understanding and recognition of the structural features and the effects achieved by the present invention, a preferred embodiment and accompanying drawings are used for detailed description as follows:

[0071] like Figure 1 and Figure 2 As shown, the method for identifying agitated videos of ICU patients based on spatiotemporal motion detection according to the present invention comprises the following steps:

[0072] The first step is to obtain and preprocess the data set: obtain the ICU patient video data set, convert it into jpg format and preprocess it.

[0073] (1) Obtain ICU patient video data, extract frames from it using FFmpeg technology, and save the extracted frames in jpg format.

[0074] (2) Perform data enhancement processing: First, use the image processing library OpenCV to flip the image horizontally, vertically, and horizontally and vertically respectively. Then use the Python script to add Gaussian noise and perform shift, scaling, and random cropping on all the photos in the dataset.

[0075] (3) The data-enhanced images were divided into a training set and a validation set in a ratio of 8:2, and the data set was labeled. The labeling refers to decomposing the agitated movements of ICU patients into five categories: head movement, shoulder movement, manual movement, standing up, and leg flexion. The VIA3.0 tool was used to label the human bodies and movements in the experimental data set.

[0076] The annotated dataset format is converted to AVA format using a python script; the training set is used to directly participate in the training of the SlowFast network model for feature extraction, and the validation set is used to adjust the hyperparameters of the SlowFast network model.

[0077] The second step is to build an ICU patient agitation recognition model: build an ICU patient agitation recognition model based on SlowFast.

[0078] In order to more accurately identify the agitated atomic movements of ICU patients, the present invention uses the SlowFast model to construct an agitated recognition model for ICU patients. SlowFast is a powerful recognition model in video action classification and spatiotemporal action detection. It is built based on a 3D convolutional neural network. It processes the spatial and temporal information in the video respectively through two parallel CNN streams, the slow branch (Slowpath) and the fast stream (Fastpath). The slow branch focuses on capturing global spatial information, such as scene layout and human targets; while the fast branch focuses on capturing short-term time information, such as the movement trajectory and speed change of the human body. The backbone network Backbone of the two branches is composed of 3D ResNet. The SlowFast network realizes information fusion through multiple lateral connections, from Fastpatht to Slowpath, which means that Fastpath will not receive any information about Slowpath, but Slowpath can contain information in Fastpath. This design makes the SlowFast network perform well in video understanding tasks.

[0079] (1) Figure 4As shown in the figure, an ICU patient agitation recognition model is set based on SlowFast, which includes two branches, namely the fast branch Fastpath and the slow branch Slowpath. The fast branch Fastpath and the slow branch Slowpath both have an input end Input, a backbone network Backbone and an output end Output. The output end Output of the fast branch Fastpath and the slow branch Slowpath is sent as input to the fully connected layer softmax for fusion.

[0080] (2) Set the input end to use Mosaic data enhancement, adaptive initial anchor frame calculation, and image scaling and cropping to preprocess the image and input it into the backbone network Backbone.

[0081] (3) The backbone networks of the fast branch Fastpath and the slow branch Slowpath are both used to extract features from the processed input images. The slow branch Slowpath uses a large time span τ=16, i.e., the number of frames skipped per second, to obtain spatial semantics. The fast branch Fastpath uses a small time span τ / α, α=8, to capture temporal semantics. The number of channels of the fast branch Fastpath backbone network is β times that of the slow branch Slowpath, β=1 / 8, to keep the fast branch Fastpath lightweight. In the slow branch Slowpath, the spatiotemporal features are extracted by stacking residual blocks and using convolution kernels to represent the action. The temporal information extracted by the fast branch Fastpath is fused with the spatial semantic information obtained by the slow branch Slowpath through lateral connections.

[0082] The fast path and the slow path work at different time rates, and the feature maps extracted are globally averaged and fused through the fully connected layer softmax.

[0083] (4) The prediction result is obtained by fusion through the fully connected layer softmax. The loss function uses the BCELoss binary cross entropy loss function to calculate the positioning and confidence loss for bounding box supervision. The optimized BCELoss is used to calculate the atomic action loss and supervise the human body position.

[0084] (5) An attention mechanism ActionNet is introduced into the backbone network of the fast path. That is, ActionNet is inserted before each residual block of the backbone network of the fast path to enhance the attention to the agitated atomic information and timing information of ICU patients.

[0085] (6) A TAdaconv convolutional module is introduced into the backbone network of the slow path to replace the ordinary residual block in the backbone network of the slow path, so as to enhance the extraction of spatial features of the agitated atomic movements of ICU patients.

[0086] The third step is to train the ICU patient agitation recognition model: use the preprocessed ICU patient video dataset to train the ICU patient recognition model.

[0087] (1) Set the training parameters of the SlowFast network model, including the number of iterations, batch size, and initial learning rate. The number of iterations epoch is 200, the batch size batch-size is 64, the initial learning rate lr0 of the model is set to 0.001, and the Adam optimizer is used to reduce the learning rate.

[0088] (2) Set the data enhancement parameters of the SlowFast network model. The data enhancement parameters include hue hsv_h, saturation hsv_s, brightness hsv_v, rotation angle degrees, up and down flip probability flipud, and left and right flip probability fliplr. Among them, hsv_h is 0.025, hsv_s is 0.6, hsv_v is 0.3, degrees is 0, flipud is 0, and fliplr is 0.6.

[0089] (3) Input the preprocessed ICU patient dataset into the ICU patient agitation recognition model:

[0090] The input module uses Mosaic data enhancement to randomly scale, crop, and arrange images.

[0091] The input modules of the fast path and the slow path send the processed training images to the improved backbone network Backbone module, such as Figure 5 and Figure 6 As shown in Figure 2, the fast path of ActionNet, which introduces the attention mechanism, takes high-frequency sampled image frames as input to capture the motion information in the video, such as Figure 7 As shown in the figure, the slowpath backbone network with the TAdaconv convolution module takes sparsely sampled image frames as input to extract the spatial feature information of the training image.

[0092] The feature information of different scales extracted by the backbone network Backbone module of the fast branch Fastpath is sent to the slow branch Slowpath through the lateral connection to achieve the fusion of temporal feature information and spatial feature information. Then the two branches output different temporal and spatial feature information of the training image respectively.

[0093] The feature vectors of different scales output by the last two branches are concatenated and fused in the fully connected softmax layer, and category prediction is performed to achieve feature extraction, feature fusion and atomic action detection.

[0094] The fourth step is to obtain the video stream data to be identified: obtain the video data of the ICU patients to be identified and perform preprocessing.

[0095] The fifth step is to identify the agitated atomic movements of ICU patients: the complex agitated movements of ICU patients are decomposed into five atomic movements of head movement, hand movement, shoulder movement, standing up and leg bending according to the RASS scoring table. The agitated recognition model of ICU patients then identifies the atomic movements based on the changes in the frames before and after the key frames.

[0096] (1) Obtain the video stream data to be identified.

[0097] (2) The video stream data to be identified is input into the trained ICU patient agitation recognition model to perform spatiotemporal motion detection, feature extraction, feature fusion and atomic motion detection.

[0098] If no human target is detected, no spatiotemporal action detection is performed and None is output;

[0099] If a human target is detected, the agitated atomic motion of the ICU patient is detected.

[0100] (3) After detecting the agitated atomic motion of the ICU patient, the human target frame data information and the agitated atomic motion information of the ICU patient are output;

[0101] Take the key frame as the reference frame, use IoU to measure the overlap between each border B and the true label G, and calculate the difference between the key frame and the current frame and the reference distance between the previous and next frames as the motion information prediction threshold d i , perform atomic action recognition, threshold d i The calculation formula is as follows, where f i is the current frame, f r is the reference frame, d(f r ) is the reference distance;

[0102]

[0103] (4) Atomic motion recognition includes: no atomic motion corresponding threshold is within 50%, small-amplitude atomic motion corresponding threshold is within 50% to 70%, medium-amplitude atomic motion corresponding threshold is within 70% to 85%, and large-amplitude atomic motion corresponding threshold is above 85%;

[0104] (5) After the atomic action is identified, agitation recognition and classification based on the RASS scoring table is performed.

[0105] The sixth step is to quantify the agitation classification based on the RASS scoring table: use the corresponding RASS scoring table to give the motion scores of different atomic movements, and use the agitation quantification scoring formula to sum up all the scores to classify the patient's agitation.

[0106] (1) Agitation is defined based on the RASS scoring tool, which is divided into four categories: restless, agitated, very agitated, and dangerously agitated.

[0107] (2) The identified atomic movements were scored for restlessness: 1 point for head movement, 1 point for shoulder movement, 1 point for manual movement, 1 point for standing up, and 1 point for leg flexion. The total score was less than or equal to 4 points.

[0108] (3) The above atomic action scores are weighted and summed using the agitation quantification calculation formula, and the patient's agitation state is determined based on the RASS score table;

[0109] If the recognition result is less than the set threshold, the system does not output the agitation warning signal;

[0110] If it is greater than the set threshold, the system will output a restlessness warning signal and its judgment basis. The warning signal includes restlessness, agitation, extreme agitation, and dangerous agitation. The judgment basis is the five atomic movement types of head movement, shoulder movement, manual movement, standing up, and leg bending.

[0111] Among them, the quantitative calculation formula of agitation based on judgment is as follows:

[0112] R total =R h (S1)+R a (S2)+R S (S3)+R g (S4)+R l (S5), R total <=4,

[0113] Among them, R total is the total score of restlessness quantification, R h , R a , R s , R g , R lThey are head movement, hand movement, shoulder movement, standing up, and leg bending corresponding to different thresholds s i Quantitative score of restlessness, S i The corresponding thresholds are S1, S2, S3, S4, and S5 respectively.

[0114] like Figure 8 As shown in the figure, SlowFast, which is based on the two improvements of ActionNet attention mechanism and TAdaconv module, can accurately identify the various atomic actions of ICU patients' agitation in video streaming data.

[0115] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for identifying agitated videos of ICU patients based on spatiotemporal motion detection, characterized in that: The following steps are involved: 11) Dataset acquisition and preprocessing: Obtain ICU patient video dataset, convert it into jpg format and perform preprocessing; 12) Construction of ICU patient agitation recognition model: Construction of ICU patient agitation recognition model based on SlowFast; 13) Training of ICU patient agitation recognition model: Use the preprocessed ICU patient video dataset to train the ICU patient recognition model; 14) Acquisition of video stream data to be identified: Acquisition of video data of ICU patients to be identified and preprocessing; 15) Identification of agitated atomic movements of ICU patients: The complex agitated movements of ICU patients are decomposed into five atomic movements: head movement, hand movement, shoulder movement, standing up, and leg bending according to the RASS score table. The agitated recognition model of ICU patients then recognizes the atomic movements according to the changes of the previous and next frames of the key frame. 16) Quantitative classification of agitation based on the RASS scoring table: Use the RASS scoring table to give motion scores for different atomic movements, and use the agitation quantification scoring formula to sum up all scores to classify the patient's agitation.

2. According to the method of claim 1, the method is characterized in that: The acquisition and preprocessing of the data set includes the following steps: 21) Obtain ICU patient video data, extract frames from it using FFmpeg technology, and save the extracted frames in jpg format; 22) Perform data enhancement processing: First, use the image processing library OpenCV to flip the image horizontally, vertically, and horizontally and vertically respectively, and then use the python script to add Gaussian noise, shift, scale, and randomly crop all the photos in the dataset; 23) The data-enhanced video images were divided into training set and validation set in a ratio of 8:2, and the data sets were annotated. Annotation refers to decomposing the agitated movements of ICU patients into five categories: head movement, shoulder movement, manual movement, standing up and leg bending. The VIA3.0 tool was used to annotate the human body and movements in the experimental data set. The annotated dataset format is converted to AVA format using a python script; the training set is used to directly participate in the training of the SlowFast network model for feature extraction, and the validation set is used to adjust the hyperparameters of the SlowFast network model.

3. According to the method of claim 1, the method is characterized in that: The construction of the ICU patient agitation recognition model includes the following steps: 31) An ICU patient agitation recognition model is set based on SlowFast, which includes two branches, namely the fast branch Fastpath and the slow branch Slowpath. The fast branch Fastpath and the slow branch Slowpath both have an input terminal Input, a backbone network Backbone and an output terminal Output. The output terminals of the fast branch Fastpath and the slow branch Slowpath are sent as input to the fully connected layer softmax for fusion; 32) Set the input end Input to use Mosaic data enhancement, adaptive initial anchor frame calculation, image scaling and cropping to pre-process the image and input it into the backbone network Backbone; 33) The backbone network Backbone of the fast branch Fastpath and the slow branch Slowpath is set to extract features from the processed input image, wherein the slow branch Slowpath uses sparsely sampled frames with a large time span τ=16 as input, that is, the number of frames skipped per second, to obtain spatial semantics, and the fast branch Fastpath uses densely sampled frames with a small time span τ / α, α=8 as input to capture temporal semantics; The number of channels in the Fastpath backbone network is β times that of the Slowpath network, β = 1 / 8, to keep the Fastpath network lightweight. In the Slowpath network, residual blocks are stacked and convolution kernels are used to extract spatiotemporal features to characterize actions. The temporal information extracted by the Fastpath network is fused with the spatial semantic information obtained by the Slowpath network through lateral connections. The fast path and the slow path work at different time rates, and the extracted spatiotemporal feature information is globally averaged and fused through the fully connected softmax layer. 34) Set the prediction result after fusion through the fully connected layer softmax. The loss function uses the BCELoss binary cross entropy loss function to calculate the positioning and confidence loss for the supervision of the bounding box. The optimized BCELoss is used to calculate the atomic action loss and supervise the human body position. 35) Introduce the attention mechanism ActionNet in the backbone network of the fast path, that is, insert ActionNet before each residual block of the backbone network of the fast path to enhance the attention to the agitation atomic information and timing information of ICU patients; 36) The TAdaconv convolution module is introduced into the backbone network of the slow path to replace the ordinary residual block in the backbone network of the slow path to enhance the extraction of spatial features of the agitated atomic movements of ICU patients.

4. According to the method of claim 1, the method is characterized in that: The training of the ICU patient agitation recognition model includes the following steps: 41) Set the training parameters of the SlowFast network model, including the number of iterations, batch size, and initial learning rate. The number of iterations epoch is 200, the batch size batch-size is 64, the initial learning rate lr0 of the model is set to 0.001, and the Adam optimizer is used to reduce the learning rate; 42) Set the data enhancement parameters of the SlowFast network model, including hue hsv_h, saturation hsv_s, brightness hsv_v, rotation angle degrees, up and down flip probability flipud and left and right flip probability fliplr, where hsv_h is 0.025, hsv_s is 0.6, hsv_v is 0.3, degrees is 0, flipud is 0, and fliplr is 0.6; 43) The training set and validation set of the preprocessed ICU patient dataset are input into the ICU patient agitation recognition model using the data loader DataLoader and batch processed: The input module uses Mosaic data enhancement to randomly scale, crop, and arrange images. The input modules of the fast branch Fastpath and the slow branch Slowpath both send the processed training images to the improved backbone network Backbone module. The fast branch Fastpath of the ActionNet with the attention mechanism uses high-frequency sampled image frames, that is, densely sampled frames, as input to capture the timing information in the video. The slow branch Slowpath backbone network with the TAdaconv convolution module uses sparsely sampled image frames, that is, sparsely sampled frames, as input to extract the spatial feature information of the training images. The fast path network Backbone module extracts temporal feature information of different scales and sends it to the slow path network Slowpath through lateral connections to achieve the fusion of temporal semantic features and spatial semantic features. Then the two branches output different temporal and spatial feature information of the training image respectively. The feature vectors of different scales output by the last two branches are concatenated and fused in the fully connected softmax layer, and the category prediction is performed to achieve feature extraction, feature fusion and atomic action detection; 44) Calculate the loss value based on the prediction result and the true label, back-propagate the loss value, update the network parameters, and repeat the above 43) steps until the preset number of training rounds is reached or other stopping conditions are met.

5. According to the method of claim 1, the method is characterized in that: The agitation atomic motion recognition of ICU patients comprises the following steps: 51) Obtain the video stream data to be identified; 52) The video stream data to be identified is input into the trained ICU patient agitation recognition model to perform spatiotemporal motion detection, feature extraction, feature fusion and atomic motion detection. If no human target is detected, no spatiotemporal action detection is performed and None is output; If a human target is detected, the agitated atomic motion of the ICU patient is detected; 53) After detecting the agitated atomic motion of the ICU patient, output the human target frame data information and the agitated atomic motion information of the ICU patient; Take the key frame as the reference frame, use IoU to measure the overlap between each border B and the true label G, and calculate the difference between the key frame and the current frame and the reference distance between the previous and next frames as the motion information prediction threshold d i , perform atomic action recognition, threshold d i The calculation formula is as follows, where f i is the current frame, f r is the reference frame, d(f r ) is the reference distance; 54) Atomic motion recognition includes: no atomic motion corresponding threshold is within 50%, small-amplitude atomic motion corresponding threshold is within 50% to 70%, medium-amplitude atomic motion corresponding threshold is within 70% to 85%, and large-amplitude atomic motion corresponding threshold is above 85%; 55) After the atomic action is identified, agitation identification and classification based on the RASS scoring table is performed.

6. The method for identifying agitated videos of ICU patients based on spatiotemporal detection according to claim 1, characterized in that: The agitation identification classification based on the RASS scoring table includes the following steps: 61) Based on the RASS scoring tool, agitation is defined as restless, agitated, very agitated, and dangerously agitated; 62) The identified atomic movements were scored for restlessness, with 1 point for head movement, 1 point for shoulder movement, 1 point for manual movement, 1 point for standing up, and 1 point for leg flexion. The total score was less than or equal to 4 points. 63) The above atomic action scores are weighted and summed using the agitation quantification calculation formula, and the patient's agitation state is determined based on the RASS score table; If the recognition result is less than the set threshold, the system does not output the agitation warning signal; If it is greater than the set threshold, the system will output a restlessness warning signal and its judgment basis. The warning signal includes restlessness, agitation, extreme agitation, and dangerous agitation. The judgment basis is the five atomic movement types of head movement, shoulder movement, manual movement, standing up, and leg bending. Among them, the quantitative calculation formula of agitation based on judgment is as follows: R total =R h (S1)+R a (S2)+R S (S3)+R g (S4)+R l (S5),R total <=4, Among them, R total is the total score of restlessness quantification, R h , R a , R s , R g , R l They are head movement, hand movement, shoulder movement, standing up, and leg bending corresponding to different thresholds s i Quantitative score of restlessness, S i The corresponding thresholds are S1, S2, S3, S4, and S5 respectively.

Images