A method for identifying patient agitation behavior based on visual change analysis

Abstract

This invention discloses a method for recognizing patient agitation based on visual change analysis. The method utilizes a visual encoder network V of a visual-text model to extract image features from video frames in parallel. Then, a similarity function D is used to calculate the change map between adjacent frames, resulting in an overall change map C for that time period. This change map is then used as the feature map of attention-weighted video frames, and the features of each frame are aggregated through a temporal fusion network A to obtain the overall video representation Z. Representation Z is passed through a linear layer classification network H to obtain the predicted probability of agitation category and calculate the classification loss L1. Simultaneously, a text encoder T of the visual-text model converts the corresponding text description into a text representation t. A distance loss L2 is calculated between the video representation and the text representation, allowing the model to utilize text to enrich the learned visual features. This invention can achieve patient agitation recognition, improving generalization across different environments and the accuracy of agitation recognition.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and pattern recognition, and specifically relates to a method for recognizing patient agitation behavior based on visual change analysis. Background Technology

[0002] In recent years, as cities have continuously promoted digital empowerment reforms, the monitoring and early warning of target behaviors can effectively reduce the occurrence of potentially dangerous behaviors. Therefore, video-based action recognition technology has gradually received widespread attention. Video action recognition mainly infers the ongoing logical action by analyzing target changes or motion information between consecutive video frames. Thus, acquiring video data of specific scenes using video capture interfaces and then using video action recognition technology to identify related actions is one of the important technologies driving the development of smart nursing, intelligent security, and other fields. However, existing video action recognition technologies mainly focus on experiments with publicly available datasets and have achieved good performance improvements. However, the dataset size required for training existing models is beyond the reach of everyday applications, and the data distribution in real-world scenarios may differ significantly from publicly available datasets, with substantial differences in action category characteristics. This presents certain practical challenges to video action recognition technology. Current mainstream deep learning methods generally involve designing network modules that model spatiotemporal relationships and using large-scale data for implicit video representation learning, which is difficult to train in practical application scenarios with only a small amount of data.

[0003] Currently, most algorithms are trained end-to-end from scratch on publicly available video action recognition datasets. On one hand, video action recognition datasets are relatively small compared to classification task datasets, resulting in insufficient generalization performance across different scenarios. On the other hand, existing models are relatively large, making it difficult to effectively learn the backbone network on small datasets. Their feature extraction capabilities are unstable and may simply be due to overfitting the current small dataset. Furthermore, without prior knowledge guidance, learning temporal video representations is also difficult. With the emergence of cross-modal foundational models such as vision-text models, some algorithms utilize text category labels as supervision signals, enabling zero-shot inference and open-set detection. Therefore, utilizing text and other modalities to assist visual models in recognizing relevant actions is currently an important research direction. Summary of the Invention

[0004] To address the need for improvement in video action recognition technology for specific scenarios involving postoperative patient behavior recognition, this invention provides a patient behavior recognition method based on visual change analysis.

[0005] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows:

[0006] A method for identifying patient agitation behavior based on visual change analysis includes the following steps:

[0007] (1) Take any input video segment and sample it at m intervals to obtain a new video I to be processed. i i = 1, 2, ..., n;

[0008] (2) Extract video I using the pre-trained visual encoder V. i Each frame of the image features is processed, and the change map between consecutive frames is calculated to obtain the overall change map C within the time range of the video. The original image features are then weighted to obtain the updated image features. The updated image features are then fused using a temporal fusion network A to form the overall video representation Z. The video representation Z is then fed into a linear layer classification network H to obtain the predicted noise category probability. The classification loss L1 is calculated to optimize the trainable temporal fusion network A and the linear layer classification network H.

[0009] (3) Calculate the distance loss L2 between the overall video representation Z without fusion change information and the text representation t extracted by the pre-trained text encoder T. This loss assists the L1 loss in step (2) to further optimize the parameters of the temporal fusion network A, thereby improving the fusion capability of the temporal fusion network A. Finally, the representation Z is fed into the linear layer classification network H to obtain the predicted agitation category probability.

[0010] Specifically, the interval sampling in step (1) is either dense sampling or sparse sampling.

[0011] Specifically, in step (2), the pre-trained visual encoder V is used to extract video I respectively. i The image features of each frame are analyzed, and the changes between consecutive frames are calculated. The calculation expression is as follows:

[0012] z j =V(f j ), j = 1, 2, ..., N;

[0013] C=Mean(D(z1,z2),D(z2,z3),…,D(z N；1 ,z N ));

[0014] z j =Update(z j ;C),j=1,2,…,N;

[0015] Z = A(z1, z2, ... z) N );

[0016] Among them, z j Indicates video I i Image f in frame j j Extracted features, N represents video I iThe total number of frames in the image is used to obtain the change map of image features between consecutive frames through the distance metric function D, and then the average operation is used to obtain video I. i Overall change map C; updating image features z using change map C. j j = 1, 2, ..., N; Z represents the image features [z1, z2, ..., z] that the temporal fusion network A will update. N The video representation is aggregated to form a whole.

[0017] Specifically, the expression for calculating the classification loss L1 in step (2) is as follows:

[0018] L1 = CE(H(Z),y);

[0019] min A , H L1;

[0020] Where CE represents a classification loss, and y represents the category of agitated behavior corresponding to the video segment.

[0021] Specifically, in step (3), the distance loss L2 is calculated between the overall video representation Z without fused change information and the text representation t extracted by the pre-trained text encoder T. The calculation expression is as follows:

[0022] L2 = D(E(Z),t);

[0023] min A,E L2;

[0024] Where D represents the distance metric function between two vectors, t represents the corresponding text representation, and E represents mapping the video representation Z to the same dimension.

[0025] The beneficial effects of this invention are as follows: First, by utilizing a pre-trained visual-text large model for basic feature extraction, the extracted basic features are made more robust and generalizable, facilitating adaptation to different scenarios. Second, by employing visual change analysis to locate regions of action change between video frames, the network model is strengthened to focus on the action details occurring in these regions, allowing it to ignore interference from a large amount of irrelevant static background and concentrate on the action itself. Furthermore, this invention further enhances video feature learning by utilizing textual descriptions of patient behavior. Text can provide more granular descriptions, such as the location and frequency of agitation, as well as descriptive terms related to the patient's emotions, thereby significantly improving the accuracy of identifying agitated patient behavior and enhancing the intelligence of postoperative patient monitoring. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the overall structure for patient agitation behavior recognition designed in this invention;

[0027] Figure 2 This is the change information integration module of the present invention;

[0028] Figure 3 This is a flowchart of the training phase of the present invention;

[0029] Figure 4 This is a flowchart of the testing phase of the present invention;

[0030] Figure 5 This is a diagram of the VIT architecture model of the present invention. Detailed Implementation

[0031] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0032] like Figure 1 , Figure 3 and Figure 4 As shown, this invention provides a method for identifying patient agitation behavior based on visual change analysis, specifically including the following steps:

[0033] (1) Take any input video segment and sample it at m intervals to obtain a new video I to be processed. i i = 1, 2, ..., n;

[0034] (2) During the training phase, the frozen pre-trained visual encoder network V is used to train video I respectively. i Features are extracted from each frame of the video, and the change map between consecutive frames is calculated to obtain the overall change map C within the video's time range. The original image features are then weighted to obtain updated image features. A temporal fusion network A is then used to fuse the updated frame image features to form the overall video representation Z. Finally, the video representation Z is fed into a linear layer classification network H to obtain the predicted noise category probability. The classification loss L1 is calculated to optimize the trainable networks A and H, as follows: Figure 1 As shown in the flowchart, TransformerBlock represents a single-layer network module, and its specific implementation is as follows: Figure 5 As shown, Change Capture is Figure 2 The process shown;

[0035] (3) During the training phase, the distance loss L2 is calculated between the overall video representation Z without fused change information and the text representation t extracted by the frozen pre-trained text encoder T. This loss assists the classification loss in (2) to further improve the ability of the temporal fusion network A, such as Figure 1 As shown in the upper middle part;

[0036] (4) During the testing phase, features of each video frame were extracted using only the frozen pre-trained visual encoder V, and the probability of the agitated behavior category was obtained by forward inference using the temporal fusion network A and the linear layer classification network H.

[0037] Furthermore, in step (1), the interval sampling of video frames can be either dense sampling or sparse sampling.

[0038] Furthermore, in step (2), during the training phase, the frozen pre-trained visual encoder V is used to extract video I respectively. i For each frame of image features, the visual encoder network V can be any Transformer series.

[0039] z j =V(f j ), j = 1, 2, ..., N

[0040] C=Mean(D(z1,z2),D(z2,z3),…,D(z N；1 ,z N ))

[0041] z j =Update(z j ;C),j=1,2,…,N

[0042] Z = A(z1, z2, ... z) N )

[0043] Among them, z j Indicates video I i Image f in frame j j Extracted features, N represents video I i The total number of frames in the image is used to obtain the change map of image features between consecutive frames through the distance metric function D, and then the average operation is used to obtain video I. i Overall change diagram C. Updating image features z using change diagram C. j j = 1, 2, ..., N. Z represents the image features [z1, z2, ..., z] that the temporal fusion network A will update. N The video representation is aggregated to form a whole. The temporal fusion network A can be a simple average pooling network or a learnable temporal learning network such as temporal convolution, LSTM, or Transformer.

[0044] Furthermore, in step (2), during the training phase, the overall video representation Z is subjected to supervised training for classification through a linear layer classification network H to make the extracted features more class-distinguishable.

[0045] L1 = CE(H(Z),y);

[0046] min A , H L1;

[0047] Where CE represents a classification loss, and y represents the category of agitated behavior corresponding to the video segment.

[0048] Furthermore, in step (3), the video representation Z without fused change information is reduced to the same dimension as the text representation through a single mapping matrix, and the distance loss L2 between the two is calculated, so that the richer details in the text can further improve the ability of the temporal fusion network A, thereby improving the learning of video features.

[0049] L2 = D(E(Z),t);

[0050] min A,E L2;

[0051] Where D represents the distance metric function between two vectors, t represents the corresponding text representation, and E represents mapping the video representation Z to the same dimension.

[0052] Furthermore, in step (4), during the actual testing phase, the text encoder T is no longer needed. Instead, the frozen pre-trained visual encoder V is used to extract the image features of each video frame, and then the trained temporal fusion network A is used to obtain the overall video representation Z. Finally, the representation Z is fed into the linear layer classification network H to obtain the final predicted category of the agitated behavior.

[0053] Example

[0054] The implementation process of this embodiment is as follows:

[0055] (1) Take any input video segment and sample it at m intervals to obtain a new video I to be processed. i Let i = 1, 2, ..., n, and denote each new video segment as consisting of N frames, represented as [f1; f2; ...; f...]. N ]

[0056] (2) During the training phase, the frozen pre-trained visual encoder network V is used to train video I. i Features are extracted from each frame of the image. The pre-trained visual-text model is represented by the CLIP VIT-B-16 model (model structure diagram shown in Figure 1). Figure 5 Taking the image shown as an example, the forward calculation process for a single frame is as follows:

[0057]

[0058]

[0059] z l′ =MSA(LN(z) l；1 ))+z l；1 ,l=1,…,L;

[0060] z l =MLP(LN(z) l ′ ))+z l ′ ,l=1,…,L;

[0061] Wherein, the input image f∈R H×W×C Divide it into non-overlapping P×P image patches, and flatten each image patch into a one-dimensional vector to obtain Where N represents the number of image patches, P represents the size of each image patch, C represents the number of channels in the original image, and E represents the mapping matrix that maps the image patches to a specified dimension. In the formula... This indicates a learnable Eembedding, also known as a Class Token. pos Here, z represents the positional encoding vector, MSA represents the multi-head self-attention module, MLP represents the multilayer perceptron, and LN represents layer normalization. The z0 obtained from the second formula is fed into the stacked Transformer Blocks to obtain feature maps of different depths. Each Transformer Block consists of one MSA and one MLP. The VIT-B-16 model visual encoder has a total of 12 Transformer Blocks, with a feature dimension D of 768, a P size of 16, and N of 196.

[0062] (3) During the training phase, in order to better capture semantic information with strong deep features, a multi-level overall video representation Z is extracted. In this example, the features of the last four Transformer Blocks are extracted to form the overall video representation of the corresponding level. The transformation diagram C of each layer (as shown in the figure) Figure 2 The generation process (as shown) is as follows:

[0063]

[0064] in, Indicates video I i The image patch features output by the l-th layer Transformer Block of the j-th frame image, i.e., ignoring the Class Token. C l Let represent the transformation graph of the l-th layer, cos represent the cosine similarity function, and Mean represent average pooling.

[0065] (4) During the training phase, the last four Transformer Blocks yielded transformation graphs C8, C9, and C1 respectively. 10 C 11 (Taking CLIP VIT-B-16 with a total of 12 Transformer Blocks as an example), each layer updates the image patch features using the corresponding transformation graph, then concatenates the Class Token and sends it to the Transformer Block of that layer to update the Class Token, and finally merges the Class Tokens of each image frame at that layer to obtain the video feature representation of that layer. Figure 2 As shown in the figure, the calculation process is as follows:

[0066]

[0067] z l =MSA(LN(z) l ))+z l ;

[0068] z l =MLP(LN(z) l ))+z l ;

[0069] y l =z l [0];

[0070]

[0071]

[0072] Among them, z l This indicates that after updating the patch block features in the transformation graph, a Class Token is appended and then fed into a single-layer Transformer Block to update the Class Token content. N represents the number of patches, and n represents the number of video frames. l This represents the updated Class Token, Z. l This indicates that the video features output from layer l are obtained by averaging the Class Tokens of each frame in that layer. Finally, the video representations from the last four layers are aggregated to form the overall video representation Z.

[0073] (5) During the training phase, the video representation Z is used to predict the probability of agitated behavior categories through a linear layer classification network H. Network H is a single-layer linear layer, and the output dimension is the number of agitated behavior categories. The classification loss L1 is calculated as follows:

[0074]

[0075]

[0076]

[0077] in, L represents the probability of the agitated behavior category predicted by the model, and L1 is the calculated classification loss function. CE The formula for calculating classification loss.

[0078] (6) During the training phase, the video text description is passed through a pre-trained text encoder network T to obtain the text representation t. Then, the Class Tokens of each frame output by the last Transformer Block without fused change information are aggregated to obtain the video representation. The dimensionality is reduced to the same dimension as the text features using a mapping matrix, and then the distance loss function L2 is calculated. As shown below:

[0079]

[0080] y = yE, E∈R d1×d2 ;

[0081] L2 = 1 - cos(y,t);

[0082] Where n represents the total number of video frames sampled, in this case n = 8. This represents the Class Token feature output from the 11th layer Transformer Block of the first frame image, and so on. E represents the linear transformation matrix, d1 represents the visual feature dimension, and d2 represents the text feature dimension. In this example, d1 = 768 and d2 = 512. cos represents the cosine similarity, which measures the distance between two elements.

[0083] (7) During the training phase, the model parameters are updated by using both loss L1 and loss L2, as shown in the following formula:

[0084] min A；H L1+L2;

[0085] (8) During the testing phase, the text encoder network T is no longer needed. The image frame is simply extracted through the visual encoder network V, and then the video representation is obtained by following steps (3)(4)(5). The video representation is then fed into the linear layer classification network H to obtain the final predicted behavior category.

[0086] The patient agitation behavior recognition method based on visual change analysis disclosed in this invention yielded the following experimental test data on a dataset collected by the Second Affiliated Hospital of Zhejiang University. According to the degree of patient agitation behavior and the differentiation requirements, it can be divided into binary and quadruple classification behavior recognition, as shown in Table 1.

[0087] Table 1

[0088]

[0089] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for identifying patient agitation behavior based on visual change analysis, characterized in that, Includes the following steps: (1) Take any input video segment and sample it at m intervals to obtain a new video to be processed. ; (2) Extract video using the pre-trained visual encoder V. Each frame of the image features is processed, and the change map between consecutive frames is calculated to obtain the overall change map C within the time range of the video. The original image features are then weighted to obtain the updated image features. The updated image features are then fused using a temporal fusion network A to form the overall video representation Z. The video representation Z is then fed into a linear layer classification network H to obtain the predicted noise category probability. The classification loss L1 is calculated to optimize the trainable temporal fusion network A and the linear layer classification network H. The video is extracted using a pre-trained visual encoder V. The image features of each frame are analyzed, and the changes between consecutive frames are calculated. The calculation expression is as follows: ； ； in, Indicates video Image of frame j Extracted features, where N represents the video. The total number of frames in the image is used to obtain the change map of image features between consecutive frames through the distance metric function D, and then an averaging operation is used to obtain the video. Overall change diagram C; updating image features using change diagram C. The updated image features are then processed using a temporal fusion network A. The video representation is aggregated to form a holistic representation; specifically as follows: During the training phase, the last four Transformer Blocks each yielded a transformation graph. Each layer updates the image patch features using the corresponding transformation graph, then concatenates the Class Token and feeds it into the Transformer Block of the single layer to update the Class Token. Finally, it fuses the Class Tokens of each image frame at that layer to obtain the video feature representation of the single layer. The calculation process is as follows: ； ； ； ； ； ； in, This means that after updating the Patch block features in the transformation graph, a Class Token is appended and then fed into a single-layer TransformerBlock. Update the Class Token content, where N represents the number of patches and n represents the number of video frames; This indicates the updated ClassToken. This indicates that the video features output by layer l are obtained by averaging the Class Tokens of each frame in that layer; finally, the video representations of the last four layers are aggregated to form the overall video representation Z; (3) Represent the video as a whole without fused change information The distance loss L2 is calculated with the text representation t extracted by the pre-trained text encoder T. This loss assists the L1 loss in step (2) to further optimize the parameters of the temporal fusion network A, thereby improving the fusion capability of the temporal fusion network A. In the actual testing stage, the text encoder T is no longer needed. It is only necessary to use the frozen pre-trained visual encoder V to extract the image features of each video frame, and then use the trained temporal fusion network A to obtain the overall video representation Z. Finally, the representation Z is fed into the linear layer classification network H to obtain the final predicted category of the agitated behavior.

2. The method for identifying patient agitation behavior based on visual change analysis according to claim 1, characterized in that, The interval sampling in step (1) is either dense sampling or sparse sampling.

3. The method for identifying patient agitation behavior based on visual change analysis according to claim 1, characterized in that, The expression for calculating the classification loss L1 in step (2) is as follows: ； ； Where CE represents a classification loss, and y represents the category of agitated behavior corresponding to the video segment.

4. The method for identifying patient agitation behavior based on visual change analysis according to claim 1, characterized in that, In step (3), the distance loss L2 is calculated between the overall video representation Z without fused change information and the text representation t extracted by the pre-trained text encoder T. The calculation expression is as follows: ； ； Where D represents the distance metric function between two vectors, t represents the corresponding text representation, and E represents the video representation. Mapped to the same dimension.

Images