A method for intraoperative state visual monitoring of an anesthetized patient based on a deep convolutional neural network

Abstract

This invention discloses a visual monitoring method for intraoperative status of anesthetized patients based on deep convolutional neural networks, belonging to the field of computer vision and medical monitoring technology. The method includes: simultaneously acquiring videos of the patient's face, body movements, and the surgical scene to obtain facial region images, body movement region images, and surgical scene images; extracting facial feature vectors, body movement spatiotemporal feature vectors, and scene feature vectors modulated by illumination gating through facial, body movement, and scene branches respectively; obtaining a fused feature vector; concatenating the fused feature vector with the modulated scene feature vector and inputting it into a gated recurrent unit for temporal modeling; and outputting the probability distributions of body movement risk, facial expression state, airway state, and sedation depth based on hidden states through multi-task output, and comparing them with preset thresholds to trigger an early warning. This invention achieves multi-dimensional, accurate, and robust monitoring of the patient's intraoperative status in complex surgical environments through multi-branch collaboration, illumination-adaptive gating, and asymmetric feature fusion.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of computer vision and medical monitoring, and in particular to a method for visual monitoring of the intraoperative status of anesthetized patients based on deep convolutional neural networks. Background Technology

[0002] During general anesthesia, continuous monitoring of the patient's condition is crucial for ensuring their safety. Anesthesiologists need to assess the patient's sedation depth, analgesia level, respiratory status, and motor responses in real time throughout the procedure, adjusting the infusion rate and type of anesthetic drugs accordingly. However, existing intraoperative monitoring methods suffer from several technical limitations in terms of monitoring scope, real-time performance, and accuracy.

[0003] Currently, the widely used anesthesia depth monitoring equipment in clinical practice mainly includes monitoring technologies based on electroencephalogram (EEG) signals and those based on autonomic nervous system function indicators. EEG-based monitoring technologies, such as bispectral index monitors and entropy index monitors, collect EEG signals from the patient's frontal lobe and perform frequency domain analysis to output a single numerical anesthesia depth index. However, EEG signal acquisition sensors are highly sensitive to electromagnetic interference from surgical devices such as electrosurgical units, resulting in a significant decrease in signal quality during orthopedic and electrosurgical procedures. More importantly, these EEG-based monitoring technologies only reflect the electrical activity state of the cerebral cortex and cannot provide visual information about the patient's intraoperative status, such as facial expressions, body movement responses, and airway status. Monitoring technologies based on autonomic nervous system function indicators, such as heart rate variability analysis and pulse wave analysis, also suffer from limitations, providing only indirect physiological indicators and lacking the ability to directly observe the patient's intraoperative behavioral state.

[0004] In computer vision-assisted anesthesia monitoring, existing technologies exhibit significant fragmentation. For facial micro-movement detection during anesthesia recovery, some studies have employed improved single-point multi-box detectors for target detection; however, this method is only applicable to specific micro-movement recognition during the recovery phase and cannot cover the diverse visual states throughout the entire surgical process. Regarding facial pain expression assessment, existing technologies have established pain intensity assessment models based on deep learning and facial expression images. However, these methods rely on static analysis of discrete image frames, lacking the ability to process continuous temporal state changes in intraoperative video streams, and cannot collaboratively analyze facial expression information with multi-dimensional visual information such as body movement and airway status. In body movement detection, existing motion perception methods based on optical flow networks show a significant decrease in accuracy in estimating local regions under surgical instrument occlusion scenarios and fail to achieve feature-level fusion with facial expression analysis.

[0005] In summary, the core problem with existing technologies lies in the lack of a multi-dimensional visual monitoring method that integrates facial expression features, body movement features, and surgical scene features for monitoring the intraoperative status of anesthetized patients. Existing single-dimensional visual analysis technologies operate in isolation, failing to form synergy and complementarity at the feature level. Consequently, the accuracy and robustness of intraoperative status assessment cannot meet clinical requirements. Furthermore, existing methods generally lack modeling and utilization of the correlation between changes in shadowless lamp illumination in the surgical scene and the patient's visual state, ignoring the anesthetic status discrimination information inherent in the scene's lighting characteristics themselves. Summary of the Invention

[0006] To achieve the above objectives, this invention provides a method for visual monitoring of the intraoperative status of anesthetized patients based on deep convolutional neural networks, comprising the following steps:

[0007] Step 1: Use at least 3 clock-synchronized high-definition cameras to capture video streams of the patient's face, patient's body movements, and the surgical scene, respectively.

[0008] Step 2: Obtain a facial region image from the patient's frontal facial video stream, obtain a body movement region image from the patient's body movement video stream, and obtain a surgical scene image from the surgical scene video stream. The facial region image, the body movement region image, and the surgical scene image have the same preset size.

[0009] Step 3: Input the facial region image, the body motion region image, and the surgical scene image into a pre-trained multi-branch collaborative deep convolutional neural network; the multi-branch collaborative deep convolutional neural network includes a facial branch deep convolutional neural network, a body motion branch deep convolutional neural network, a surgical scene branch deep convolutional neural network, an illumination feature gating module, a cross-attention feature fusion module, and a temporal recursive fusion module;

[0010] Step 4: The facial branch deep convolutional neural network extracts facial feature vectors from the facial region image. The deep convolutional neural network with the motion branch extracts motion feature vectors from a motion input sequence consisting of multiple consecutive frames of the motion region images. The surgical scene branch deep convolutional neural network extracts an initial scene feature vector from the surgical scene image. And from the illumination feature gating module Extracting illumination feature subvectors Generate a gating weight vector G, and then combine G with... The middle corresponds to The feature components are multiplied element-wise to obtain the scene feature vector after gating modulation. ;

[0011] Step 5: The cross-attention feature fusion module will... The query vector Q is mapped to the low-dimensional joint feature space through the first trainable transformation matrix. The key vector K in the low-dimensional joint feature space is mapped to the second trainable transformation matrix. The value vector V is mapped through a third trainable transformation matrix; cross-attention weights are calculated in the low-dimensional joint feature space. ,Will Multiplying by V yields the cross-attention output vector. The cross-attention output vector is then linearly projected and multiplied by V. Add them together to obtain the fused feature vector. ;

[0012] Step 6: The time-series recursive fusion module will... and After concatenating the vectors, input them into the gated recurrent unit to obtain the hidden state vector at the current time step. ;

[0013] Step 7: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full The input multi-task output module outputs the probability distribution of the corresponding categories through the body movement risk binary classification head, facial expression state three-classification head, airway state binary classification head, and sedation depth three-classification head, respectively.

[0014] Step 8: Compare the probability distribution output by each classification head with the corresponding preset alarm threshold, and trigger an early warning signal when the probability of any category exceeds the corresponding threshold.

[0015] Preferably, in step 1, the first high-definition camera is deployed above the head area of ​​the operating table with its lens pointing vertically downwards toward the patient's face; the second high-definition camera is deployed to the side of the operating table with its lens pointing toward the patient's torso and limbs; and the third high-definition camera is deployed on the ceiling of the operating room with its lens pointing toward the entire operating table. The three high-definition cameras are clock-synchronized using a hardware-triggered synchronization line. The acquisition frame rate is set to 30 frames per second, and the acquisition resolution is 1920 pixels × 1080 pixels. The three high-definition cameras are connected to an edge computing device containing at least one graphics processing unit via a wired data transmission interface with a transmission rate of not less than 1 gigabits per second.

[0016] Preferably, in step 2, the facial region image is acquired by: performing facial region detection on each frame of the patient's frontal video stream using a multi-task cascaded convolutional neural network, outputting facial bounding box coordinates, and cropping a facial region image of size 224 pixels × 224 pixels based on the facial bounding box coordinates; the body movement region image is acquired by: extracting a region of size 224 pixels × 224 pixels centered on the center of the image from the patient's body movement video stream as the body movement region image; the surgical scene image is acquired by: scaling the frames of the surgical scene video stream to 224 pixels × 224 pixels.

[0017] Preferably, in step 4, the facial branch deep convolutional neural network adopts the ResNet-50 architecture. After removing the last fully connected classification layer of the ResNet-50 architecture, it is used as the facial branch feature extractor to perform layer-by-layer convolution and pooling operations on the facial region image, and outputs the facial feature vector with a dimension of 2048. .

[0018] Preferably, in step 4, the motion branch deep convolutional neural network adopts an R(2+1)D-18 architecture pre-trained on the Kinetics-400 dataset, and the motion input sequence consists of 16 consecutive frames of the motion region images; the motion branch deep convolutional neural network decomposes the spatial dimension 2D convolution and the temporal dimension 1D convolution to extract spatiotemporal features from the motion input sequence, and outputs a 512-dimensional motion feature vector. .

[0019] Preferably, in step 4, the operation of the illumination feature gating module includes: the initial scene feature vector The dimension is 2048, and the first 256 scalar components are taken as the illumination feature sub-vector. The illumination feature gating module includes an illumination attention network, which consists of a first fully connected layer and a second fully connected layer. The mapping is a 256-dimensional gated weight vector G, where the i-th gated weight... The calculation formula is:

[0020]

[0021] In the formula, This is the first fully connected weight matrix. This is the second fully connected weight matrix. This is the first bias vector. This is the second bias vector. It is a linear rectified activation function. The Sigmoid activation function is used; the gate weight vector G is then combined with... The first 256 scalar components are multiplied element by element. The remaining 1792 scalar components remain unchanged, resulting in the gated modulated scene feature vector. .

[0022] Preferably, in step 5, the first trainable transformation matrix The second trainable transformation matrix The third trainable transformation matrix The dimension of the low-dimensional joint feature space The cross-attention output vector has a dimension of 256. A projection matrix that performs a linear projection on the cross-attention output vector maps the 256 dimensions to 2048 dimensions. The projection result is... Element-wise addition yields the 2048-dimensional fused feature vector. .

[0023] Preferably, in step 6, the vector concatenation operation involves concatenating 2048-dimensional vectors... With 2048 dimensions The concatenated vector is concatenated into a 4096-dimensional vector; this dimensionality is then reduced to 256 dimensions using a fully connected layer, and used as the input vector for the current time step. The hidden state of the gated recurrent unit has a dimension of 256, and is determined based on the input vector of the current time step and the hidden state vector of the previous time step. Calculate the hidden state vector at the current time step. Steps 2 through 6 are executed cyclically at the same frequency as the image acquisition in step 2, with each time step generating one... .

[0024] Preferably, in step 7, each classification head of the multi-task output module uses the Softmax activation function; the probability distribution output by the body movement risk binary classification head corresponds to the two labels "body movement risk exists" and "body movement risk does not exist"; the probability distribution output by the facial expression state ternary classification head corresponds to the three labels "normal expression", "painful expression" and "unconscious expression"; the probability distribution output by the airway state binary classification head corresponds to the two labels "airway open" and "airway abnormal"; and the probability distribution output by the sedation depth ternary classification head corresponds to the three labels "sedation too shallow", "sedation moderate", and "sedation too deep".

[0025] Preferably, in step 8, the preset alarm thresholds include: a probability threshold of 0.7 for the "risk of body movement" label, a probability threshold of 0.6 for the "pain expression" label, a probability threshold of 0.5 for the "airway abnormality" label, a probability threshold of 0.7 for the "too shallow sedation" label, and a probability threshold of 0.7 for the "too deep sedation" label; when an alarm signal is triggered, the alarm signal is transmitted to the anesthesia monitoring display screen through the hospital's internal network, and the classification results, probability distribution values, and alarm signal status of each classification head at the current time step are recorded as an intraoperative monitoring log.

[0026] The beneficial effects of this invention are:

[0027] 1. This invention constructs facial, body movement, and surgical scene branches, and employs an asymmetric cross-attention fusion mechanism to map high-dimensional facial features into query vectors and low-dimensional body movement features into key and value vectors. This enables cross-dimensional interaction of heterogeneous features in a low-dimensional joint feature space, allowing facial details to guide the selective aggregation of body movement spatiotemporal patterns. This extracts facial-body movement collaborative discrimination information that cannot be obtained by a single branch, significantly reducing misjudgment of facial expressions caused by surgical stimuli and greatly improving the accuracy of intraoperative state recognition.

[0028] 2. This invention extracts the light-sensitive component that is highly correlated with the change in the illuminance of the shadowless lamp from the initial scene feature vector through the light feature gating module. It uses a two-layer fully connected network to generate gating weights, modulates only the light-sensitive dimension element by element while retaining non-light semantic features, effectively suppressing the scene feature distortion caused by sudden changes in lighting such as the switching of surgical lights. It transforms the lighting changes that are regarded as noise by traditional methods into features with discriminative value, greatly enhancing the robustness of the model to complex surgical lighting environments.

[0029] 3. This invention uses a temporal recursive fusion module to splice the joint features obtained from cross-attention fusion with the scene features after gating modulation and to model the gating loop unit. This makes the state discrimination at each moment not only depend on the current multimodal visual input, but also integrates historical temporal dependencies. This allows the continuous evolution trend of intraoperative sedation, analgesia and body movement to be captured, improving the detection sensitivity of slowly progressive state changes and realizing multi-dimensional trend monitoring and timely early warning throughout the entire surgical cycle. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a flowchart of the steps of the method of the present invention. Detailed Implementation

[0032] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should also be noted that, to make the embodiments more comprehensive, the following embodiments are the best and preferred embodiments, and those skilled in the art can use other alternative methods to implement some well-known technologies; moreover, the accompanying drawings are only for more specific description of the embodiments and are not intended to specifically limit the present invention.

[0033] Please see Figure 1 This invention provides a method for visual monitoring of the intraoperative state of anesthetized patients based on a deep convolutional neural network. The core concept lies in constructing a multi-branch collaborative deep convolutional neural network architecture. This architecture performs asymmetric feature-level fusion and gating modulation of facial expression information, spatiotemporal information of body movement, and surgical scene illumination information from different perspectives. Through temporal recursive modeling, it achieves continuous and accurate monitoring of the patient's multi-dimensional intraoperative state. The method of this invention will be described in detail below with reference to embodiments.

[0034] In this invention, "asymmetric feature transformation" refers to the process of cross-attention fusion when fusing feature vectors from two sources with different dimensions (e.g., high-dimensional facial features and low-dimensional body movement features). Instead of mapping them to the same dimensional space beforehand, the high-dimensional features are directly mapped to a query vector in a low-dimensional joint feature space using their respective trainable transformation matrices. The low-dimensional features are then mapped to key and value vectors in the same low-dimensional joint feature space. This approach preserves the rich expressive power of the high-dimensional features while enabling cross-dimensional interaction of heterogeneous features. This avoids the information loss caused by traditional methods that first reduce the dimensionality to a low-dimensional representation of the high-dimensional features and then align them, allowing facial detail information to fully guide the selective aggregation of body movement spatiotemporal patterns.

[0035] The core idea behind the illumination feature gating module is that surgical scene images not only contain environmental information such as the state of the shadowless lamp and the layout of surgical instruments, but also that certain dimensions of their feature representation are highly sensitive to changes in light intensity. This invention utilizes the first 256 scalar components of the initial scene feature vector as illumination feature sub-vectors because experiments have shown that among the 2048-dimensional feature vectors extracted from surgical scene images by ResNet-50, the response variance in the first 256 dimensions has the most significant correlation with changes in shadowless lamp illuminance, effectively representing situations such as the surgical lamp being turned on, switched, or occluded. Based on this, the illumination feature gating module generates a gating weight vector and modulates only these 256 dimensions of the scene features element-wise, while keeping the remaining 1792 dimensions of non-illumination-related features unchanged. This achieves the goal of suppressing the interference of sudden illumination changes on scene representation without destroying environmental semantic features, transforming illumination changes from noise into valuable discriminative information.

[0036] The temporal recursive fusion module employs a gated loop unit, which temporally integrates the fused multimodal features with the gated modulated scene features. This module leverages the continuity of intraoperative state evolution, such as the gradual shift from moderate sedation to shallow sedation, and through the recursion of hidden states, enables the model to make comprehensive judgments based not only on the visual information of the current frame but also on historical state sequences. This significantly improves the detection sensitivity for slowly changing states and the robustness to instantaneous interference. The following detailed implementation process, from hardware deployment to early warning output, is described in detail through specific examples.

[0037] Example 1

[0038] This embodiment corresponds to a general anesthesia surgery scenario. The monitoring system of the present invention is deployed in a standard operating room environment and the visual monitoring method is implemented. The specific steps are as follows.

[0039] Step 1: Camera Deployment and Clock Synchronization

[0040] Before the surgery begins, three high-definition cameras are deployed as follows: The first high-definition camera is fixed to the surgical light bracket directly above the head area of ​​the operating table, with the lens pointing vertically downwards towards the patient's face, ensuring the field of view covers the unobstructed facial area from the brow bone to the chin. The second high-definition camera is fixed to the right side of the operating table using an adjustable bracket, 1.8 m horizontally from the edge of the operating table, with the lens 1.35 m above the ground, facing the patient's torso and limbs, capturing a field of view including the patient's chest, abdomen, and both upper limbs. The third high-definition camera is installed on the operating room ceiling, 2.5 m horizontally from the center of the operating table, with the lens pointing downwards at a 45° angle towards the entire operating table area, capturing a field of view including the patient's entire body, the surgical light, and part of the area surrounding the operating table.

[0041] Three high-definition cameras are connected via hardware-triggered synchronization lines with BNC interfaces. A synchronization signal generator sends square wave pulses at a frequency of 30 Hz to ensure strict alignment of the exposure times of each frame in the three video streams, with a timestamp deviation of less than 1 ms. The frame rate of all three high-definition cameras is set to 30 fps, and the resolution is set to 1920 pixels × 1080 pixels in YUV 4:2:2 format. The three high-definition cameras are connected via CAT6A Gigabit Ethernet cables to an edge computing device deployed with a deep convolutional neural network model. This edge computing device is equipped with one NVIDIA Tesla T4 graphics processing unit, 32 GB of system memory, and 1 TB of solid-state storage. The Ethernet link transmission rate is 1 Gbps, meeting the real-time transmission requirements of the three uncompressed video streams.

[0042] Step 2: Synchronous reception of video stream and image preprocessing

[0043] During the surgery, the edge computing device synchronously received video streams from three high-definition cameras via a gigabit Ethernet interface. All three video streams had a frame rate of 30 fps and a resolution of 1920×1080 per frame. The edge computing device extracted keyframes corresponding to the current timestamp from each of the three aligned video streams once per second for preprocessing. The selection of keyframes followed the sequence number of the same hardware trigger pulse in all three video streams to ensure spatiotemporal consistency.

[0044] For the patient's frontal facial video stream captured by the first high-definition camera, the edge computing device loads a pre-trained multi-task cascaded convolutional neural network (MTCNN), which consists of three cascaded stages: P-Net, R-Net, and O-Net. For the current keyframe, MTCNN first generates candidate facial region windows through P-Net, then performs fine regression through R-Net and O-Net to output the final five facial feature points and the coordinates of the top-left corner of the facial bounding box. and the coordinates of the bottom right corner Based on the bounding box coordinates, a facial region image of size 224 pixels × 224 pixels is cropped from the original 1920×1080 frame using bilinear interpolation. During the cropping process, the center position of the facial region remains unchanged. When the aspect ratio of the bounding box is not 1:1, black pixels are uniformly filled outward from the long side of the bounding box to make the cropped area a square, and then scaled to 224×224.

[0045] For the patient's body movement video stream captured by the second high-definition camera, the edge computing device directly extracts a 224-pixel × 224-pixel square region from the center of the current keyframe as the body movement region image, with the center of the extracted region coinciding with the geometric center of the frame. Because the field of view of the body movement camera has been pre-adjusted to cover the patient's torso and the main areas of limb movement, this fixed extraction method can stably acquire a temporal image sequence reflecting the patient's body movements. The edge computing device maintains a 16-frame first-in-first-out queue in memory. Each acquired body movement region image is pushed into the queue. When the queue is full (16 frames), it constitutes a body movement input sequence with the shape (16, 3, 224, 224).

[0046] For the surgical scene video stream captured by the third high-definition camera, the edge computing device scales the entire frame image to 224 pixels × 224 pixels using bicubic interpolation to obtain the surgical scene image.

[0047] Step 3: Input the multi-branch collaborative deep convolutional neural network model

[0048] The edge computing device takes the facial region image (shape 1×3×224×224), the body motion region image sequence (shape 16×3×224×224), and the surgical scene image (shape 1×3×224×224) obtained in step 2 and feeds them into a set of three-channel input data, which is then fed into a pre-trained multi-branch collaborative deep convolutional neural network model. This model is implemented using the PyTorch 1.12 framework, and all network parameters have been converged through offline training. It has also been deployed in the graphics processing unit of the edge computing device using FP16 half-precision format to accelerate inference.

[0049] The multi-branch collaborative deep convolutional neural network model specifically includes: a facial branch deep convolutional neural network, a body motion branch deep convolutional neural network, a surgical scene branch deep convolutional neural network, an illumination feature gating module, a cross-attention feature fusion module, and a temporal recursive fusion module. Steps 4 to 6 below describe the detailed working process of each module.

[0050] Step 4: Multi-branch feature extraction and illumination gating modulation

[0051] The deep convolutional neural network for the facial branch uses a ResNet-50 architecture as its backbone. This ResNet-50 consists of one 7×7 convolutional layer, one max-pooling layer, and four bottleneck residual modules. The feature map size output by the bottleneck residual modules decreases progressively, ultimately resulting in a 2048-dimensional feature vector after passing through a global average pooling layer. This invention removes the 1000-class fully connected classification layer at the end of the ResNet-50, directly using the output of the global average pooling layer as the output of the facial branch feature extractor. When the 224×224×3 facial region image obtained in step 2 is input into the facial branch, it undergoes layer-by-layer forward propagation, outputting a 2048-dimensional facial feature vector at the global average pooling layer. .

[0052] The motion branch deep convolutional neural network employs an R(2+1)D-18 architecture pre-trained on the Kinetics-400 human motion dataset. R(2+1)D-18 decomposes traditional 3D convolution into spatial 2D convolution and temporal 1D convolution. Each decomposed convolutional block contains a 2D spatial convolutional layer (kernel size 1×3×3) and a 1D temporal convolutional layer (kernel size 3×1×1), with ReLU activation and batch normalization inserted between them. The motion input sequence is a tensor formed by 16 consecutive frames of 224×224×3 motion region images constructed in step 2, with a shape of (16,3,224,224). After receiving this sequence, the motion branch progressively reduces the spatiotemporal resolution through five decomposed convolutional stages, finally obtaining a 512-dimensional motion feature vector through spatiotemporal global average pooling. .

[0053] The deep convolutional neural network for the surgical scene branch also uses the ResNet-50 architecture as the feature extractor. It performs forward computation on the 224×224×3 surgical scene image obtained in step 2 to obtain an initial scene feature vector with a dimension of 2048. .Will Input illumination feature gating module. This module first retrieves... The first 256 scalar components constitute the illumination feature vector. This sub-vector encodes lighting-sensitive information such as the illuminance of the surgical lamp and the light field distribution in the current surgical scene. The lighting feature gating module internally contains a lighting attention network, which consists of two fully connected layers: the first fully connected layer has a weight matrix... The size is 128×256, and the first bias vector It has 128 dimensions; the weight matrix of the second fully connected layer. The size is 256×128, and the second bias vector It is 256-dimensional. The illumination attention network will... The mapping is a 256-dimensional gated weight vector G, where the i-th component is calculated using the following formula:

[0054]

[0055] In the formula, It is a linear rectified activation function. The sigmoid activation function has an output range between (0,1). Each element in the gated weight vector G essentially represents the confidence or significance of the corresponding dimension's illumination feature in the current frame. When illumination changes abruptly (such as a change in surgical lights), the dimensions most severely affected will receive lower weights, thus suppressing their interference with subsequent tasks. The gated weight vector G is then compared with the initial scene feature vector. The first 256 scalar components are multiplied element-wise, while The remaining 1792 scalar components remain unchanged, ultimately yielding the scene feature vector after gating modulation. This selective modulation mechanism ensures that semantic content in the scene features unrelated to the shadowless lamp illumination (such as surgical instruments, patient positioning, etc.) is not unnecessarily attenuated, thus preserving the complete scene context.

[0056] Step 5: Cross-attention feature fusion

[0057] The cross-attention feature fusion module receives the facial feature vector output from step 4. and body motion feature vector As input, an asymmetric feature transformation strategy is employed to interact within the low-dimensional joint feature space. Specifically, the dimension of the low-dimensional joint feature space is defined. Construct three trainable fully connected transformation matrices:

[0058] First trainable transformation matrix High-dimensional facial features Mapped to query vector , ;

[0059] Second trainable transformation matrix Low-dimensional body movement characteristics Mapped to key vector , ;

[0060] Third trainable transformation matrix Low-dimensional body movement characteristics Mapped to value vector , The dimension of the value vector .

[0061] The module computes the cross-attention weight vector in the low-dimensional joint feature space. The formula for its calculation is:

[0062]

[0063] in This represents the transpose of the key vector. We obtain a 1×1 scalar attention score (since both Q and K are 128-dimensional vectors, the inner product is a scalar), which is actually a simplified representation in vector form. More precisely, in the batch implementation, Q is of shape (1, 128), K is of shape (1, 128), and the dot product is... To be a scalar, then divide by Scalar attention weights are obtained through softmax normalization. After that, Multiplying with the value vector V yields the cross-attention output vector. This operation means that facial features are adaptively weighted by the values ​​of body motion features through a query mechanism, amplifying the contribution of the spatiotemporal patterns of body motion most relevant to the current facial state.

[0064] 256-dimensional Subsequently, the cross-attention feature fusion module uses a trainable linear projection matrix. Will Map back to 2048 dimensions and combine with the original facial feature vector. Perform element-wise addition to obtain the fused feature vector:

[0065]

[0066] It simultaneously maintains strong discriminative power of facial detail features and effectively supplements the spatiotemporal patterns from the body movement branch, achieving joint representation of facial expressions and body movements. The addition operation, acting as a residual connection, can stably train and ensure that facial information always dominates, avoiding feature degradation during cross-modal fusion.

[0067] Step 6: Temporal Recursive Fusion

[0068] The temporal recursive fusion module receives the fusion feature vector obtained in step 5. (Dimension 2048) and the gated modulated scene feature vector obtained in step 4 (Dimension 2048). First, the two vectors are concatenated along the feature dimension to obtain a 4096-dimensional concatenated vector. Then, a fully connected layer reduces the dimensionality of this 4096-dimensional concatenated vector to 256 dimensions, which is then used as the input vector for the current time step. .

[0069] This input vector The hidden state is sent to the Group Regulator Unit (GRU). In this embodiment, the hidden state dimension of the GRU is set to 256. The forward propagation formula of the GRU is as follows:

[0070]

[0071]

[0072]

[0073]

[0074] in To update the door, To reset the door, Let this be the hidden state vector from the previous time step. This indicates element-wise multiplication. This is the weight matrix. This is the bias vector. In the first time step, Initialize it as an all-zero vector. Through a gating mechanism, GRU can adaptively determine how much historical state information to retain and how much current input information to fuse, thereby effectively capturing the long-range dependencies of intraoperative state changes.

[0075] Starting from the beginning of the surgery, the edge computing device continuously executes steps 2 through 6 at a frequency of once per second, generating a hidden state vector for the current time step with each execution. .

[0076] Step 7: Multi-task output

[0077] The hidden state vector at each time step Input-multi-task output module. This module first contains a shared fully connected hidden layer, and a weight matrix... bias After ReLU activation, the shared feature representation is obtained. .after, Four parallel output classification heads are fed into the system, each consisting of a fully connected layer with a Softmax activation function.

[0078] Body movement risk binary classification head: The fully connected layer outputs a 2D vector, which is then processed by Softmax to obtain the probability values ​​of "body movement risk exists" and "body movement risk does not exist";

[0079] Facial expression state three-classification head: The fully connected layer outputs a 3D vector, which is then passed through Softmax to obtain the probability values ​​of "normal expression", "painful expression" and "unconscious expression";

[0080] Airway status binary classification head: The fully connected layer outputs a 2D vector, which is then processed by Softmax to obtain the probability values ​​of "airway open" and "airway abnormal".

[0081] Sedation Depth Three-Classification Head: The fully connected layer outputs a 3D vector, which is then processed by Softmax to obtain the probability values ​​of "sedation too shallow", "sedation moderate", and "sedation too deep".

[0082] Step 8: Alarm Decision and Log Recording

[0083] The edge computing device compares the probability distribution results output in step 7 with preset alarm thresholds. The thresholds for each label are set as follows: "Risk of body movement present" threshold is 0.7, "Painful expression" threshold is 0.6, "Airway abnormality" threshold is 0.5, "Insufficient sedation" threshold is 0.7, and "Insufficient sedation" threshold is 0.7. These thresholds were determined by maximizing the Youden index using the receiver operating characteristic curve on a validation set containing 120 surgical videos.

[0084] When the probability of any label exceeds its corresponding threshold, the edge computing device immediately generates a corresponding warning signal: body movement risk warning, insufficient analgesia warning, airway warning, or sedation depth warning. The warning signal, along with the current timestamp, all classification results and probability values ​​of the four classification heads, is packaged into structured data. This data is transmitted in real time to the anesthesia monitoring display screen in the operating room via the hospital's gigabit local area network for visualization, and simultaneously written to the intraoperative monitoring log file in the solid-state storage of the edge computing device for postoperative review and quality assessment.

[0085] To further verify the technical effectiveness of this invention, a performance comparison was conducted with an ablation control group on the same test set. The test set contained full-process video data from 25 general anesthesia surgeries, each lasting approximately 2.5 hours, totaling approximately 225,000 time steps of evaluation samples. All samples were cross-labeled with binary or ternary labels for each state by three senior anesthesiologists, serving as the gold standard. The evaluation metrics used were the macro-average F1 score and the classification accuracy for each category. The control group method was constructed as follows:

[0086] Comparative Example 1 (No Lighting Gating): The lighting feature gating module is removed, and the initial scene feature vector is used directly. As It participates in subsequent temporal recursive fusion, and the remaining network structure, training strategy and hyperparameters are exactly the same as in Example 1.

[0087] Comparative Example 2 (without cross-attention fusion): The cross-attention feature fusion module is removed, and facial features are... and physical characteristics Directly concatenating the vectors yields a 2560-dimensional vector, which is then reduced to 2048 dimensions using a fully connected layer as the fusion feature. The rest is the same as in Example 1.

[0088] Comparative Example 3 (Single Branch Only - Face): Extracted using only the facial branch. Discarding motion branch and scene branch, After direct dimensionality reduction, the data is input into a GRU for sequence modeling and connected to the same multi-task output head.

[0089] Table 1 Performance comparison between Example 1 and the comparative example.

[0090] method Physical risk F1 Facial Expressions F1 Airway status F1 Sedation Depth F1 Macro average F1 Example 1 0.912 0.886 0.935 0.857 0.898 Comparative Example 1 (Gated without light) 0.885 0.857 0.914 0.822 0.870 Comparative Example 2 (No Cross-Attention Fusion) 0.871 0.831 0.902 0.795 0.850 Comparative Example 3 (facial branch only) 0.802 0.764 0.853 0.731 0.788

[0091] As shown in Table 1, Example 1 outperforms Comparative Examples 1-3 in all evaluation metrics. Compared to Comparative Example 1, Example 1 shows a 2.8 percentage point improvement in the macro-average F1 score, indicating that the illumination feature gating module effectively suppresses illumination variation interference by dynamically modulating scene features, enhancing the model's adaptability to surgical scenarios. Example 1 shows a 4.8 percentage point improvement in the macro-average F1 score compared to Comparative Example 2, demonstrating that feature fusion based on asymmetric cross-attention is better at capturing the collaborative discrimination pattern between facial and body movements than direct splicing fusion, avoiding redundancy and noise caused by simple feature stacking. Example 1 shows an 11 percentage point improvement in the macro-average F1 score compared to Comparative Example 3, fully validating the significant superiority of multi-branch multimodal collaborative monitoring over a single visual branch.

[0092] This invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To provide the public with a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments; however, those skilled in the art will fully understand the invention even without these details. Furthermore, to avoid unnecessary misunderstanding of the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0093] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for visual monitoring of the intraoperative status of anesthetized patients based on deep convolutional neural networks, characterized in that, Includes the following steps: Step 1: Use at least 3 clock-synchronized high-definition cameras to capture video streams of the patient's face, patient's body movements, and the surgical scene, respectively. Step 2: Obtain a facial region image from the patient's frontal facial video stream, obtain a body movement region image from the patient's body movement video stream, and obtain a surgical scene image from the surgical scene video stream. The facial region image, the body movement region image, and the surgical scene image have the same preset size. Step 3: Input the facial region image, the body motion region image, and the surgical scene image into a pre-trained multi-branch collaborative deep convolutional neural network; the multi-branch collaborative deep convolutional neural network includes a facial branch deep convolutional neural network, a body motion branch deep convolutional neural network, a surgical scene branch deep convolutional neural network, an illumination feature gating module, a cross-attention feature fusion module, and a temporal recursive fusion module; Step 4: The facial branch deep convolutional neural network extracts facial feature vectors from the facial region image. The deep convolutional neural network with the motion branch extracts motion feature vectors from a motion input sequence consisting of multiple consecutive frames of the motion region images. The surgical scene branch deep convolutional neural network extracts an initial scene feature vector from the surgical scene image. And from the illumination feature gating module Extracting illumination feature subvectors Generate a gating weight vector G, and then combine G with... The middle corresponds to The feature components are multiplied element-wise to obtain the scene feature vector after gating modulation. ; Step 5: The cross-attention feature fusion module will... The query vector Q is mapped to the low-dimensional joint feature space through the first trainable transformation matrix. The key vector K in the low-dimensional joint feature space is mapped to the second trainable transformation matrix. The value vector V is mapped through a third trainable transformation matrix; cross-attention weights are calculated in the low-dimensional joint feature space. ,Will Multiplying by V yields the cross-attention output vector. The cross-attention output vector is then linearly projected and multiplied by V. Add them together to obtain the fused feature vector. ; Step 6: The time-series recursive fusion module will... and After concatenating the vectors, input them into the gated recurrent unit to obtain the hidden state vector at the current time step. ; Step 7: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full The input multi-task output module outputs the probability distribution of the corresponding categories through the body movement risk binary classification head, facial expression state three-classification head, airway state binary classification head, and sedation depth three-classification head, respectively. Step 8: Compare the probability distribution output by each classification head with the corresponding preset alarm threshold, and trigger an early warning signal when the probability of any category exceeds the corresponding threshold.

2. The method for visual monitoring of the intraoperative status of anesthetized patients based on a deep convolutional neural network according to claim 1, characterized in that, In step 1, the first high-definition camera is deployed above the head area of ​​the operating table with its lens pointing vertically downwards toward the patient's face; the second high-definition camera is deployed to the side of the operating table with its lens pointing toward the patient's torso and limbs; and the third high-definition camera is deployed on the ceiling of the operating room with its lens pointing toward the entire operating table. The three high-definition cameras are clock-synchronized using a hardware-triggered synchronization line; the acquisition frame rate is set to 30 frames per second, and the acquisition resolution is 1920 pixels × 1080 pixels; the three high-definition cameras are connected to an edge computing device containing at least one graphics processing unit via a wired data transmission interface with a transmission rate of not less than 1 gigabit per second.

3. The method for visual monitoring of the intraoperative status of anesthetized patients based on a deep convolutional neural network according to claim 2, characterized in that, In step 2, the facial region image is acquired by: performing facial region detection on each frame of the patient's frontal video stream using a multi-task cascaded convolutional neural network, outputting facial bounding box coordinates, and cropping a facial region image of size 224 pixels × 224 pixels based on the facial bounding box coordinates; the body movement region image is acquired by: extracting a 224-pixel × 224-pixel region centered on the center of the image from the patient's body movement video stream as the body movement region image; the surgical scene image is acquired by: scaling the frames of the surgical scene video stream to 224 pixels × 224 pixels.

4. The method for visual monitoring of the intraoperative status of anesthetized patients based on a deep convolutional neural network according to claim 1, characterized in that, In step 4, the facial branch deep convolutional neural network adopts the ResNet-50 architecture. After removing the last fully connected classification layer of the ResNet-50 architecture, it is used as the facial branch feature extractor. Layer-by-layer convolution and pooling operations are performed on the facial region image to output the facial feature vector with a dimension of 2048. .

5. The method for visual monitoring of the intraoperative status of anesthetized patients based on a deep convolutional neural network according to claim 1, characterized in that, In step 4, the motion branch deep convolutional neural network adopts an R(2+1)D-18 architecture pre-trained on the Kinetics-400 dataset. The motion input sequence consists of 16 consecutive frames of the motion region images. The motion branch deep convolutional neural network decomposes the spatial dimension 2D convolution and the temporal dimension 1D convolution to extract spatiotemporal features from the motion input sequence, and outputs a 512-dimensional motion feature vector. .

6. The method for visual monitoring of the intraoperative status of anesthetized patients based on a deep convolutional neural network according to claim 1, characterized in that, In step 4, the operation of the illumination feature gating module includes: the initial scene feature vector The dimension is 2048, and the first 256 scalar components are taken as the illumination feature sub-vector. The illumination feature gating module includes an illumination attention network, which consists of a first fully connected layer and a second fully connected layer. The mapping is a 256-dimensional gated weight vector G, where the i-th gated weight... The calculation formula is: In the formula, This is the first fully connected weight matrix. This is the second fully connected weight matrix. This is the first bias vector. This is the second bias vector. It is a linear rectified activation function. The Sigmoid activation function is used; the gate weight vector G is then combined with... The first 256 scalar components are multiplied element by element. The remaining 1792 scalar components remain unchanged, resulting in the gated modulated scene feature vector. .

7. The method for visual monitoring of the intraoperative status of anesthetized patients based on a deep convolutional neural network according to claim 1, characterized in that, In step 5, the first trainable transformation matrix The second trainable transformation matrix The third trainable transformation matrix The dimension of the low-dimensional joint feature space The cross-attention output vector has a dimension of 256. A projection matrix that performs a linear projection on the cross-attention output vector maps the 256 dimensions to 2048 dimensions. The projection result is... Element-wise addition yields the 2048-dimensional fused feature vector. .

8. The method for visual monitoring of the intraoperative status of anesthetized patients based on a deep convolutional neural network according to claim 1, characterized in that, In step 6, the vector concatenation operation involves concatenating the 2048-dimensional vectors... With 2048 dimensions The concatenated vector is concatenated into a 4096-dimensional vector; this dimensionality is then reduced to 256 dimensions using a fully connected layer, and used as the input vector for the current time step. The hidden state of the gated recurrent unit has a dimension of 256, and is determined based on the input vector of the current time step and the hidden state vector of the previous time step. Calculate the hidden state vector at the current time step. Steps 2 through 6 are executed cyclically at the same frequency as the image acquisition in step 2, with each time step generating one... .

9. The method for visual monitoring of the intraoperative status of anesthetized patients based on a deep convolutional neural network according to claim 1, characterized in that, In step 7, each classification head of the multi-task output module uses the Softmax activation function; the probability distribution output by the body movement risk binary classification head corresponds to the two labels "body movement risk exists" and "body movement risk does not exist"; the probability distribution output by the facial expression state ternary classification head corresponds to the three labels "normal expression", "painful expression" and "unconscious expression"; the probability distribution output by the airway state binary classification head corresponds to the two labels "airway open" and "airway abnormal"; and the probability distribution output by the sedation depth ternary classification head corresponds to the three labels "sedation too shallow", "sedation moderate", and "sedation too deep".

10. A method for visual monitoring of the intraoperative status of anesthetized patients based on a deep convolutional neural network according to claim 9, characterized in that, In step 8, the preset alarm thresholds include: a probability threshold of 0.7 for the "risk of body movement" label, a probability threshold of 0.6 for the "pain expression" label, a probability threshold of 0.5 for the "airway abnormality" label, a probability threshold of 0.7 for the "too shallow sedation" label, and a probability threshold of 0.7 for the "too deep sedation" label. When an alarm signal is triggered, the alarm signal is transmitted to the anesthesia monitoring display screen through the hospital's internal network, and the classification results, probability distribution values, and alarm signal status of each classification head at the current time step are recorded as an intraoperative monitoring log.

Images