An open airway assist device operating at the edge

Abstract

The application provides an open airway auxiliary device running at an edge end, image sensor suspended on the upper part of a sickbed is used to collect image data of the head and chest region of a patient, an intelligent visual processing model and method are designed, the model and method run on an edge device built in a camera, analyze the image data of the head and chest region collected by the image sensor, establish feature mapping of the image data, establish a feature model of head deviation, can distinguish head deviation at multiple angles in a dark light and blocked environment, and send a warning signal according to the analysis result, so that automatic post-intubation monitoring is realized.

Description

Technical Field

[0001] This invention pertains to the field of intelligent medical devices, and in particular, relates to an edge-operated open airway assist device and its monitoring method. Background Technology

[0002] In some cases of sudden illness, emergency surgery is often required. The most crucial aspect before, during, and after surgery is establishing the airway, such as through endotracheal intubation, suctioning, and anesthesia. Especially after intubation, it is essential for medical staff to continuously monitor the patient's head to ensure it remains in the optimal position; otherwise, medical risks may arise. However, real-time monitoring requires a significant amount of manpower. Even after the patient returns to the ward or ICU, in some cases, endotracheal intubation must be maintained, making it difficult to have sufficient manpower for continuous patient monitoring. Therefore, automated monitoring methods are urgently needed.

[0003] In recent years, with technological advancements, intelligent processing devices have been increasingly used in industries such as healthcare. Among these, the use of image vision methods for automated monitoring has become a trend and is gaining traction in engineering applications. Many applications of image vision methods for monitoring human behavior, particularly head movements, have emerged, such as liveness detection in the financial sector, which determines liveness by detecting specific actions performed by the individual. However, existing methods also have many limitations. For example, in facial liveness detection, edge devices like mobile phones can only detect a few simple actions, such as turning the head or opening the mouth, and are subject to strict requirements regarding ambient lighting and head position. Achieving better detection results requires more complex methods with higher computational demands, which are difficult to implement on edge devices. Adding servers and communication equipment is not only expensive but also difficult to implement in special hospital settings, failing to meet the needs of medical care. Furthermore, even in medical settings, current methods for detecting patient head posture typically use common neural network models, without specific designs tailored to the highly complex, high-security, multi-occluded, and low-computing-power environments of the medical field. Therefore, there is an urgent need for an airway assist device that can accurately, cost-effectively, and with high precision automatically monitor patient head posture. Summary of the Invention

[0004] To address this problem, this invention proposes an edge-operated airway assist device and monitoring method. It utilizes an image sensor suspended above the hospital bed to collect image data of the patient's head and chest region. An intelligent visual processing model and method are designed and invented, running on an edge device embedded with a camera. The device analyzes the head and chest image data collected by the image sensor, establishes a feature mapping of the image data (step 1), and establishes a feature model of head tilt (step 2). It can distinguish head tilt at multiple angles in low-light environments and issue warning signals based on the analysis results, thus achieving automated post-intubation monitoring.

[0005] An open airway assist device operating at the edge includes an image sensor, a fixing device, a processor, an alarm, and a transmission device;

[0006] Image sensors acquire images of the patient's head and chest. Where u and v represent the spatial dimensions of the image, and n represents the number of frames;

[0007] The processor processes the images acquired by the image sensor as follows:

[0008] (1) Let the two adjacent frames be respectively and N is a positive integer, and N=0 represents the first frame in the temporal sequence. Spatially, the positions offset from u and v in two directions by i and j are denoted as u and v, respectively. i v j Then, calculate the local change features of adjacent frames:

[0009]

[0010] in,

[0011] and

[0012]

[0013] (2) Calculate the spatiotemporal feature map in the finite spatiotemporal domain based on the local change characteristics.

[0014] The spatiotemporal feature map consists of temporal feature vectors mapped to each two-dimensional coordinate point u, v in space. The method for calculating the temporal feature vectors is as follows:

[0015]

[0016] N1, N2, N3, and N4 are adjacent frames, and

[0017]

[0018] Therefore, the spatiotemporal feature map is represented as a graph composed of feature vectors f|u,v

[0019] (3) Convert the spatiotemporal feature map into a 64-dimensional spatiotemporal sequence feature vector; and input it into a neural network. When the output of the neural network is positive, it indicates that the patient’s head is tilted. When the output is negative, it indicates that no tilting has occurred.

[0020] Image sensors are used to acquire images of the patient's head and chest.

[0021] The fixation device is used to secure the image sensor above the patient.

[0022] The processor is mounted on a fixed device.

[0023] The alarm is used to receive alarm signals sent by the processor and then sound an alarm.

[0024] The transmission device is used to transmit alarm signals to a remote location.

[0025] The sensor is a CCD camera.

[0026] The alarms are sound, light, and display alarms.

[0027] The neural network training process uses offline training.

[0028] The cost function for training a neural network is:

[0029]

[0030] in, γ represents the manually labeled value of the above sample, and γ is the network output value.

[0031] The inventive points and technical effects of this invention:

[0032] 1. The present invention is characterized by the innovative proposal of an edge-end airway assist device and monitoring method. It uses an image sensor to collect head and chest image data of patients after endotracheal intubation, and designs and invents a set of intelligent machine vision processing models and processing methods to realize automated and intelligent post-endotracheal intubation monitoring.

[0033] 2. The key feature of this invention is the proposed method for acquiring spatiotemporal sequence features of head and chest image data from patients undergoing airway intubation, establishing a feature mapping for the image data. From image data of the patient's head and chest region acquired using an image sensor suspended above the hospital bed, image processing computational methods are used to calculate and statistically analyze the source image data, obtaining statistical features related to the spatiotemporal variations of the images. These obtained statistical features are used for subsequent modeling of the visual appearance of the head and chest, thereby enabling the detection of postoperative head tilting behavior in patients. Compared with existing classical methods, by establishing a spatiotemporal feature map with a finite domain and building a global statistical model of local features, the data is mapped from the image space to the feature space, significantly reducing the data volume, improving processing efficiency, and meeting the needs of running on current mainstream edge computing devices.

[0034] 3. The key feature of this invention is the proposal of a patient head deviation monitoring model and method based on spatiotemporal sequence feature vectors. This model models spatiotemporal sequence features, calculates the regression output of head deviation based on the model, and determines whether head deviation has occurred based on the output. Compared with existing methods based on neural network models, this invention's model has fewer connection parameters, shorter processing latency, and equivalent recognition performance. Compared with relatively simple classical methods, this invention achieves equivalent processing latency, higher computational efficiency, higher recognition rate, and lower false alarm rate. Detailed Implementation

[0035] This invention proposes an edge-operated airway assist device and monitoring method. It utilizes an image sensor suspended above the bed to collect image data of the patient's head area, and designs and invents an intelligent visual processing model and method that runs on an edge device with a built-in camera. This allows for the differentiation of head tilt at multiple angles in low-light environments and the issuance of warning signals based on the analysis results, achieving automated post-intubation monitoring. Specifically:

[0036] An airway assist device operating at the edge includes an image sensor, a fixing device, a processor, an alarm, and a transmission device.

[0037] The image sensor is used to acquire images of the patient's head and chest; the sensor is a CCD camera.

[0038] Fixing devices are used to secure the image sensor above the patient, such as to the wall above the head of the bed, or to a bracket next to the bed or operating table.

[0039] The processor is mounted on a fixed device and is used to control the image sensor to acquire images, process the images, and obtain alarm signals, as shown in the following method.

[0040] The alarm is used to receive alarm signals sent by the processor and provide sound, light, and display alarms.

[0041] Transmission devices are used to transmit alarm signals to remote locations, such as nursing stations, nurse handheld terminals, or monitoring systems.

[0042] Step 1: Acquisition of spatiotemporal sequence features of head and chest images of patients with airway intubation.

[0043] This method uses image processing algorithms to calculate and statistically analyze the source image data of patients' head and chest images captured by cameras, obtaining statistical features related to the temporal and spatial variations of the images. These statistical features are then used to model the appearance of the head and chest of intubated patients, thereby enabling the monitoring of early warning behaviors related to head tilting.

[0044] Current classic behavior monitoring and recognition methods mostly employ frame difference analysis, statistically analyzing changes between adjacent frames. Limited by computational power, these methods only count local changes while ignoring global changes, leading to sensitivity to noise and a high false alarm rate. Newer models, such as those using Convolutional Neural Networks (CNNs), employ multi-layered neural networks to synthesize local features and achieve global feature modeling. However, low layer counts result in poor recognition performance, while excessively high layer counts lead to high computational costs, making them unacceptable on edge devices. Therefore, this invention proposes an innovative method for acquiring spatiotemporal sequence features from head and chest images of intubated patients. By establishing a finite-domain spatiotemporal feature map and creating a global statistical model of local features, the method maps data from image space to feature space, significantly reducing data volume and improving processing efficiency.

[0045] The patient's head and chest image data are expressed in a combined temporal and spatial dimension as follows:

[0046]

[0047] Where u and v represent the spatial dimensions of the image, and n represents the temporal dimension of the image (i.e., the number of frames). Within a finite spatiotemporal domain, i.e., u, v, and n each take a finite number of values, the method and steps for obtaining spatiotemporal sequence features are as follows.

[0048] Step 1.1 Acquire local variation features between two adjacent frames in time.

[0049] Let the two adjacent frames be respectively and N is a positive integer, and N=0 represents the first frame in the temporal sequence. Spatially, with u and v as the center, the positions offset from u and v in two directions by i and j are denoted as u and v, respectively. i ,v j The local change features of adjacent frames are then calculated as follows:

[0050]

[0051] in,

[0052]

[0053] and

[0054]

[0055] Step 1.2 Based on local change characteristics, calculate the spatiotemporal feature map within a finite spatiotemporal domain.

[0056] The spatiotemporal feature map consists of temporal feature vectors mapped to each two-dimensional coordinate point u,v in space. The temporal feature vectors are calculated based on the local change features of adjacent frames obtained in step 1.1, as follows.

[0057]

[0058] in

[0059]

[0060] Therefore, the spatiotemporal feature map is represented as a graph composed of feature vectors f|u,v

[0061]

[0062] Step 1.3 Calculate the spatiotemporal sequence features based on the spatiotemporal feature map.

[0063] For spatiotemporal feature maps For each point u, v:

[0064] a. Find the largest component in its temporal feature vector f|u,v. (Middle maximum value)

[0065] b. Based on the dimensions of the largest component in step a, i.e., the 1st, 2nd, 3rd, and 4th dimensions respectively, record the maximum dimension of that point.

[0066] c. Obtain the maximum component dimension and value for each of the above points. For the 1st, 2nd, 3rd, and 4th dimensions, take the top K largest values. Each dimension is grouped together, and the values ​​within each group are arranged in descending order to form a feature vector, called the spatiotemporal sequence feature vector. The value of K is empirically chosen as K=16, at which point the spatiotemporal sequence feature vector has a dimension of 4*16=64.

[0067] If, in step c, any one of the dimensions 1, 2, 3, or 4 has fewer than K values, then obtain the component of the second largest value according to step a, record the second largest value and dimension in step b, and supplement it into step c.

[0068] If step c still has fewer than K values ​​for a certain dimension after the above processing, then the missing values ​​are padded with 0 values.

[0069] The acquisition of spatiotemporal sequence feature vectors is now complete.

[0070] Step 2: Method for constructing a patient head deviation monitoring model based on spatiotemporal sequence feature vectors.

[0071] The spatiotemporal sequence feature vector is a 64-dimensional vector, denoted as φ(k).

[0072] Define the first hidden layer of the model:

[0073]

[0074] Based on the first hidden layer described above, each spatiotemporal sequence feature vector is divided according to the four dimensions of Equation 5, and each is mapped to the output vector β1 of the first hidden layer in a linear fully connected form. ω11 ω 12 ω 13 ω 14 Each of these represents a fully connected mapping coefficient, b0 is the linear bias of the first hidden layer, and ξ is the activation function. Due to the characteristics of neural networks, the activation function ξ is non-linear and is defined as follows.

[0075]

[0076] Where k1 and k2 satisfy

[0077] k2k1=1,k1,k2>0

[0078] Compared to classic nonlinear functions such as the sigmoid function, the above-mentioned function 9 can improve the classification performance of neural networks on spatiotemporal sequence feature vectors. Based on experimental data, k1 = 15 is optimally selected.

[0079] Furthermore, we define the second hidden layer of the model.

[0080]

[0081] Based on the second hidden layer described above, the first hidden layer is further divided into two parts, each mapped to the output vector β2 of the second hidden layer in a linear fully connected form. ω 21 ω 22 Each represents a fully connected mapping coefficient, b1 is the linear bias of the second hidden layer, and ξ is the activation function as described above.

[0082] Furthermore, we define the third hidden layer of the model.

[0083]

[0084] Based on the third hidden layer described above, the second hidden layer is mapped to the output vector β3 of the third hidden layer in a linear fully connected form. ω3 represents the fully connected mapping coefficients, b2 is the linear bias of the third hidden layer, and ξ is the activation function as described above.

[0085] Note that the first to third hidden layers establish a mapping connection from local dimension to all dimension of the spatiotemporal sequence feature vector in a folded manner. Compared with the classic fully connected network, the method of the present invention has fewer connection parameters and has the same recognition performance.

[0086] Furthermore, define the output layer of the model:

[0087]

[0088] ω4 represents the fully connected mapping coefficients, b3 is the linear bias of the output layer, ξ is the activation function as above, and e is the natural exponential function.

[0089] Image data from the post-endotracheal intubation nursing stage of the patient were collected. Spatiotemporal sequence features were obtained according to the method in step 1. Simultaneously, the image data were manually labeled. The label value was the output of the aforementioned neural network model. A positive value of γ (1) indicates that the patient's head tilt has occurred, and a negative value of γ (-1) indicates that tilt has not occurred. The cost function Φ used during training was:

[0090]

[0091] in, γ represents the manually labeled value of the above sample, and γ is the network output value.

[0092] A camera is installed above the hospital bed, along with an edge terminal system implemented using the model and calculation method described in this invention. After the camera captures images, they are transmitted to the edge terminal for calculation. When the output γ value is positive, a head tilt warning is issued. The training process is conducted offline; after training is completed on the server side, the model parameters obtained during training are written to the edge terminal.

[0093] Experimental results show that the method of the present invention has accurate identification, low false alarm rate, and real-time computing response capability on edge terminals.

[0094]

[0095]

Claims

1. An open airway assist device operating at the edge, characterized in that: Includes image sensors, mounting devices, processors, alarms, and transmission devices; Image sensors acquire images of the patient's head and chest. Where u and v represent the spatial dimensions of the image, and n represents the number of frames; The processor processes the images acquired by the image sensor as follows: (1) Let the two adjacent frames be respectively and N is a positive integer, and N=0 represents the first frame in the temporal sequence. Spatially, the positions of u and v, offset from them by i and j in two directions, are denoted as u and v, respectively. i ,v j Then, calculate the local change features of adjacent frames: in, and (2) Calculate the spatiotemporal feature map in the finite spatiotemporal domain based on the local change characteristics. The spatiotemporal feature map consists of temporal feature vectors mapped to each two-dimensional coordinate point u,v in space. The method for calculating the temporal feature vectors is as follows: N1, N2, N3, and N4 are adjacent frames, and Therefore, the spatiotemporal feature map is represented as a graph composed of feature vectors f|u,v (3) Convert the spatiotemporal feature map into a 64-dimensional spatiotemporal sequence feature vector; and input it into a neural network. When the output of the neural network is positive, it indicates that the patient’s head is tilted. When the output is negative, it indicates that no tilting has occurred.

2. The open airway assist device as described in claim 1, characterized in that: Image sensors are used to acquire images of the patient's head and chest.

3. The open airway assist device as described in claim 1, characterized in that: The fixation device is used to secure the image sensor above the patient.

4. The open airway assist device as described in claim 1, characterized in that: The processor is mounted on a fixed device.

5. The open airway assist device as described in claim 1, characterized in that: The alarm is used to receive alarm signals sent by the processor and then sound an alarm.

6. The open airway assist device as described in claim 1, characterized in that: The transmission device is used to transmit alarm signals to a remote location.

7. The open airway assist device as described in claim 2, characterized in that: The sensor is a CCD camera.

8. The open airway assist device as described in claim 5, characterized in that: The alarms are sound, light, and display alarms.

9. The open airway assist device as described in claim 1, characterized in that: The neural network training process uses offline training.

10. The open airway assist device as described in claim 1, characterized in that: The cost function for training a neural network is: in, γ represents the manually labeled value of the above sample, and γ is the network output value.

Images