Multi-mode difficult airway evaluation method and system, electronic equipment and medium

Abstract

The invention discloses a multi-mode difficult airway evaluation method and system, electronic equipment and a medium, and the method comprises the steps: obtaining historical sampling data of a plurality of patients, including basic information, head and neck activity videos, neck ultrasonic images and glottis exposure pictures; acquiring facial image features of the patient based on the head and neck activity video; acquiring ultrasonic features of the patient based on the neck ultrasonic image; acquiring an airway label of the patient based on the glottis exposure picture; taking basic information, facial image features, ultrasonic features and airway labels of a patient as a sample, and constructing a sample set; constructing a difficult airway classifier based on a machine learning algorithm, and training the difficult airway classifier by adopting the data in the sample set to obtain a trained difficult airway classifier; and evaluating the airway sampling data of the current patient by adopting the trained difficult airway classifier to obtain an airway evaluation result. The difficult airway prediction accuracy of the patient can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of difficult airway assessment, and in particular to a multimodal difficult airway assessment method, system, electronic equipment and medium. Background Art

[0002] In clinical work, patient airway assessment and management play an important role in anesthesiology, critical care medicine, and emergency medicine. They are also the last line of defense to ensure the life safety of perioperative patients and the rescue of critically ill patients. Poor airway assessment and management can lead to serious complications such as poor ventilation, reflux aspiration, airway injury, hypoxic-ischemic encephalopathy, and even death. Difficult airway is the focus and difficulty of airway assessment and management, but there is currently no reliable and accurate predictive indicator in clinical practice.

[0003] Traditional assessments mainly include: mouth opening, thyromental distance, head and neck range of motion, temporomandibular joint range of motion, etc. These appearance indicators are simple and easy to obtain. Although they indicate difficult airway to a certain extent, the appearance assessment indicators cannot accurately reflect the abnormal anatomical structure inside the airway, so their accuracy is not high. The accuracy of prediction of difficult airway can be significantly improved with the help of imaging indicators such as X-ray, CT, and MRI, but such examinations are not necessary in routine clinical practice, and patients are often required to go to specific examination sites, which not only increases the patient's movement, prolongs hospital stay, increases medical expenses, but also increases the patient's radiation exposure. Ultrasound, as a mobile, portable and non-destructive examination instrument, is widely used in clinical practice. Airway ultrasound has been a research direction that has received much attention in the field of anesthesia in recent years. At present, ultrasound has been partially used in the clinical diagnosis of difficult airway. It can not only provide real-time images, but also provide dynamic airway structural changes, clearly visualize the tongue, epiglottis, vocal cords, hyoid bone, thyrohyoid membrane, thyroid cartilage, cricoid cartilage, cricothyroid membrane, tracheal cartilage rings and other airway anatomical tissues, and measure the distance between each tissue to predict difficult airways, but the accuracy still cannot meet the needs. It is found that it is difficult to obtain accurate evaluation results based on the existing single factor, single angle, and single dimension evaluation.

[0004] As an emerging discipline, artificial intelligence is a technology that integrates the research and development of theories, methods, and applications for simulating, extending, and enhancing human intelligence. It can autonomously learn structured and unstructured data combinations to generate corresponding decision support. By combining basic patient information, head and neck facial information, and head and neck ultrasound images with artificial intelligence, a multimodal difficult airway assessment model is established, which will be of great help in clinical preoperative airway assessment. Summary of the invention

[0005] The present invention provides a multimodal difficult airway assessment method, system, electronic equipment and medium, which can significantly improve the accuracy of patient difficult airway prediction.

[0006] According to one aspect of the present invention, a multimodal difficult airway assessment method is provided, comprising:

[0007] Acquire historical sampling data of several patients, wherein the historical sampling data includes basic information, head and neck activity videos, neck ultrasound images, and glottis exposure pictures;

[0008] Acquire the facial image features of the patient based on the head and neck activity video; acquire the ultrasound features of the patient based on the neck ultrasound image; and acquire the airway label of the patient based on the glottis exposure image;

[0009] Taking the basic information, facial image features, ultrasound features and airway labels of a patient as a sample, constructing a sample set;

[0010] Constructing a difficult airway classifier based on a machine learning algorithm, and using the data in the sample set to train the difficult airway classifier to obtain the trained difficult airway classifier;

[0011] The trained difficult airway classifier is used to evaluate the airway sampling data of the current patient to obtain an airway evaluation result.

[0012] Optionally, acquiring facial image features of the patient based on the head and neck activity video includes:

[0013] The XFeat model is used to construct a facial image feature extraction network, and the head and neck activity video is input into the facial image feature extraction network to extract the facial image features.

[0014] Optionally, the adopting of the XFeat model to construct a facial image feature extraction network, inputting the head and neck activity video into the facial image feature extraction network to extract the facial image features, comprises:

[0015] Converting the head and neck activity video into a grayscale image;

[0016] Separating a face region image from the grayscale image using an XFeat model;

[0017] Facial image features are acquired based on the face area image; the facial image features at least include mandibular protrusion, maximum head backward position, maximum head forward position, mouth opening, and neck circumference.

[0018] Optionally, before constructing a difficult airway classifier based on a machine learning algorithm and training the difficult airway classifier using the data in the sample set to obtain the trained difficult airway classifier, the method further comprises:

[0019] The data in the sample set is preprocessed; the preprocessing includes at least data standardization, normalization and data cleaning.

[0020] Optionally, the preprocessing of the data in the sample set includes:

[0021] Performing data standardization and normalization processing on the multimodal data in the sample set;

[0022] Based on multiple key indicators of the patient's head and neck imaging data, data cleaning is performed on the data in the sample set after standardization and normalization;

[0023] Multiple interpolation method is used to fill the missing values ​​in the sample set after the data cleaning to obtain the filled sample set.

[0024] Optionally, constructing a difficult airway classifier based on a machine learning algorithm, and using the data in the sample set to train the difficult airway classifier to obtain the trained difficult airway classifier includes:

[0025] Constructing the difficult airway classifier of the Diffusion-MAE model;

[0026] The K-fold cross-validation method is used to divide the sample set into two categories: a training set and a test set. The difficult airway classifier is trained by randomly dividing the data set multiple times to comprehensively evaluate the performance of the difficult airway classifier; and a trained difficult airway classifier is obtained.

[0027] Optionally, the ultrasonic characteristics include at least the distance from skin to epiglottis, the distance from skin to hyoid bone, the width and thickness of tongue, the distance from mandible to hyoid bone and physiological indicators of tongue area.

[0028] According to another aspect of the present invention, there is provided a multimodal difficult airway assessment system, comprising:

[0029] A module is used to obtain historical sampling data of several patients, wherein the historical sampling data includes basic information, head and neck activity videos, neck ultrasound images, and glottis exposure pictures;

[0030] A feature extraction module, used to obtain facial image features of the patient based on the head and neck activity video; to obtain ultrasound features of the patient based on the neck ultrasound image; and to obtain airway labels of the patient based on the glottis exposure image;

[0031] A sample construction module, used to construct a sample set by taking the basic information, facial image features, ultrasound features and airway labels of a patient as a sample;

[0032] A training module, used to construct a difficult airway classifier based on a machine learning algorithm, and train the difficult airway classifier using the data in the sample set to obtain the trained difficult airway classifier;

[0033] The evaluation module is used to evaluate the airway sampling data of the current patient using the trained difficult airway classifier to obtain an airway evaluation result.

[0034] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0035] at least one processor; and

[0036] a memory communicatively connected to the at least one processor; wherein,

[0037] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the multimodal difficult airway assessment method described in any embodiment of the present invention.

[0038] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the multimodal difficult airway assessment method described in any embodiment of the present invention when executed.

[0039] The technical solution of the embodiment of the present invention obtains the patient's historical sampling data, including basic information, head and neck activity videos, neck ultrasound images, and glottis exposure pictures, based on facial image features obtained from the head and neck activity videos, and ultrasound features obtained from the neck ultrasound images, and then uses the multimodal data combining text images, ultrasound images, etc. as a sample set to train a difficult airway classifier based on machine learning, thereby improving the recognition accuracy of the difficult airway classifier. That is, the present application only requires three types of data that are easy to collect: basic patient information, head and neck activity videos, and neck ultrasound images, to accurately identify whether the patient has a difficult airway, thereby improving the recognition accuracy and reducing the patient's expenses.

[0040] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 is a flow chart of a multimodal difficult airway assessment method provided according to Embodiment 1 of the present invention;

[0043] Figure 2 This is a schematic diagram of the Cormack-Lehane difficult airway classification under video laryngoscopy;

[0044] Figure 3 A schematic diagram of the principle of an Xfeat model provided in an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of constructing a three-dimensional human head and neck model and key indicator detection for head and neck activity videos in a multimodal difficult airway assessment method provided in an embodiment of the present invention;

[0046] Figure 5 The neck ultrasound collection site and ultrasound schematic diagram provided in an embodiment of the present invention;

[0047] Figure 6 It is a schematic diagram of classification analysis of multiple data indicators based on the Diffusion-MAE model and optimization of the Diffusion-MAE algorithm integrating grid search (GS) and coordinate descent method in an embodiment of the present invention;

[0048] Figure 7 This is a visualization platform display diagram in an embodiment of the present invention;

[0049] Figure 8 is an architecture diagram of a multi-modal difficult airway assessment system provided according to Embodiment 2 of the present invention;

[0050] Fig. 9 It is a structural schematic diagram of an electronic device for implementing a multimodal difficult airway assessment method provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0052] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0053] Embodiment 1

[0054] Figure 1 A flowchart of a multi-modal difficult airway assessment method is provided for Embodiment 1 of the present invention. Figure 1 As shown, the method includes:

[0055] S101. Acquire historical sampling data of several patients, wherein the historical sampling data includes basic information, head and neck activity videos, neck ultrasound images, and glottis exposure pictures.

[0056] In this embodiment, sampling data of multiple patients in the past can be obtained to train the difficult airway classifier. For example, historical sampling data of thousands of patients (including difficult airway and non-difficult airway) can be obtained. The historical sampling data can include basic information, head and neck movement videos, neck ultrasound images and glottis exposure pictures, among which basic information, head and neck movement videos, and neck ultrasound images are relatively easy to obtain. The head and neck movement video can be captured by a camera; the neck ultrasound image can be captured using a mobile portable non-destructive ultrasound examination instrument. The glottis exposure picture is collected under a visual laryngoscope during endotracheal intubation, which can be used as the patient's airway data label to verify the patient's airway classification. The Cormack-Lehane (CL) difficult airway grade under visual laryngoscope (see Figure 2 ) as the standard to judge whether the patient's glottis exposure picture represents a difficult airway. Figure 2 CL 1-2 is not considered a difficult airway, while CL 3-4 is considered a difficult airway.

[0057] In one embodiment, the basic information includes gender, age, height, weight, neck surgery history, neck radiotherapy history, and snoring. As shown in Table 1, the gender, age, height, weight, neck surgery history, neck radiotherapy history, etc. of the patient are collected as text information in the model.

[0058]

[0059]

[0060] S102, acquiring facial image features of the patient based on the head and neck activity video; acquiring ultrasound features of the patient based on the neck ultrasound image; and acquiring airway labels of the patient based on the glottis exposure image.

[0061] In this embodiment, facial image features can be extracted from the patient's head and neck activity video through a facial feature extraction algorithm; the ultrasonic features of the patient's head and neck can be directly calculated through the neck ultrasound image; and the glottis exposure image can be compared with the difficult airway grading standard to determine whether the patient's airway is a difficult airway or a non-difficult airway.

[0062] S103: construct a sample set by taking the basic information, facial image features, ultrasound features, and airway labels of a patient as a sample.

[0063] S104, constructing a difficult airway classifier based on a machine learning algorithm, and using the data in the sample set to train the difficult airway classifier to obtain the trained difficult airway classifier.

[0064] This embodiment can use a machine learning module as a difficult airway classifier, and the machine learning model used is not limited. The difficult airway classifier is trained based on a multimodal sample set, and the sample label is used as a learning target to improve the accuracy of the difficult airway classifier, and finally a trained difficult airway classifier is obtained.

[0065] S105 , using the trained difficult airway classifier to evaluate the airway sampling data of the current patient to obtain an airway evaluation result.

[0066] In this embodiment, the airway sampling data of the current patient may include one or more of the patient's basic information, head and neck activity video, and neck ultrasound image. The airway sampling data of the current patient may be input into the difficult airway classifier to obtain the airway assessment result to assess whether the patient has a difficult airway.

[0067] The present invention obtains the patient's historical sampling data, including basic information, head and neck activity videos, neck ultrasound images, and glottis exposure pictures, based on facial image features obtained from the head and neck activity videos, and ultrasound features obtained from the neck ultrasound images, and then uses multimodal data combining text images, ultrasound images, etc. as a sample set for training a difficult airway classifier based on machine learning, thereby improving the recognition accuracy of the difficult airway classifier. That is, the present application only requires three types of data that are easy to collect: basic patient information, head and neck activity videos, and neck ultrasound images, to accurately identify whether the patient has a difficult airway, thereby improving the recognition accuracy and reducing the patient's expenses.

[0068] In one embodiment, step S102 acquires facial image features of the patient based on the head and neck activity video, including: constructing a facial image feature extraction network using an XFeat model, and inputting the head and neck activity video into the facial image feature extraction network to extract the facial image features.

[0069] Specifically, Figure 3-Figure 4 As shown in the figure, the head and neck activity video can be converted into image frames, and then the image frames can be converted into grayscale images. The local image of the face area can be separated from the grayscale image through the XFeat model to realize face recognition and face key point detection. At the same time, the ROI area is set for the detected face to mark the contours of various parts of the face (left and right eyes, left and right eyebrows, nose, and mouth).

[0070] Specifically, the RGB image of the head and neck video is first input into the computer and the grayscale transformation is performed on the RGB image to obtain a grayscale image of the same scale. The image is processed to meet the input requirements and maintain sufficient resolution. The key points of the image are quickly identified through the model key point detection branch, and the features of the area around the detected key points are extracted to generate local descriptors, which are used for feature matching. Based on the prior knowledge of the head and neck, the model parameters are estimated by the inverse rendering method based on traditional optimization. Then, MRFP (multi-resolution feature perturbation) is used to extract the head semantic segmentation information in all multi-view portrait images at multiple perspectives to avoid unreasonable facial organ shapes and unclear head geometric structures during reconstruction. MRFP is a technology that improves multi-view image processing and semantic segmentation by perturbing features at different resolution levels. MRFP receives multi-view images from the input and extracts feature maps at different resolution levels. Then, at each resolution level, the feature maps are perturbed by adding noise, random occlusion, etc. By introducing some small changes, the model can better adapt to changes in different perspectives and lighting conditions. Finally, the perturbed feature maps at different resolution levels are fused to generate the final feature representation. This enables the model to better cope with input images under different spans. After extracting semantic information, under the guidance of prior knowledge, by introducing a priori loss function term into the implicit neural rendering network, a dynamic implicit volume rendering network guided by prior information is constructed, and the rough three-dimensional FLAME (deep learning face modeling framework) model and camera parameters are restored frame by frame from the input video sequence, and the Neural Implicit Representations neural implicit representation algorithm is used to construct a movable mandibular joint three-dimensional head and neck model.

[0071] In a specific embodiment, the method for extracting the mandibular protrusion is as follows: first, the video RGB image is input into a computer, and the RGB image is gray-scale transformed to obtain a gray-scale image of the same scale, and a partial image of the face area image is separated from the gray-scale image according to the XFeat model. Subsequently, the partial image is input into the Openpose face posture model, and the facial landmarks are identified based on the Openpose face posture model, and the facial landmarks are located to determine the two-dimensional coordinates of the facial landmarks, wherein the facial landmarks include the outer contour of each eye, the inner contour of each eyebrow, the nasolabial contour, and the chin vertex. According to the range of change of the facial landmarks during the movement of the mandibular edge, the two-dimensional angle of the left and right corners of the mouth and the MAR value (a quantitative measure of the degree of mouth opening, which is measured by calculating the Euclidean distance between specific points around the mouth to measure the degree of mouth opening) are determined, and the mandibular protrusion is calculated. The XFeat model is a feature extraction method designed for local feature detection in images. Its main function is to complete face detection and face key point detection by quickly extracting key points and feature descriptors of the image. Figure 3 The schematic diagram of the Xfeat model shown in the figure shows that the whole process is divided into three stages: In the first stage, XFeat uses its shallow convolutional network to extract features from images of different scales and generate candidate regions. In this stage, the model extracts basic features such as edges and textures in the image to determine potential face areas. In the second stage, the XFeat model uses its deep convolutional network to further process these candidate regions and select candidate frames with higher accuracy. Through the feature matching algorithm, this stage can significantly reduce the number of candidate regions and improve the accuracy of detection. In the third stage, the model uses a refined feature extraction module to ultimately output high-precision facial frames and facial key points.

[0072] In a specific embodiment, the method for extracting the maximum backward tilt position / maximum forward tilt position of the head is as follows: after completing the face detection task, extract the face feature information, and use the extracted face features in key point positioning. The face rectangular frame obtained in the above face detection process is used to extract the position, shape and other information of the feature parts such as the left and right eyes, left and right eyebrows, nose, mouth and facial contour in the image using a feature extraction template to obtain a digital information set of the feature points of the face in the image, which contains the coordinate information of each key point, and the coordinate values ​​of the eyes, eyebrows, nose, mouth and facial contour are extracted from the set. In the video frame, the maximum backward tilt (or forward tilt) position problem of the head is treated as a multi-point projection problem (perspective projection refers to the figure obtained by projecting an object from a certain projection center onto a single projection surface, and the transformation of the 2D face key point coordinates to the 3D coordinates is equivalent to using a camera to obtain the coordinates after the 3D coordinates are transformed, and then the transformed coordinates are projected into the 2D image coordinate system). In actual operation, the key coordinates of the face are obtained through the head posture estimation algorithm to calculate the rotation matrix and translation matrix, and the rotation matrix is ​​used to obtain the head posture angle.

[0073] In a specific embodiment, the method for extracting the mouth opening is: firstly, the lips are roughly detected by using the face geometric features and the Gentle-Adaboost classifier, and then the lips are segmented by using the chromaticity difference between the lip color and the skin color. The lips are located in the lower third of the face. By setting the ROI area for the detected face, the lips are detected in the area by using the lip classifier. Since the brightness of the lips in the grayscale image is lower than that of the surrounding skin, the lips are segmented by using the chromaticity difference feature and setting the corresponding threshold, so that the lips can be accurately located from the roughly obtained area, and then the grayscale image is obtained by the chromaticity difference and processed by the median filter, histogram equalization and other methods, and the Otsu (maximum inter-class variance method) algorithm is used to perform threshold segmentation on the detected lip area, and it is eroded and expanded at the same time. The complete lips can be segmented by using the chromaticity difference between the skin color and the lip color, and the height and width of the lips are calculated by the outer contour.

[0074] In a specific embodiment, the neck circumference extraction method is as follows: after completing the face detection task, the input head and neck moving image is converted into a grayscale image, the grayscale image is converted into a binary image through a certain threshold operation, and then the obtained binary image is inverted and the background noise is removed by corrosion and expansion methods. Before the corrosion and expansion are used to remove the background noise, all contours are first whitened to prevent the small black holes near the edge of the head from being removed as background noise and affecting the smoothness of the edge. The contour with the largest area in the image is retained as the human body part, and its rectangular spatial position is calibrated and recorded. Finally, the black area is cleared by the traversal threshold method, and the largest contour of the head is searched. The obtained image is measured according to the image recognition method from top to bottom (taking the neck as an example: the neck is a line segment narrower than the head, so as to obtain the length in the corresponding image), and then the patient's neck circumference is measured according to the geometric relationship between the focal length of the camera and the distance parameters of the shooting.

[0075] In a specific embodiment, the method for constructing a three-dimensional head and neck model is as follows: first, multi-view images are extracted from different frames in the head and neck activity video, including the head and neck prior knowledge, the head semantic segmentation information, etc. For the head and neck prior knowledge, the given N multi-view images are used as input, and the model parameters are estimated by the inverse rendering method based on the traditional optimization according to the photometric consistency to obtain accurate geometric information. For the head semantic segmentation information, the main purpose is to avoid the unreasonable facial organ shape and unclear head geometric structure that appear during reconstruction. The face parsing network in MRFP is used to extract the head semantic segmentation information in all multi-view portrait images, and the unreasonable facial organ shape and blurred head geometric structure that appear during reconstruction are avoided by multi-resolution feature perturbation, and the fine-grained features and coarse-grained style features are perturbed at different scales, so as to achieve robust semantic segmentation in multi-view data. Through semantic-based mapping, the pre-trained model is used to achieve the goal in the face operation to extract the head semantic segmentation information in all multi-view portrait images, including hair, face, eyes, eyebrows, nose and lips, etc. Guided by prior knowledge, a loss function based on prior knowledge is introduced into the implicit neural rendering network to improve the reconstruction accuracy and robustness of the 3D head model. By constructing a dynamic implicit volume rendering network guided by prior information, a rough 3D FLAME model and camera parameters are restored frame by frame from the input video sequence using an optimization-based method. This prior information can provide a reasonable 3D structure and initial depth range for the implicit space; then the 2D semantic segmentation information obtained by the facial parsing model is used to maintain the accuracy of reconstruction. By using these auxiliary information, the head geometry in the canonical space is estimated as the zero level set of the signed distance field represented by the implicit volume rendering network. Finally, the Neural Implicit Representations algorithm is used to extract the 3D head and neck model with movable mandibular joint frame by frame. The signed distance field is represented as an implicit function, which is learned and approximated by a neural network as a continuous function. The function outputs the distance from each spatial point to the surface, and when the distance is zero, it is defined as the surface of the object. Then the implicit neural network learns the input image data, optimizes the parameters of the network frame by frame, and generates the corresponding 3D shape for each frame. By learning the geometric information hidden in multi-view images, the implicit representation of the model is gradually updated so that it can accurately fit the head surface structure corresponding to each frame of the image. For the active mandibular joint model, the algorithm generates a dynamic geometric model consistent with the actual anatomical structure by analyzing the prior information of the head and mandibular movement. The 3D model surface is extracted using the zero level set through the implicit function optimized frame by frame. Finally, the algorithm integrates all implicit representations frame by frame through a time series to generate a continuous dynamic three-dimensional head and neck model. For details, please refer to Figure 3 The diagram shown is a schematic diagram of building a three-dimensional human head and neck model and detecting key indicators based on head and neck activity videos.

[0076] In one embodiment, the method further includes: preprocessing the data in the sample set; the preprocessing includes at least data standardization, normalization and data cleaning.

[0077] Performing data standardization and normalization processing on the multimodal data in the sample set;

[0078] Based on multiple key indicators of the patient's head and neck imaging data, data cleaning is performed on the data in the sample set after standardization and normalization;

[0079] Multiple interpolation method is used to fill the missing values ​​in the sample set after the data cleaning to obtain the filled sample set.

[0080] Specifically, each type of multimodal data acquired needs to be standardized and normalized. For example, for text data of basic information, the text data can be standardized and normalized, and other video data and ultrasound data can also be standardized and normalized accordingly.

[0081] In addition, for the data in the dataset, data that obviously do not meet the index requirements can be cleaned based on multiple key indicators from the patient's head and neck imaging data, such as the distance from the skin to the epiglottis, the distance from the skin to the hyoid bone, the width and thickness of the tongue, the distance from the mandible to the hyoid bone, and the area of ​​the tongue. For example, the outliers in the dataset can be identified by the quasi-difference method. By calculating the mean and standard deviation of the data, data points that exceed a certain threshold of the mean deviation are regarded as outliers and replaced with missing values ​​to reduce their impact on subsequent analysis. Secondly, multiple interpolation methods are used to fill missing values. Multiple complete datasets are generated by estimating the possible values ​​of missing values ​​multiple times, and then the results are merged to improve the accuracy of filling by using the correlation between features. In addition, to solve the problem of sample imbalance, stratified sampling is used to stratify the original dataset according to a certain feature, and then random sampling is performed in each stratum to ensure that the sampling distribution is similar to the original data, thereby improving the accuracy and reliability of the model.

[0082] In one embodiment, the ultrasound characteristics include at least the distance from skin to epiglottis, the distance from skin to hyoid bone, the width and thickness of tongue, the distance from mandible to hyoid bone, and the physiological index of tongue area. Figure 5 As shown, the physiological indicators of the skin-epiglottis distance, skin-hyoid bone distance, tongue width and thickness, mandibular-hyoid bone distance, and tongue area can be directly calculated from the neck ultrasound image.

[0083] In one embodiment, step S104 also includes: constructing the difficult airway classifier of the Diffusion-MAE model; using the K-fold cross-validation method to divide the sample set into two categories: a training set and a test set, and training the difficult airway classifier by randomly dividing the data set multiple times to comprehensively evaluate the performance of the difficult airway classifier; and obtaining a trained difficult airway classifier.

[0084] In this embodiment, the K-fold cross-validation method can be used to divide the sample set into two categories: training set and test set. By randomly dividing the data set multiple times, the model performance is comprehensively evaluated to avoid deviations caused by improper data division, so as to ensure the accuracy and consistency of the data set. In addition, the middle layer in the deep learning architecture fuses multi-modal data and trains the model in an iterative manner so that the models of different modalities enhance each other to form fused data.

[0085] Specific as Figure 6 As shown, Figure 6In this paper, a Diffusion-MAE (Diffusion-Mask Autoencoder) model is constructed as a difficult airway classifier to identify whether there is a difficult airway and analyze the relationship and classification of each indicator feature. The masked autoencoder is used to learn to capture the complex interactions between different features from the image. Through the step-by-step denoising process of the diffusion model, the separability of the features can be enhanced, facilitating more refined classification. In the feature encoding stage, the input medical image data is divided into multiple feature subsets using a mask mechanism based on self-supervised learning, and high-level features are extracted through mask prediction and reconstruction processes. Diffusion-MAE encodes the image features and divides the features into different levels of representation through noise level adjustment, helping the model to better understand the relationship between the features and the target variable (such as airway status). Subsequently, in the feature interaction, the Diffusion-MAE model captures complex interactive features through a multi-layer autoencoder structure and a mask mechanism. Combined with the cross-feature capture strategy, the model uses the features of different stages in the diffusion process to explore the nonlinear relationship between the features and the target variable, improving the generalization ability and prediction performance of the model. Finally, the model uses the gradient estimation method based on the diffusion process to generate gradient estimates for each sample through the training process under different noise levels, thereby reducing model overfitting and generating a robust final model. After that, the grid search (GS) traverses the preset parameter combinations to find the best model parameters. In the grid search, according to the Diffusion-MAE characteristics, the parameters that need to be tuned, such as learning rate, loss function, category weight, etc., are selected. The candidate values ​​of these parameters are formed into a parameter grid, and each grid point represents a hyperparameter combination, which is arranged in the form of Cartesian product. Subsequently, the coordinate descent method is used for rapid tuning. Its idea is similar to that of grid search, but the coordinate descent method can converge to the best parameter combination faster. In the coordinate descent method, the cross-validation technique is used to divide the data set into K subsets. K-1 subsets are used for training each time, and the remaining subset is used for verification. This process is repeated K times, and then the parameter value is adjusted according to the average value of the K verification results to optimize each hyperparameter. Evaluation indicators (such as accuracy, F1 score, mean square error, etc.) are obtained through cross-validation as evaluation criteria for model performance. The parameter combination with the best performance was selected through cross-validation results as the hyperparameters of the final model to improve the performance and generalization ability of the model and ensure the accuracy of predicting patients' difficult airway in clinical practice.

[0086] In this process, the following formula is used for calculation:

[0087] D k ={x j :δ(j)<δ(k)}

[0088] Among them, δ(j) is used to measure the sample x jPriority or gradient size in coordinate descent, δ(j) is a metric function, D k It indicates that in the tuning process, the updated sample set that meets δ(j)<δ(k) is preferentially selected.

[0089]

[0090] q represents the probability, x t represents the image state at time t, I represents the diagonal covariance matrix, β t is the diffusion rate, α t are the denoising rates, which determine the degree of preservation of the current state relative to the previous state in different state transition processes, and N represents the normal distribution.

[0091] The present invention uses the Cormack-Lehane (CL) difficult airway classification under visual laryngoscope as the standard to judge whether it is a difficult airway. The final evaluation result will be displayed on the visualization platform, such as Figure 7 . Build a visualization platform based on user-friendly principles, and implement functions such as charts, dashboards, and image displays on the platform to intuitively display patient data and warning information. Implement warning prompts based on patient data and warning levels. Generate trend charts and statistical data through historical data analysis to display the changing trend of patients' health status and help doctors make more accurate diagnosis and treatment decisions. After the platform is developed, conduct system testing and collect user feedback to optimize and improve the platform.

[0092] Embodiment 2

[0093] Figure 8 This is a schematic diagram of the structure of a multi-modal difficult airway assessment system provided in Embodiment 3 of the present invention. Figure 8 As shown, the system includes:

[0094] Module 801 is used to obtain historical sampling data of several patients, wherein the historical sampling data includes basic information, head and neck activity videos, neck ultrasound images, and glottis exposure pictures;

[0095] The feature extraction module 802 is used to obtain the facial image features of the patient based on the head and neck activity video; obtain the ultrasound features of the patient based on the neck ultrasound image; and obtain the airway label of the patient based on the glottis exposure image;

[0096] A sample construction module 803 is used to construct a sample set by taking the basic information, facial image features, ultrasound features and airway labels of a patient as a sample;

[0097] A training module 804 is used to construct a difficult airway classifier based on a machine learning algorithm, and train the difficult airway classifier using the data in the sample set to obtain the trained difficult airway classifier;

[0098] The evaluation module 805 is used to evaluate the airway sampling data of the current patient using the trained difficult airway classifier to obtain an airway evaluation result.

[0099] In one embodiment, the feature extraction module 802 is further used to construct a facial image feature extraction network using an XFeat model, and input the head and neck activity video into the facial image feature extraction network to extract the facial image features.

[0100] In one embodiment, the feature extraction module 802 is further used to convert the head and neck activity video into a grayscale image;

[0101] Separating a face region image from the grayscale image using an XFeat model;

[0102] Facial image features are acquired based on the face area image; the facial image features at least include mandibular protrusion, maximum head backward position, maximum head forward position, mouth opening, and neck circumference.

[0103] In one embodiment, a preprocessing module is further included, which is used to preprocess the data in the sample set; the preprocessing at least includes data standardization, normalization and data cleaning.

[0104] In one embodiment, the preprocessing module is further used to perform data standardization and normalization processing on the multimodal data in the sample set;

[0105] Based on multiple key indicators of the patient's head and neck imaging data, data cleaning is performed on the data in the sample set after standardization and normalization;

[0106] Multiple interpolation method is used to fill the missing values ​​in the sample set after the data cleaning to obtain the filled sample set.

[0107] In one embodiment, the training module 804 is further used to construct the difficult airway classifier of the Diffusion-MAE model;

[0108] The K-fold cross-validation method is used to divide the sample set into two categories: a training set and a test set. The difficult airway classifier is trained by randomly dividing the data set multiple times to comprehensively evaluate the performance of the difficult airway classifier; and a trained difficult airway classifier is obtained.

[0109] The message data processing resource scheduling device provided in the embodiment of the present invention can execute the message data processing resource scheduling device method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0110] Embodiment 3

[0111] Fig. 9 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0112] like Fig. 9 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0113] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0114] The processor 11 may be a variety of general and / or dedicated processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a multimodal difficult airway assessment method.

[0115] In some embodiments, a multimodal difficult airway assessment method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the multimodal difficult airway assessment method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform a multimodal difficult airway assessment method in any other appropriate manner (e.g., by means of firmware).

[0116] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0117] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0118] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0119] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0120] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0121] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0122] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0123] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal difficult airway assessment method, characterized in that: include: Acquire historical sampling data of several patients, wherein the historical sampling data includes basic information, head and neck activity videos, neck ultrasound images, and glottis exposure pictures; Acquiring facial image features of the patient based on the head and neck activity video; Acquiring ultrasound characteristics of the patient based on the neck ultrasound image; Obtain the patient's airway label based on the glottis exposure image; Taking the basic information, facial image features, ultrasound features and airway labels of a patient as a sample, constructing a sample set; Constructing a difficult airway classifier based on a machine learning algorithm, and using the data in the sample set to train the difficult airway classifier to obtain the trained difficult airway classifier; The trained difficult airway classifier is used to evaluate the airway sampling data of the current patient to obtain an airway evaluation result.

2. The multimodal difficult airway assessment method according to claim 1, characterized in that: The step of acquiring facial image features of the patient based on the head and neck activity video comprises: The XFeat model is used to construct a facial image feature extraction network, and the head and neck activity video is input into the facial image feature extraction network to extract the facial image features.

3. The multimodal difficult airway assessment method according to claim 2, characterized in that: The method of using the XFeat model to construct a facial image feature extraction network and inputting the head and neck activity video into the facial image feature extraction network to extract the facial image features includes: Converting the head and neck activity video into a grayscale image; Separating a face region image from the grayscale image using an XFeat model; Facial image features are acquired based on the face area image; the facial image features at least include mandibular protrusion, maximum head backward position, maximum head forward position, mouth opening, and neck circumference.

4. The multimodal difficult airway assessment method according to claim 1, characterized in that: Before constructing a difficult airway classifier based on a machine learning algorithm and training the difficult airway classifier using the data in the sample set to obtain the trained difficult airway classifier, the method includes: The data in the sample set is preprocessed; the preprocessing includes at least data standardization, normalization and data cleaning.

5. The multimodal difficult airway assessment method according to claim 4, characterized in that: The preprocessing of the data in the sample set includes: Performing data standardization and normalization processing on the multimodal data in the sample set; Based on multiple key indicators of the patient's head and neck imaging data, data cleaning is performed on the data in the sample set after standardization and normalization; Multiple interpolation method is used to fill the missing values ​​in the sample set after the data cleaning to obtain the filled sample set.

6. The multimodal difficult airway assessment method according to claim 1, characterized in that: The constructing of a difficult airway classifier based on a machine learning algorithm and training the difficult airway classifier using the data in the sample set to obtain the trained difficult airway classifier includes: Constructing the difficult airway classifier of the Diffusion-MAE model; The K-fold cross-validation method is used to divide the sample set into two categories: a training set and a test set. The difficult airway classifier is trained by randomly dividing the data set multiple times to comprehensively evaluate the performance of the difficult airway classifier; and a trained difficult airway classifier is obtained.

7. The multimodal difficult airway assessment method according to claim 1, characterized in that: The ultrasonic characteristics at least include the distance from skin to epiglottis, the distance from skin to hyoid bone, the width and thickness of tongue, the distance from mandible to hyoid bone and physiological index of tongue area.

8. A multimodal difficult airway assessment system, characterized in that: include: A module is used to obtain historical sampling data of several patients, wherein the historical sampling data includes basic information, head and neck activity videos, neck ultrasound images, and glottis exposure pictures; A feature extraction module, used to obtain facial image features of the patient based on the head and neck activity video; and to obtain ultrasonic features of the patient based on the neck ultrasonic image; Obtain the patient's airway label based on the glottis exposure image; A sample construction module, used to construct a sample set by taking the basic information, facial image features, ultrasound features and airway labels of a patient as a sample; A training module, used to construct a difficult airway classifier based on a machine learning algorithm, and train the difficult airway classifier using the data in the sample set to obtain the trained difficult airway classifier; The evaluation module is used to evaluate the airway sampling data of the current patient using the trained difficult airway classifier to obtain an airway evaluation result.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the multimodal difficult airway assessment method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the multimodal difficult airway assessment method according to any one of claims 1 to 7 when executed.

Images