Automatic tracheal intubation method and system based on artificial intelligence

Abstract

The invention discloses an automatic tracheal intubation method and system based on artificial intelligence, and relates to the field of medical treatment, and the method comprises the steps: constructing a double-branch convolutional neural network according to a mode feature space, respectively processing and fusing an endoscopic image and a pressure distribution cloud picture by using a visual branch and a tactile branch in the double-branch convolutional neural network, and obtaining a fused endoscopic image and a fused pressure distribution cloud picture; obtaining depth prediction values of the tip of the catheter and the glottis; capturing the dynamic deformation of the respiratory tract of the patient according to the depth predicted values of the catheter tip and the glottis, generating a respiratory tract cross section image, and reconstructing the respiratory tract cross section image and the respiratory tract three-dimensional digital model into a three-dimensional dynamic deformation model; according to the three-dimensional dynamic deformation model, utilizing a linear trend averaging method to predict the motion trail of the glottis of the patient, calculating the motion displacement coordinate of the tip of the catheter, and outputting the final positioning of the tip of the catheter. According to the invention, rapid and accurate positioning of the tip of the catheter is achieved, and intubation failure caused by airway movement of a patient and positioning errors caused by airway movement of the patient are reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical treatment, in particular to an automatic tracheal intubation method and system based on artificial intelligence. BACKGROUND

[0002] Tracheal intubation is widely used in anesthesia management and intensive care, with the progress of medical imaging technology, biomechanical modeling and artificial intelligence technology, the automatic intubation method has been greatly improved. Early intubation technology relies on manual operation of doctors, combined with laryngoscope and anatomical experience to locate the glottis, but limited by the experience difference of operators and the complexity of patient anatomy, CT scanning, respiratory monitoring equipment are introduced to construct three-dimensional model.

[0003] There are still some problems to be solved in the existing tracheal intubation method. The conventional intubation method ignores the deformation caused by patient's breathing and cough when constructing the three-dimensional model, only relying on static CT data or air flow model, which leads to the inability to accurately reflect the real-time state of the patient's airway. In predicting the glottis motion trajectory of the patient, Kalman filter or polynomial fitting is often used, but these methods are very complex in calculation, and the adaptability to the patient's cough airflow is insufficient, which cannot meet the real-time requirement. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides an automatic tracheal intubation method based on artificial intelligence to solve the problems of insufficient dynamic deformation capture and poor real-time glottis trajectory prediction in existing automatic intubation methods.

[0006] To solve the above technical problems, the present application provides the following technical solutions: In a first aspect, the present application provides an automatic tracheal intubation method based on artificial intelligence, which comprises collecting preoperative data of a patient and constructing a three-dimensional digital model of the respiratory tract; Collecting endoscopic images and pressure distribution cloud maps and mapping them with the three-dimensional digital model of the respiratory tract in a three-dimensional coordinate system to output a mode feature space; According to the mode feature space, a double-branch convolutional neural network is constructed, and the visual branch and the tactile branch in the double-branch convolutional neural network are used to process and fuse the endoscopic images and the pressure distribution cloud maps respectively to obtain the depth prediction value of the catheter tip and the glottis; According to the depth prediction value of the catheter tip and the glottis, the dynamic deformation of the patient's respiratory tract is captured, the respiratory tract cross-section image is generated and reconstructed into a three-dimensional dynamic deformation model with the three-dimensional digital model of the respiratory tract; According to the three-dimensional dynamic deformation model, the linear trend average method is used to predict the movement trajectory of the glottis of the patient, the movement displacement coordinates of the catheter tip are calculated, and the final positioning of the catheter tip is output.

[0007] As a preferred scheme of the automatic tracheal intubation method based on artificial intelligence, wherein: the preoperative data of the patient includes the glottis position, tracheal diameter, anatomical markers, respiratory rate, tidal volume and cough airflow velocity of the patient.

[0008] As a preferred scheme of the automatic tracheal intubation method based on artificial intelligence, wherein: the three-dimensional digital model of the respiratory tract is constructed, specifically including the following steps, The glottis position, tracheal diameter and anatomical markers of the patient are arranged into a three-dimensional structure data set of the respiratory tract, and the respiratory rate, tidal volume and cough airflow velocity of the patient are arranged into a respiratory parameter data set; ANSYS software is selected as the execution software, the three-dimensional structure data set of the respiratory tract is imported into the ANSYS software, and a geometric grid unit of the patient's respiratory tract is converted using a stereomicroscope method; According to the geometric grid unit of the patient's respiratory tract, geometric boundary conditions are set and Boolean operation is performed on the geometric grid unit of the patient's respiratory tract, and a three-dimensional geometric model of the respiratory tract is output; All parameters of the respiratory parameter data set are set as airflow dynamic boundary conditions, an airflow velocity field is set, biomechanical parameters are obtained from a biomechanics database, and the biomechanical parameters and the airflow velocity field are added to the three-dimensional geometric model of the respiratory tract to obtain a dynamic three-dimensional geometric model of the respiratory tract; The dynamic three-dimensional geometric model of the respiratory tract is used to simulate the dynamic deformation of the patient's respiratory tract under respiration and cough, the displacement between the glottis and the tracheal wall is calculated, and a space-time data set is generated; The space-time data set and the geometric grid unit in the dynamic three-dimensional geometric model of the respiratory tract are integrated to obtain a three-dimensional digital model of the respiratory tract.

[0009] As a preferred scheme of the automatic tracheal intubation method based on artificial intelligence, wherein: the output mode feature space specifically includes the following steps, A flexible pressure sensor and an endoscopic camera are used to collect a pressure distribution cloud map and an endoscopic image, and a three-dimensional coordinate of the catheter tip is determined according to the geometric center of the pressure distribution cloud map; Inverse distance weighting method is used to calculate the pressure value between each pressure point in the pressure distribution cloud map and the geometric grid unit in the three-dimensional digital model of the respiratory tract, and is assigned to the three-dimensional digital model of the respiratory tract to generate a mapped pressure distribution cloud map. An edge gradient threshold is set, the endoscope image is preprocessed, when the edge gradient of the preprocessed endoscope image is greater than the edge gradient threshold, a strong and weak edge point is judged, and a binary edge image is output; A contour area threshold is set, a contour area of the binary edge image greater than the contour area threshold is marked as a feature point, an Euclidean distance between a three-dimensional coordinate of each feature point and a geometric grid unit in the three-dimensional digital model of the respiratory tract is calculated, and a matching result is obtained; The mapped pressure distribution cloud map, the matching result and the three-dimensional coordinate of the catheter tip are integrated to obtain a mode feature space.

[0010] As a preferred scheme of the automatic tracheal intubation method based on artificial intelligence, the depth prediction value of the catheter tip and the glottis is obtained, specifically including the following steps, The pressure distribution mode in the mode feature space, the endoscope image and the position of the catheter tip are respectively interpolated into a pressure distribution tensor, cropped into an endoscope image tensor and normalized into a three-dimensional point cloud coordinate to form a mode data tensor; ResNet and MobileNet are selected as core architectures, and the mode data tensor is used for training to obtain a visual branch convolutional neural network and a tactile branch convolutional neural network; The endoscope image tensor is input into the visual branch convolutional neural network for convolution processing and pooling processing to obtain a visual feature vector; The pressure distribution tensor is input into the tactile branch convolutional neural network to calculate the pressure gradient amplitude of the catheter tip and the respiratory tract wall of the patient, and generate a tactile feature vector; The visual feature vector and the tactile feature vector are dimensionally cropped, the dimensionally cropped visual feature vector and tactile feature vector are assigned weights, and a fusion feature vector is output; According to the fusion feature vector, the depth prediction value of the catheter tip and the glottis of the patient is calculated.

[0011] As a preferred scheme of the automatic tracheal intubation method based on artificial intelligence, the three-dimensional dynamic deformation model is reconstructed, specifically including the following steps, According to the depth prediction value of the catheter tip and the glottis of the patient, the cross-sectional images of the glottis and the upper segment of the trachea of the patient are collected and preprocessed to obtain a preprocessed OCT image sequence; The preprocessed OCT image sequence is rigidly registered to generate a registered OCT image sequence; The pixel displacement amount in the registered OCT image sequence is calculated using the inverse distance weighted interpolation method, a deformation field sequence is generated and mapped to the three-dimensional digital model of the respiratory tract to form a three-dimensional dynamic deformation model.

[0012] As a preferred solution of the artificial intelligence-based automatic tracheal intubation method, the final positioning of the catheter tip comprises the following steps, locating the center of the glottis of the patient in the three-dimensional dynamic deformation model, generating a sequence of glottis center coordinates and calculating the average moving speed of the glottis center of the patient; adjusting the average moving speed of the glottis center of the patient using the respiratory frequency of the patient in the respiratory parameter data set to obtain the adjusted average moving speed of the glottis center of the patient; According to the adjusted average moving speed of the glottis center of the patient, the glottis motion trajectory coordinates of the patient are calculated; According to the glottis motion trajectory coordinates of the patient and the three-dimensional coordinates of the catheter tip, the motion displacement coordinates of the catheter tip are calculated as the final positioning of the catheter tip.

[0013] In a second aspect, the present application provides an artificial intelligence-based automatic tracheal intubation system, comprising a data acquisition module for acquiring preoperative data of a patient and constructing a three-dimensional digital model of the respiratory tract; The mapping module acquires endoscopic images and pressure distribution cloud images and maps them with the three-dimensional digital model of the respiratory tract in a three-dimensional coordinate system to output a mode feature space; The fusion module constructs a double-branch convolutional neural network according to the mode feature space, processes and fuses the endoscopic images and pressure distribution cloud images using the visual branch and tactile branch of the double-branch convolutional neural network to obtain a depth prediction value of the catheter tip and the glottis; The reconstruction module captures the dynamic deformation of the patient's respiratory tract according to the depth prediction value of the catheter tip and the glottis, generates a respiratory tract cross-sectional image and reconstructs it into a three-dimensional dynamic deformation model with the three-dimensional digital model of the respiratory tract; The output module uses the linear trend average method to predict the motion trajectory of the patient's glottis according to the three-dimensional dynamic deformation model, calculates the motion displacement coordinates of the catheter tip, and outputs the final positioning of the catheter tip.

[0014] In a third aspect, the present application provides a computer device comprising a memory and a processor, the memory storing a computer program, wherein the computer program is executed by the processor to implement any step of the artificial intelligence-based automatic tracheal intubation method according to the first aspect of the present application.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement any step of the artificial intelligence-based automatic tracheal intubation method according to the first aspect of the present application.

[0016] The present application has the beneficial effects that: the scanning range is optimized by collecting the cross-sectional image of the glottis and the upper segment of the trachea of the patient combined with the predicted value of the depth of the catheter tip, the SIFT algorithm registration and the inverse distance weighted interpolation, a three-dimensional dynamic deformation model is generated, the dynamic limitations of the preoperative model are made up, the adaptability to the dynamic deformation of the airway of the patient is enhanced, the real-time deformation tracking of the glottis and the upper segment of the trachea of the patient is realized, the intubation failure caused by the airway movement of the patient and the positioning error caused by the airway movement of the patient are reduced, the motion displacement coordinates of the catheter tip are calculated, the glottis trajectory is predicted through the efficient linear trend average method, the rapid and accurate positioning of the catheter tip is achieved, the success rate of intubation is greatly increased, and the operation delay and the risk of misinsertion are reduced. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Fig. 1 The flow chart for constructing the three-dimensional digital model of the respiratory tract.

[0019] Fig. 2 The schematic diagram for generating the mode feature space.

[0020] Fig. 3 The architecture diagram of the double-branch convolutional neural network.

[0021] Fig. 4 The flow chart of the three-dimensional dynamic deformation model and positioning output. DETAILED DESCRIPTION

[0022] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings of the specification.

[0023] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0024] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an independent or alternative embodiment that excludes other embodiments.

[0025] Reference Figs. 1-4 For one embodiment of the present application, the embodiment provides an artificial intelligence-based automatic tracheal intubation method, comprising the following steps: S1. Collect preoperative data of the patient, and construct a three-dimensional digital model of the respiratory tract.

[0026] Comprising the following steps: S1.1, before the intubation operation, use Siemens SOMATOM Force CT scanner to comprehensively scan the respiratory tract of the patient, the scanning range covers the patient's larynx, trachea and glottis, the respiratory tract data obtained by scanning is exported in DICOM format and separated automatically using the automatic segmentation algorithm in 3DSlicer software, to obtain a three-dimensional structure data set of the respiratory tract, including the patient's glottis position, tracheal diameter and anatomical markers (such as tracheal rings and laryngeal cartilage).

[0027] Select Philips Respironics NM3 respiratory monitor, integrated with chest and abdominal belt sensor, nasal flow meter and peak flow meter. The patient wears the chest and abdominal belt sensor and the nasal flow meter in a quiet environment, records the respiratory rate and tidal volume under calm breathing, and guides the patient to perform cough simulation, and uses the peak flow meter to record the cough airflow velocity of the patient when coughing, and stores the respiratory rate, tidal volume and cough airflow velocity of the patient as CSV format, forming a respiratory parameter data set.

[0028] S1.2, select ANSYS software as the execution software, import the three-dimensional structure data set of the respiratory tract into the ANSYS software, and use the stereomicroscope method to convert the imported three-dimensional structure data set of the respiratory tract into a geometric grid unit of the patient's respiratory tract.

[0029] Specifically, after importing the three-dimensional structure data set of the respiratory tract into the ANSYS software, the respiratory tract data is displayed in the form of vertex coordinates to show the surface of the patient's respiratory tract, and the interpolation algorithm in the stereomicroscope method, i.e. Delaunay triangulation algorithm, is used to connect the respiratory tract data in sequence into a continuous surface to generate a preliminary stereoscopic structure. The Delaunay triangulation algorithm automatically calculates the spatial adjacency relationship between each data point in the preliminary stereoscopic structure, constructs a triangular facet to cover the outer wall of the respiratory tract, and forms a closed respiratory tract surface structure. The closed respiratory tract surface structure is expanded into an actual stereoscopic structure through stereoscopic filling function, i.e. setting the target resolution and the total number of grid units, the total number of grid units should ensure that the fine structure of the respiratory tract, such as tracheal ring protrusion, can be captured, and the grid density also needs to be set, the grid unit density is increased in the glottis area, such as one unit per 0.2mm, and the grid unit density is reduced in the straight section of the trachea, to balance the calculation efficiency.

[0030] According to the geometric grid unit of the patient's respiratory tract, the geometric boundary condition is set and the Boolean operation is performed on the geometric grid unit of the patient's respiratory tract, and the three-dimensional geometric model of the respiratory tract is output.

[0031] Specifically, the definition of the geometric boundary condition depends on the geometric grid unit of the patient's respiratory tract, and the behavior of the respiratory tract in the subsequent dynamic simulation is determined by specifying the properties of the grid unit, such as fixed or deformable. The geometric boundary condition is specifically set to mark the geometric grid unit of the patient's glottis region as a deformable boundary, allowing deformation to simulate expansion and contraction during patient breathing or coughing, and marking the grid unit of the tracheal wall as a fixed boundary to limit displacement to conform to anatomical rigidity.

[0032] The Boolean operation refers to merging overlapping geometric grid units or removing redundant geometric grid units, such as redundant facets at the junction of the glottis and trachea, but must be performed under the constraints of the boundary conditions to preserve the definition of the glottis and tracheal wall. That is, the Boolean operation will not remove the deformable nodes of the glottis region, nor will it change the fixed properties of the tracheal wall. The execution process is as follows: by comparing the vertex coordinates and facets of adjacent units, overlapping facets at the junction of the glottis and trachea are detected and marked as overlapping regions. Perform a Boolean merge operation to integrate the facets of the overlapping regions into a single continuous facet, reducing the number of geometric grid units. Scan the geometric grid units to identify redundant grid unit nodes, i.e. grid units whose distance due to overlap is less than the grid density, remove these nodes through the merge function to further reduce the number of geometric grid units, and output the three-dimensional geometric model of the respiratory tract.

[0033] S1.3, all parameters in the respiratory parameter dataset are set as airflow dynamic boundary conditions, and an airflow velocity field is set according to the airflow dynamic boundary conditions.

[0034] Specifically, in the ANSYS software, the patient's breathing frequency is set as the basis for the periodic change of airflow, a fixed length of breathing cycle is defined, for example 5 seconds, to simulate the periodic fluctuations of calm breathing, and the cough airflow velocity and tidal volume are set as instantaneous peaks to simulate sudden airflow impact.

[0035] The set airflow velocity field is: in the Fluent unit of ANSYS, the throat is specified as the airflow inlet, i.e. the velocity boundary. According to the patient's breathing frequency, a periodic velocity curve is assigned, such as a sinusoidal fluctuation for calm breathing and a short-time pulse for coughing.

[0036] S1.4, obtain biomechanical parameters from a biomechanics database (e.g. IT'IS Foundation database), and add the biomechanical parameters and airflow velocity field to the three-dimensional geometric model of the respiratory tract synchronously to obtain a dynamic three-dimensional geometric model of the respiratory tract.

[0037] Specifically, the obtained biomechanical parameters include cartilage elastic modulus, mucosa elastic modulus, and airflow pressure during coughing. In the ANSYS software, the airflow velocity field is applied to the inner wall surface of the three-dimensional geometric model of the respiratory tract through the nodes of the geometric grid elements, reflecting the stress state of breathing and coughing. The biomechanical parameters are used to assign material properties to the geometric grid elements of the three-dimensional geometric model of the respiratory tract, i.e., the glottis region dominated by mucosa is assigned a lower mucosa elastic modulus to allow flexible deformation, while the tracheal wall dominated by cartilage is assigned a higher cartilage elastic modulus to maintain rigidity. Finally, the airflow pressure during coughing is applied to the inner wall surface of the three-dimensional geometric model of the respiratory tract as a mechanical load, and the dynamic three-dimensional geometric model of the respiratory tract is output.

[0038] S1.5, in the ANSYS software, the dynamic three-dimensional geometric model of the respiratory tract is used to simulate the dynamic deformation of the patient's respiratory tract under breathing and coughing, generate a spatiotemporal data set and integrate it with the geometric grid elements of the patient's respiratory tract, and obtain a three-dimensional digital model of the respiratory tract.

[0039] Specifically, in the ANSYS software, the explicit dynamic finite element analysis method is configured, and the simulation period and time step are set to simulate multiple breathing and coughing cycles. The process of simulating the dynamic deformation of the patient's respiratory tract under breathing and coughing is to simulate the mechanical response of the airflow pressure to the grid element nodes through iteration, calculate the displacement between the glottis and the tracheal wall, and the expression is: ; wherein, represents the displacement between the glottis and the tracheal wall, i.e., the deformation of the patient's glottis or tracheal wall under the airflow pressure during coughing, represents the airflow pressure during coughing of the patient, represents the elastic modulus, including the cartilage elastic modulus and the mucosa elastic modulus.

[0040] The displacement of the glottis and the tracheal wall is recorded to generate a spatiotemporal data set containing the deformation amplitude of the glottis and the tracheal wall. The spatiotemporal data set is automatically integrated with the geometric grid elements of the patient's respiratory tract using the post-processing tool of CFD-Post, and the static anatomical structure such as the glottis position, tracheal diameter and dynamic displacement of the glottis and tracheal wall is combined to obtain a three-dimensional digital model of the respiratory tract.

[0041] S2. Collect endoscopic images and pressure distribution cloud maps and map them with the three-dimensional digital model of the respiratory tract in a three-dimensional coordinate system to output a pattern feature space.

[0042] including the following steps, S2.1, deploy a flexible pressure sensor and an endoscopic camera at the tip of the cannula, collect the contact pressure distribution of the catheter tip and the respiratory tract wall, generate a pressure distribution cloud chart containing the three-dimensional coordinates and pressure values of each pressure point. Start the endoscopic camera and collect the internal image of the respiratory tract to generate an endoscopic image.

[0043] S2.2, determine the three-dimensional coordinates of the catheter tip according to the geometric center of the pressure distribution cloud chart, calculate the pressure values between each pressure point in the pressure distribution cloud chart and the geometric grid cells in the three-dimensional digital model of the respiratory tract using the inverse distance weighting method and assign them to the three-dimensional digital model of the respiratory tract, and generate a mapped pressure distribution cloud chart.

[0044] Specifically, the pressure values between each pressure point in the pressure distribution cloud chart and the geometric grid cells in the three-dimensional digital model of the respiratory tract are calculated, and the expression is: ; Wherein, represents the pressure value between the pressure point and the geometric grid cell in the three-dimensional digital model of the respiratory tract, represents the number of pressure points, represents the pressure value of the th pressure point, represents the Euclidean distance between the three-dimensional coordinates of the geometric grid cells in the three-dimensional digital model of the respiratory tract and the three-dimensional coordinates of the pressure points.

[0045] The pressure values between the pressure points and the geometric grid cells in the three-dimensional digital model of the respiratory tract are assigned to all geometric grid cells in the three-dimensional digital model of the respiratory tract one by one to generate a mapped pressure distribution cloud chart, and mark the pressure hotspot area, such as the geometric grid cell of the glottis or tracheal wall >1N.

[0046] S2.3, set an edge gradient threshold, pre-process the endoscopic image, and when the edge gradient of the pre-processed endoscopic image is greater than the edge gradient threshold, determine the strong and weak edge points, and output a binary edge image.

[0047] Specifically, pre-processing the endoscopic image means converting the endoscopic image to a grayscale image and using Gaussian blur to remove noise in the grayscale image to enhance edge detection stability.

[0048] The edge strength threshold is set according to the typical gradient value of the glottis and tracheal ring, for example, 100-200. When the edge gradient of the preprocessed endoscopic image is greater than the edge gradient threshold, the pixel point of the preprocessed endoscopic image is marked as a strong edge point. When the edge gradient of the preprocessed endoscopic image is less than or equal to the edge gradient threshold, and is connected to the strong edge point, the pixel point of the preprocessed endoscopic image is marked as a weak edge point and is removed. All strong edge points are connected by edge connection to obtain a binary edge image containing the edge of the anatomical marker, such as the glottis contour.

[0049] S2.4, set the contour area threshold, and mark the effective pixel contour of the contour area of the binary edge image greater than the contour area threshold as a feature point.

[0050] Specifically, the contour area threshold is set according to the typical contour area of the glottis and tracheal ring, for example, the glottis contour is about 500 pixels, and the tracheal ring is about 300 pixels. When the contour area of the binary edge image is greater than the contour area threshold, it is determined that the pixel contour is noise or an irrelevant structure such as a small pleat edge, and is removed. The effective pixel contour is marked as a feature point, including the three-dimensional coordinates of the feature point.

[0051] S2.5, using a distance minimization algorithm, calculating the Euclidean distance between the three-dimensional coordinates of each feature point and the geometric grid cell in the three-dimensional digital model of the respiratory tract, expressed as: ; wherein, represents the Euclidean distance between the three-dimensional coordinates of the feature point and the geometric grid cell in the three-dimensional digital model of the respiratory tract, represents the horizontal coordinate of the feature point, represents the vertical coordinate of the feature point, represents the depth coordinate of the feature point, represents the horizontal coordinate of the geometric grid cell in the three-dimensional digital model of the respiratory tract, represents the vertical coordinate of the geometric grid cell in the three-dimensional digital model of the respiratory tract, represents the depth coordinate of the geometric grid cell in the three-dimensional digital model of the respiratory tract.

[0052] The geometric grid cell with the smallest Euclidean distance is selected as the matching result, and the matching result contains multiple matching pairs.

[0053] The mapped pressure distribution cloud map, the matching result, and the three-dimensional coordinates of the catheter tip are integrated and stored in JSON format, and an output mode feature space is obtained, containing the endoscopic image, the position of the catheter tip, and the pressure distribution mode.

[0054] S3. Constructing a double-branch convolutional neural network according to the mode feature space, processing and fusing the endoscope image and the pressure distribution cloud map using the visual branch and the tactile branch in the double-branch neural convolutional network respectively to obtain the depth prediction value of the catheter tip and the glottis.

[0055] comprising the following steps, S3.1, interpolating the pressure distribution mode and the endoscope image in the mode feature space into a pressure distribution tensor and an endoscope image tensor respectively to form a mode data tensor.

[0056] Specifically, the three-dimensional coordinates and pressure values of the pressure points are extracted from the mode feature space, and a two-dimensional grid is created to represent the planar projection of the catheter tip circumference. The two-dimensional distance (i.e., straight-line distance) between each two-dimensional grid point and all pressure points is calculated by interpolation, and each pressure point is assigned a weight according to the straight-line distance, i.e., the closer the pressure point to the two-dimensional grid point, the higher the weight value. The pressure values of the pressure points are weighted and averaged according to the assigned weights to obtain the pressure value of the two-dimensional grid point. The interpolation calculation is repeated for all two-dimensional grid points in the two-dimensional grid to generate multiple pressure distribution values, which form a two-dimensional pressure distribution tensor with a numerical range of 0.1N~2N, reflecting the pressure distribution of the catheter tip.

[0057] The center cropping method is used to crop a fixed size endoscope image (such as 512x512) with the center pixel of the endoscope image as the center, and the pixel values of the cropped endoscope image are normalized to the range of 0~1 to form an endoscope image tensor.

[0058] The pressure distribution tensor and the endoscope image tensor are uniformly formatted to obtain a mode data tensor.

[0059] S3.2, selecting ResNet (Residual Network) and MobileNet (Mobile Network) as the core architecture and training using the mode data tensor to obtain a visual branch convolutional neural network and a tactile branch convolutional neural network.

[0060] Specifically, the architecture of ResNet includes 18 layers, i.e., 4 residual blocks, each containing two convolutional layers, with the input being the endoscope image tensor, the first layer using a 7x7 convolutional kernel, and each subsequent layer using a 3x3 max pooling. The channel numbers of the 4 residual blocks are 64, 128, 256, and 512 respectively, each residual block contains 2 3x3 convolutional layers, and the last layer performs global average pooling to output a visual feature vector.

[0061] The structure of MobileNet includes 17 inverted residual blocks, the input is a pressure distribution tensor, the first layer is a 3x3 convolution layer, and the subsequent inverted residual blocks use 1x1 expansion convolution, 3x3 depth separable convolution and 1x1 compression convolution, the channel number is 1280, and the last layer is global average pooling, and the output is a tactile feature vector.

[0062] 80% of the mode data tensor is divided into a training set, and 20% is a validation set. The straight-line distance between the three-dimensional coordinates of the catheter tip and the center of the patient's glottis is calculated as the actual label. The process of training the double-branch convolutional neural network is as follows: setting the batch size to 32 frames, each batch containing an endoscopic image tensor, a pressure distribution tensor and an actual label, initializing the optimizer to Adam, setting the learning rate to 0.001, inputting the endoscopic image tensor and the pressure distribution tensor into ResNet and MobileNet respectively, ResNet and MobileNet will calculate the mean square error between the distance between the catheter tip and the patient's glottis and the actual label, and record the mean square error on the validation set after each training. The total training is 100 batches, but the early stopping mechanism is set, that is, when the mean square error on the validation set does not decrease significantly for 5 consecutive batches, it is considered that the double-branch convolutional neural network has reached convergence, and the training is ended.

[0063] S3.3, input the endoscopic image tensor into the visual branch convolutional neural network for convolution processing and pooling processing to obtain a visual feature vector.

[0064] Specifically, the endoscopic image tensor is input into the first layer of the visual branch convolutional neural network, and the pixel values of each endoscopic image are weighted and summed using a 7x7 convolution kernel to extract low-level features such as glottis edges and textures, and generate an initial feature map. For each channel of the initial feature map, perform batch normalization, that is, calculate the mean and variance of each channel in the batch and standardize it to a dimension with a mean of 0 and a variance of 1. After normalization, pass through 4 residual blocks to extract high-level features such as glottis contour shapes, and repeat the normalization process for each channel of the initial feature map in each convolution layer to generate a visual feature map.

[0065] Apply 3x3 max pooling to the visual feature map to compress the spatial dimension to 128x128x64 and retain high-level features. Use residual blocks to apply adaptive pooling, pass the feature map of the last residual block through global average pooling to calculate the grid average value of each channel, and generate a multi-dimensional visual feature vector representing the edge intensity and tracheal ring texture of the glottis contour.

[0066] Input the pressure distribution tensor into the tactile branch convolutional neural network to calculate the pressure gradient amplitude of the catheter tip and the patient's respiratory tract wall to generate a tactile feature vector.

[0067] Specifically, the pressure distribution tensor is input into the first layer of the haptic branch convolutional neural network, a 3x3 convolution kernel (32 channels, stride 2, padding 1) is used, and the pressure values of each two-dimensional coordinate point of the pressure distribution tensor are weighted and summed. In order to enhance the contact between the catheter and the patient's respiratory tract, the pressure gradient amplitude of the catheter tip and the patient's respiratory tract wall needs to be calculated, and a pressure gradient amplitude map is generated, expressed as: ; wherein, represents the pressure gradient amplitude of the catheter tip and the patient's respiratory tract wall, represents the horizontal coordinate of the pressure distribution tensor, represents the vertical coordinate of the pressure distribution tensor, represents the mathematical symbol of partial derivative, indicating the local change in or direction, represents the pressure value of the two-dimensional coordinate point of the pressure distribution tensor, i.e. the contact pressure of the catheter tip and the respiratory tract wall, represents the partial derivative of, i.e. the small change amount of the pressure value of the two-dimensional coordinate point of the pressure distribution tensor in or direction, represents the partial derivative variable of, i.e. the small position change in the pressure distribution tensor along the axis direction, represents the partial derivative variable of, i.e. the small position change in the pressure distribution tensor along the axis direction.

[0068] The pressure gradient amplitude map is normalized to the range of 0-1 to ensure consistency with the value range of the pressure distribution tensor. The normalized gradient amplitude map is spliced with the pressure distribution tensor to generate an input tensor. The splicing method is to stack along the dimension of the channel, i.e. the first channel is the pressure value of the two-dimensional coordinate point of the pressure distribution tensor, the second channel is the pressure gradient amplitude, and so on. The input tensor is input into the haptic branch convolutional neural network, and the input tensor is standardized by using the reverse residual block, i.e. the mean and variance of the input tensor are normalized, to generate a haptic feature map. A 2x2 max pooling is applied to the haptic feature map to compress the haptic feature map to a fixed size (such as 8x8x64) to retain the pressure features, and output a haptic feature vector.

[0069] S3.4, the dimensions of the visual feature vector and the haptic feature vector are dimensionally cropped, and the visual feature vector and the haptic feature vector after dimensionally cropping are assigned weights, and a fusion feature vector is output.

[0070] Specifically, since the visual feature vector and the tactile feature vector are different features, the dimensions are different. In order to unify the dimensions, the dimension of the higher one is cut and kept consistent with the dimension of the other one. For example, the visual feature vector has a higher dimension. According to the visual feature map of the visual branch convolutional neural network, high-intensity features such as the glottal contour are preferentially retained, and low-intensity features are discarded, while the dimension of the tactile feature vector remains unchanged.

[0071] The visual and tactile features of the cut visual feature vector and the tactile feature vector are assigned weights, and the weights are assigned according to the visual features and the tactile features. For example, the visual feature of the glottal contour contributes more to the depth of the catheter tip and the glottis, so a higher weight is assigned to the visual feature.

[0072] The fusion values of the visual features and the tactile features after assigning the weights are calculated and weighted summed to generate a fusion feature vector, and the expression is: ; wherein, represents the fusion value of the visual feature and the tactile feature after assigning the weights, represents the visual feature value, represents the tactile feature value, represents the weight of the visual feature, represents the weight of the tactile feature. For example, if the visual feature value is 0.9, the tactile feature value is 0.6, the weight of the visual feature is 0.7, and the weight of the tactile feature is 0.3, the fusion value is 0.81.

[0073] S3.5, according to the fusion feature vector, the depth prediction value of the catheter tip and the glottis is calculated, and the expression is: ; wherein, represents the depth prediction value of the catheter tip and the glottis of the patient, represents the horizontal coordinate of the catheter tip, represents the vertical coordinate of the catheter tip, represents the depth coordinate of the catheter tip, represents the horizontal coordinate of the center of the glottis of the patient, represents the vertical coordinate of the center of the glottis of the patient, represents the depth coordinate of the center of the glottis of the patient, represents the control fusion feature vector affecting factor of the distance between the catheter tip and the glottis, which is determined according to the alignment ability of the fusion feature vector to the edge of the glottis and the pressure hotspot. For example, when the glottal edge and the pressure hotspot are aligned, the value of the Euclidean distance between the catheter tip and the glottis will be slightly higher, and at this time To be corrected to an exact value, such as 0.1, denotes the L2 norm of the fusion feature vector, reflecting the feature intensity of the tactile feature and the visual feature.

[0074] S4. According to the predicted value of the depth of the catheter tip and the glottis, capture the dynamic deformation of the patient's respiratory tract, generate the cross-sectional image of the respiratory tract and reconstruct it into a three-dimensional dynamic deformation model with the three-dimensional digital model of the respiratory tract.

[0075] comprising the steps of, S4.1, collect the cross-sectional images of the patient's glottis and upper trachea and pre-process them to obtain the pre-processed OCT image sequence.

[0076] Specifically, the frequency domain optical coherence tomography (OCT) device at the front end of the laryngoscope is activated, and according to the predicted value of the depth of the catheter tip and the glottis, the three-dimensional coordinates of the catheter tip and the center of the glottis, the scanning frequency of the OCT device is set and covers the patient's glottis and upper trachea, the cross-sectional images of the patient's glottis and upper trachea are collected, including the patient's mucosal boundary, tracheal ring and airway cavity, reflecting the dynamic deformation of the patient's respiratory tract. Here, the cross-sectional images are collected, and without the predicted value of the depth of the catheter tip and the glottis, the three-dimensional coordinates of the catheter tip and the center of the glottis as the basis, direct collection may cause the laser beam of the OCT to deviate from the glottis, such as aiming at the lower trachea or esophagus, generating irrelevant cross-sectional images, due to the limited field of view of the OCT, and the real-time depth prediction value reflects the dynamic change of the catheter insertion depth, especially when the patient is breathing or coughing.

[0077] The pre-processing process is: applying Gaussian filtering to each frame of the cross-sectional images of the patient's glottis and upper trachea to remove laser speckle noise and enhance the clarity of the cross-sectional images. The contrast of the cross-sectional images after removing noise is enhanced, and the gray value of the cross-sectional images after removing noise is normalized to the range of 0~255 through linear stretching, highlighting the gray value of the mucosal boundary and the airway cavity.

[0078] S4.2, use the SIFT algorithm to extract SIFT feature points in the pre-processed OCT image sequence, and according to the SIFT feature points, perform rigid registration on the pre-processed OCT image sequence to generate a registered OCT image sequence.

[0079] Specifically, a scale space is constructed, i.e. a Gaussian filter is applied to each frame of the pre-processed OCT image sequence to form a scale space pyramid, which is used to detect regions with significant intensity changes in the pre-processed OCT image sequence, such as the gray scale of the mucosal boundary. In the scale space, each pixel and its adjacent pixels of the pre-processed OCT image sequence are compared one by one to identify local extreme points, such as the gray scale difference between two pixels being greater than 10, i.e. within the normal gray scale range, and a gradient direction is assigned to each local extreme point to generate SIFT feature points, including feature descriptors, to describe the texture around the local extreme points.

[0080] The process of rigid registration is as follows: taking the first frame of the pre-processed OCT image sequence as the reference, comparing the feature descriptors of the SIFT feature points of each frame with those of the first frame, calculating the Euclidean distance between the SIFT feature point descriptors point by point, finding the point with the smallest Euclidean distance in the first frame for each SIFT feature point of each frame, i.e. the nearest neighbor distance, and checking the second nearest neighbor distance. A feature matching threshold is set, which is dynamically set according to the contrast of the OCT image sequence. When the ratio between the nearest neighbor distance and the second nearest neighbor distance is less than the feature matching threshold, the SIFT feature points are matched as matching point pairs, otherwise no matching is performed.

[0081] The coordinates of the matching point pairs are compared one by one to determine the rotation angle of the matching point pairs. The displacement of the matching point pairs is calculated, i.e. the average difference of the coordinates of the matching point pairs, such as the average difference of the horizontal coordinates of all matching point pairs being 0.2 mm and the average difference of the vertical coordinates being 0.1 mm, then the displacement of the matching point pairs is (0.2, 0.1) mm. According to the rotation angle and displacement of the matching point pairs, a 3x3 rigid transformation matrix is constructed, including a 2x2 rotation matrix, a 1x2 translation, and the last row of the rigid transformation matrix [0, 0, 1].

[0082] For each frame of the pre-processed OCT image sequence, the pixel coordinates of each frame are converted to homogeneous coordinates, such as a pixel coordinate of (100, 150) being converted to homogeneous coordinates of [100, 150, 1], i.e. extended to a three-dimensional vector, with 1 representing the homogeneous term, to facilitate uniform translation and rotation transformation. The 3x3 rigid transformation matrix is multiplied by each pixel coordinate respectively to obtain new pixel coordinates. Since the new pixel coordinates are not integers, the gray scale values of the adjacent pixels are extracted from each frame of the pre-processed OCT image sequence and weighted averaged with the distance weight between the new pixel coordinates and the adjacent pixel coordinates (set according to the Euclidean distance between the new pixel coordinates and the adjacent pixel coordinates, the closer the distance, the higher the weight) to obtain the interpolated gray scale values.

[0083] The preprocessed OCT image sequence is processed one by one by applying interpolated gray values ​​and adjusting pixel coordinates to generate a registered OCT image sequence. For example, the pixel coordinates of the glottic mucosa boundary are (100, 150) with a gray value of 200. After registration, it is transformed into (108.6905, 155.867) with a gray value of 200.5.

[0084] S4.3. Use inverse distance weighted interpolation to calculate the pixel displacement in the registered OCT image sequence and generate a deformation field sequence.

[0085] Specifically, the displacement of SIFT feature points in the registered OCT image sequence is calculated. For example, if the coordinates of one SIFT feature point are (100, 150) and the coordinates of another SIFT feature point are (102, 151), the displacement of these two SIFT feature points is calculated by the difference between their horizontal and vertical coordinates and converted into millimeters. That is, the displacement vector of the generated SIFT feature point is (0.04, 0.02) millimeters, which means that it moves 0.04 millimeters horizontally and 0.02 millimeters vertically.

[0086] Based on the displacement of SIFT feature points in the registered OCT image sequence, the pixel displacement is calculated using the following expression: ; ; in, This represents the horizontal displacement of pixels in the registered OCT image sequence. Indicates the number of SIFT feature points. Indicates the first The weights of each SIFT feature point are determined by the proximity of the SIFT feature point to the pixel coordinates. The higher the value, the greater the influence of SIFT feature points on pixel displacement. Indicates the first Horizontal displacement of SIFT feature points This represents the vertical displacement of pixels in the registered OCT image sequence. Indicates the first Vertical displacement of each SIFT feature point.

[0087] The pixel displacement is stored for each frame in the registered OCT image sequence, and the deformation field sequence is output to describe the dynamic displacement of the glottis and upper trachea. For example, the horizontal displacement of the patient's glottic mucosal boundary region is 0.03 mm, reflecting an expansion of 0.1 mm.

[0088] S4.4 Map the deformation field sequence to the three-dimensional digital model of the respiratory tract to form a three-dimensional dynamic deformation model.

[0089] Specifically, the geometric grid unit is initialized, that is, the coordinates of the center of the glottis of the patient are aligned to the coordinate system of the OCT image sequence, and the alignment is achieved by translating the geometric grid unit so that the center of the glottis of the respiratory tract three-dimensional digital model coincides with the center of the glottis of the OCT image sequence coordinate system.

[0090] After initialization, the deformation field sequence is a two-dimensional cross section, so the depth coordinates of the geometric grid unit of the respiratory tract three-dimensional digital model need to be made to correspond to the deformation field plane, that is, the z-axis of the geometric grid unit is discretized into the longitudinal depth of the OCT image sequence, and each frame of image is spaced by a fixed millimeter.

[0091] Taking any one node in the geometric grid unit, for example, a coordinate located at the mucosal boundary of the glottis of the patient, for example, (2, 3, 5), the horizontal and vertical coordinates of the node, that is, 2 and 3, are converted into pixel coordinates in the deformation field sequence. Since each pixel represents 20 microns, 2 millimeters is 100 pixels, and 3 millimeters is 150 pixels, the node corresponds to the pixel position (100, 150). If the pixel position is an integer, such as (100, 150), the displacement of the pixel from the deformation field sequence is directly taken as the displacement of the node, that is, how much to move in the horizontal direction and how much to move in the vertical direction. However, if the node position is not an integer pixel, such as (100.5, 150.2) after conversion, several nearest integer pixels are found, each pixel has its own displacement, which records the movement amount in the horizontal and vertical directions. The node displacement synthesizes the displacement of several nearest integer pixels, and the node is observed from the near and far of each pixel. For example, the node is close to 100 pixels in the horizontal direction and far from 101 pixels, so it refers to the displacement of 100 pixels more. The vertical direction is the same, the node is close to 150 pixels, so it refers to the displacement of 150 pixels. The output node displacement is the movement amount of the geometric grid unit node in the horizontal and vertical directions.

[0092] Take any one node coordinate in the geometric grid unit, add the node displacement to the node coordinate to form a new node coordinate, such as node coordinate (2, 3, 5), and assume that the node displacement is (0.0308, 0.0154) millimeters. The updated node coordinates are (2.0308, 3.0154, 5), and the depth coordinate 5 remains unchanged.

[0093] Repeat the operation of updating the node coordinates for each frame of the deformation field sequence until all the nodes are traversed, generate a three-dimensional grid, integrate the three-dimensional grid according to the updated time sequence, and form a three-dimensional dynamic deformation model for the patient's glottis.

[0094] S5. According to the three-dimensional dynamic deformation model, the linear trend average method is used to predict the movement trajectory of the glottis of the patient, the movement displacement vector of the catheter tip is calculated, and the final positioning of the catheter tip is output.

[0095] comprising the steps of, S5.1, find the glottis boundary of the patient from the three-dimensional grid of the three-dimensional dynamic deformation model, and calculate the three-dimensional coordinates of the glottis boundary center of each three-dimensional grid to form a glottis center coordinate sequence, such as [(1.8, 1.6, 0.2), (1.81, 1.61, 0.21),...].

[0096] S5.2, according to the glottis center coordinate sequence, calculate the average moving speed of the glottis center and adjust it using the patient's breathing frequency in the respiratory parameter data set to obtain the adjusted average moving speed of the glottis center.

[0097] Specifically, take several glottis center coordinates, calculate the moving speed of the glottis center for the horizontal, vertical and depth coordinates of the glottis center coordinates respectively, for example, the glottis center coordinate 1 is (1.79, 1.59, 0.19), the glottis center coordinate 2 is (1.81, 1.61, 0.21), the difference between the horizontal coordinates of the glottis center coordinate 1 and the glottis center coordinate 2 is 0.02mm, then according to the time span between the glottis center coordinates (such as 0.1s), the average moving speed of the glottis center is 0.2mm / s, and the calculation process of the vertical and depth coordinates is the same.

[0098] Select the patient's breathing frequency from the previously output respiratory parameter data set as an adjustment parameter, and adjust the average moving speed of the glottis center, the adjustment formula is: ; wherein, represents the adjusted average moving speed of the glottis center, represents a speed increment factor, which is determined according to the patient's breathing frequency and breathing period, the higher the patient's breathing frequency, the shorter the breathing period, the faster the displacement change of the glottis center, and vice versa, the displacement change of the glottis center is slower, which is defined as the ratio of the patient's breathing frequency to the normal breathing frequency, when the patient's breathing frequency is higher than the normal breathing frequency, the speed increment factor increases, and vice versa, represents the average moving speed of the glottis center.

[0099] S5.3, calculate the glottis movement trajectory coordinates of the patient using the adjusted average moving speed of the glottis center, the expression is: ; ; ; wherein, represents the horizontal coordinate of the glottal center of the patient at time , represents the horizontal coordinate of the glottal center of the patient, represents the vertical coordinate of the glottal center of the patient at time , represents the vertical coordinate of the glottal center of the patient, represents the depth coordinate of the glottal center of the patient at time , represents the depth coordinate of the glottal center of the patient.

[0100] According to the glottal motion trajectory coordinates of the patient and the three-dimensional coordinates of the catheter tip, the motion displacement coordinates of the catheter tip are calculated, and the expression is: ; : : wherein, represents the horizontal coordinate of the catheter tip at time , represents the vertical coordinate of the catheter tip at time , the depth coordinate of the catheter tip at time , represents the horizontal coordinate of the motion of the catheter tip at time , represents the vertical coordinate of the motion of the catheter tip at time , represents the depth coordinate of the motion of the catheter tip at time . For example, the glottal motion trajectory coordinates of a patient are (1.865, 1.615, 0.224), and the three-dimensional coordinates of the catheter tip are (2, 1.5, 0.5), and the motion displacement coordinates of the catheter tip are (-0.135, 0.115, -0.276), which represent the distance that the catheter tip needs to move, and output the final positioning of the catheter tip.

[0101] The embodiment also provides an artificial intelligence-based automated tracheal intubation system, comprising: a data acquisition module that acquires preoperative data of a patient and constructs a three-dimensional digital model of the respiratory tract; a mapping module that acquires endoscopic images and pressure distribution cloud images and maps them with the three-dimensional digital model of the respiratory tract in a three-dimensional coordinate system to output a mode feature space; The fusion module constructs a double-branch convolutional neural network according to a mode feature space, processes and fuses the endoscope image and the pressure distribution cloud chart using a visual branch and a tactile branch in the double-branch convolutional neural network to obtain a depth prediction value of the catheter tip and the glottis; The reconstruction module captures dynamic deformation of the respiratory tract of the patient according to the depth prediction value of the catheter tip and the glottis, generates a respiratory tract cross-section image and reconstructs the respiratory tract three-dimensional digital model into a three-dimensional dynamic deformation model; The output module predicts a motion trajectory of the glottis of the patient using a linear trend average method according to the three-dimensional dynamic deformation model, calculates a motion displacement coordinate of the catheter tip and outputs a final positioning of the catheter tip.

[0102] The embodiment also provides a computer device suitable for the automatic tracheal intubation method based on artificial intelligence, which comprises a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions to realize the automatic tracheal intubation method based on artificial intelligence provided in the above embodiment.

[0103] The computer device can be a terminal, which comprises a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, a trackball or a touchpad arranged on the shell of the computer device, or an external keyboard, a touchpad or a mouse, etc.

[0104] The embodiment also provides a storage medium having a computer program stored thereon, the program being executed by a processor to implement the method for realizing automatic tracheal intubation based on artificial intelligence as proposed in the above embodiment; the storage medium can be realized by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0105] To sum up, the present application optimizes the scanning range by collecting the cross-sectional images of the glottis and the upper segment of the trachea and combining the predicted value of the depth of the catheter tip, generates a three-dimensional dynamic deformation model by SIFT algorithm registration and inverse distance weighted interpolation, makes up for the dynamic limitations of the preoperative model, enhances the adaptability to the dynamic deformation of the patient's airway, realizes real-time deformation tracking of the patient's glottis and upper segment of the trachea, reduces the intubation failure caused by the movement of the patient's airway and the positioning error caused by the movement of the patient's airway, calculates the motion displacement coordinates of the catheter tip, predicts the glottis trajectory by the efficient linear trend average method, achieves rapid and accurate positioning of the catheter tip, greatly increases the success rate of intubation, and reduces the operation delay and the risk of misinsertion.

[0106] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application but not limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application, and all of them should be covered in the scope of the claims of the present application.

Claims

1. An automated endotracheal intubation method based on artificial intelligence, characterized in that: include, Collect patients' preoperative data and construct a three-dimensional digital model of the airway; Acquire endoscopic images and pressure distribution cloud maps and map them with a three-dimensional digital model of the airway in a three-dimensional coordinate system, and output the pattern feature space; Based on the pattern feature space, a two-branch convolutional neural network is constructed. The visual branch and tactile branch in the two-branch convolutional neural network are used to process and fuse the endoscopic image and pressure distribution cloud map respectively to obtain the depth prediction value of the catheter tip and glottis. Based on the predicted depth of the catheter tip and glottis, the dynamic deformation of the patient's airway is captured, and the cross-sectional image of the airway is generated and reconstructed into a three-dimensional dynamic deformation model with the three-dimensional digital model of the airway. Based on the three-dimensional dynamic deformation model, the linear trend averaging method is used to predict the motion trajectory of the patient's glottis, calculate the motion displacement coordinates of the catheter tip, and output the final position of the catheter tip.

2. The automated endotracheal intubation method based on artificial intelligence as described in claim 1, characterized in that: The patient's preoperative data included glottic position, tracheal diameter, anatomical landmarks, respiratory rate, tidal volume, and cough airflow velocity.

3. The automated endotracheal intubation method based on artificial intelligence as described in claim 2, characterized in that: The construction of the three-dimensional digital model of the respiratory tract specifically includes the following steps. The patient's glottic position, tracheal diameter and anatomical landmarks were compiled into a three-dimensional respiratory structure dataset, and the patient's respiratory rate, tidal volume and cough airflow velocity were compiled into a respiratory parameter dataset. ANSYS software was selected as the execution software. The three-dimensional structure dataset of the airway was imported into ANSYS software and converted into geometric mesh elements of the patient's airway using stereomicroscopy. Based on the geometric mesh elements of the patient's airway, set geometric boundary conditions and perform Boolean operations on the geometric mesh elements of the patient's airway to output a three-dimensional geometric model of the airway; All parameters of the respiratory parameter dataset are used as airflow dynamic boundary conditions. An airflow velocity field is set, biomechanical parameters are obtained from the biomechanical database, and the biomechanical parameters and airflow velocity field are added to the three-dimensional geometric model of the respiratory tract to obtain a dynamic three-dimensional geometric model of the respiratory tract. The dynamic airway three-dimensional geometric model is used to simulate the dynamic deformation of the patient's airway under breathing and coughing, calculate the displacement between the glottis and the tracheal wall, and generate a spatiotemporal dataset. By integrating the spatiotemporal dataset and the geometric mesh units in the dynamic three-dimensional geometric model of the respiratory tract, a three-dimensional digital model of the respiratory tract is obtained.

4. The automated endotracheal intubation method based on artificial intelligence as described in claim 3, characterized in that: The output mode feature space specifically includes the following steps. Using a flexible pressure sensor and an endoscope camera, pressure distribution cloud maps and endoscope images are acquired, and the three-dimensional coordinates of the catheter tip are determined based on the geometric center of the pressure distribution cloud map. The pressure value between each pressure point in the pressure distribution cloud map and the geometric grid cell in the three-dimensional digital model of the respiratory tract is calculated using the inverse distance weighting method, and then assigned to the three-dimensional digital model of the respiratory tract to generate the mapped pressure distribution cloud map. Set an edge gradient threshold to preprocess the endoscopic image. When the edge gradient of the preprocessed endoscopic image is greater than the edge gradient threshold, determine the strong and weak edge points and output a binarized edge image. Set a contour area threshold, mark the contour area of ​​the binarized edge image that is greater than the contour area threshold as feature points, calculate the Euclidean distance between the three-dimensional coordinates of each feature point and the geometric grid unit in the three-dimensional digital model of the respiratory tract, and obtain the matching result. The mapped pressure distribution cloud map, matching results, and three-dimensional coordinates of the catheter tip are integrated to obtain the pattern feature space.

5. The automated endotracheal intubation method based on artificial intelligence as described in claim 4, characterized in that: The predicted depth values ​​between the duct tip and the glottis are obtained, specifically... Includes the following steps, The pressure distribution pattern and the endoscopic image in the pattern feature space are interpolated into a pressure distribution tensor and cropped into an endoscopic image tensor, respectively, to form a pattern data tensor. ResNet and MobileNet were chosen as the core architectures and trained using pattern data tensors to obtain visual branch convolutional neural networks and tactile branch convolutional neural networks. The endoscopic image tensor is input into the visual branch convolutional neural network for convolution and pooling processing to obtain the visual feature vector. The pressure distribution tensor is input into the tactile branch convolutional neural network to calculate the pressure gradient amplitude between the catheter tip and the patient's airway wall, and generate a tactile feature vector. The visual and tactile feature vectors are dimensionally clipped, weights are assigned to the dimensionally clipped visual and tactile feature vectors, and a fused feature vector is output. Based on the fused feature vector, the predicted depth between the catheter tip and the patient's glottis is calculated.

6. The automated endotracheal intubation method based on artificial intelligence as described in claim 5, characterized in that: The reconstruction is a three-dimensional dynamic deformation model, specifically including the following steps. Based on the predicted depth of the catheter tip and the patient's glottis, cross-sectional images of the patient's glottis and upper trachea are acquired and preprocessed to obtain a preprocessed OCT image sequence. Rigid registration is performed on the preprocessed OCT image sequence to generate a registered OCT image sequence. The pixel displacement in the registered OCT image sequence is calculated using the inverse distance weighted interpolation method, and a deformation field sequence is generated and mapped to the three-dimensional digital model of the respiratory tract to form a three-dimensional dynamic deformation model.

7. The automated endotracheal intubation method based on artificial intelligence as described in claim 6, characterized in that: The final positioning of the tip of the output catheter specifically includes the following steps: Locate the patient's glottis center in a three-dimensional dynamic deformation model, generate a glottis center coordinate sequence, and calculate the average movement velocity of the patient's glottis center. The average velocity of the patient's glottic center was adjusted by using the respiratory rate of the patient in the respiratory parameter dataset to obtain the adjusted average velocity of the patient's glottic center. Calculate the coordinates of the patient's glottic movement trajectory based on the average movement speed of the patient's glottic center after adjustment; Based on the patient's glottic motion trajectory coordinates and the three-dimensional coordinates of the catheter tip, the motion displacement coordinates of the catheter tip are calculated as the final positioning of the catheter tip.

8. An automated endotracheal intubation system based on artificial intelligence, based on the automated endotracheal intubation method based on artificial intelligence as described in any one of claims 1 to 7, characterized in that: include, The data acquisition module collects the patient's preoperative data and constructs a three-dimensional digital model of the respiratory tract; The mapping module acquires endoscopic images and pressure distribution cloud maps and maps them to the three-dimensional digital model of the airway in a three-dimensional coordinate system, outputting the pattern feature space. The fusion module constructs a dual-branch convolutional neural network based on the pattern feature space. The visual and tactile branches in the dual-branch convolutional neural network are used to process and fuse the endoscopic image and pressure distribution cloud map respectively to obtain the depth prediction value of the catheter tip and glottis. The reconstruction module captures the dynamic deformation of the patient's airway based on the predicted depth of the catheter tip and glottis, generates a cross-sectional image of the airway, and reconstructs it into a three-dimensional dynamic deformation model with the three-dimensional digital model of the airway. The output module predicts the motion trajectory of the patient's glottis using the linear trend averaging method based on the three-dimensional dynamic deformation model, calculates the motion displacement coordinates of the catheter tip, and outputs the final position of the catheter tip.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the automated endotracheal intubation method based on artificial intelligence as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the automated endotracheal intubation method based on artificial intelligence as described in any one of claims 1 to 7.

Images