A visual intelligent intubation assistance system for automatic identification of left and right bronchi

Abstract

The present invention provides a visual intelligent intubation auxiliary system for automatically identifying the left and right bronchi, the system comprising a data acquisition unit, a host computer and a human-computer interaction unit, wherein: the data acquisition unit comprises a camera and a position sensor arranged at the front end of the intubation catheter probe, for collecting video data and position information of the position of the front end of the intubation catheter; the host computer is used to determine the best observation position based on the video data and position information, and based on the video data and position information collected at the best observation position, perform feature extraction and classification prediction through a pre-trained left and right bronchial discrimination model to identify the left and right bronchus; the human-computer interaction unit is used to display an image of the position of the front end of the intubation catheter probe and the left and right bronchial identification results, to assist the tracheal intubation operation. The present invention solves the problem that the bronchial intubation technology in the prior art relies on the subjective judgment of the doctor, which easily leads to the wrong identification of the left and right bronchus.

Description

Technical Field

[0001] The present invention belongs to the technical field of clinical medicine, and in particular relates to a visual intelligent intubation auxiliary system for automatically identifying left and right bronchi. Background Art

[0002] Endotracheal intubation is a commonly used intubation technique for achieving single-lung ventilation and lung isolation. The key to the successful implementation of this technology lies in accurately distinguishing the left and right main bronchi. At present, the commonly used methods of endotracheal intubation in clinical practice include double-lumen endotracheal tube combined with fiberoptic bronchoscope positioning, visual double-lumen endotracheal tube and single-lumen endotracheal tube combined with occluder. The identification of the left and right main bronchi is mainly confirmed by physicians through traditional methods such as identification of the patient's airway anatomical landmarks under visual tools and auscultation. Such methods are subjective and have a certain error rate. Moreover, due to the similar opening characteristics of the left and right bronchi, coupled with the influence of catheter shaping and rotation during intubation, even with the assistance of visual tools, it is sometimes difficult to accurately distinguish the left and right main bronchi, and the left and right main bronchi may be incorrectly distinguished. Repeated operations in the airway can cause a series of problems such as airway damage and intubation failure, and even threaten the patient's life safety. Summary of the invention

[0003] In view of the above analysis, the present invention aims to provide a visual intelligent intubation assistance system for automatically identifying the left and right bronchi, so as to solve the problem that the bronchial intubation technology in the prior art relies on the subjective judgment of the doctor, which easily leads to the misidentification of the left and right bronchus.

[0004] The purpose of the present invention is mainly achieved through the following technical solutions:

[0005] On the one hand, the present invention provides a visual intelligent intubation auxiliary system for automatically identifying the left and right bronchi, the system comprising a data acquisition unit, a host computer and a human-computer interaction unit, wherein:

[0006] The data acquisition unit includes a camera and a position sensor disposed at the front end of the intubation catheter probe, and is used to collect video data and position information of the position of the front end of the intubation catheter;

[0007] The host computer is used to determine the best observation position based on the video data and position information, and to identify the left and right bronchi by extracting features and classifying and predicting the left and right bronchi through a pre-trained left and right bronchial discrimination model based on the video data and position information collected at the best observation position;

[0008] The human-computer interaction unit is used to display an image of the location of the front end of the intubation catheter probe and the left and right bronchial identification results to assist the tracheal intubation operation.

[0009] Furthermore, the host computer uses a pre-trained optimal position recognition model to make a real-time determination of the optimal observation position during the intubation process; the training of the optimal position recognition model includes:

[0010] Acquire video data and position information data during the intubation process of multiple patients, extract multiple frames of images based on the video data, annotate the extracted images containing all objects of interest with labels of optimal observation positions, and annotate the images lacking at least one object of interest with labels of non-optimal observation positions;

[0011] The labeled images are respectively fused with the corresponding position information data to construct a first training sample set;

[0012] The CNN model is trained using the first training sample set, and an optimal location recognition model is obtained through iterative updating.

[0013] Further, the image annotated with the best observation position label is an image containing all the objects of interest, and the objects of interest include the tracheal carina, the C-shaped cartilage ring, the left bronchial opening, and the right bronchial opening.

[0014] Furthermore, the location information data is point cloud data including a horizontal angle, a vertical angle, and a distance from the obstacle to the camera; and the step of fusing the labeled images with the corresponding location information data includes:

[0015] According to the pre-calibrated relative position and posture of the position sensor and camera, the point cloud data acquired by the sensor is preliminarily aligned to the image coordinate system;

[0016] Use the ICP algorithm to accurately align the initially aligned point cloud data and image feature points;

[0017] Projecting the precisely aligned point cloud data into an image coordinate system;

[0018] The projected point cloud data is matched and fused with the image feature points through SIFT or ORB descriptors to obtain the fused data.

[0019] Furthermore, the left and right bronchus discrimination model includes an encoder module, a memory module, a reconstruction adversarial learning module, a contrastive learning module and a decoder module;

[0020] The encoder module uses an encoder based on the Transformer model to extract features from the received data and obtain a high-dimensional feature representation of the input data;

[0021] The memory module is arranged after the encoder and is used to capture and memorize key features of the high-dimensional feature representation of the continuous multiple frames of images corresponding to the best observation position video data;

[0022] The reconstruction adversarial learning module and the contrastive learning module are arranged in parallel after the memory module, and are used to enhance and optimize the extracted key features by means of generative adversarial and contrastive learning methods respectively;

[0023] The decoder module adopts the decoder structure of the Transformer model to decode and classify the enhanced and optimized feature representation.

[0024] Furthermore, the training of the left and right bronchus discrimination model includes:

[0025] Acquire continuous multiple frames of image data of optimal observation positions during intubation of multiple patients, image data of non-optimal observation positions, and position information data corresponding to each image data;

[0026] Label the target of interest in the image data at the best observation position with a category label;

[0027] The acquired non-optimal observation position image is used as a negative sample, and the optimal observation position image with the category label is used as a positive sample, and data fusion is performed with the corresponding position information data to construct a second training sample set; the positive sample data after data fusion in the second training sample set includes all objects of interest, the category labels and position information corresponding to all objects of interest;

[0028] The left and right bronchus discrimination model is trained using the second training sample set, and a converged left and right bronchus discrimination model is obtained through iterative updating of the comprehensive loss; during the training process, the left and right bronchus discrimination model learns the corresponding relationship between each target of interest, the category label corresponding to the target of interest, and the position information, so as to identify the left and right bronchi;

[0029] The comprehensive loss is obtained by weighting the Transformer loss function, the adversarial loss function, and the contrast loss function, and is expressed as:

[0030] ;

[0031] in, is the Transformer loss function; To combat the loss function; is the contrast loss function; , , is the weight coefficient.

[0032] Furthermore, the memory module captures and memorizes the key features of the high-dimensional feature representation of the continuous multiple frames of images corresponding to each segment of video data by the following method:

[0033] S1: Randomly initialize the memory matrix M of the memory module;

[0034] S2: Generate a query vector q based on the high-dimensional feature representation corresponding to the current image;

[0035] S3: Calculate the similarity between the query vector corresponding to the current image and the memory matrix in the memory unit;

[0036] S4: obtaining memory information content according to the similarity; performing residual connection between the memory information content and the high-dimensional feature representation corresponding to the current image as a residual module; and updating the memory matrix using the memory information obtained after the residual connection;

[0037] S5: receiving the next frame of image, repeating steps S2-S4, and obtaining the key features of the high-dimensional feature representation through multiple iterative updates of multiple frames of image.

[0038] Furthermore, the reconstruction adversarial learning module includes a generator and a discriminator;

[0039] The generator is constructed based on a convolutional neural network or a deconvolutional neural network, and is used to generate multiple realistic images based on the key features output by the memory module;

[0040] The discriminator is constructed based on a convolutional neural network and is used to distinguish generated images from real images; the generator and the discriminator are iteratively trained through an adversarial loss function to optimize the images generated by the generator.

[0041] Furthermore, the contrastive learning module calculates the feature similarity between the positive and negative samples and the input best observation position samples containing the left and right bronchial opening features based on the key feature representations corresponding to the positive and negative samples output by the memory module; it performs iterative optimization through the contrast loss function to minimize the feature distance between the positive sample and the input best observation position sample, and maximize the feature distance between the negative sample and the input best observation position sample, so as to enhance the features of the positive sample.

[0042] Furthermore, the system also includes an inertial navigation module, which is arranged at the front end of the intubation catheter probe and includes an accelerometer, a magnetometer and a gyroscope; the inertial navigation module is communicatively connected to the host computer via a data transmission line arranged in the intubation catheter, and is used to obtain the posture information of the catheter probe in the catheter probe coordinate system and send it to the host computer; the posture information in the catheter probe coordinate system is converted to the earth coordinate system by the host computer to obtain the catheter probe posture information in the earth coordinate system, and the posture of the image in the earth coordinate system is obtained based on the catheter probe posture information in the earth coordinate system, and sent to the human-computer interaction unit for display.

[0043] Beneficial effects of this technical solution:

[0044] 1. The present invention aims at the problem that the distinguishing features of the left and right bronchial openings are not obvious, which easily leads to the misjudgment of the left and right bronchial openings during bronchial intubation. By utilizing the characteristics that the left and right bronchial openings, tracheal carina and C-shaped cartilage ring can be displayed simultaneously in the image during the intubation process, the present invention adopts an image that not only contains the features of the left and right bronchial openings, but also contains the best observation positions of the tracheal carina and C-shaped cartilage ring, and fuses the position information of each feature point. The present invention realizes the auxiliary positioning of the left and right bronchial openings by utilizing the tracheal carina and C-shaped cartilage ring with obvious features and the position information of each feature point, thereby ensuring the accuracy of the identification of the left and right bronchial openings.

[0045] 2. The left and right bronchus discrimination model adopted in the present invention integrates a memory module based on the traditional Transformer architecture, dynamically updates the key features of the extracted image, and optimizes the quality of feature extraction of the left and right bronchus discrimination model and the accuracy of left and right bronchus discrimination through reconstruction of adversarial learning and contrastive learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;

[0047] Figure 1 This is a block diagram of a visual intelligent intubation auxiliary system for automatically identifying left and right bronchi according to an embodiment of the present invention;

[0048] Figure 2 is a schematic diagram of an image including all objects of interest collected at the best observation position of an embodiment of the present invention;

[0049] Figure 3 4 is a structural diagram of the left and right bronchus discrimination model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the implementation cases of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0051] One embodiment of the present invention provides a visual intelligent intubation auxiliary system for automatically identifying the left and right bronchi. Figure 1 As shown, the system includes a data acquisition unit, a host computer and a human-computer interaction unit, wherein:

[0052] The data acquisition unit includes a camera and a position sensor disposed at the front end of the intubation catheter probe, and is used to collect video data and position information of the position of the front end of the intubation catheter;

[0053] The host computer is used to determine the best observation position based on the video data and position information, and to identify the left and right bronchi by extracting features and classifying and predicting the video data and position information collected at the best observation position through a pre-trained left and right bronchial discrimination model;

[0054] The human-computer interaction unit is used to display an image of the location of the front end of the intubation catheter probe and the left and right bronchial identification results to assist in the tracheal intubation operation.

[0055] Specifically, the data acquisition unit can communicate with the host computer in a wired or wireless manner to send the collected video data and location information to the host computer for processing; the host computer divides the received video data into frames and performs frame sampling to reduce data redundancy and ensure coverage of key scenes. Based on the image data and position information sampled during the endobronchial intubation process, the host computer first determines the optimal observation position. After determining the optimal observation position, it uses the image data and position information sampled at the optimal observation position to identify the left and right bronchial openings through the pre-trained left and right bronchial discrimination model, and simultaneously identifies the tracheal carina and C-shaped cartilage ring. The host computer identifies the identified left and right bronchial openings, tracheal carina and C-shaped cartilage ring as targets, and sends them to the human-computer interaction unit for display, so that the endobronchial intubation operator can determine the patient's ventral and dorsal sides according to the positions of the tracheal carina and the C-shaped cartilage ring, and further determine the positions of the left and right bronchi, so as to further confirm the positions of the left and right bronchial openings identified by the host computer, ensure the accuracy of the left and right bronchial intubation, and avoid misoperation.

[0056] Preferably, in order to improve the accuracy of left and right bronchial identification, the host computer of this embodiment first determines the best observation position in real time during the bronchial intubation process, that is, Figure 2 As shown, the positions of the images of the left and right bronchial openings, tracheal carina and C-shaped cartilage ring are observed simultaneously for detection, and before the left and right bronchial distinction is made using continuous multiple-frame images extracted from the video collected at the best observation position, the positional relationship between the tracheal carina and C-shaped cartilage ring and the left and right bronchial openings is used to assist in the judgment of the left and right bronchial openings, thereby avoiding the problem of misjudgment of the left and right bronchus due to the similar opening features of the left and right bronchus.

[0057] More specifically, the host computer of this embodiment adopts a pre-trained optimal position recognition model to make real-time judgment on the optimal observation position during the intubation process; that is, during the intubation process, based on the image and position information of the front end of the tracheal intubation probe acquired in real time by the camera and the position sensor, the optimal observation position recognition model is used to judge in real time whether the current position is the optimal observation position.

[0058] Preferably, the training of the optimal position recognition model includes:

[0059] Obtain video data and position information data of multiple patients during intubation, and extract multiple frames of images based on the video data and perform preprocessing. The preprocessing includes using convolution filters or other denoising algorithms to remove noise from the image, and applying data enhancement techniques such as rotation, translation, scaling, contrast adjustment, etc. to generate more diverse training data, and normalize the image pixel values ​​to the [0, 1] interval to ensure stable model training;

[0060] Label the preprocessed images, that is, label the images containing all the objects of interest with the best observation position image label, and label the images lacking at least one object of interest with the non-best observation position image label; wherein the objects of interest include tracheal carina, C-shaped cartilage ring, left bronchial opening and right bronchial opening;

[0061] The labeled images are respectively fused with the corresponding position information data to construct a first training sample set;

[0062] The CNN model is trained using the first training sample set, and an optimal location recognition model is obtained through iterative updating.

[0063] In particular, the aforementioned location information data is point cloud data including horizontal angle, vertical angle and distance from the obstacle to the camera; the labeled images are respectively fused with the corresponding location information data, including:

[0064] According to the pre-calibrated relative position and posture of the position sensor and camera, the point cloud data acquired by the sensor is preliminarily aligned to the image coordinate system;

[0065] Use the ICP algorithm to accurately align the initially aligned point cloud data and image feature points;

[0066] Projecting the precisely aligned point cloud data into an image coordinate system;

[0067] The projected point cloud data is matched and fused with the image feature points through SIFT or ORB descriptors to obtain the fused data.

[0068] It should be noted that the present embodiment first identifies the best observation position, and uses the image collected at the best observation position to determine the left and right bronchi, so as to ensure that all targets of interest are included in the image used for determining the left and right bronchi, and to use the features and position information of each target of interest to accurately determine the left and right bronchi; and the corresponding position information data is fused to the image data in the training data, and the relative distance between each feature point and the front end of the probe is learned during the training of the best position recognition model, and the accuracy of the best observation position judgment is improved by combining the features of the best observation position image. In the subsequent training of the left and right bronchial discrimination model, the relative distance and azimuth angle between each feature point are learned, and the auxiliary judgment of the left and right bronchial openings with similar features based on the relative position of each feature point is realized, which solves the problem of easy misjudgment of the left and right bronchial openings in the existing method.

[0069] Further, such as Figure 3 As shown, the left and right bronchus discrimination model includes an encoder module, a memory module, a reconstruction adversarial learning module, a contrastive learning module and a decoder module;

[0070] The encoder module uses an encoder based on the Transformer model to extract features from the received data and obtain a high-dimensional feature representation of the input data;

[0071] The memory module is arranged after the encoder, and is used to capture and memorize the key features of the high-dimensional feature representation of the continuous multiple frames of images corresponding to the best observation position video data;

[0072] The reconstruction adversarial learning module and the contrastive learning module are arranged in parallel after the memory module, and are used to enhance and optimize the extracted key features by means of generative adversarial and contrastive learning methods respectively;

[0073] The decoder module adopts the decoder structure of the Transformer model to decode and classify the enhanced and optimized feature representation.

[0074] Specifically, the memory module captures and memorizes the key features of the high-dimensional feature representation of the continuous multiple frames of images corresponding to each segment of video data by the following method:

[0075] S1: Randomly initialize the memory matrix M of the memory module;

[0076] S2: Generate a query vector q based on the high-dimensional feature representation corresponding to the current image;

[0077] S3: Calculate the similarity between the query vector corresponding to the current image and the memory matrix in the memory unit;

[0078] S4: obtaining memory information content according to the similarity, specifically, obtaining the memory information content by multiplying the similarity with the query vector or by convolution learning; performing residual connection between the memory information content and the high-dimensional feature representation corresponding to the current image as a residual module; and updating the memory matrix using the memory information obtained after the residual connection;

[0079] S5: receiving the next frame of image, repeating steps S2-S4, and obtaining the key features of the high-dimensional feature representation through multiple iterative updates of multiple frames of image.

[0080] The reconstruction adversarial learning module includes a generator and a discriminator; the generator is constructed based on a convolutional neural network or a deconvolutional neural network, and is used to generate multiple realistic images based on the key features output by the memory module; the discriminator is constructed based on a convolutional neural network, and is used to distinguish between generated images and real images; the generator and the discriminator are iteratively trained through an adversarial loss function to optimize the images generated by the generator.

[0081] The contrastive learning module is used to calculate the similarity between the key feature representations corresponding to the positive and negative samples output by the memory module and the original positive samples; iterative optimization is performed through the contrast loss function to minimize the feature distance between the key features corresponding to the positive samples and the original positive samples, and to maximize the feature distance between the negative samples and the original positive samples, so as to enhance the features of the positive samples.

[0082] Preferably, the training of the left and right bronchus discrimination model includes:

[0083] Acquire continuous multiple frames of image data of optimal observation positions during intubation of multiple patients, image data of non-optimal observation positions, and position information data corresponding to each image data;

[0084] Label the targets of interest in the image data at the best observation position, including: using existing deep learning models for preliminary automatic labeling, such as Mask R-CNN or U-Net models for target detection and segmentation. The specific labeling process can use two image segmentation models, namely, Mask R-CNN model and U-Net model for image segmentation respectively, for model training. After target detection and segmentation of the image, use labeling tools (such as LabelImg) to label the specific positions of the left and right bronchial openings, tracheal carinas, and C-shaped cartilage rings frame by frame. After labeling, the automatic labeling results can be manually corrected to ensure the accuracy of the labeling;

[0085] The acquired non-optimal observation position image is used as a negative sample, and the optimal observation position image with the category label is used as a positive sample, and data fusion is performed with the corresponding position information data to construct a second training sample set; the positive sample data after data fusion in the second training sample set includes all objects of interest, the category labels and position information corresponding to all objects of interest;

[0086] The left and right bronchial discrimination model is trained using the second training sample set, and after iterative updating with comprehensive loss, a converged left and right bronchial discrimination model is obtained; during the training process, the left and right bronchial discrimination model learns the correspondence between each target of interest, the category label corresponding to the target of interest, and the position information, and performs left and right bronchi identification. That is, the positive sample in the second training sample set is an image containing all targets of interest obtained at the best observation position, and after image segmentation and annotation, each target of interest is annotated with a category label, wherein the category label includes the left bronchial opening, the right bronchial opening, the tracheal carina, and the C-shaped cartilage ring. In practical applications, the left and right bronchial discrimination model trained using the second training sample set can identify the left bronchial opening, the right bronchial opening, the tracheal carina, and the C-shaped cartilage ring based on the image of the best observation position acquired in real time, and annotate the target recognition result in the image, and display it through the human-computer interaction unit to assist the bronchial intubation operation.

[0087] In the process of training the left and right bronchus discrimination model, in order to enhance the recognition ability of the model, the model can also be enhanced through the masking strategy, that is, randomly masking part of the image area to force the model to predict the masked part through the remaining information. A random mask is applied to the image to generate a masked input image. The model receives the masked image and outputs the prediction result of the masked part. The model is trained by calculating the difference between the prediction result and the real image.

[0088] Preferably, the aforementioned comprehensive loss is obtained by weighting the Transformer loss function, the adversarial loss function, and the contrast loss function, and is expressed as:

[0089] ;

[0090] in, is the Transformer loss function; To combat the loss function; is the contrast loss function; , , is the weight coefficient, obtained through training;

[0091] Among them, the Transformer loss function is expressed as:

[0092] ;

[0093] in, is the true label, is the predicted value, is the sample size.

[0094] The adversarial loss function is expressed as:

[0095] ;

[0096] in, is the original image, To generate images.

[0097] The contrast loss function is expressed as:

[0098] ;

[0099] in, and are the feature vectors corresponding to positive samples and negative samples respectively, and D is the distance metric function.

[0100] Furthermore, the visualized intubation auxiliary system of the present embodiment also includes an inertial navigation module, which is arranged at the front end of the intubation catheter probe, including an accelerometer, a magnetometer and a gyroscope; the inertial navigation module is communicatively connected to the host computer via a data transmission line arranged in the intubation catheter, and is used to obtain the posture information of the catheter probe in the catheter probe coordinate system and send it to the host computer; the posture information in the catheter probe coordinate system is converted to the earth coordinate system by the host computer to obtain the catheter probe posture information in the earth coordinate system, and the posture of the image in the earth coordinate system is obtained based on the catheter probe posture information in the earth coordinate system, and sent to the human-computer interaction unit for display, so as to facilitate the tracheal intubation operator to assist in judging the position of the left and right bronchi.

[0101] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0102] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A visual intelligent intubation auxiliary system for automatically identifying the left and right bronchi, characterized in that: The system includes a data acquisition unit, a host computer and a human-computer interaction unit, wherein: The data acquisition unit includes a camera and a position sensor disposed at the front end of the intubation catheter probe, and is used to collect video data and position information of the position of the front end of the intubation catheter; The host computer is used to determine the best observation position based on the video data and position information, and to identify the left and right bronchi by extracting features and classifying and predicting based on the video data and position information collected at the best observation position through a pre-trained left and right bronchial discrimination model; the best observation position is a position where images including the left and right bronchial openings, tracheal carinas and C-shaped cartilage rings can be collected; the left and right bronchial discrimination model includes an encoder module, a memory module, a reconstruction adversarial learning module, a contrast learning module and a decoder module; The encoder module uses an encoder based on the Transformer model to extract features from the received data and obtain a high-dimensional feature representation of the input data; The memory module is arranged after the encoder, and is used to capture and memorize the key features of the high-dimensional feature representation of the multiple frames of images corresponding to the best observation position video data; The reconstruction adversarial learning module and the contrastive learning module are arranged in parallel after the memory module, and are used to enhance and optimize the extracted key features by means of generative adversarial and contrastive learning methods respectively; The decoder module adopts the decoder structure of the Transformer model to decode and classify the enhanced and optimized feature representation; The training of the left and right bronchus discrimination model includes: Acquire continuous multiple frames of image data of optimal observation positions during intubation of multiple patients, image data of non-optimal observation positions, and position information data corresponding to each image data; Label the target of interest in the image data at the best observation position with a category label; The acquired non-optimal observation position image is used as a negative sample, and the optimal observation position image with the category label is used as a positive sample, and data fusion is performed with the corresponding position information data to construct a second training sample set; the positive sample data after data fusion in the second training sample set includes all objects of interest, the category labels and position information corresponding to all objects of interest; The left and right bronchus discrimination model is trained using the second training sample set, and a converged left and right bronchus discrimination model is obtained through iterative updating of the comprehensive loss; during the training process, the left and right bronchus discrimination model learns the corresponding relationship between each target of interest, the category label corresponding to the target of interest, and the position information, so as to identify the left and right bronchi; The comprehensive loss is obtained by weighting the Transformer loss function, the adversarial loss function, and the contrast loss function, and is expressed as: ； in, is the Transformer loss function; To combat the loss function; is the contrast loss function; α, β, γ are weight coefficients; The human-computer interaction unit is used to display an image of the location of the front end of the intubation catheter probe and the left and right bronchial identification results to assist the tracheal intubation operation.

2. The visual intelligent intubation auxiliary system for automatically identifying the left and right bronchi according to claim 1 is characterized in that: The system also includes an inertial navigation module, which is arranged at the front end of the intubation catheter probe and includes an accelerometer, a magnetometer and a gyroscope; the inertial navigation module is communicatively connected with the host computer via a data transmission line arranged in the intubation catheter, and is used to obtain the posture information of the catheter probe in the catheter probe coordinate system and send it to the host computer; the posture information in the catheter probe coordinate system is converted to the earth coordinate system by the host computer to obtain the catheter probe posture information in the earth coordinate system, and the posture of the image in the earth coordinate system is obtained based on the catheter probe posture information in the earth coordinate system, and sent to the human-computer interaction unit for display.

3. The visual intelligent intubation auxiliary system for automatically identifying the left and right bronchi according to claim 1 is characterized in that: The host computer uses a pre-trained optimal position recognition model to make real-time determination of the optimal observation position during the intubation process; The training of the optimal position recognition model includes: Acquire video data and position information data during the intubation process of multiple patients, extract multiple frames of images based on the video data, annotate the extracted images containing all objects of interest with optimal observation position image labels, and annotate the images lacking at least one object of interest with non-optimal observation position image labels; The labeled images are respectively fused with the corresponding position information data to construct a first training sample set; The CNN model is trained using the first training sample set, and an optimal location recognition model is obtained through iterative updating.

4. The visual intelligent intubation auxiliary system for automatically identifying the left and right bronchi according to claim 3 is characterized in that: The targets of interest include the tracheal carina, C-shaped cartilage rings, left bronchial opening, and right bronchial opening.

5. The visual intelligent intubation auxiliary system for automatically identifying the left and right bronchi according to claim 3 is characterized in that: The position information data is point cloud data including horizontal angle, vertical angle and distance from the obstacle to the camera; The step of fusing the labeled images with the corresponding position information data includes: According to the pre-calibrated relative position and posture of the position sensor and camera, the point cloud data acquired by the sensor is preliminarily aligned to the image coordinate system; Use the ICP algorithm to accurately align the initially aligned point cloud data and image feature points; Projecting the precisely aligned point cloud data into an image coordinate system; The projected point cloud data is matched and fused with the image feature points through SIFT or ORB descriptors to obtain the fused data.

6. The visual intelligent intubation auxiliary system for automatically identifying the left and right bronchi according to claim 1, characterized in that: The memory module captures and memorizes the key features of the high-dimensional feature representation of the continuous multi-frame images corresponding to the best observation position video data by the following method: S1: Randomly initialize the memory matrix M of the memory module; S2: Generate a query vector q based on the high-dimensional feature representation corresponding to the current image; S3: Calculate the similarity between the query vector corresponding to the current image and the memory matrix in the memory unit; S4: obtaining memory information content according to the similarity; performing residual connection between the memory information content and the high-dimensional feature representation corresponding to the current image as a residual module; and updating the memory matrix using the memory information obtained after the residual connection; S5: receiving the next frame of image, repeating steps S2-S4, and obtaining the key features of the high-dimensional feature representation through multiple iterative updates of multiple frames of image.

7. The visual intelligent intubation auxiliary system for automatically identifying the left and right bronchi according to claim 1 is characterized in that: The reconstruction adversarial learning module includes a generator and a discriminator; The generator is constructed based on a convolutional neural network or a deconvolutional neural network, and is used to generate multiple realistic images based on the key features output by the memory module; The discriminator is constructed based on a convolutional neural network and is used to distinguish generated images from real images; the generator and the discriminator are iteratively trained through an adversarial loss function to optimize the images generated by the generator.

8. The visual intelligent intubation auxiliary system for automatically identifying the left and right bronchi according to claim 1, characterized in that: The contrastive learning module calculates the feature similarity between the positive and negative samples and the input best observation position samples containing the left and right bronchial opening features based on the key feature representations corresponding to the positive and negative samples output by the memory module; it performs iterative optimization through the contrast loss function to minimize the feature distance between the positive sample and the input best observation position sample, and maximize the feature distance between the negative sample and the input best observation position sample, so as to enhance the features of the positive sample.

Images