A method for constructing a left-right bronchus discrimination model

Abstract

The present invention provides a method for constructing a left and right bronchus discrimination model, comprising: constructing an initial left and right bronchus discrimination model including a data fusion module, an optimal observation position identification module and a left and right bronchus opening identification module; the left and right bronchus discrimination model is used to fuse data based on the image and position information of the optimal observation position obtained by the optimal observation position identification, and identify the left and right bronchus openings; obtaining the catheter probe front end images and position information of multiple optimal observation positions and non-optimal observation positions during the tracheal intubation of multiple patients, constructing a training sample set, and training the left and right bronchus discrimination model, and obtaining a converged left and right bronchus discrimination model through iterative updating of the loss function. The present invention solves the problem that the bronchial intubation technology in the prior art relies on the subjective judgment of the doctor, which easily leads to the wrong identification of the left and right bronchi.

Description

Technical Field

[0001] The invention belongs to the technical field of clinical medicine, and in particular relates to a method for constructing a left-right bronchus discrimination model. Background Art

[0002] Endotracheal intubation technology is a commonly used technical means to achieve single-lung ventilation and lung isolation in clinical medicine. The key to the successful implementation of this technology lies in accurately distinguishing the left and right main bronchi. At present, the commonly used methods of endotracheal intubation in clinical practice include double-lumen endotracheal tube combined with fiberoptic bronchoscope positioning, visual double-lumen endotracheal tube and single-lumen endotracheal tube combined with occluder. The identification of the left and right main bronchi is mainly confirmed by physicians through traditional methods such as identification of the patient's airway anatomical landmarks under visual tools and auscultation. Such methods are subjective and have a certain error rate. Moreover, due to the similar opening characteristics of the left and right bronchi, coupled with the influence of catheter shaping and rotation during intubation, even with the assistance of visual tools, it is sometimes difficult to accurately distinguish the left and right main bronchi, and the left and right main bronchi may be misidentified. Repeated operations in the airway can cause a series of problems such as airway damage and intubation failure, and even threaten the patient's life safety. Therefore, with the assistance of artificial intelligence technology, it has become an urgent problem to realize an intelligent diagnosis and treatment system that combines online and offline endotracheal intubation. Summary of the invention

[0003] In view of the above analysis, the present invention aims to provide a method for constructing a left and right bronchus discrimination model to solve the problem that the bronchial intubation technology in the prior art relies on the subjective judgment of the doctor, which easily leads to the wrong discrimination of the left and right bronchus.

[0004] The purpose of the present invention is mainly achieved through the following technical solutions:

[0005] The present invention provides a method for constructing a left-right bronchus discrimination model, comprising: constructing an initial left-right bronchus discrimination model; the left-right bronchus discrimination model comprises a data fusion module, an optimal observation position recognition module and a left-right bronchus opening recognition module; the data fusion module is used to perform data fusion on images collected during bronchial intubation and corresponding position information; the left-right bronchus opening recognition module is used to perform left-right bronchus opening recognition based on the image of the optimal observation position obtained by the optimal observation position recognition module and the position information fusion data;

[0006] Acquire catheter probe front end images and position information at multiple optimal observation positions and non-optimal observation positions during endotracheal intubation of multiple patients, and construct a first training sample set and a second training sample set respectively;

[0007] Based on the first training sample set and the second training sample set, the corresponding optimal observation position recognition module and the left and right bronchial opening recognition module are trained respectively, and a converged left and right bronchial discrimination model is obtained through iterative updating of the loss function.

[0008] Further, the construction of the first training sample set includes: acquiring video data and position information data of the intubation process of multiple patients, and extracting multiple frames of images based on the video data, annotating the extracted images containing all the objects of interest with the best observation position image label, annotating the images lacking at least one object of interest with the non-best observation position image label, and constructing the first training sample set based on the labeled multiple images and the corresponding position information; the objects of interest include tracheal carina, C-shaped cartilage ring, left bronchial opening and right bronchial opening;

[0009] The construction of the second training sample set includes: using Mask R-CNN and U-Net models to segment the target of interest in the sampled best observation position images respectively; using labeling tools to label the target category of interest in the image after target segmentation; using the best observation position image after category labeling as a positive sample and the non-best observation position image as a negative sample to construct the second training sample set.

[0010] Furthermore, the optimal observation position recognition module includes a convolutional layer, a pooling layer and a fully connected layer;

[0011] The convolution layer is used to extract features from input data;

[0012] The pooling layer is used to reduce the dimension of the extracted feature data by a pooling method to reduce the amount of calculation;

[0013] The fully connected layer is used to perform linear transformation through weights and biases based on the feature data output by the convolution layer and the pooling layer, and to perform nonlinear transformation through an activation function to obtain a classification result of whether the input image is an optimal observation position image or a non-optimal observation position image.

[0014] Furthermore, the left and right bronchial opening recognition module includes an encoder module, a memory module, a reconstruction adversarial learning module, a contrastive learning module and a decoder module;

[0015] The encoder module uses an encoder based on the Transformer model to extract features from the received data and obtain a high-dimensional feature representation of the input data;

[0016] The memory module is arranged after the encoder, and is used to capture and memorize the key features of the high-dimensional feature representation of the multiple frames of images corresponding to the best observation position video data;

[0017] The reconstruction adversarial learning module and the contrastive learning module are arranged in parallel after the memory module, and are used to enhance and optimize the extracted key features by means of generative adversarial and contrastive learning methods respectively;

[0018] The decoder module adopts the decoder structure of the Transformer model to decode and classify the enhanced and optimized feature representation.

[0019] Furthermore, the left and right bronchial opening recognition module is iteratively trained using a comprehensive loss including a Transformer loss function, an adversarial loss function, and a contrast loss function; during the training process, the left and right bronchial opening recognition module learns to obtain the correspondence between each target of interest, the category label corresponding to the target of interest, and the position information, and performs left and right bronchial recognition;

[0020] The Transformer loss function is expressed as:

[0021]

[0022] Among them, y i is the true label, is the predicted value, N is the number of samples;

[0023] The adversarial loss function is expressed as:

[0024]

[0025] Among them, x i is the original image, To generate images;

[0026] The contrast loss function is expressed as:

[0027]

[0028] Among them, z i and are the feature vectors corresponding to positive samples and negative samples respectively, and D is the distance metric function.

[0029] Furthermore, the comprehensive loss is expressed as:

[0030] L total =αL transformer +βL reconstruction +γL self-superυised

[0031] Among them, L transformer is the Transformer loss function; L reconstruction is the adversarial loss function; L self-superυisedis the contrast loss function; α, β, γ are weight coefficients.

[0032] Furthermore, the memory module captures and memorizes the key features of the high-dimensional feature representation of the continuous multiple frames of images corresponding to the best observation position video data by the following method:

[0033] S1: Randomly initialize the memory matrix M of the memory module;

[0034] S2: Generate a query vector q based on the high-dimensional feature representation corresponding to the current image;

[0035] S3: Calculate the similarity between the query vector corresponding to the current image and the memory matrix in the memory unit;

[0036] S4: obtaining memory information content according to the similarity; performing residual connection between the memory information content and the high-dimensional feature representation corresponding to the current image as a residual module; and updating the memory matrix using the memory information obtained after the residual connection;

[0037] S5: receiving the next frame of image, repeating steps S2-S4, and obtaining the key features of the high-dimensional feature representation through multiple iterative updates of multiple frames of image.

[0038] Furthermore, the reconstruction adversarial learning module includes a generator and a discriminator;

[0039] The generator is constructed based on a convolutional neural network or a deconvolutional neural network, and is used to generate multiple realistic images based on the key features output by the memory module;

[0040] The discriminator is constructed based on a convolutional neural network and is used to distinguish generated images from real images; the generator and the discriminator are iteratively trained through an adversarial loss function to optimize the images generated by the generator.

[0041] Furthermore, the contrastive learning module calculates the feature similarity between the positive and negative samples and the input best observation position samples containing the left and right bronchial opening features based on the key feature representations corresponding to the positive and negative samples output by the memory module; it performs iterative optimization through the contrast loss function to minimize the feature distance between the positive sample and the input best observation position sample, and maximize the feature distance between the negative sample and the input best observation position sample, so as to enhance the features of the positive sample.

[0042] Furthermore, the position information data is point cloud data including horizontal angle, vertical angle and distance from the obstacle to the camera acquired by the position sensor; the data fusion module fuses the acquired images with the corresponding position information data respectively by the following method:

[0043] According to the pre-calibrated relative position and posture of the position sensor and the camera, the point cloud data obtained by the position sensor is preliminarily aligned to the image coordinate system;

[0044] Use the ICP algorithm to accurately align the initially aligned point cloud data and image feature points;

[0045] Projecting the precisely aligned point cloud data into an image coordinate system;

[0046] The projected point cloud data is matched and fused with the image feature points through SIFT or ORB descriptors to obtain the fused data.

[0047] Beneficial effects of this technical solution:

[0048] 1. The present invention aims to solve the problem that the distinguishing features of the left and right bronchial openings are not obvious, which easily leads to the misjudgment of the left and right bronchi during bronchial intubation. By utilizing the fact that the left and right bronchial openings, tracheal carina and C-shaped cartilage ring can be displayed simultaneously in the image during intubation, in the process of model construction, an image containing not only the features of the left and right bronchial openings but also the best observation positions of the tracheal carina and the C-shaped cartilage ring is used, and the position information of each feature point is integrated. The model not only learns the features of the left and right bronchial openings, but also learns the relative position features of the left and right bronchi and the tracheal carina and the C-shaped cartilage ring. The model uses the tracheal carina and C-shaped cartilage ring with obvious features and the position information of each feature point to assist in positioning the left and right bronchial openings, thereby ensuring the accuracy of the identification of the left and right bronchial openings.

[0049] 2. The left and right bronchial opening recognition module adopted by the present invention is trained with images collected at the optimal observation position, and on the basis of the traditional Transformer architecture, a memory module is integrated to dynamically update the key features of the extracted image, and the quality of feature extraction of the left and right bronchial opening recognition module and the accuracy of left and right bronchial opening recognition are optimized through reconstruction of adversarial learning and contrastive learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;

[0051] Figure 1 is a flow chart of a method for constructing a left and right bronchus discrimination model according to an embodiment of the present invention;

[0052] Figure 2 is a schematic diagram of an image including all objects of interest collected at the best observation position of an embodiment of the present invention;

[0053] Figure 34 is a structural diagram of a left and right bronchial opening identification module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the implementation cases of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0055] One embodiment of the present invention provides a method for constructing a left-right bronchus discrimination model, such as Figure 1 As shown, the following steps are included:

[0056] Step S1: constructing an initial left and right bronchial discrimination model; the left and right bronchial discrimination model comprises a data fusion module, an optimal observation position recognition module and a left and right bronchial opening recognition module; the data fusion module is used to perform data fusion on the images and corresponding position information collected during the bronchial intubation process; the left and right bronchial opening recognition module is used to perform left and right bronchial opening recognition based on the image and position information fusion data of the optimal observation position obtained by the optimal observation position recognition module;

[0057] Specifically, in order to improve the accuracy of left and right bronchus identification, the left and right bronchus discrimination model constructed in this embodiment is based on the image data and position information sampled during the bronchial intubation process. The optimal observation position is first identified. After the optimal observation position is identified, the image data and position information are sampled using the optimal observation position. The left and right bronchial openings are identified through the pre-trained left and right bronchial opening recognition modules, and the tracheal carina and C-shaped cartilage ring are simultaneously identified. The identified left and right bronchial openings, tracheal carina and C-shaped cartilage ring are sent to the human-computer interaction unit for display, so that the tracheal intubation operator can determine the patient's ventral and dorsal sides according to the position of the tracheal carina and the C-shaped cartilage ring, and further determine the position of the left and right bronchi, so as to further confirm the identified left and right bronchial opening positions, ensure the accuracy of left and right bronchial intubation, and avoid misoperation.

[0058] Preferably, in order to enable the model to learn the relative positions between the objects of interest, the left and right bronchus discrimination model of this embodiment first performs data fusion on the image data and the corresponding position information data through a fusion module. Specifically, the position information data is point cloud data including horizontal angle, vertical angle and distance from the obstacle to the camera acquired by a position sensor (such as an ultrasonic sensor); the data fusion module performs data fusion on the acquired images and the corresponding position information data respectively through the following method:

[0059] According to the pre-calibrated relative position and posture of the position sensor and the camera, the point cloud data obtained by the position sensor is preliminarily aligned to the image coordinate system;

[0060] Use the ICP algorithm to accurately align the initially aligned point cloud data and image feature points;

[0061] Projecting the precisely aligned point cloud data into an image coordinate system;

[0062] The projected point cloud data is matched and fused with the image feature points through SIFT or ORB descriptors to obtain the fused data.

[0063] During the endotracheal intubation process, the optimal observation position is first determined in real time based on the fusion data of image and position information. That is, during the intubation process, the image and position information of the front end of the endotracheal intubation probe acquired in real time by the camera and position sensor are used to determine in real time whether the current position is the optimal observation position through the optimal observation position recognition module. The optimal observation position is the position where the image and position information of the front end of the endotracheal intubation probe can be acquired. Figure 2 As shown, the positions of the left and right bronchial openings, tracheal carina and C-shaped cartilage ring images are observed at the same time, and the left and right bronchi are identified by using continuous multi-frame images extracted from the video collected at the best observation position. The positional relationship between the tracheal carina and C-shaped cartilage ring and the left and right bronchial openings is used to assist in judging the left and right bronchial openings, avoiding the problem of misjudgment of the left and right bronchus due to the similar opening features of the left and right bronchus.

[0064] More specifically, the optimal observation position recognition module used in this embodiment includes a convolutional layer, a pooling layer, and a fully connected layer;

[0065] The convolution layer is used to extract features from input data;

[0066] The pooling layer is used to reduce the dimension of the extracted feature data by a pooling method to reduce the amount of calculation;

[0067] The fully connected layer is used to perform linear transformation through weights and biases based on the feature data output by the convolution layer and the pooling layer, and to perform nonlinear transformation through an activation function to obtain a classification result of whether the input image is an optimal observation position image or a non-optimal observation position image.

[0068] It should be noted that the present embodiment first identifies the best observation position, and uses the image collected at the best observation position to determine the left and right bronchi, so as to ensure that all targets of interest are included in the image used for determining the left and right bronchi, and use the features and position information of each target of interest to accurately determine the left and right bronchi; and the corresponding position information data is fused to the image data in the training data, and the relative distance between each feature point and the front end of the probe is learned during the training of the best position recognition module, and the accuracy of the best observation position judgment is improved by combining the features of the best observation position image. In the subsequent training of the left and right bronchial opening recognition module, the relative distance and azimuth angle between each feature point are learned, and the auxiliary judgment of the left and right bronchial openings with similar features based on the relative position of each feature point is realized, which solves the problem of easy misjudgment of the left and right bronchial openings in the existing method.

[0069] After the best observation position is identified by the best observation position recognition module, the video data of the best observation position is collected, and continuous frame sampling is performed and sent to the left and right bronchial opening recognition module for left and right bronchial opening recognition.

[0070] Specifically, the left and right bronchial opening recognition module includes an encoder module, a memory module, a reconstruction adversarial learning module, a contrastive learning module and a decoder module;

[0071] The encoder module uses an encoder based on the Transformer model to extract features from the received data and obtain a high-dimensional feature representation of the input data;

[0072] The memory module is arranged after the encoder, and is used to capture and memorize the key features of the high-dimensional feature representation of the multiple frames of images corresponding to the best observation position video data;

[0073] The reconstruction adversarial learning module and the contrastive learning module are arranged in parallel after the memory module, and are used to enhance and optimize the extracted key features by means of generative adversarial and contrastive learning methods respectively;

[0074] The decoder module adopts the decoder structure of the Transformer model to decode and classify the enhanced and optimized feature representation.

[0075] The memory module captures and memorizes the key features of the high-dimensional feature representation of the continuous multiple frames of images corresponding to the video data at the best observation position by the following method:

[0076] S1: Randomly initialize the memory matrix M of the memory module;

[0077] S2: Generate a query vector q based on the high-dimensional feature representation corresponding to the current image;

[0078] S3: Calculate the similarity between the query vector corresponding to the current image and the memory matrix in the memory unit;

[0079] S4: obtaining memory information content according to the similarity; performing residual connection between the memory information content and the high-dimensional feature representation corresponding to the current image as a residual module; and updating the memory matrix using the memory information obtained after the residual connection;

[0080] S5: receiving the next frame of image, repeating steps S2-S4, and obtaining the key features of the high-dimensional feature representation through multiple iterative updates of multiple frames of image.

[0081] Furthermore, the reconstruction adversarial learning module includes a generator and a discriminator;

[0082] The generator is constructed based on a convolutional neural network or a deconvolutional neural network, and is used to generate multiple realistic images based on the key features output by the memory module;

[0083] The discriminator is constructed based on a convolutional neural network and is used to distinguish generated images from real images; the generator and the discriminator are iteratively trained through an adversarial loss function to optimize the images generated by the generator.

[0084] Preferably, the contrastive learning module is used to calculate the similarity between the key feature representations corresponding to the positive and negative samples output by the memory module and the original positive samples; iterative optimization is performed through the contrast loss function to minimize the feature distance between the key features corresponding to the positive samples and the original positive samples, and to maximize the feature distance between the negative samples and the original positive samples, so as to enhance the features of the positive samples.

[0085] The left and right bronchial opening recognition module of this embodiment integrates a memory module based on the traditional Transformer architecture, dynamically updates the key features of the extracted image, and optimizes the quality of feature extraction of the left and right bronchial opening recognition module and the accuracy of left and right bronchial opening recognition through reconstruction of adversarial learning and contrastive learning.

[0086] Step S2: acquiring images and position information of the front end of the catheter probe at multiple optimal observation positions and non-optimal observation positions during the endotracheal intubation of multiple patients, and constructing a first training sample set and a second training sample set respectively;

[0087] Specifically, the construction of the first training sample set includes: acquiring video data and position information data of multiple patients during intubation, and extracting multiple frames of images based on the video data, annotating the extracted images containing all objects of interest with optimal observation position image labels, annotating the images lacking at least one object of interest with non-optimal observation position image labels, and constructing the first training sample set based on the labeled multiple images and corresponding position information; the objects of interest include tracheal carina, C-shaped cartilage ring, left bronchial opening, and right bronchial opening;

[0088] The construction of the second training sample set includes: using Mask R-CNN and U-Net models to segment the target of interest in the sampled best observation position images respectively; using labeling tools to label the target category of interest in the image after target segmentation; using the best observation position image after category labeling as a positive sample and the non-best observation position image as a negative sample to construct the second training sample set.

[0089] Preferably, the video data and position information of the position of the front end of the intubation catheter can be collected by a camera and a position sensor arranged at the front end of the intubation catheter probe;

[0090] After acquiring the video data and position information data, multiple frames of images are extracted based on the video data and preprocessed; that is, the received video data is divided into frames and frame sampling is performed to reduce data redundancy and ensure coverage of key scenes; the preprocessing shown includes using convolution filters or other denoising algorithms to remove noise in the image, and applying data enhancement techniques such as rotation, translation, scaling, contrast adjustment, etc. to generate more diverse training data, and normalize the image pixel values ​​to the [0, 1] interval to ensure stable model training. Label the preprocessed images, that is, label the best observation position image labels for images containing all objects of interest, and label the non-best observation position image labels for images lacking at least one object of interest, and construct a first training sample set in combination with the position information data corresponding to the image. The best observation position recognition module is trained using the first training sample set to learn the ability to judge whether the input image is the best observation position image or the non-best observation position image.

[0091] When constructing the second training sample set, firstly, the target of interest in the image data of the best observation position is labeled with category labels, including: using the existing deep learning model for preliminary automatic labeling, such as using the Mask R-CNN or U-Net model for target detection and segmentation. The specific labeling process can use two image segmentation models, that is, using the Mask R-CNN model and the U-Net model for image segmentation respectively, for model training. After the target detection and segmentation of the image, the specific positions of the left and right bronchial openings, tracheal carinas and C-shaped cartilage rings are labeled frame by frame using a labeling tool (such as LabelImg). After the labeling is completed, the automatic labeling results can be manually corrected to ensure the accuracy of the labeling; finally, the best observation position image after category labeling is used as a positive sample, and the non-best observation position image is used as a negative sample to obtain the second training sample set; the left and right bronchial opening recognition module is trained using the second training sample set to learn the ability to accurately judge the characteristics of the left and right bronchial openings. That is, the positive sample in the second training sample set is an image containing all the targets of interest obtained at the best observation position, and after image segmentation and annotation, each target of interest is labeled with a category label, wherein the category labels include the left bronchial opening, the right bronchial opening, the tracheal carina, and the C-shaped cartilage ring. In practical applications, the left and right bronchial opening recognition modules trained with the second training sample set can identify the left bronchial opening, the right bronchial opening, the tracheal carina, and the C-shaped cartilage ring based on the image of the best observation position acquired in real time, and annotate the target recognition results in the image, which is displayed through the human-computer interaction module to assist the bronchial intubation operation and improve the accuracy of bronchial intubation.

[0092] Step S3: Based on the first training sample set and the second training sample set, the corresponding optimal observation position recognition module and the left and right bronchial opening recognition module are trained respectively, and a converged left and right bronchial discrimination model is obtained through iterative updating of the loss function.

[0093] Specifically, before using the first training sample set and the second training sample set for model training respectively, the image data and the position information data are first fused through the data fusion layer. Using the data of the first training sample set, the image data is fused with the corresponding position information data. During the training process of the optimal position recognition module, the relative distance between each feature point and the front end of the probe is learned, and the accuracy of the optimal observation position judgment is improved by combining the features of the optimal observation position image. After using the data in the second training sample set for data fusion, when training the left and right bronchial opening recognition module, the relative distance and azimuth angle between each feature point are learned, and the auxiliary judgment of the left and right bronchial openings with similar features based on the relative position of each feature point is realized, which solves the problem of easy misjudgment of the left and right bronchial openings in the existing method.

[0094] Furthermore, the left and right bronchial opening recognition module is iteratively trained using a comprehensive loss including a Transformer loss function, an adversarial loss function, and a contrast loss function; the left and right bronchial opening recognition module learns to obtain the corresponding relationship between each object of interest, the category label corresponding to the object of interest, and the position information during the training process, and performs left and right bronchial recognition;

[0095] The Transformer loss function is expressed as:

[0096]

[0097] Among them, y i is the true label, is the predicted value, N is the number of samples;

[0098] The adversarial loss function is expressed as:

[0099]

[0100] Among them, x i is the original image, To generate images;

[0101] The contrast loss function is expressed as:

[0102]

[0103] Among them, z i and are the feature vectors corresponding to positive samples and negative samples respectively, and D is the distance metric function.

[0104] L total =αL transformer +βL reconstruction +γL self-superυised

[0105] The comprehensive loss is expressed as:

[0106] Among them, L transformer is the Transformer loss function; L reconstruction is the adversarial loss function; L self-superυised is the contrast loss function; α, β, γ are weight coefficients, which can be obtained through training.

[0107] The left and right bronchial opening recognition modules are trained using the second training sample set, and after iterative updating with comprehensive loss, a converged left and right bronchial opening recognition module is obtained; during the training process, the left and right bronchial opening recognition module learns the corresponding relationship between each object of interest, the category label corresponding to the object of interest, and the position information, and performs left and right bronchi recognition.

[0108] During the training of the left and right bronchial opening recognition module, in order to enhance the recognition ability of the model, the model can also be enhanced through the masking strategy, that is, randomly masking parts of the image to force the model to predict the masked parts through the remaining information. A random mask is applied to the image to generate a masked input image. The model receives the masked image and outputs the prediction result of the masked part. The model is trained by calculating the difference between the prediction result and the real image.

[0109] In summary, the present invention aims at the problem that the distinguishing features of the left and right bronchial openings are not obvious, which easily leads to the misjudgment of the left and right bronchi during bronchial intubation. By utilizing the fact that the left and right bronchial openings, tracheal carina and C-shaped cartilage ring can be displayed simultaneously in the image during intubation, in the process of model construction, an image that not only contains the features of the left and right bronchial openings, but also contains the best observation positions of the tracheal carina and the C-shaped cartilage ring is used, and the position information of each feature point is integrated, so that the model not only learns the features of the left and right bronchial openings, but also learns the relative position features of the left and right bronchi and the tracheal carina and the C-shaped cartilage ring, and realizes the auxiliary positioning of the left and right bronchial openings by utilizing the feature-clear tracheal carina and C-shaped cartilage ring and the position information of each feature point, thereby ensuring the accuracy of the identification of the left and right bronchial openings.

[0110] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0111] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for constructing a left and right bronchus discrimination model, characterized in that: include: Constructing an initial left and right bronchial discrimination model; the left and right bronchial discrimination model includes a data fusion module, an optimal observation position identification module and a left and right bronchial opening identification module; the data fusion module is used to perform data fusion on the images and corresponding position information collected during the bronchial intubation process; the left and right bronchial opening identification module is used to fuse data based on the image and position information of the optimal observation position obtained by the optimal observation position identification module to identify the left and right bronchial openings; the left and right bronchial opening identification module includes an encoder module, a memory module, a reconstruction adversarial learning module, a contrastive learning module and a decoder module; the encoder module adopts an encoder based on the Transformer model to extract features from the received data to obtain a high-dimensional feature representation of the input data; the memory module is arranged after the encoder to capture and memorize the key features of the high-dimensional feature representation of multiple frames of images corresponding to the optimal observation position video data; the reconstruction adversarial learning module and the contrastive learning module are arranged in parallel after the memory module, and are used to enhance and optimize the extracted key features by generating adversarial and contrastive learning methods respectively; the decoder module adopts the decoder structure of the Transformer model to decode and classify and predict the enhanced and optimized feature representation; Acquire catheter probe front end images and position information at multiple optimal observation positions and non-optimal observation positions during endotracheal intubation of multiple patients, and construct a first training sample set and a second training sample set respectively; Based on the first training sample set and the second training sample set, the corresponding optimal observation position recognition module and the left and right bronchial opening recognition module are trained respectively, and a converged left and right bronchial discrimination model is obtained through iterative updating of the loss function.

2. The method for constructing a left and right bronchus discrimination model according to claim 1, characterized in that: The construction of the first training sample set includes: acquiring video data and position information data of multiple patients during intubation, extracting multiple frames of images based on the video data, annotating the extracted images containing all objects of interest with optimal observation position image labels, annotating the images lacking at least one object of interest with non-optimal observation position image labels, and constructing the first training sample set based on the labeled multiple images and corresponding position information; the objects of interest include tracheal carina, C-shaped cartilage ring, left bronchial opening and right bronchial opening; The construction of the second training sample set includes: using Mask R-CNN and U-Net models to segment the target of interest in the sampled best observation position images respectively; using labeling tools to label the target category of interest in the image after target segmentation; using the best observation position image after category labeling as a positive sample and the non-best observation position image as a negative sample to construct the second training sample set.

3. The method for constructing a left and right bronchus discrimination model according to claim 1, characterized in that: The optimal observation position recognition module includes a convolution layer, a pooling layer and a fully connected layer; The convolution layer is used to extract features from input data; The pooling layer is used to reduce the dimension of the extracted feature data by a pooling method to reduce the amount of calculation; The fully connected layer is used to perform linear transformation through weights and biases based on the feature data output by the convolution layer and the pooling layer, and to perform nonlinear transformation through an activation function to obtain a classification result of whether the input image is an optimal observation position image or a non-optimal observation position image.

4. The method for constructing a left and right bronchus discrimination model according to claim 3, characterized in that: The left and right bronchial opening recognition module is iteratively trained using a comprehensive loss including a Transformer loss function, an adversarial loss function, and a contrast loss function; during the training process, the left and right bronchial opening recognition module learns the corresponding relationship between each target of interest, the category label corresponding to the target of interest, and the position information, and performs left and right bronchial recognition; The Transformer loss function is expressed as: Among them, y i is the true label, is the predicted value, N is the number of samples; The adversarial loss function is expressed as: Among them, x i is the original image, To generate images; The contrast loss function is expressed as: Among them, z i and are the feature vectors corresponding to positive samples and negative samples respectively, and D is the distance metric function.

5. The method for constructing a left and right bronchus discrimination model according to claim 4, wherein L total =αL transformer +βL reconstruction +γL self-superuised In other words, the comprehensive loss is expressed as: Among them, L transformer is the Transformer loss function; L reconstruction is the adversarial loss function; L self-supervised is the contrast loss function; α, β, γ are weight coefficients.

6. The method for constructing a left and right bronchus discrimination model according to claim 3, characterized in that: The memory module captures and memorizes the key features of the high-dimensional feature representation of the continuous multi-frame images corresponding to the best observation position video data by the following method: S1: Randomly initialize the memory matrix M of the memory module; S2: Generate a query vector q based on the high-dimensional feature representation corresponding to the current image; S3: Calculate the similarity between the query vector corresponding to the current image and the memory matrix in the memory unit; S4: obtaining memory information content according to the similarity; performing residual connection between the memory information content and the high-dimensional feature representation corresponding to the current image as a residual module; and updating the memory matrix using the memory information obtained after the residual connection; S5: receiving the next frame of image, repeating steps S2-S4, and obtaining the key features of the high-dimensional feature representation through multiple iterative updates of multiple frames of image.

7. The method for constructing a left and right bronchus discrimination model according to claim 3, characterized in that: The reconstruction adversarial learning module includes a generator and a discriminator; The generator is constructed based on a convolutional neural network or a deconvolutional neural network, and is used to generate multiple realistic images based on the key features output by the memory module; The discriminator is constructed based on a convolutional neural network and is used to distinguish generated images from real images; the generator and the discriminator are iteratively trained through an adversarial loss function to optimize the images generated by the generator.

8. The method for constructing a left and right bronchus discrimination model according to claim 3, characterized in that: The contrastive learning module calculates the feature similarity between the positive and negative samples and the input best observation position samples containing the left and right bronchial opening features based on the key feature representations corresponding to the positive and negative samples output by the memory module; it performs iterative optimization through the contrast loss function to minimize the feature distance between the positive sample and the input best observation position sample, and maximize the feature distance between the negative sample and the input best observation position sample, so as to enhance the features of the positive sample.

9. The method for constructing a left and right bronchus discrimination model according to claim 2, characterized in that: The position information data is point cloud data including horizontal angle, vertical angle and distance from the obstacle to the camera acquired by the position sensor; the data fusion module fuses the acquired images with the corresponding position information data respectively by the following method: According to the pre-calibrated relative position and posture of the position sensor and the camera, the point cloud data obtained by the position sensor is preliminarily aligned to the image coordinate system; Use the ICP algorithm to accurately align the initially aligned point cloud data and image feature points; Projecting the precisely aligned point cloud data into an image coordinate system; The projected point cloud data is matched and fused with the image feature points through SIFT or ORB descriptors to obtain the fused data.

Images