Combined narrow-band and fluorescence imaging method and system for bronchoscopic lesion detection

Abstract

The present application relates to the technical field of medical image processing, and particularly relates to a combined narrowband and fluorescence imaging method and system for bronchoscopy lesion detection. The method provided by the present application realizes selection and extraction of features with discriminative ability in narrowband imaging images and autofluorescence imaging images based on deep learning, and realizes fusion of advantages of different imaging technologies for lesion detection through decision fusion, thereby improving the speed and accuracy of lesion recognition. Meanwhile, the method provided by the present application can better adapt to different scenes and lesion characteristics, has better generalization ability, improves the specificity and sensitivity of lesion detection, and is beneficial to improving the diagnostic accuracy and efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, and particularly relates to a combined narrowband and fluorescence imaging method and system for bronchoscopy lesion detection. BACKGROUND

[0002] Lung cancer is one of the most common cancer types worldwide and is also the leading cause of cancer-related deaths. The survival time of lung cancer patients is highly related to the stage of the disease and the size of the tumor at the time of diagnosis, and early diagnosis is the most important step in the diagnosis and treatment of lung cancer. Current lung cancer screening mainly relies on low-dose chest CT and chest X-ray. The characteristics of chest CT and chest X-ray examination determine that the current chest imaging technology is more sensitive to peripheral lung cancer, and has poor sensitivity for central bronchial lung cancer with squamous cell carcinoma and small cell lung cancer as the main pathological type, and tobacco exposure leads to central lung cancer with squamous cell carcinoma as the main pathological type. In order to improve the early detection rate of central lung cancer, the current bronchoscopy technology for diagnosing lung cancer usually uses narrow-band imaging (NBI) and auto fluorescence imaging (AFI) technology.

[0003] Narrow-band imaging technology and auto fluorescence imaging technology are improved based on white light bronchoscopy imaging (WLB), and the purpose is to better help doctors detect early lung cancer conditions. These two technologies are currently widely used for the diagnosis of precancerous lesions and early central lung cancer (ECLC) under bronchoscopy. The results of most clinical trials show that the use of AFI and NBI has potential benefits for the diagnosis of lung cancer, especially for the detection of precancerous lesions.

[0004] NBI narrow band imaging technology, that is, by filter filtering out the ordinary endoscope light source emitted red, blue, green in the wide band spectrum, select 415 nm, 540 nm narrow band light wave. 415 nm blue light wavelength is short, penetrate the superficial mucosa, the absorption peak of hemoglobin to light wave is also located at about 415 nm, thus conducive to display bronchial mucosa surface layer blood vessels. 540 nm green light penetration is strong, and the effect of displaying submucosal blood vessels is good. It can display the fine structure of the mucosal surface and the submucosal blood vessels, which is clearer than the traditional white light mode endoscope and has stronger stereoscopic effect. Through high contrast observation of the mucosal surface, important fine patterns can be provided for the diagnosis of vascular lesions. However, NBI focuses on detecting vascular structure, so it may limit other important information in the general field of view, and the color of the image produced by NBI is special, so the requirements for endoscopic imaging quality are high, and the lesion characteristics are not obvious, so a high-definition endoscope with magnification function is often used to display the mucosal microvessel morphology and glandular tube morphology more clearly and accurately judge the nature of the lesion, which makes the doctor spend extra time and effort to observe the lesion, and the doctor using NBI for diagnosis needs to be trained and practiced to correctly interpret the NBI image.

[0005] AFI autofluorescence bronchoscopy technology is a new type of bronchoscope developed by combining cell autofluorescence and computer image analysis technology. By observing the fluorescence changes of the tracheal and bronchial mucosa, it can determine whether there is abnormal lesion, and according to the display of the fluorescence image, the mucosa of the abnormal fluorescence part can be brushed or biopsied, which increases the detection rate of early lung cancer, especially precancerous lesions or carcinoma in situ and local mucosal invasive carcinoma. However, due to its low specificity, it is easy to confuse normal bronchial mucosal tissue with abnormal tissue during endoscopic observation, and it is difficult to distinguish the part that really needs pathological tissue.

[0006] The combined use of the two imaging techniques can improve sensitivity and specificity. The results of most clinical trials show that the use of AFI and NBI has potential benefits for the diagnosis of lung cancer, but AFI and NBI use different bronchoscopes, which requires the use of two different bronchoscopes on the same patient, increasing the examination time and requiring at least two times of entering and exiting the glottis. This is the main reason why there are few applications and literatures on the combined use of AFI and NBI for the diagnosis of lung cancer. SUMMARY

[0007] The present application aims to solve the problem that the existing lesion detection method is difficult to combine different imaging techniques to improve the detection accuracy.

[0008] To solve the above technical problems, in a first aspect, the present application provides a combined narrow band and fluorescence imaging method for bronchoscope lesion detection, comprising the following steps:

[0009] Collect white light imaging images, narrow band imaging images and autofluorescence imaging images of a lesion area of a bronchoscope respectively, and apply a preset image enhancement method for image preprocessing respectively;

[0010] The white light imaging image is taken as an input of a preset generative adversarial network model, and narrow band imaging generated images and autofluorescence imaging generated images are output through the preset generative adversarial network model, wherein the preset generative adversarial network model comprises a generator network and a discriminator network;

[0011] A lesion detection model is trained based on the narrow band imaging generated images and the autofluorescence imaging generated images, and the lesion detection model is used for lesion detection on the narrow band imaging images and the autofluorescence imaging images to obtain narrow band imaging detection results and autofluorescence imaging detection results respectively, and a preset weighted fusion method is used for fusing the narrow band imaging detection results and the autofluorescence imaging detection results to obtain lesion detection results.

[0012] Further, the preset image enhancement method comprises at least one of image quality enhancement and image color correction.

[0013] Further, in the preset generative adversarial network model, the probability that the generator network generates z images is defined as G(z), the probability that the discriminator network discriminates a real image is defined as D(x), and the process that the generator network generates images and the discriminator network discriminates images satisfies a value function V(G, D):

[0014]

[0015] wherein:

[0016]

[0017] In the above expression, is an initial data distribution, z is a random vector of Gaussian white noise, is a random initialization noise distribution, θ d is a network parameter of the discriminator network, θ g is a network parameter of the generator network, and m is a sample number.

[0018] Further, the lesion detection model is based on a YOLOv8 model.

[0019] Further, in the step of training a lesion detection model based on the narrow band imaging generated images and the autofluorescence imaging generated images, a classification loss function and a bounding box loss function are set for the lesion detection model, and the classification loss function satisfies:

[0020]

[0021] wherein x i is a current category prediction value, y i is a probability of the current category after an activation function, is a real value of the current category, and N is a sample number;

[0022] The bounding box loss function satisfies:

[0023]

[0024] wherein b i is an actual bounding box, is a predicted bounding box.

[0025] Further, in the step of training the lesion detection model based on the narrowband imaging generated image and the autofluorescence imaging generated image, the classification loss function and the bounding box loss function for the narrowband imaging generated image are defined as and the classification loss function and the bounding box loss function for the autofluorescence imaging generated image are and The joint loss function of the lesion detection model satisfies:

[0026]

[0027] wherein α and β are weight parameters of the narrowband imaging and the autofluorescence imaging respectively, and λ is a weight parameter of the weighted loss.

[0028] Further, the preset weighted fusion method is specifically:

[0029] The narrowband imaging detection result and the autofluorescence imaging detection result are defined as D NBI and D AFI , which satisfy:

[0030] D NBI ={(B i ,C i )∣i=1,2,…,n};

[0031] D AFI ={(B j ,C j )∣j=1,2,…,m};

[0032] wherein B i and B j represent the bounding boxes of the lesion regions in the narrowband imaging detection result and the autofluorescence imaging detection result respectively, and C i and C jconfidence C of the narrow-band imaging detection result and the autofluorescence image detection result NBI , C AFI ;

[0033] define the lesion detection result as satisfying:

[0034] D fused = {(B i , C fused,i ) | i = 1, 2, …, n};

[0035] C fused = alpha * C NBI + beta * C AFI ;

[0036] Wherein, alpha and beta are weight parameters, and alpha + beta = 1.

[0037] In a second aspect, the present application also provides a combined narrow-band and fluorescence imaging system for bronchoscope lesion detection, comprising:

[0038] An imaging module is configured to acquire white light imaging images, narrow-band imaging images and autofluorescence imaging images of a bronchoscope lesion area respectively, and apply a preset image enhancement method for image preprocessing;

[0039] A processing module is configured to input the white light imaging images into a preset generative adversarial network model, and output narrow-band imaging generated images and autofluorescence imaging generated images through the generative adversarial network model, wherein the generative adversarial network model comprises a generator network and a discriminator network.

[0040] A detection module is configured to train a lesion detection model based on the narrow-band imaging generated images and the autofluorescence imaging generated images, and perform lesion detection on the narrow-band imaging images and the autofluorescence imaging images using the lesion detection model, to obtain narrow-band imaging detection results and autofluorescence image detection results respectively, and fuse the narrow-band imaging detection results and the autofluorescence image detection results through a preset weighted fusion method to obtain a lesion detection result.

[0041] The present application has the advantages that a combined narrow-band and fluorescence imaging method for bronchoscope lesion detection is provided, which realizes the selection and extraction of features with discriminative ability in narrow-band imaging images and autofluorescence imaging images based on deep learning, and realizes the fusion of advantages of different imaging technologies through decision fusion, thereby improving the speed and accuracy of lesion recognition. At the same time, the method provided by the present application can better adapt to different scenes and lesion characteristics, has better generalization ability, improves the specificity and sensitivity of lesion detection, and is beneficial to improve the diagnostic accuracy and efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a step flow diagram of a combined narrowband and fluorescence imaging method for bronchoscopy lesion detection provided by an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0044] Please refer to Figure 1 , Figure 1 is a step flow diagram of a combined narrowband and fluorescence imaging method for bronchoscopy lesion detection provided by an embodiment of the present application, the combined narrowband and fluorescence imaging method for bronchoscopy lesion detection comprises the following steps:

[0045] S1, white light imaging images, narrowband imaging images and autofluorescence imaging images of a bronchoscopy lesion area are respectively collected, and a preset image enhancement method is respectively applied for image preprocessing;

[0046] S2, the white light imaging image is taken as an input of a preset generative adversarial network model (GAN), and narrowband imaging generated images and autofluorescence imaging generated images are output through the preset generative adversarial network model, wherein the preset generative adversarial network model comprises a generator network and a discriminator network;

[0047] S3, a lesion detection model is obtained based on the narrowband imaging generated images and the autofluorescence imaging generated images, and the lesion detection model is used for lesion detection on the narrowband imaging images and the autofluorescence imaging images, to respectively obtain narrowband imaging detection results and autofluorescence image detection results, and a preset weighted fusion method is used for fusion of the narrowband imaging detection results and the autofluorescence image detection results, to obtain lesion detection results.

[0048] The preset image enhancement method comprises at least one of image quality enhancement and image color correction.

[0049] Specifically, whether it is narrowband imaging or autofluorescence imaging technology, it has a higher requirement on endoscope imaging quality. When narrowband imaging detection is performed, a high-definition endoscope with magnification function is often used, so that the mucosal microvessel morphology and glandular morphology can be displayed more clearly, and the nature of the lesion can be judged more accurately. In the autofluorescence imaging endoscope mode, normal tissue is light green, inflammation is dark green or light purple, tumor-like lesions are purple or brown, and tumor tissue is magenta or reddish brown.

[0050] Therefore, the imaging quality of the bronchoscope is crucial for detecting and diagnosing lesions. When collecting images using bronchoscopes with different imaging technologies, image enhancement and color correction are required to improve image quality and ensure color accuracy.

[0051] In the embodiments of the present application, the images with narrowband imaging and AFI imaging effects are generated by the generative adversarial network, which can effectively compensate for the difficulty that the two imaging technologies cannot collect images simultaneously in the same bronchoscope. Further, in the preset generative adversarial network model, the probability of the generator network generating z images is defined as G(z), the probability of the discriminator network discriminating real images is defined as D(x), and the process of the generator network generating images and the discriminator network discriminating images satisfies the value function V(G, D):

[0052]

[0053] Wherein:

[0054]

[0055] In the above expression, is the initial data distribution, z is a random vector of Gaussian white noise, is a random initialization noise distribution, θ d is the network parameter of the discriminator network, θ g is the network parameter of the generator network, and m is the sample number.

[0056] The training process is as follows: first, train the discriminator D, and take m samples from the data set and the random noise distribution P data (z), update the network parameters θ z in the discriminator D through the gradient ascent algorithm, and optimize the cost function D(x) of the discriminator, that is, maximize the value function V(G, D); d

[0057] In the second step of training the generator G, take m samples from the random noise distribution P z (z), update the network parameters θ g in the generator G through the gradient descent algorithm, and optimize the cost function D(G(z)) of the generator, that is, minimize the value function V(G, D).

[0058] ​In the whole processing process of the generative adversarial network model, the generator network sets the style transfer imaging parameters, inputs the original white light imaging image into the generator network, respectively generates data samples with narrowband imaging and autofluorescence imaging effect images; and then inputs the narrowband imaging and autofluorescence imaging effect images into the discriminator for iterative adversarial training. The adversarial training process includes alternating training of the generator and the discriminator. In each training, the discriminator network distinguishes between generated images and real imaging images, and the generator network adjusts the generated image parameters so that the imaging effect of the generated image tends to the real image. When the number of iterations of the generative adversarial network model reaches a threshold value, the generated image and the real image cannot be distinguished, and the adversarial learning training is completed.

[0059] The lesion detection model is based on a YOLOv8 model, which is a deep learning-based detection model.

[0060] Each channel of the YOLOv8 model processes corresponding image features. In the first few layers of the YOLOv8 model, narrowband imaging and autofluorescence imaging features are extracted. Then, for the image features of the two imaging technologies, a joint loss function is applied to balance the detection tasks of narrowband imaging and autofluorescence imaging, which can help the model to learn and optimize the features of the two imaging modes at the same time. The joint loss function is usually composed of multiple sub-loss functions, each of which corresponds to a specific task or data source.

[0061] Specifically, in the step of training the lesion detection model based on the narrowband imaging generated image and the autofluorescence imaging generated image, a classification loss function and a bounding box loss function are set for the lesion detection model, the classification loss function satisfies:

[0062]

[0063] where x i is the current class prediction value, y i is the probability of the current class after passing through the activation function, is the true value of the current class, and N is the number of samples;

[0064] The bounding box loss function satisfies:

[0065]

[0066] where b i is the actual bounding box, is the predicted bounding box.

[0067] In the step of training the lesion detection model based on the narrowband imaging generated image and the autofluorescence imaging generated image, the classification loss function and the bounding box loss function for the narrowband imaging generated image are defined as and The classification loss function and the bounding box loss function for generating images for the autofluorescence imaging are and The joint loss function of the lesion detection model satisfies:

[0068]

[0069] Wherein, α and β are weight parameters of narrowband imaging and autofluorescence imaging respectively, and λ is a weight parameter of the weighted loss.

[0070] In the above process, the image data set of narrowband imaging and autofluorescence imaging generated by the generative adversarial network model in step S2 provides data source for YOLOv8 model training, and the lesions in each image are manually labeled using a labeling tool to determine the exact position and category of the lesions. In the training process, the weight parameters α and β need to be dynamically adjusted, and the model parameters are updated according to the loss value. Through the joint loss function, the model can optimize the detection performance of narrowband imaging and autofluorescence imaging images at the same time, and improve the overall detection effect.

[0071] The preset weighted fusion method is specifically:

[0072] Define the narrowband imaging detection result and the autofluorescence image detection result as D NBI and D AFI , which satisfy:

[0073] D NBI ={(B i ,C i )∣i=1,2,…,n};

[0074] D AFI ={(B j ,C j )∣j=1,2,…,m};

[0075] Wherein, B i and B j represent the bounding box of the lesion area in the narrowband imaging detection result and the autofluorescence image detection result respectively, and C i and C j represent the confidence C NBI and C AFI of the narrowband imaging detection result and the autofluorescence image detection result.

[0076] Define the lesion detection result as D fused , which satisfies:

[0077] D fused ={(B i ,c fused,i)|i=1,2,...,n};

[0078] C fused =α·C NBI +β·C AFI ;

[0079] Wherein, α and β are weight parameters, and α+β=1.

[0080] The beneficial effect achieved by the present invention lies in proposing a combined narrow-band and fluorescence imaging method for bronchoscopic lesion detection. The method selects and extracts discriminative features in narrow-band imaging images and autofluorescence imaging images based on deep learning, and integrates the advantages of different imaging technologies for lesion detection through decision fusion, thereby improving the speed and accuracy of lesion identification. At the same time, the method provided by the present invention can better adapt to different scenarios and lesion characteristics, has better generalization ability, improves the specificity and sensitivity of lesion detection, and is conducive to improving diagnostic accuracy and efficiency.

[0081] The present invention also provides a combined narrow-band and fluorescence imaging system for bronchoscopic lesion detection, comprising:

[0082] An imaging module is used to respectively acquire white light imaging images, narrow band imaging images and autofluorescence imaging images of the lesion area of ​​the bronchoscopy, and to perform image preprocessing using preset image enhancement methods;

[0083] a processing module, configured to use the white light imaging image as input to a preset generative adversarial network model, and output a narrowband imaging generated image and an autofluorescence imaging generated image through the preset generative adversarial network model, wherein the preset generative adversarial network model includes a generator network and a discriminator network;

[0084] A detection module is used to train a lesion detection model based on the narrowband imaging-generated image and the autofluorescence imaging-generated image, and use the lesion detection model to perform lesion detection on the narrowband imaging image and the autofluorescence imaging image to obtain narrowband imaging detection results and autofluorescence image detection results, respectively, and fuse the narrowband imaging detection results and the autofluorescence image detection results through a preset weighted fusion method to obtain a lesion detection result.

[0085] The combined narrow-band and fluorescence imaging system for bronchoscopic lesion detection can implement the same steps as the combined narrow-band and fluorescence imaging method for bronchoscopic lesion detection described in the above embodiment when executed, and can achieve the same technical effects. To avoid repetition, they will not be repeated here.

[0086] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer readable storage medium. When the program is executed, the processes of the above-mentioned embodiment methods can be included. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM).

[0087] It should be noted that, in this document, the terms "comprising", "containing" or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or device that includes a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0088] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and a necessary general hardware platform, and of course, they can also be realized by hardware, but in many cases, the former is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disc, an optical disc) and includes a plurality of instructions for making a terminal (which can be a mobile phone, a computer, a server, an air conditioner or a network device) execute the methods described in the various embodiments of the present application.

[0089] The embodiments of the present application are described above in combination with the accompanying drawings, and the disclosed are only the preferred embodiments of the present application, but the present application is not limited to the above-mentioned specific embodiments. The above-mentioned specific embodiments are only illustrative, not restrictive, and those skilled in the art can make many equivalent changes under the inspiration of the present application without departing from the scope of the present application and the protection scope of the claims.

Claims

1. A combined narrow-band and fluorescence imaging method for bronchoscopic lesion detection, characterized in that: The following steps are involved: White light imaging images, narrow band imaging images, and autofluorescence imaging images of the lesion area under bronchoscopy were collected respectively, and the images were preprocessed using the preset image enhancement methods. Using the white light imaging image as input to a preset generative adversarial network model, and outputting a narrowband imaging generated image and an autofluorescence imaging generated image through the preset generative adversarial network model, wherein the preset generative adversarial network model includes a generator network and a discriminator network; A lesion detection model is obtained based on the training of the narrow-band imaging generated image and the autofluorescence imaging generated image, and the lesion detection model is used to perform lesion detection on the narrow-band imaging image and the autofluorescence imaging image to obtain narrow-band imaging detection results and autofluorescence image detection results respectively. The narrow-band imaging detection results and the autofluorescence image detection results are fused by a preset weighted fusion method to obtain a lesion detection result.

2. The combined narrow-band and fluorescence imaging method for bronchoscopic lesion detection according to claim 1, characterized in that: The preset image enhancement method includes at least one of image quality enhancement and image color correction.

3. The combined narrow-band and fluorescence imaging method for bronchoscopic lesion detection according to claim 1, characterized in that: The lesion detection model is the YOLOv8 model.

4. The combined narrow-band and fluorescence imaging method for bronchoscopic lesion detection according to claim 3, characterized in that: In the step of training a lesion detection model based on the narrowband imaging-generated image and the autofluorescence imaging-generated image, a classification loss function and a bounding box loss function are set for the lesion detection model, and the classification loss function satisfies: ； ； in, is the predicted value of the current category, is the probability of the current category after the activation function, is the true value of the current category, N is the number of samples; The bounding box loss function satisfies: ； in, is the actual bounding box, is the predicted bounding box.

5. The combined narrow-band and fluorescence imaging method for bronchoscopic lesion detection according to claim 4, characterized in that: In the step of training a lesion detection model based on the narrowband imaging generated image and the autofluorescence imaging generated image, the classification loss function and the bounding box loss function for the narrowband imaging generated image are defined as follows: and , the classification loss function and bounding box loss function for the autofluorescence imaging generated image are respectively and , the joint loss function of the lesion detection model satisfies: ； in, and are the weight parameters for narrowband imaging and autofluorescence imaging, is the weight parameter of the weighted loss.

6. A combined narrow-band and fluorescence imaging system for bronchoscopic lesion detection, characterized in that: include: An imaging module is used to respectively acquire white light imaging images, narrow band imaging images and autofluorescence imaging images of the lesion area of ​​the bronchoscopy, and to perform image preprocessing using preset image enhancement methods; a processing module, configured to use the white light imaging image as input to a preset generative adversarial network model, and output a narrowband imaging generated image and an autofluorescence imaging generated image through the preset generative adversarial network model, wherein the preset generative adversarial network model includes a generator network and a discriminator network; A detection module is used to train a lesion detection model based on the narrowband imaging-generated image and the autofluorescence imaging-generated image, and use the lesion detection model to perform lesion detection on the narrowband imaging image and the autofluorescence imaging image to obtain narrowband imaging detection results and autofluorescence image detection results, respectively, and fuse the narrowband imaging detection results and the autofluorescence image detection results through a preset weighted fusion method to obtain a lesion detection result.

Images