A Semi-Supervised Method and System for Pharyngeal Lesion Segmentation Based on Electronic Laryngeal Endoscopy Images
By employing a semi-supervised pharyngeal lesion segmentation method and utilizing data augmentation and dynamic threshold training to train the model, the problem of insufficient experience among laryngoscopy physicians in NBI mode diagnosis was addressed, thereby improving the accurate judgment of pharyngeal lesions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- THE FIRST AFFILIATED HOSPITAL OF SUN YAT SEN UNIV
- Filing Date
- 2024-07-31
- Publication Date
- 2026-05-26
Smart Images
Figure CN119067988B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of deep learning technology, and in particular to a semi-supervised method and system for segmenting pharyngeal lesions based on electronic laryngeal endoscopy images. Background Technology
[0002] Due to its unique anatomical location, lesions in the pharynx and larynx are often hidden and difficult to visualize directly. To understand the intricate anatomical structures and pathological phenomena within the pharynx and larynx, specialized tools such as laryngoscopy are necessary. Electronic laryngoscopy, as a new generation of flexible endoscopes, has become the most convenient and widely used tool in clinical pharyngeal diagnosis and treatment due to its higher resolution, superior structure, and wider field of view. More importantly, narrow-band imaging (NBI) endoscopic mode filters out blue and green light from ordinary broad-spectrum illumination that is absorbed by blood vessels but not widely scattered. This causes the blue light in the microvessels on the mucosal surface to be absorbed and appear brownish, allowing for better observation of vascular morphology in the mucosa and tissue structures. This improves the laryngoscopy physician's ability to detect pharyngeal lesions, especially early cancerous changes occurring on the mucosal surface, significantly enhancing the accuracy of differentiating between benign and malignant lesions.
[0003] However, the correct use of NBI mode and the accurate interpretation of results require a long learning curve in endoscopy knowledge and the accumulation and summarization of long-term experience. For laryngoscopy physicians who lack training and sufficient experience, the accuracy of diagnosis using NBI is not high. Summary of the Invention
[0004] In view of this, embodiments of the present invention provide a semi-supervised pharyngeal lesion segmentation method and system based on electronic laryngeal endoscopy images, so as to reduce the cost of manual annotation, train a high-performance model with a small amount of data to help determine the location and malignancy of tumors, and improve the generalization ability of the model under different endoscopic imaging.
[0005] One aspect of this invention provides a semi-supervised method for segmenting pharyngeal lesions based on electronic laryngeal endoscopy images, comprising the following steps:
[0006] Collect endoscopic laryngeal cancer dataset;
[0007] The endoscopic laryngeal cancer dataset was labeled to obtain a labeled dataset;
[0008] The endoscopic laryngeal cancer dataset and the labeled dataset are input into the model to be trained. Data augmentation of different degrees is performed through weak perturbation branch, strong perturbation branch and style perturbation branch respectively. Based on the dynamic threshold and the preset loss function, the parameters of the model to be trained are adjusted to obtain the target model.
[0009] Based on the target model, the electronic laryngeal endoscope image to be identified is predicted to obtain the prediction result of pharyngeal lesions.
[0010] Optionally, the data augmentation using different degrees of weak perturbation branches, strong perturbation branches, and style perturbation branches includes the following steps:
[0011] The original dataset is subjected to a first data augmentation process through a weak perturbation branch. The first data augmentation process includes random cropping and horizontal variation.
[0012] The original dataset is subjected to a second data augmentation process through a strong perturbation branch, the second data augmentation process including color enhancement processing;
[0013] The original dataset is subjected to third data augmentation processing through a style perturbation branch;
[0014] The training dataset is obtained based on the results of the first data augmentation process, the second data augmentation process, and the third data augmentation process.
[0015] Optionally, the third data augmentation process performed on the original dataset through the style perturbation branch includes the following steps:
[0016] A style transfer method is used to synthesize light-transfer images through a style transfer network;
[0017] The optically migrated image is used as a perturbation factor and incorporated into the image data in the original dataset to complete the third data augmentation process of the original dataset.
[0018] Optionally, the training process of the model to be trained includes three types of loss: strong and weak perturbation consistency loss based on dynamic threshold, style consistency loss based on dynamic threshold, and feature consistency loss based on dynamic threshold. The parameter adjustment of the model to be trained based on the dynamic threshold and a preset loss function includes the following steps:
[0019] Construct a style transfer network based on an autoencoder;
[0020] The style transfer network uses an encoder to map both white light images and narrowband light images into a shared feature space, and then uses two different decoders to generate reconstructed images with different styles.
[0021] A dynamic threshold is used instead of a fixed high threshold, and the reconstructed image is used to train the model to be trained, predicting the pseudo-labels corresponding to each training data.
[0022] Optionally, the step of using the style transfer network to map both the white light image and the narrowband light image into a shared feature space using an encoder, and then using two different decoders to generate reconstructed images with different styles, includes the following steps:
[0023] The encoder maps both the white light image and the narrowband light image into a shared feature space, resulting in the first latent representation of the white light image after the encoder and the second latent representation of the narrowband light image after the encoder.
[0024] The first latent representation and the second latent representation are input into the white light decoder to obtain the reconstructed white light image and the migrated white light image; the first latent representation and the second latent representation are input into the narrowband light decoder to obtain the reconstructed narrowband light image and the migrated narrowband light image.
[0025] The transferred white light image and the transferred narrowband light image are input into the encoder to obtain the latent representation of the transferred white light image and the latent representation of the transferred narrowband light image after the encoder.
[0026] Optionally, the step of replacing the fixed high threshold with a dynamic threshold and training the model to be trained using the reconstructed image to predict the pseudo-labels corresponding to each training data point includes the following steps:
[0027] For each image input to the model to be trained, the average confidence of the corresponding segmented region is obtained through the model to be trained;
[0028] The global and local thresholds for the training process are determined based on the average confidence level.
[0029] Calculate the final threshold based on the global threshold and the local threshold;
[0030] Based on the final threshold, the model to be trained is trained using the reconstructed image, and the pseudo-labels corresponding to each training data are predicted.
[0031] Optionally, the loss function includes a feature alignment loss function, an image reconstruction loss function, a structural similarity loss function, and a style adversarial loss function;
[0032] The feature alignment loss function is used to enable the decoder to sample from the same distribution;
[0033] The image reconstruction loss function is used to enable the decoder to synthesize images of the corresponding style;
[0034] The structural similarity loss function is used to maximize the structural similarity between the latent representations of different images;
[0035] The style adversarial loss function is used to make the images generated by the two decoders conform to the corresponding style.
[0036] Another aspect of this invention provides a semi-supervised pharyngeal lesion segmentation system based on electronic laryngeal endoscopy images, comprising:
[0037] The first module is used to collect endoscopic laryngeal cancer datasets.
[0038] The second module is used to annotate the endoscopic laryngeal cancer dataset to obtain a labeled dataset;
[0039] The third module is used to input the endoscopic laryngeal cancer dataset and the label dataset into the model to be trained, and perform data augmentation of different degrees through weak perturbation branch, strong perturbation branch and style perturbation branch respectively. Based on the dynamic threshold and the preset loss function, the parameters of the model to be trained are adjusted to obtain the target model.
[0040] The fourth module is used to predict the electronic laryngeal endoscope image to be identified based on the target model, and obtain the prediction result of the pharyngeal lesion.
[0041] Another aspect of the present invention provides an electronic device, including a processor and a memory;
[0042] The memory is used to store programs;
[0043] The processor executes the program to implement the method described above.
[0044] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.
[0045] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.
[0046] The embodiments of this invention first collect an endoscopic laryngeal cancer dataset; then, the endoscopic laryngeal cancer dataset is labeled to obtain a labeled dataset; the endoscopic laryngeal cancer dataset and the labeled dataset are input into a model to be trained, and data augmentation is performed to different degrees through weak perturbation branches, strong perturbation branches, and style perturbation branches, respectively. Based on a dynamic threshold and a preset loss function, the parameters of the model to be trained are adjusted to obtain a target model; finally, the target model is used to predict the electronic laryngeal endoscopy image to be identified to obtain the prediction result of the pharyngeal lesion. The embodiments of this invention can reduce the cost of manual annotation, train a high-performance model with a small amount of data to assist in judging the location and malignancy of tumors, and improve the generalization ability of the model under different endoscopic imaging conditions. Attached Figure Description
[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0048] Figure 1 This is a schematic diagram of an implementation environment provided for an embodiment of the present invention;
[0049] Figure 2 A flowchart illustrating the overall steps of an embodiment of the present invention;
[0050] Figure 3 This is a network model architecture diagram provided in an embodiment of the present invention;
[0051] Figure 4 An architecture diagram of a style transfer network provided in an embodiment of the present invention;
[0052] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.
[0054] It is understood that the terms “first,” “second,” etc., used in this invention may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to determination” as used herein may be interpreted as “when…” or “when…” or “in response to determination.”
[0055] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.
[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.
[0057] The semi-supervised pharyngeal lesion segmentation method based on electronic laryngeal endoscopy images provided in this invention relates to the field of deep learning technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the semi-supervised pharyngeal lesion segmentation method based on electronic laryngeal endoscopy images, but is not limited to the above forms.
[0058] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0059] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided by an embodiment of the invention. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.
[0060] Server 101 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.
[0061] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.
[0062] Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc. It can also be a vehicle-mounted terminal of the various device types described above, but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.
[0063] To address the problems existing in the prior art, embodiments of the present invention provide a semi-supervised method for segmenting pharyngeal lesions based on electronic laryngeal endoscopy images, such as... Figure 2 As shown, it includes the following steps:
[0064] Collect endoscopic laryngeal cancer dataset;
[0065] The endoscopic laryngeal cancer dataset was labeled to obtain a labeled dataset;
[0066] The endoscopic laryngeal cancer dataset and the labeled dataset are input into the model to be trained. Data augmentation of different degrees is performed through weak perturbation branch, strong perturbation branch and style perturbation branch respectively. Based on the dynamic threshold and the preset loss function, the parameters of the model to be trained are adjusted to obtain the target model.
[0067] Based on the target model, the electronic laryngeal endoscope image to be identified is predicted to obtain the prediction result of pharyngeal lesions.
[0068] Optionally, the data augmentation using different degrees of weak perturbation branches, strong perturbation branches, and style perturbation branches includes the following steps:
[0069] The original dataset is subjected to a first data augmentation process through a weak perturbation branch. The first data augmentation process includes random cropping and horizontal variation.
[0070] The original dataset is subjected to a second data augmentation process through a strong perturbation branch, the second data augmentation process including color enhancement processing;
[0071] The original dataset is subjected to third data augmentation processing through a style perturbation branch;
[0072] The training dataset is obtained based on the results of the first data augmentation process, the second data augmentation process, and the third data augmentation process.
[0073] Optionally, the third data augmentation process performed on the original dataset through the style perturbation branch includes the following steps:
[0074] A style transfer method is used to synthesize light-transfer images through a style transfer network;
[0075] The optically migrated image is used as a perturbation factor and incorporated into the image data in the original dataset to complete the third data augmentation process of the original dataset.
[0076] Optionally, the training process of the model to be trained includes three types of loss: strong and weak perturbation consistency loss based on dynamic threshold, style consistency loss based on dynamic threshold, and feature consistency loss based on dynamic threshold. The parameter adjustment of the model to be trained based on the dynamic threshold and a preset loss function includes the following steps:
[0077] Construct a style transfer network based on an autoencoder;
[0078] The style transfer network uses an encoder to map both white light images and narrowband light images into a shared feature space, and then uses two different decoders to generate reconstructed images with different styles.
[0079] A dynamic threshold is used instead of a fixed high threshold, and the reconstructed image is used to train the model to be trained, predicting the pseudo-labels corresponding to each training data.
[0080] Optionally, the step of using the style transfer network to map both the white light image and the narrowband light image into a shared feature space using an encoder, and then using two different decoders to generate reconstructed images with different styles, includes the following steps:
[0081] The encoder maps both the white light image and the narrowband light image into a shared feature space, resulting in the first latent representation of the white light image after the encoder and the second latent representation of the narrowband light image after the encoder.
[0082] The first latent representation and the second latent representation are input into the white light decoder to obtain the reconstructed white light image and the migrated white light image; the first latent representation and the second latent representation are input into the narrowband light decoder to obtain the reconstructed narrowband light image and the migrated narrowband light image.
[0083] The transferred white light image and the transferred narrowband light image are input into the encoder to obtain the latent representation of the transferred white light image and the latent representation of the transferred narrowband light image after the encoder.
[0084] Optionally, the step of replacing the fixed high threshold with a dynamic threshold and training the model to be trained using the reconstructed image to predict the pseudo-labels corresponding to each training data point includes the following steps:
[0085] For each image input to the model to be trained, the average confidence of the corresponding segmented region is obtained through the model to be trained;
[0086] The global and local thresholds for the training process are determined based on the average confidence level.
[0087] Calculate the final threshold based on the global threshold and the local threshold;
[0088] Based on the final threshold, the model to be trained is trained using the reconstructed image, and the pseudo-labels corresponding to each training data are predicted.
[0089] Optionally, the loss function includes a feature alignment loss function, an image reconstruction loss function, a structural similarity loss function, and a style adversarial loss function;
[0090] The feature alignment loss function is used to enable the decoder to sample from the same distribution;
[0091] The image reconstruction loss function is used to enable the decoder to synthesize images of the corresponding style;
[0092] The structural similarity loss function is used to maximize the structural similarity between the latent representations of different images;
[0093] The style adversarial loss function is used to make the images generated by the two decoders conform to the corresponding style.
[0094] The specific implementation process of this invention will be described in detail below using a specific application scenario as an example:
[0095] First, semi-supervised learning was proposed to address the limited availability of labeled data in deep learning applications. Currently, most existing deep learning algorithms are based on two main methods: pseudo-labeling and consistency learning. Pseudo-labeling involves using the model's predictions of unlabeled data to generate labels to supervise model training. To ensure the quality of pseudo-labels, existing methods often set a fixed high threshold (0.95) to filter them. Existing consistency learning methods are based on the idea that the model should make consistent predictions in the face of different disturbances. Although they have achieved good results, they do not consider the special characteristics of medical endoscopic images.
[0096] A fixed high threshold (0.95) may lead to low data utilization in the early training phase of deep learning models and fail to account for differences in learning difficulty between different classes, resulting in poor model training performance. This issue is addressed by setting a dynamic threshold that varies with the model training phase and by setting a specific threshold for each class.
[0097] Existing technologies do not take into account the stylistic differences in medical endoscopic images, such as white light and narrowband light. This invention improves the generalization ability of the model based on the concepts of consistency and style transfer.
[0098] like Figure 3 The diagram shown is a network model architecture diagram of an embodiment of the present invention. The network model input of this embodiment is divided into three branches: a weak perturbation branch, a strong perturbation branch, and a style perturbation branch. The encoder and decoder can be either classic residual networks or deeplabv3.
[0099] for Figure 3 The relevant variables and terms appearing in the text are explained below:
[0100] Light Transfer Network: A style transfer neural network that enables white light and narrowband light transfer.
[0101] Dynamic threshold filtering: Uses thresholds to filter out pseudo-labels with low confidence.
[0102] Weak perturbations consist of simple data augmentations such as random cropping, horizontal transformation, blurring, etc.
[0103] Strong perturbations: operations such as color enhancement and shearing blending.
[0104] Feature perturbations, such as dropout.
[0105] P l The model's predictions for labeled data.
[0106] P w : The model's prediction of the weak perturbation branch.
[0107] P s : The model's prediction of the strong perturbation branch.
[0108] P t The model's predictions on style-transferred data.
[0109] P f The model's predictions are based on the features decoded after dropout following the encoder.
[0110] Based on the concept of consistency in semi-supervised learning, this embodiment of the invention categorizes perturbation levels into three types: fw(·) represents weak perturbation, consisting of simple data augmentation such as random cropping, horizontal transformation, blurring, etc.; fs(·) represents strong perturbation, such as color enhancement, cropping and blending operations; and Ffp(·) represents level perturbation, such as dropout. To improve the model's generalization ability to endoscopic images and enhance its understanding of two different types of light (white light and narrowband light), a style transfer scheme is employed, using a style transfer network to synthesize light-transfer images as perturbations. Finally, this embodiment of the invention uses pseudo-labels obtained from the weak perturbation branch to supervise the predictions of other branches, thereby training the semi-supervised segmentation model.
[0111] This invention transforms the original unlabeled data After the different enhancement perturbations mentioned above, w(x) u ) and S(w(x u The weakly perturbated image x is obtained. w and strongly perturbated image x s And the image x after style transformation t As input to the model, the encoder and decoder output the model's prediction p. u ={p w ,p f ,p s ,p tThe model uses a dynamic threshold for filtering. The goal is to minimize the total loss function L, which can be defined as: L = Ll + Lu, where Ll is the supervised loss and p. t The cross-entropy loss with labels, Lu is the unsupervised loss, which can be specifically defined as p w and {p f p s p t The cross-entropy loss of}.
[0112] The style transfer network, one of the key improvements in the embodiments of the present invention, is described below:
[0113] To achieve consistent learning between narrowband light and white light and improve the model's generalization ability, this embodiment of the invention designs a style transfer network based on an autoencoder. This network first uses an encoder to map both the white light image and the narrowband light image into a shared feature space, and then uses two different decoders to generate images with different styles, such as... Figure 4 As shown.
[0114] Figure 4 The meanings of the relevant variables and terms in the text are explained below:
[0115] Discriminator: The discriminator in an adversarial neural network.
[0116] x wl Photographs of type white light.
[0117] x nbi Photographs of the narrowband light type.
[0118] z wl The potential representation of a white light photograph after it has been encoded.
[0119] z nbi The potential representation of a narrowband light photograph after it has been encoded.
[0120] y wl : Reconstructed white light image.
[0121] y nbi : Reconstructed narrowband light image.
[0122] y wl t : Migrating white light image.
[0123] y nbi t Narrowband light image of migration.
[0124] z wl tThe latent representation of the migrated white light image after passing through the encoder.
[0125] z nbi t The latent representation of the narrowband light image after the encoder.
[0126] In embodiments of the present invention, such as Figure 4 As shown, the style transfer network uses two types of images x obtained under different lighting conditions. wl and x nbi After passing through the decoder, the corresponding image feature z will be obtained. wl and z nbi These two features, after being processed by two different decoders, will yield the reconstructed image y. wl y nbi and the migration image y t nbi y t wl .
[0127] This invention uses four different loss functions to train the network, namely: 1) Feature alignment loss function: to ensure that the features z of images from different lighting conditions are aligned. wl and z nbi Following the same distribution, in this embodiment of the invention, z is trained... wl and z nbi Alignment with second-order statistical features allows the decoder to sample from the same distribution, resulting in better synthesis of the transfer image y. t nbi y t wl 2) Image reconstruction loss: This embodiment of the invention uses SSIM (Structure Similarity Index Measure) loss to make y wl y nbi Reconstructed into x wl and x nbi This allows the decoder to better synthesize images of the corresponding style. 3) Structural similarity loss: After obtaining the synthesized transfer image y t nbi y t wl In this embodiment of the invention, a decoder is then used to map it to the feature space to obtain z. t nbi z t wl The embodiments of the present invention can maximize z t nbi z t wl and z nbiz wl The structural similarity between features is calculated using self-cosine similarity (calculating the cosine similarity between pairs of locations in the feature map). 4) Style adversarial loss: This embodiment of the invention uses the idea of adversarial networks, employing two discriminators to supervise two decoders respectively, so that the images generated by the two decoders can conform to the corresponding styles.
[0128] In addition, the dynamic threshold design method, another important improvement of the embodiments of the present invention, will be described in detail:
[0129] In existing semi-supervised techniques, pseudo-labeling is a classic learning paradigm. To make fuller use of pseudo-labels and improve model performance, this embodiment of the invention designs a dynamic threshold instead of a fixed high threshold (0.95). The number of pseudo-labels the model can use for training varies depending on the model training stage. This embodiment uses the confidence score of the model's segmented regions, a metric from the model training stage, to update the threshold used for filtering pseudo-labels. Considering the differences in learning difficulty between the dataset and each class, a global threshold (for the dataset) and a local threshold (for each class) are set. Specifically, for each input image... After modeling, the average confidence score conf(1, 2, ..., c) for the corresponding segmented region is obtained, where c is the number of semantic segmentation classes. For the global threshold, Global_threshold(i) = Global_threshold(i-1)*ɑ + (1-ɑ)*max(conf(i)), with an initial global threshold of 1 / c. The global threshold increases with the number of model iterations, providing sufficient samples for model learning in the early stages using more pseudo-labels, and filtering out noisy pseudo-labels in the later stages to prevent misleading model learning. For the local threshold, the formula can be expressed as: local_threshold(i)j = local_threshold*ɑ + (1-ɑ)*conf(i)j, j = {j1, j2, ..., jc}. More pseudo-labels are provided to assist training for classes that are difficult to learn (with low model prediction confidence). The final threshold formula can be expressed as threshold(j) = Global_threshold*local_threshold(j) / max(local_threshold).
[0130] The semi-supervised pharyngeal lesion segmentation method based on electronic laryngeal endoscopy images provided in embodiments of the present invention can be implemented in other embodiments as follows:
[0131] The data is divided into different proportions, such as 1 / 16 and 1 / 8, and input into a ResNet-DeepLabv3+ basic segmentation model. The specific implementation plan includes data loading, data partitioning, model training, model evaluation, and optimization. For labeled data, a conventional approach is used. For unlabeled data, based on the consistency principle and the characteristics of endoscopic instruments (white light, narrowband light), the model input is divided into a weak perturbation branch, a strong perturbation branch, and a style transfer branch. The weak perturbation branch performs random cropping and transformation operations on the image to generate higher-quality pseudo-labels for model training. The strong perturbation branch performs color jittering, Gaussian blur, random grayscale transformation, and cropping mixing operations on the input image. This is used for consistency loss with the pseudo-labels generated by the weak perturbation branch, pushing the model decision boundary to a low-density region and improving model performance. The style transfer branch aligns features based on the different imaging modalities of endoscopic images (white light, narrowband light) through a style transfer network, enhancing the model's robustness.
[0132] Style Transfer Network: This embodiment of the invention uses all data in the training set to train the style transfer network, including both labeled and unlabeled images. After training, this embodiment uses the style transfer network to generate a style transformation branch. The white light endoscopic image is input into the style transfer encoder to obtain the corresponding latent representation. This latent representation is then input into the narrowband light decoder to obtain the narrowband light image. The same process applies to the narrowband light endoscopic image; it is input into the style transfer encoder, and the encoder's output is input into the white light decoder to obtain the transferred image. The transferred image is used as a style perturbation for one branch of the training process.
[0133] In general, this embodiment of the invention uses the model's predictions for the weakly perturbated branch as pseudo-labels, and applies a self-training loss to the prediction results for other branches. Furthermore, to better utilize the pseudo-labels, this embodiment sets a dynamic threshold based on the model's training phase instead of a fixed high threshold to filter available pseudo-labels.
[0134] The following section will explain how the model is trained, starting with the loss function. The formula can be expressed as:
[0135] L = 0.5 * L l +0.5*L u
[0136] The overall loss consists of two parts: the loss from labeled data and the loss from unlabeled data. Cross-entropy loss is used as the loss function. For the unlabeled loss L... u The calculation method is as follows.
[0137]
[0138] Where, p wIt is the model's prediction for the weak perturbation branch, p s It is the model's prediction for the strong perturbation branch, p st This is the model's prediction result for the style-transferred image. conf(p w ) represents the model's prediction confidence level for pseudo-labels, used to filter out reliable pseudo-labels, while γ is a threshold that changes as the model trains. By constraining the consistency of the model's predictions for different perturbations, a large amount of unlabeled data can be fully utilized.
[0139] Finally, in this embodiment of the invention, the real-time acquired endoscopic laryngeal cancer data is input into a trained semi-supervised model to obtain the location and condition of the tumor.
[0140] In summary, this invention can reduce the cost of manual annotation, train a high-performance model with a small amount of data to help determine the location and malignancy of tumors, and improve the model's generalization ability under different endoscopic imaging conditions.
[0141] Another aspect of this invention provides a semi-supervised pharyngeal lesion segmentation system based on electronic laryngeal endoscopy images, comprising:
[0142] The first module is used to collect endoscopic laryngeal cancer datasets.
[0143] The second module is used to annotate the endoscopic laryngeal cancer dataset to obtain a labeled dataset;
[0144] The third module is used to input the endoscopic laryngeal cancer dataset and the label dataset into the model to be trained, and perform data augmentation of different degrees through weak perturbation branch, strong perturbation branch and style perturbation branch respectively. Based on the dynamic threshold and the preset loss function, the parameters of the model to be trained are adjusted to obtain the target model.
[0145] The fourth module is used to predict the electronic laryngeal endoscope image to be identified based on the target model, and obtain the prediction result of the pharyngeal lesion.
[0146] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.
[0147] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned semi-supervised pharyngeal lesion segmentation method based on electronic laryngeal endoscopy images. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.
[0148] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0149] Please see Figure 5 , Figure 5 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:
[0150] The processor 501 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.
[0151] The memory 502 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 502 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and is called and executed by the processor 501 to execute the semi-supervised pharyngeal lesion segmentation method based on electronic laryngeal endoscopy images of the embodiments of this invention.
[0152] The input / output interface 503 is used to implement information input and output;
[0153] The communication interface 504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).
[0154] Bus 505 transmits information between various components of the device (e.g., processor 501, memory 502, input / output interface 503, and communication interface 504);
[0155] The processor 501, memory 502, input / output interface 503, and communication interface 504 are connected to each other within the device via bus 505.
[0156] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described semi-supervised pharyngeal lesion segmentation method based on electronic laryngeal endoscopy images.
[0157] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.
[0158] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0159] It should be noted that in various specific embodiments of the present invention, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of the present invention require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data for the normal operation of the embodiments of the present invention acquired.
[0160] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.
[0161] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.
[0162] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0163] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.
[0164] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.
[0165] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0166] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0167] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
[0168] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.
Claims
1. A semi-supervised method for segmenting pharyngeal lesions based on electronic laryngeal endoscopy images, characterized in that, Includes the following steps: Collect endoscopic laryngeal cancer dataset; The endoscopic laryngeal cancer dataset was labeled to obtain a labeled dataset; The endoscopic laryngeal cancer dataset and the labeled dataset are input into the model to be trained. Data augmentation of different degrees is performed through weak perturbation branch, strong perturbation branch and style perturbation branch respectively. Based on the dynamic threshold and the preset loss function, the parameters of the model to be trained are adjusted to obtain the target model. Based on the target model, the electronic laryngeal endoscope image to be identified is predicted to obtain the prediction result of pharyngeal lesions. The training process of the model to be trained includes three types of loss: dynamic threshold-based strong / weak perturbation consistency loss, dynamic threshold-based style consistency loss, and dynamic threshold-based feature consistency loss. The parameter adjustment of the model to be trained based on the dynamic threshold and a preset loss function includes the following steps: Construct a style transfer network based on an autoencoder; The style transfer network uses an encoder to map both white light images and narrowband light images into a shared feature space, and then uses two different decoders to generate reconstructed images with different styles. A dynamic threshold is used instead of a fixed high threshold, and the reconstructed image is used to train the model to be trained, predicting the pseudo-labels corresponding to each training data.
2. The semi-supervised pharyngeal lesion segmentation method based on electronic laryngeal endoscopy images according to claim 1, characterized in that, The data augmentation, performed using weak perturbation branches, strong perturbation branches, and style perturbation branches, includes the following steps: The original dataset is subjected to a first data augmentation process through a weak perturbation branch. The first data augmentation process includes random pruning and horizontal variation. The original dataset is subjected to a second data augmentation process through a strong perturbation branch, the second data augmentation process including color enhancement processing; The original dataset is subjected to third data augmentation processing through a style perturbation branch; The training dataset is obtained based on the results of the first data augmentation process, the second data augmentation process, and the third data augmentation process.
3. The semi-supervised pharyngeal lesion segmentation method based on electronic laryngeal endoscopy images according to claim 2, characterized in that, The third data augmentation process performed on the original dataset through style perturbation branches includes the following steps: A style transfer method is used to synthesize light-transfer images through a style transfer network; The optically migrated image is used as a perturbation factor and incorporated into the image data in the original dataset to complete the third data augmentation process of the original dataset.
4. The semi-supervised pharyngeal lesion segmentation method based on electronic laryngeal endoscopy images according to claim 1, characterized in that, The process involves using the style transfer network to map both the white light image and the narrowband light image into a shared feature space using an encoder, and then using two different decoders to generate reconstructed images with different styles. This includes the following steps: The encoder maps both the white light image and the narrowband light image into a shared feature space, resulting in the first latent representation of the white light image after the encoder and the second latent representation of the narrowband light image after the encoder. The first latent representation and the second latent representation are input into the white light decoder to obtain the reconstructed white light image and the migrated white light image; the first latent representation and the second latent representation are input into the narrowband light decoder to obtain the reconstructed narrowband light image and the migrated narrowband light image. The transferred white light image and the transferred narrowband light image are input into the encoder to obtain the latent representation of the transferred white light image and the latent representation of the transferred narrowband light image after the encoder.
5. The semi-supervised pharyngeal lesion segmentation method based on electronic laryngeal endoscopy images according to claim 1, characterized in that, The process of replacing a fixed high threshold with a dynamic threshold and training the model to be trained using the reconstructed image to predict the pseudo-labels corresponding to each training data point includes the following steps: For each image input to the model to be trained, the average confidence of the corresponding segmented region is obtained through the model to be trained; The global and local thresholds for the training process are determined based on the average confidence level. Calculate the final threshold based on the global threshold and the local threshold; Based on the final threshold, the model to be trained is trained using the reconstructed image, and the pseudo-labels corresponding to each training data are predicted.
6. The semi-supervised pharyngeal lesion segmentation method based on electronic laryngeal endoscopy images according to claim 1, characterized in that, The loss functions include feature alignment loss function, image reconstruction loss function, structural similarity loss function, and style adversarial loss function; The feature alignment loss function is used to enable the decoder to sample from the same distribution; The image reconstruction loss function is used to enable the decoder to synthesize images of the corresponding style; The structural similarity loss function is used to maximize the structural similarity between the latent representations of different images; The style adversarial loss function is used to make the images generated by the two decoders conform to the corresponding style.
7. A system for implementing the semi-supervised pharyngeal lesion segmentation method based on electronic laryngeal endoscopy images as described in any one of claims 1-6, characterized in that, include: The first module is used to collect endoscopic laryngeal cancer datasets. The second module is used to annotate the endoscopic laryngeal cancer dataset to obtain a labeled dataset; The third module is used to input the endoscopic laryngeal cancer dataset and the label dataset into the model to be trained, and perform data augmentation of different degrees through weak perturbation branch, strong perturbation branch and style perturbation branch respectively. Based on the dynamic threshold and the preset loss function, the parameters of the model to be trained are adjusted to obtain the target model. The fourth module is used to predict the electronic laryngeal endoscope image to be identified based on the target model, and obtain the prediction result of the pharyngeal lesion.
8. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 6.