Semi-supervised throat focus segmentation method and system based on electronic throat endoscope image
Through semi-supervised learning methods and data augmentation technology, the model used for electronic laryngeal endoscopic images is trained, which solves the problem of low diagnostic accuracy in narrowband imaging endoscopic mode, and improves the segmentation accuracy of throat lesions and model generalization ability of the generalization of the model .
Patent Information
- Application Number
- CN202411038995.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-07-31
AI Technical Summary
The prior art requires a long endoscopic knowledge learning curve and long-term experience when diagnosing throat lesions using narrowband imaging endoscopic mode, resulting in inaccurate application of physicians who lack training and experience, especially when identifying early cancerous and malignant conditions. Low accuracy.
Using a semi-supervised learning method based on electronic laryngeal endoscopic images, data augmentation is performed through weak perturbation, strong perturbation and style perturbation branches, combining dynamic thresholds and preset loss functions, the target model is trained to assist in judging tumor location and malignant conditions, and Improve the generalization ability of the model under different endoscopic imaging.
The cost of manual labeling is reduced, and the excellent performance model is trained using a small amount of data, which improves the accuracy of judging tumor location and malignant conditions, and enhances the generalization ability of the model under different endoscopic imaging.
Smart Images

Figure CN119067988A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning technology, and in particular to a semi-supervised throat lesion segmentation method and system based on electronic laryngeal endoscope images. Background Art
[0002] Due to its special anatomical location, the lesions inside the throat are often hidden deep and difficult to directly see. If you need to understand the delicate anatomical structure and pathological phenomena inside the throat, you must use special tools such as laryngoscopes. As a new generation of flexible tube endoscopes, electronic laryngoscopes have become the most convenient and widely used tools for clinical throat diagnosis and treatment due to their higher-definition images, better structures, and wider fields of view. More importantly, the narrow band imaging (NBI) endoscopy mode filters the blue and green light in ordinary broad-spectrum illumination that is absorbed by blood vessels but not widely scattered, so that the blue light in the microvessels on the surface of the mucosa is absorbed and appears brown. This allows for better observation of the vascular morphology in the mucosa and tissue structure, improves the laryngoscope physician's detection of throat lesions, especially early canceration occurring on the surface of the mucosa, and significantly improves the accuracy of distinguishing benign and malignant lesions.
[0003] However, the correct use of the NBI mode and the correct interpretation of the results require a long endoscopic knowledge learning curve and the accumulation and summary of long-term experience. For laryngoscopists who lack training and sufficient experience, the accuracy of diagnosis using NBI is not high. Summary of the invention
[0004] In view of this, an embodiment of the present invention provides a semi-supervised laryngeal lesion segmentation method and system based on electronic laryngeal endoscope images to reduce the cost of manual labeling, use a small amount of data to train a model with excellent performance to assist in determining the location and malignancy of the tumor, and improve the generalization ability of the model under different endoscopic imaging.
[0005] An aspect of an embodiment of the present invention provides a semi-supervised laryngeal lesion segmentation method based on electronic laryngeal endoscope images, comprising the following steps:
[0006] Acquisition of endoscopic laryngeal cancer dataset;
[0007] Annotating the endoscopic laryngeal cancer dataset to obtain a labeled dataset;
[0008] The endoscopic laryngeal cancer dataset and the label dataset are input into the model to be trained, different degrees of data enhancement are performed through a weak perturbation branch, a strong perturbation branch, and a style perturbation branch, and parameters of the model to be trained are adjusted based on a dynamic threshold and a preset loss function to obtain a target model;
[0009] The electronic laryngeal endoscope image to be identified is predicted according to the target model to obtain a laryngeal lesion prediction result.
[0010] Optionally, performing different degrees of data enhancement through a weak perturbation branch, a strong perturbation branch, and a style perturbation branch comprises the following steps:
[0011] Performing a first data enhancement process on the original data set through a weak perturbation branch, wherein the first data enhancement process includes a random cropping process and a level change process;
[0012] Performing a second data enhancement process on the original data set through a strong perturbation branch, wherein the second data enhancement process includes a color enhancement process;
[0013] Performing a third data enhancement process on the original data set through a style perturbation branch;
[0014] A data set to be trained is obtained according to the results of the first data enhancement processing, the second data enhancement processing and the third data enhancement processing.
[0015] Optionally, performing a third data enhancement process on the original data set through a style perturbation branch comprises the following steps:
[0016] Using the style transfer method, the light transfer image is synthesized through the style transfer network;
[0017] The light migration image is used as a disturbance factor and combined with the image data in the original data set to complete the third data enhancement processing of the original data set.
[0018] Optionally, the training process of the model to be trained includes three types of losses, namely, strong and weak perturbation consistency loss based on a dynamic threshold, style consistency loss based on a dynamic threshold, and feature consistency loss based on a dynamic threshold. The parameter adjustment of the model to be trained based on the dynamic threshold and a preset loss function includes the following steps:
[0019] Construct a style transfer network based on autoencoder;
[0020] Through the style transfer network, an encoder is used to map both the white light image and the narrow-band light image into a shared feature space, and then two different decoders are used to generate reconstructed images with different styles respectively;
[0021] A dynamic threshold is used to replace a fixed high threshold, and the model to be trained is trained in combination with the reconstructed image to predict the pseudo labels corresponding to each data to be trained.
[0022] Optionally, the style transfer network is used to map both the white light image and the narrow-band light image into a shared feature space using an encoder, and then two different decoders are used to generate reconstructed images with different styles, including the following steps:
[0023] Use an encoder to map both the white light image and the narrow-band light image into a shared feature space, thereby obtaining a first potential representation of the white light image after passing through the encoder and a second potential representation of the narrow-band light image after passing through the encoder;
[0024] Inputting the first latent representation and the second latent representation into a white light decoder to obtain a reconstructed white light image and a migrated white light image; inputting the first latent representation and the second latent representation into a narrow-band light decoder to obtain a reconstructed narrow-band light image and a migrated narrow-band light image;
[0025] The migrated white light image and the migrated narrow-band light image are input into an encoder to obtain a latent representation of the migrated white light image after passing through the encoder and a latent representation of the migrated narrow-band light image after passing through the encoder.
[0026] Optionally, the method of using a dynamic threshold to replace a fixed high threshold, training a model to be trained in combination with the reconstructed image, and predicting pseudo labels corresponding to each piece of data to be trained includes the following steps:
[0027] For each image input into the model to be trained, an average confidence of the corresponding segmented area is obtained by the model to be trained;
[0028] Determining a global threshold and a local threshold of a training process according to the average confidence;
[0029] Calculate a final threshold value according to the global threshold value and the local threshold value;
[0030] The model to be trained is trained according to the final threshold value and combined with the reconstructed image to predict pseudo labels corresponding to each data to be trained.
[0031] Optionally, the loss function includes a feature alignment loss function, an image reconstruction loss function, a structural similarity loss function, and a style adversarial loss function;
[0032] Wherein, the feature alignment loss function is used to enable the decoder to sample from the same distribution;
[0033] The image reconstruction loss function is used to enable the decoder to synthesize an image of a corresponding style;
[0034] The structural similarity loss function is used to maximize the structural similarity between the latent representations of different images;
[0035] The style adversarial loss function is used to make the pictures generated by the two decoders conform to the corresponding styles.
[0036] Another aspect of the embodiment of the present invention further provides a semi-supervised throat lesion segmentation system based on electronic laryngeal endoscope images, comprising:
[0037] The first module is used to collect endoscopic laryngeal cancer datasets;
[0038] The second module is used to label the endoscopic laryngeal cancer dataset to obtain a label dataset;
[0039] The third module is used to input the endoscopic laryngeal cancer dataset and the label dataset into the model to be trained, perform different degrees of data enhancement through the weak perturbation branch, the strong perturbation branch and the style perturbation branch, and adjust the parameters of the model to be trained based on the dynamic threshold and the preset loss function to obtain the target model;
[0040] The fourth module is used to predict the electronic laryngeal endoscope image to be identified according to the target model to obtain a laryngeal lesion prediction result.
[0041] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory;
[0042] The memory is used to store programs;
[0043] The processor executes the program to implement the method described above.
[0044] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.
[0045] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.
[0046] The embodiment of the present invention first collects an endoscopic laryngeal cancer dataset; then labels the endoscopic laryngeal cancer dataset to obtain a label dataset; inputs the endoscopic laryngeal cancer dataset and the label dataset into the model to be trained, performs different degrees of data enhancement through weak perturbation branches, strong perturbation branches and style perturbation branches, respectively, and adjusts the parameters of the model to be trained based on dynamic thresholds and preset loss functions to obtain a target model; finally, the electronic laryngeal endoscope image to be identified is predicted according to the target model to obtain a laryngeal lesion prediction result. The embodiment of the present invention can reduce the cost of manual labeling, use a small amount of data to train a model with excellent performance to assist in judging the location of tumors and malignancy, and improve the generalization ability of the model under different endoscopic imaging. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0048] Figure 1 A schematic diagram of an implementation environment provided for an embodiment of the present invention;
[0049] Figure 2 An overall step flow chart provided for an embodiment of the present invention;
[0050] Figure 3 A network model architecture diagram provided for an embodiment of the present invention;
[0051] Figure 4 An architectural diagram of a style transfer network provided by an embodiment of the present invention;
[0052] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0053] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.
[0054] It is understood that the terms "first", "second", etc. used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0055] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.
[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.
[0057] The semi-supervised throat lesion segmentation method based on electronic laryngeal endoscope images provided in an embodiment of the present invention relates to the field of deep learning technology. The semi-supervised throat lesion segmentation method based on electronic laryngeal endoscope images provided in an embodiment of the present invention can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The server can also be a node server in a blockchain network; the software can be an application that implements a semi-supervised throat lesion segmentation method based on electronic laryngeal endoscope images, etc., but is not limited to the above forms.
[0058] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0059] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.
[0060] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.
[0061] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.
[0062] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 102 may also be a vehicle-mounted terminal of various device types described above, but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the present invention.
[0063] In view of the problems existing in the prior art, the embodiment of the present invention provides a semi-supervised throat lesion segmentation method based on electronic laryngeal endoscope images, such as Figure 2 As shown, the following steps are included:
[0064] Acquisition of endoscopic laryngeal cancer dataset;
[0065] Annotating the endoscopic laryngeal cancer dataset to obtain a labeled dataset;
[0066] The endoscopic laryngeal cancer dataset and the label dataset are input into the model to be trained, different degrees of data enhancement are performed through a weak perturbation branch, a strong perturbation branch, and a style perturbation branch, and parameters of the model to be trained are adjusted based on a dynamic threshold and a preset loss function to obtain a target model;
[0067] The electronic laryngeal endoscope image to be identified is predicted according to the target model to obtain a laryngeal lesion prediction result.
[0068] Optionally, performing different degrees of data enhancement through a weak perturbation branch, a strong perturbation branch, and a style perturbation branch comprises the following steps:
[0069] Performing a first data enhancement process on the original data set through a weak perturbation branch, wherein the first data enhancement process includes a random cropping process and a level change process;
[0070] Performing a second data enhancement process on the original data set through a strong perturbation branch, wherein the second data enhancement process includes a color enhancement process;
[0071] Performing a third data enhancement process on the original data set through a style perturbation branch;
[0072] A data set to be trained is obtained according to the results of the first data enhancement processing, the second data enhancement processing and the third data enhancement processing.
[0073] Optionally, performing a third data enhancement process on the original data set through a style perturbation branch comprises the following steps:
[0074] Using the style transfer method, the light transfer image is synthesized through the style transfer network;
[0075] The light migration image is used as a disturbance factor and combined with the image data in the original data set to complete the third data enhancement processing of the original data set.
[0076] Optionally, the training process of the model to be trained includes three types of losses, namely, strong and weak perturbation consistency loss based on a dynamic threshold, style consistency loss based on a dynamic threshold, and feature consistency loss based on a dynamic threshold. The parameter adjustment of the model to be trained based on the dynamic threshold and a preset loss function includes the following steps:
[0077] Build a style transfer network based on autoencoder;
[0078] Through the style transfer network, an encoder is used to map both the white light image and the narrow-band light image into a shared feature space, and then two different decoders are used to generate reconstructed images with different styles respectively;
[0079] A dynamic threshold is used to replace a fixed high threshold, and the model to be trained is trained in combination with the reconstructed image to predict the pseudo labels corresponding to each data to be trained.
[0080] Optionally, the style transfer network is used to map both the white light image and the narrow-band light image into a shared feature space using an encoder, and then two different decoders are used to generate reconstructed images with different styles, including the following steps:
[0081] Use an encoder to map both the white light image and the narrow-band light image into a shared feature space, thereby obtaining a first potential representation of the white light image after passing through the encoder and a second potential representation of the narrow-band light image after passing through the encoder;
[0082] Inputting the first latent representation and the second latent representation into a white light decoder to obtain a reconstructed white light image and a migrated white light image; inputting the first latent representation and the second latent representation into a narrow-band light decoder to obtain a reconstructed narrow-band light image and a migrated narrow-band light image;
[0083] The migrated white light image and the migrated narrow-band light image are input into an encoder to obtain a latent representation of the migrated white light image after passing through the encoder and a latent representation of the migrated narrow-band light image after passing through the encoder.
[0084] Optionally, the method of using a dynamic threshold to replace a fixed high threshold, training a model to be trained in combination with the reconstructed image, and predicting a pseudo label corresponding to each data to be trained includes the following steps:
[0085] For each image input into the model to be trained, an average confidence of the corresponding segmented area is obtained by the model to be trained;
[0086] Determining a global threshold and a local threshold of a training process according to the average confidence;
[0087] Calculate a final threshold value according to the global threshold value and the local threshold value;
[0088] The model to be trained is trained according to the final threshold value and combined with the reconstructed image to predict pseudo labels corresponding to each data to be trained.
[0089] Optionally, the loss function includes a feature alignment loss function, an image reconstruction loss function, a structural similarity loss function, and a style adversarial loss function;
[0090] Wherein, the feature alignment loss function is used to enable the decoder to sample from the same distribution;
[0091] The image reconstruction loss function is used to enable the decoder to synthesize an image of a corresponding style;
[0092] The structural similarity loss function is used to maximize the structural similarity between the latent representations of different images;
[0093] The style adversarial loss function is used to make the pictures generated by the two decoders conform to the corresponding styles.
[0094] The following describes the specific implementation process of the present invention in detail by taking a specific application scenario as an example:
[0095] First, semi-supervised learning was proposed to solve the problem of limited class label data in deep learning applications. Most of the existing deep learning algorithms are mainly based on two methods: pseudo-labeling technology and consistency learning. Pseudo-labeling technology refers to using the model's prediction of unlabeled data to generate labels to supervise model training. To ensure the quality of pseudo-labels, existing methods often set a fixed high threshold (0.95) to filter pseudo-labels. Existing consistency learning methods are based on the idea that the model should make consistent predictions in the face of different interferences. Although good results have been achieved, they do not take into account the particularity of medical endoscopic images.
[0096] A fixed high threshold (0.95) may lead to low data utilization in the early training stage of the deep learning model, and does not take into account the difference in learning difficulty between different categories, resulting in poor model training results. This problem is solved by setting a dynamic threshold that changes with the model training stage and setting a specific threshold for each class.
[0097] The prior art does not consider the style differences of medical endoscopy images, such as white light and narrow-band light. The present invention is based on the idea of consistency and style transfer to improve the generalization ability of the model.
[0098] like Figure 3 As shown, it is a network model architecture diagram of an embodiment of the present invention. The network model input of the embodiment of the present invention is divided into three branches, a weak perturbation branch, a strong perturbation branch, and a style perturbation branch. The encoder and decoder can choose the classic residual network and deeplabv3.
[0099] for Figure 3 The relevant variables and nouns appearing in are explained as follows:
[0100] Light Transfer Network: A style transfer neural network that implements white light and narrowband light transfer.
[0101] Dynamic threshold filtering: Apply a threshold to filter out low-confidence pseudo-labels.
[0102] Weak perturbations: consist of some simple data augmentations such as random cropping, horizontal transformation, blurring, etc.
[0103] Strong perturbations: color enhancement, shear mixing, etc.
[0104] Feature perturbations, such as dropout.
[0105] P l : The model's predictions for labeled data.
[0106] P w : The model's predictions for weakly perturbed branches.
[0107] P s : The model's predictions for strongly perturbed branches.
[0108] P t : The model’s predictions for the style-transferred data.
[0109] P f : The model's predictions for feature decoding after dropout after the encoder.
[0110] According to the idea of consistency in semi-supervised learning, the embodiment of the present invention divides the level of perturbation into three categories, fw(·) represents weak perturbation, which consists of some simple data enhancements such as random cropping, horizontal transformation, blurring, etc. fs(·) represents strong perturbation, such as color enhancement, shear mixing and other operations. Ffp(·) level perturbation, such as dropout. In order to make the model have better generalization ability for endoscopic images and at the same time improve the model's ability to understand two different lights (white light and narrow-band light), a style transfer scheme is adopted, and a style transfer network is used to synthesize light migration images as perturbations. Finally, the embodiment of the present invention uses the pseudo-labels obtained by the weak perturbation branch to supervise the predictions of other branches, so as to train the semi-supervised segmentation model.
[0111] The present invention converts the original unlabeled data After the different enhanced perturbations mentioned above, w(x u ) and S(w(x u ))Get the weakly perturbed image x w and the strongly perturbed image x s And the image x after style transformation t As the input of the model. After the encoder and decoder, the model's prediction p is obtained u ={p w ,p f ,p s ,p t}, and use dynamic thresholds to filter them. The goal of the model is to minimize the total loss function of L, which can be defined as: L = Ll + Lu, where Ll is the supervised loss, which is p t and the cross entropy loss of the label, Lu is the unsupervised loss, which can be defined as p w and {p f , p s , p t} cross entropy loss.
[0112] The following is a description of the style transfer network, which is one of the key improvements of the embodiment of the present invention:
[0113] In order to achieve the consistency learning of narrow-band light and white light and improve the generalization ability of the model, the embodiment of the present invention designs a style transfer network based on an autoencoder. The network first uses an encoder to map both the white light image and the narrow-band light image to a shared feature space, and then uses two different decoders to generate images with different styles. Figure 4 shown.
[0114] Figure 4 The meanings of the relevant variables and nouns in are explained as follows:
[0115] Discriminator: The discriminator in adversarial neural networks.
[0116] x wl :Photos of type white light.
[0117] x nbi : This type of photo is a narrow-band light photo.
[0118] z wl : The latent representation of a white light photo after passing through the encoder.
[0119] z nbi : The latent representation of a narrowband light photo after it passes through the encoder.
[0120] y wl :Reconstructed white light image.
[0121] y nbi :Reconstructed narrow-band light image.
[0122] y wl t : White light image of migration.
[0123] y nbi t : Narrow-band light image of migration.
[0124] z wl t: The latent representation of the transferred white-light image after passing through the encoder.
[0125] z nbi t : The latent representation of the transferred narrowband light image after passing through the encoder.
[0126] In the embodiment of the present invention, Figure 4 As shown, the style transfer network uses two types of pictures x obtained with different lighting wl and x nbi , after passing through the decoder, the corresponding image feature z will be obtained wl and z nbi , and after these two features pass through two different decoders respectively, the reconstructed image y wl ,y nbi And the migration image y t nbi ,y t wl .
[0127] The embodiment of the present invention uses four different loss functions to train the network, namely: 1) Feature alignment loss function: In order to ensure that the feature z of the images from different illuminations is wl and z nbi Following the same distribution, the embodiment of the present invention performs training on z wl and z nbi The second-order statistical features are aligned, which enables the decoder to sample from the same distribution and better synthesize the migration image y t nbi ,y t wl 2) Image reconstruction loss: The present invention uses SSIM (Structure Similarity Index Measure) loss to make y wl ,y nbi Rebuild to x wl and x nbi , so that the decoder can better synthesize images of corresponding styles. 3) Structural similarity loss: After obtaining the synthesized migration image y t nbi ,y t wl , the embodiment of the present invention uses a decoder to map it to the feature space to obtain z t nbi 、z t wl , the embodiment of the present invention can maximize z t nbi 、z t wl and z nbi、z wl The structural similarity between the two features, where the structure of the feature is calculated by self cosine similarity (calculating the cosine similarity between two positions in the feature map). 4) Style adversarial loss: The embodiment of the present invention uses the idea of adversarial network, using two discriminators to supervise the two decoders respectively, so that the pictures generated by the two decoders can meet the corresponding style.
[0128] In addition, a dynamic threshold design method of another important improvement of the embodiment of the present invention is described in detail:
[0129] In the existing semi-supervised technology, the pseudo-label method is a classic learning paradigm. In order to make better use of pseudo-labels and improve the performance of the model, the embodiment of the present invention designs a dynamic threshold instead of a fixed high threshold (0.95). Depending on the model training stage, the number of pseudo-labels that the model can use for training also varies. The embodiment of the present invention uses the confidence index of the model segmentation area in the model training stage, and uses it to update the threshold used to filter pseudo-labels. Taking into account the differences in the difficulty of learning the data set and each class, a global threshold (for the data set) and a local threshold (for each class) are set. Specifically, for each input image After passing through the model, the average confidence of the corresponding segmented area is obtained, conf(1, 2, ..., c), where c is the number of semantic segmentation classes. For the global threshold, Global_threshold(i) = Global_threshold(i-1)*ɑ+(1-ɑ)*max(conf(i)), and the initial global threshold is 1 / c. The global threshold will increase with the increase in the number of model iterations. In the early stage, more pseudo-labels are used to provide sufficient samples for model learning, and in the later stage, noisy pseudo-labels are filtered to prevent misleading model learning. For the local threshold, the formula can be expressed as: local_threshold(i)j = local_threshold*ɑ+(1-ɑ)*conf(i)j, j = {j1, j2, ..., jc}. For classes that are more difficult to learn (low model prediction confidence), more pseudo-labels are provided to assist training. The final threshold formula can be expressed as threshold(j) = Global_threshold*local_threshold(j) / max(local_threshold).
[0130] According to the semi-supervised throat lesion segmentation method based on electronic laryngeal endoscope images provided by the embodiment of the present invention, in other embodiments, it can be implemented in the following manner:
[0131] Divide the data into different proportions, such as 1 / 16, 1 / 8. Input into the ResNet-DeepLabv3+ based segmentation model. The specific implementation plan includes data loading, data partitioning, model training, model evaluation, and adjustment and optimization. For labeled data, the conventional method for unlabeled data is used. According to the consistency idea and the characteristics of endoscope instruments (white light, narrow-band light), the input of the model is divided into weak perturbation branch, strong perturbation branch, and style transformation branch. Weak perturbation branch: Perform random cropping, random transformation and other enhancement operations on the image. Used to generate higher quality pseudo labels for model training. Strong perturbation branch: Perform color jitter, Gaussian blur, random grayscale transformation and cropping mixing operations on the input image, which is used to make consistency loss with the pseudo labels generated by the weak perturbation branch, push the model decision boundary to the low-density area, and improve the model performance. Style transformation branch: According to the different imaging modalities of the endoscopic image (white light, narrow-band light), the features are aligned through the style transfer network to enhance the robustness of the model.
[0132] Style transfer network: The embodiment of the present invention uses all the data in the training set to train the style transfer network, which includes labeled images and unlabeled images. After the training is completed, the embodiment of the present invention uses the style transfer network to generate a style transformation branch, inputs the white light endoscopic image into the style transfer encoder, obtains the corresponding latent representation, and then inputs the latent representation into the narrow-band light decoder to obtain the narrow-band light image; the same is true for the narrow-band light endoscopic image, which is input into the style transfer encoder, and the output of the encoder is input into the white light decoder to obtain the migration image. Among them, the migrated image will be used as a style perturbation as a branch of training.
[0133] In general, the embodiment of the present invention uses the model's prediction of the weakly perturbed branch as a pseudo-label, and performs self-training loss on the prediction results of other branches. At the same time, in order to better utilize the pseudo-label, the embodiment of the present invention sets a dynamic threshold instead of a fixed high threshold to filter the available pseudo-labels according to the training stage of the model.
[0134] The following will introduce how the model is trained from the perspective of the loss function. The formula can be expressed as:
[0135] L=0.5*L l +0.5*L u
[0136] The overall loss consists of two parts: labeled data loss and unlabeled data loss, and the cross entropy loss is used as the loss function. u , the calculation method is as follows.
[0137]
[0138] Among them, p wis the model’s prediction for the weakly perturbed branch, p s is the model’s prediction for the strongly perturbed branch, p st is the model's prediction result for the image after style transfer. w ) is the model’s prediction confidence for pseudo labels, which is used to screen credible pseudo labels, and γ is a threshold that changes with the model training stage. By constraining the consistency of the model’s predictions for different perturbations, a large amount of unlabeled data can be fully utilized.
[0139] Finally, the embodiment of the present invention inputs the real-time acquired endoscopic laryngeal cancer data into the trained semi-supervised model to obtain the location and condition of the tumor.
[0140] In summary, the present invention can reduce the cost of manual labeling, use a small amount of data to train a model with excellent performance to assist in determining the location and malignancy of the tumor, and improve the generalization ability of the model under different endoscopic imaging conditions.
[0141] Another aspect of the embodiment of the present invention further provides a semi-supervised throat lesion segmentation system based on electronic laryngeal endoscope images, comprising:
[0142] The first module is used to collect endoscopic laryngeal cancer datasets;
[0143] The second module is used to label the endoscopic laryngeal cancer dataset to obtain a label dataset;
[0144] The third module is used to input the endoscopic laryngeal cancer dataset and the label dataset into the model to be trained, perform different degrees of data enhancement through the weak perturbation branch, the strong perturbation branch and the style perturbation branch, and adjust the parameters of the model to be trained based on the dynamic threshold and the preset loss function to obtain the target model;
[0145] The fourth module is used to predict the electronic laryngeal endoscope image to be identified according to the target model to obtain a laryngeal lesion prediction result.
[0146] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0147] The embodiment of the present invention further provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned semi-supervised throat lesion segmentation method based on electronic laryngeal endoscope images when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.
[0148] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0149] See also Figure 5 , Figure 5 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0150] The processor 501 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention;
[0151] The memory 502 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 502, and the processor 501 calls and executes the semi-supervised throat lesion segmentation method based on electronic laryngeal endoscope images of the embodiment of the present invention;
[0152] Input / output interface 503, used to implement information input and output;
[0153] Communication interface 504, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);
[0154] A bus 505 that transmits information between the various components of the device (e.g., the processor 501, the memory 502, the input / output interface 503, and the communication interface 504);
[0155] The processor 501 , the memory 502 , the input / output interface 503 and the communication interface 504 are connected to each other in communication within the device via the bus 505 .
[0156] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned semi-supervised laryngeal lesion segmentation method based on electronic laryngeal endoscope images.
[0157] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0158] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0159] It should be noted that in various specific embodiments of the present invention, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present invention needs to obtain the user's sensitive personal information, it will obtain the user's separate permission or consent through a pop-up window or jump to a confirmation page, and after clearly obtaining the user's separate permission or consent, it will obtain the necessary user-related data for the normal operation of the embodiment of the present invention.
[0160] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.
[0161] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.
[0162] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.
[0163] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.
[0164] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.
[0165] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0166] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0167] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.
[0168] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.
Claims
1. A semi-supervised laryngeal lesion segmentation method based on electronic laryngeal endoscope images, characterized in that: The following steps are involved: Acquisition of endoscopic laryngeal cancer dataset; Annotating the endoscopic laryngeal cancer dataset to obtain a labeled dataset; The endoscopic laryngeal cancer dataset and the label dataset are input into the model to be trained, different degrees of data enhancement are performed through a weak perturbation branch, a strong perturbation branch, and a style perturbation branch, and parameters of the model to be trained are adjusted based on a dynamic threshold and a preset loss function to obtain a target model; The electronic laryngeal endoscope image to be identified is predicted according to the target model to obtain a laryngeal lesion prediction result.
2. According to the semi-supervised laryngeal lesion segmentation method based on electronic laryngeal endoscope images according to claim 1, it is characterized in that: The method of performing different degrees of data enhancement through a weak perturbation branch, a strong perturbation branch, and a style perturbation branch includes the following steps: Performing a first data enhancement process on the original data set through a weak perturbation branch, wherein the first data enhancement process includes a random cropping process and a level change process; Performing a second data enhancement process on the original data set through a strong perturbation branch, wherein the second data enhancement process includes a color enhancement process; Performing a third data enhancement process on the original data set through a style perturbation branch; A data set to be trained is obtained according to the results of the first data enhancement processing, the second data enhancement processing and the third data enhancement processing.
3. The semi-supervised laryngeal lesion segmentation method based on electronic laryngeal endoscope images according to claim 2 is characterized in that: The performing a third data enhancement process on the original data set by using a style perturbation branch comprises the following steps: Using the style transfer method, the light transfer image is synthesized through the style transfer network; The light migration image is used as a disturbance factor and combined with the image data in the original data set to complete the third data enhancement processing of the original data set.
4. The semi-supervised laryngeal lesion segmentation method based on electronic laryngeal endoscope images according to claim 1 is characterized in that: The training process of the model to be trained includes three types of losses, namely, strong and weak disturbance consistency loss based on dynamic threshold, style consistency loss based on dynamic threshold, and feature consistency loss based on dynamic threshold. The parameter adjustment of the model to be trained based on the dynamic threshold and the preset loss function includes the following steps: Construct a style transfer network based on autoencoder; Through the style transfer network, an encoder is used to map both the white light image and the narrow-band light image into a shared feature space, and then two different decoders are used to generate reconstructed images with different styles respectively; A dynamic threshold is used to replace a fixed high threshold, and the model to be trained is trained in combination with the reconstructed image to predict the pseudo labels corresponding to each data to be trained.
5. The semi-supervised laryngeal lesion segmentation method based on electronic laryngeal endoscope images according to claim 4 is characterized in that: The method uses the style transfer network to map both the white light image and the narrow-band light image into a shared feature space using an encoder, and then uses two different decoders to generate reconstructed images with different styles, including the following steps: Use an encoder to map both the white light image and the narrow-band light image into a shared feature space, thereby obtaining a first potential representation of the white light image after passing through the encoder and a second potential representation of the narrow-band light image after passing through the encoder; Inputting the first latent representation and the second latent representation into a white light decoder to obtain a reconstructed white light image and a migrated white light image; inputting the first latent representation and the second latent representation into a narrow-band light decoder to obtain a reconstructed narrow-band light image and a migrated narrow-band light image; The migrated white light image and the migrated narrow-band light image are input into an encoder to obtain a latent representation of the migrated white light image after passing through the encoder and a latent representation of the migrated narrow-band light image after passing through the encoder.
6. The semi-supervised throat lesion segmentation method based on electronic laryngeal endoscope images according to claim 4 is characterized in that: The method of using a dynamic threshold to replace a fixed high threshold, training the model to be trained in combination with the reconstructed image, and predicting the pseudo labels corresponding to each data to be trained includes the following steps: For each image input into the model to be trained, an average confidence of the corresponding segmented area is obtained by the model to be trained; Determining a global threshold and a local threshold of a training process according to the average confidence; Calculate a final threshold value according to the global threshold value and the local threshold value; The model to be trained is trained according to the final threshold value and combined with the reconstructed image to predict pseudo labels corresponding to each data to be trained.
7. The semi-supervised throat lesion segmentation method based on electronic laryngeal endoscope images according to claim 1 is characterized in that: The loss functions include feature alignment loss function, image reconstruction loss function, structural similarity loss function, and style adversarial loss function; Wherein, the feature alignment loss function is used to enable the decoder to sample from the same distribution; The image reconstruction loss function is used to enable the decoder to synthesize an image of a corresponding style; The structural similarity loss function is used to maximize the structural similarity between the latent representations of different images; The style adversarial loss function is used to make the pictures generated by the two decoders conform to the corresponding styles.
8. A semi-supervised throat lesion segmentation system based on electronic laryngeal endoscope images, characterized in that: include: The first module is used to collect endoscopic laryngeal cancer datasets; The second module is used to label the endoscopic laryngeal cancer dataset to obtain a label dataset; The third module is used to input the endoscopic laryngeal cancer dataset and the label dataset into the model to be trained, perform different degrees of data enhancement through the weak perturbation branch, the strong perturbation branch and the style perturbation branch, and adjust the parameters of the model to be trained based on the dynamic threshold and the preset loss function to obtain the target model; The fourth module is used to predict the electronic laryngeal endoscope image to be identified according to the target model to obtain a laryngeal lesion prediction result.
9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Method and system for generating virtual and augmented reality
CN115359567A
Semi-supervised image classification method and semi-supervised image classification system
CN116894985A
Visual communication design method based on artificial intelligence
CN117011129A
Efficient teacher-student semi-supervised segmentation method and device based on epistaxis endoscope image
CN117197472A
Semi-supervised lung cancer medical image segmentation method and device based on LesionMix and entropy minimization
CN117611596A
Cited By
Intelligent diagnosis and monitoring system for otolaryngology department based on multi-modal data fusion
CN120107271A
Gastric cancer artificial intelligence screening method based on plain-scan CT (Computed Tomography) image
CN121837278A
