A glottal state monitoring method and system based on weakly supervised learning

Abstract

The application discloses a glottal state monitoring method and system based on weak supervision learning, comprising the following steps: acquiring a glottal image and point labeling the glottis, and constructing a glottal segmentation dataset; generating pseudo-labeling of the glottal boundary from the point labeling based on an automatic seed region growing algorithm, and updating the glottal segmentation dataset; designing a dynamic structure similarity loss function based on the updated glottal segmentation dataset, and training a convolutional neural network model under weak supervision; extracting a glottal image in real time from a video stream, and segmenting the real-time extracted glottal image based on the trained convolutional neural network model; displaying the segmentation result based on an operation display platform, drawing a glottal area change curve, and giving an early warning for abnormal conditions. The application solves the problem that a medical image dataset is difficult to construct, and overcomes the defect that a large amount of time is required for manufacturing an existing deep learning dataset.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing analysis and understanding, and particularly relates to a glottal state monitoring method and system based on weakly supervised learning. BACKGROUND

[0002] Tracheal intubation is a common surgical procedure for establishing an artificial airway for patients during clinical surgery. During the tracheal intubation procedure, anesthesiologists need to monitor the patient's glottal state, i.e., the degree of opening and closing, in real time through endoscopic images. The glottis is a valve that controls airflow into and out of the respiratory tract. There are four basic states of the glottal passage: open, vibrating, narrowing, and closed. Glottal closure can cause hypoxia and even asphyxia in patients. However, long-term interpretation of endoscopic videos can cause cognitive burden on anesthesiologists. In addition, endoscopic images often have low texture and motion blur, which increases the difficulty of image recognition. False positives can lead to surgical complications. Therefore, automatic segmentation and monitoring of the glottal state is essential in clinical surgery.

[0003] Automatic segmentation of the glottis from endoscopic images is a prerequisite for glottal state monitoring. Several solutions for glottis segmentation have been proposed in the field of computer vision, such as threshold segmentation, seed region growing, and active shape model algorithms. However, these traditional methods perform poorly on anatomical regions with complex structures because they rely too much on image contrast and image texture. In addition, feature extraction in these methods relies heavily on prior knowledge, reducing the generality of the method.

[0004] Due to its ability to learn from data, deep neural networks have been used for medical image segmentation. Several supervised learning methods have been developed to train glottis segmentation networks. For example, Bin Ji et al. used a double U-shaped convolutional neural network to segment laryngeal leukoplakia in images taken under laryngoscopy. Laves et al. used FCNN to segment a dataset of human larynx. Glottis segmentation based on supervised learning often requires a large amount of pixel-labeled dataset for training, and endoscopic images often contain complex human anatomy that requires professional doctors to label, making it difficult to obtain a large amount of labeled data.

[0005] Weakly supervised learning uses point, scribble, bounding box, and other image-level weak labels, which not only greatly saves the workload of manual labeling, but also improves the autonomy of medical image processing. Due to the availability of only a small amount of supervision signal, the implementation of weakly supervised learning is challenging, for example, self-training methods have difficulty obtaining accurate boundaries of target regions, especially on medical images, which cannot achieve accurate image segmentation, and the learning speed of neural networks is slow.

[0006] Therefore, there is an urgent need to propose a glottal state monitoring method and system based on weakly supervised learning to overcome the defect that the existing deep learning dataset preparation requires a large amount of time. SUMMARY

[0007] The purpose of the present application is to provide a glottal state monitoring method and system based on weakly supervised learning, in order to overcome the defect that the existing deep learning data set requires a lot of time, a weakly supervised deep learning method is provided for training only by marking points as annotation, which can solve the difficulty of medical image annotation requiring a lot of time, and at the same time, based on the proposed weakly supervised deep learning method, an automatic glottal state monitoring system is constructed, which can assist in understanding the anesthesia state and increase the reliability of anesthesia to solve the above-mentioned problems existing in the prior art.

[0008] In order to achieve the above-mentioned purpose, the present application provides a glottal state monitoring method based on weakly supervised learning, comprising the following steps:

[0009] Obtain glottal images and point label glottis, and construct a glottal segmentation data set;

[0010] Based on the automatic seed region growing algorithm, the pseudo-label of the glottal boundary is iteratively generated from the point label, and the glottal segmentation data set is updated;

[0011] Based on the updated glottal segmentation data set, a dynamic structure similarity loss function is designed, and a convolutional neural network model under weak supervision is trained;

[0012] Real-time extraction of glottal images from video streams, and segmentation of real-time extracted glottal images based on the trained convolutional neural network model;

[0013] Based on the operation display platform, the segmentation results are displayed, the glottal area change curve is drawn, and the abnormal situation is warned.

[0014] Optionally, the process of iteratively generating pseudo-labels of glottal boundaries from point labels based on the automatic seed region growing algorithm comprises: obtaining the probability of each pixel in the glottal image belonging to the glottis, dividing the glottal image into background region, uncertain region and glottal region based on the probability value; taking the glottal region as a seed region, obtaining the mean gray value and gray value variance of the seed region, based on the mean gray value and gray value variance of the seed region, traversing all pixels in the uncertain region, and changing the uncertain region where the pixel points meeting the preset requirements to the foreground region. The screening of all uncertain region pixel points is completed, the final foreground region is obtained, and the pseudo-label of the glottal boundary is obtained. Wherein, the background region is the region of the glottal image except the glottal region.

[0015] Optionally, the process of traversing all pixels in the uncertain region comprises: taking a pixel of a seed region as a starting point, traversing all pixels in the 8-neighborhood, and regarding a pixel as a seed region when the difference between the gray value of the pixel and the gray mean value of the seed region is less than the gray value variance of the seed region; continue traversing the 8-neighborhood of the pixel until the difference between the gray value of all pixels in the neighborhood of the pixel and the gray mean value of the seed region is not less than the gray value variance of the seed region, stop traversing, and change the uncertain region where the pixel meeting the preset requirement is located into a foreground region, wherein the seed region is an initial foreground region.

[0016] Optionally, the process of training the convolutional neural network model under weak supervision comprises: obtaining a newly diffused region through iterative diffusion of pseudo-labeling; obtaining a structure similarity coefficient of the newly diffused region and a seed region based on the dynamic structure similarity loss function; and stopping training of the convolutional neural network model when the structure similarity coefficient is less than a preset threshold, to obtain a trained convolutional neural network model.

[0017] The application further provides a glottal state monitoring system based on weak supervision learning, comprising:

[0018] An image acquisition module is configured to acquire glottal images and construct a glottal segmentation dataset based on the glottal images.

[0019] A glottal segmentation module is connected to the image acquisition module and configured to construct a convolutional neural network model, train the convolutional neural network model based on the glottal segmentation dataset, and then segment real-time extracted glottal images based on the trained convolutional neural network model.

[0020] A display module is connected to the glottal segmentation module and configured to display the segmentation results on an operation display platform and draw a glottal area change curve.

[0021] Optionally, the image acquisition module comprises:

[0022] A dataset construction unit is configured to acquire glottal images, point-label glottals, and construct a glottal segmentation dataset.

[0023] A dataset updating unit is configured to generate pseudo-labels of glottal boundaries from point-labeling through iterative generation based on an automatic seed region growing algorithm, and update the glottal segmentation dataset.

[0024] Optionally, the glottal segmentation module comprises:

[0025] A model training unit is configured to construct a convolutional neural network model, design a dynamic structure similarity loss function based on the updated glottal segmentation dataset, and train the convolutional neural network model.

[0026] A glottis segmentation unit is configured to segment the real-time extracted glottis image based on the trained convolutional neural network model.

[0027] Optionally, the display module comprises:

[0028] A display unit is configured to display the segmentation result based on the operation display platform and draw a glottis area change curve.

[0029] A warning unit is configured to warn an abnormal condition based on the glottis area change curve.

[0030] The technical effect of the present application is:

[0031] The glottis state monitoring method and system based on weakly supervised learning provided by the present application only need to make a point mark in the input image as a label for training, and an ideal glottis segmentation model can be obtained; meanwhile, an automatic glottis state monitoring system is constructed using an offline glottis segmentation model based on weakly supervised deep learning, which utilizes the idea of constantly updating the data set in the iterative learning of deep learning, including a threshold seed region growing algorithm and a dynamic structure similarity loss function, solves the problem of difficult construction of medical image data sets, and can better assist in completing the anesthesia task and improve the reliability of anesthesia. BRIEF DESCRIPTION OF DRAWINGS

[0032] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application illustrated in the drawings, and their description, are presented to add generic scope to this application. In the drawings:

[0033] Figure 1 The glottis state monitoring method based on weakly supervised learning in the embodiment of the present application is shown in the flow chart.

[0034] Figure 2 The operation flow chart of training the convolutional neural network based on weakly supervised deep learning in the embodiment of the present application is shown.

[0035] Figure 3 The structure schematic diagram of the convolutional neural network for glottis segmentation in the embodiment of the present application is shown.

[0036] Figure 4 The structure schematic diagram of the glottis state monitoring system based on weakly supervised learning in the embodiment of the present application is shown. DETAILED DESCRIPTION

[0037] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0038] It is noted that the steps shown in the flowcharts of the drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases the steps shown or described can be executed in an order different from that shown here.

[0039] Embodiment one

[0040] As shown in the figure, the embodiment provides a weakly supervised deep learning-based automatic glottis state monitoring method and system in anesthesia, comprising the following steps: Figures 1-3

[0041] A point in the glottis region of the input image is marked as a pseudo-label input network, and a convolutional neural network model (U-net) under weak supervision is trained. Specifically, the glottis in the input image is weakly labeled in the form of point labeling, and a glottis segmentation dataset is constructed; the automatic seed region growing algorithm is used to iteratively generate pseudo-labels close to the glottis boundary from the point labeling, and the glottis segmentation dataset is updated; using the updated dataset, a dynamic structure similarity loss function is designed, and a convolutional neural network model under weak supervision is trained. The convolutional neural network model uses an existing U-net network, which is iteratively trained by inputting labeled pictures and real pictures.

[0042] Using the iterative process of deep learning and combining the designed automatic seed region growing algorithm, the dataset is continuously iteratively updated, so that the pseudo-labels gradually spread from the initial one point to the entire glottis region. The steps of the automatic seed region growing algorithm are as follows:

[0043] a. The neural network model calculates the probability of each pixel of the image belonging to the glottis, and divides the image into three regions according to the probability: background region A b , uncertain region A u , and glottis region A g .

[0044] b. Take A g as the seed region, calculate the mean gray value and the gray value variance of the seed region. For each pixel x in the uncertain region A u , do the following operations: first, take the pixel x0 of the seed region as the starting point, traverse all pixels x1 in the 8-neighborhood of x0, and the gray value of x1 is If the formula is true, then the pixel x1 is considered as a seed region; continue to visit the 8-neighborhood of this pixel until a pixel point is found whose all adjacent domains do not satisfy the formula u ​Iterate through the entire process once; finally, calculate the uncertain region A. u All values ​​in the range can transform some uncertain areas into foreground areas.

[0045] The steps for calculating the dynamic structural similarity loss function based on the network segmentation results are as follows:

[0046] a. The pseudo-labels gradually spread, continuously approaching the glottal boundary. In each iteration of diffusion, the newly diffused region is represented as ΔA. g (x) t+1 =A g (x) t+1 -A g (x) t ;

[0047] b, ΔA located within the glottic region g (x) t+1 and A g (x) t The image structure, brightness, contrast, and other features are not significantly different, when ΔA g (x) t+1 When it exceeds the glottal boundary, ΔA g (x) t+1 and A g (x) t The images differ significantly in features. Using the formula: Calculate ΔA g (x) t+1 and A g (x) t Structural similarity coefficient SSIM(ΔA) g (x) t+1 A g (x) t When the calculated result is less than the threshold σ, the network stops the TSRG algorithm, and the structural similarity coefficient is set to 0, so that the training of the entire network gradually becomes stable.

[0048] Images are extracted frame by frame from the intraoperative endoscopic video stream. The glottis in the images is segmented using a trained network model, and the area of ​​the glottis is calculated. The glottis segmentation results are displayed on the operating display panel, and the curve of the change in glottis area is plotted on the display panel. If the glottis closes or jitters in the video stream, the system will provide an alarm signal to the doctor.

[0049] The steps for calculating the area of ​​the glottis are as follows:

[0050] a. Train the offline network model using the weakly supervised method described above;

[0051] b. Obtain dynamic video of the glottis from the endoscope;

[0052] c. Extract the dynamic video frame by frame, segment the glottis using a model, and calculate the glottis area.

[0053] The segmentation results are displayed on the operation display panel, and the curve of the change in glottal area is plotted on the display panel simultaneously. The steps are as follows:

[0054] a. Display the segmentation results using the operation display panel;

[0055] b. Plot the curve of glottal area change on the display panel. If the glottis closes or jitters in the video stream, the system will provide an alarm signal to the doctor.

[0056] This embodiment presents an automatic monitoring method and system for glottal state during anesthesia based on weakly supervised deep learning. During the model training phase, a point is marked as a pseudo-label in the glottal region. Then, during iterative network training, a threshold seed region growing algorithm is used to gradually spread the pseudo-label to the glottal boundary. Simultaneously, a dynamic structural similarity loss function is used to supervise network learning. In the automatic glottal state monitoring system phase, an offline network model trained using weakly supervised deep learning is used to segment the input video, calculate the glottal area, obtain a polygonal line showing changes in the glottal area, and automatically monitor the patient's anesthesia state based on these changes.

[0057] Example 2

[0058] like Figure 4 As shown, this embodiment provides a glottal state monitoring system based on weakly supervised learning, including:

[0059] An image acquisition module is used to acquire glottal images and construct a glottal segmentation dataset based on the glottal images;

[0060] The glottis segmentation module, connected to the image acquisition module, is used to construct a convolutional neural network model, train the convolutional neural network model based on the glottis segmentation dataset, and then segment the real-time extracted glottis image based on the trained convolutional neural network model.

[0061] The display module, connected to the glottis segmentation module, is used to display the segmentation results based on the operation display platform and to draw the glottis area change curve.

[0062] Implementable, the image acquisition module includes: a dataset construction unit for acquiring glottal images and labeling the glottis with points to construct a glottal segmentation dataset; and a dataset update unit for iteratively generating pseudo-labels for the glottal boundaries from the point labels based on an automatic seed region growing algorithm to update the glottal segmentation dataset.

[0063] As a specific example, the steps of the automatic seed region growth algorithm are as follows:

[0064] The neural network model calculates the probability that each pixel in the image belongs to the glottis, and divides the image into three regions according to the probabilities: 0-β, β-0.5, and 0.5-1 (β is a preset constant between 0 and 0.5): background region A. b Uncertain region A u Glottic region A g ;

[0065] With glottal region A g As a seed region, calculate the average gray value of the seed region. and gray value variance For the uncertain region A u For each pixel x in the seed region, perform the following operation: First, starting with pixel x0 in the seed region, iterate through all pixels x1 in the 8-neighborhood of x0, where the gray value of x1 is... If the formula If the condition is met, then pixel x1 will be considered the seed region; continue to visit the 8-neighbor domain of that pixel until it is found that all the neighbor domains of a certain pixel do not satisfy the formula. Or consider all uncertain regions A u Iterate through the entire process once; finally, calculate the uncertain region A. u All values ​​in the range can transform some uncertain areas into foreground areas.

[0066] The implementable glottis segmentation module includes: a model training unit for constructing a convolutional neural network model, designing a dynamic structural similarity loss function based on the updated glottis segmentation dataset, and training the convolutional neural network model; and a glottis segmentation unit for acquiring dynamic video of the glottis from the endoscope, extracting glottis images frame by frame in real time from the dynamic video, and segmenting the real-time extracted glottis images based on the trained convolutional neural network model.

[0067] As a specific example, the steps of the dynamic structural similarity loss function are as follows:

[0068] The pseudo-labels gradually spread, approaching the glottal boundary. In each iteration of diffusion, the newly diffused region is represented as ΔA. g (x) t+1 =A g (x) t+1 -A g (x) t ;

[0069] ΔA located within the glottic region g (x) t+1 and A g (x) t The image structure, brightness, contrast, and other features are not significantly different, when ΔA g (x) t+1ΔA g (x) t+1 and A g (x) t have great differences in features. Using the formula: ΔA g (x) t+1 and A g (x) t , the structural similarity coefficient SSIM(ΔA g (x) t+1 , A g (x) t ) is calculated, when the calculation result is less than the threshold σ, the network stops the TSRG algorithm, and the structural similarity coefficient is 0, so that the whole network training gradually tends to be stable.

[0070] The display module can be implemented, and the display module includes a display unit configured to display the segmentation result based on the operation and draw a glottal area change curve; and a warning unit configured to warn an abnormal situation based on the glottal area change curve.

[0071] The above merely describes a preferred specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, and all these changes or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A glottal state monitoring method based on weakly supervised learning, characterized in that, The method comprises the following steps: acquiring glottis images and performing point labeling on the glottis to construct a glottis segmentation dataset; generating pseudo-labeling of the glottis boundary from the point labeling based on an automatic seed region growing algorithm, and updating the glottis segmentation dataset; designing a dynamic structural similarity loss function based on the updated glottis segmentation dataset, and training a convolutional neural network model under weak supervision; extracting glottis images from a video stream in real time, and segmenting the real-time extracted glottis images based on the trained convolutional neural network model; displaying the segmentation results on an operation demonstration platform, drawing a glottis area change curve, and warning of abnormal conditions; The process of generating pseudo-labeling of the glottis boundary from the point labeling based on the automatic seed region growing algorithm comprises: acquiring the probability of each pixel in the glottis image belonging to the glottis, dividing the glottis image into a background region, an uncertain region and a glottis region based on the probability value; taking the glottis region as a seed region, acquiring the mean and variance of the gray value of the seed region, and based on the mean and variance of the gray value of the seed region, traversing all pixels in the uncertain region, changing the uncertain region where the pixel points meeting the preset requirements are located into a foreground region, completing the screening of all uncertain region pixel points, and obtaining the final foreground region, and further obtaining the pseudo-labeling of the glottis boundary; wherein the background region is a region in the glottis image other than the glottis region; The process of traversing all pixels in the uncertain region comprises: taking the pixel of the seed region as a starting point, traversing all pixels in the 8-neighborhood, and when the difference between the gray value of the pixel in the 8-neighborhood and the mean of the gray value of the seed region is less than the variance of the gray value of the seed region, regarding the pixel as a seed region; continue to traverse the 8-adjacent domain of the pixel until the difference between the gray value of all adjacent domain pixels of the pixel and the mean of the gray value of the seed region is not less than the variance of the gray value of the seed region, stop traversing, and change the uncertain region where the pixel points meeting the preset requirements are located into a foreground region; wherein the seed region is an initial foreground region; The process of designing a dynamic structural similarity loss function and training a convolutional neural network model under weak supervision comprises: the pseudo-labeling is iteratively diffused to obtain a newly diffused region; the structural similarity coefficient of the newly diffused region and the seed region is obtained based on the dynamic structural similarity loss function; when the structural similarity coefficient is less than a preset threshold, the training of the convolutional neural network model is stopped, and a trained convolutional neural network model is obtained.

2. A weakly supervised learning based glottal state monitoring system, characterized in that, For implementing the method of claim 1, comprising: an image acquisition module, configured to acquire glottis images and construct a glottis segmentation dataset based on the glottis images; a glottis segmentation module, connected with the image acquisition module, configured to construct a convolutional neural network model, train the convolutional neural network model based on the glottis segmentation dataset, and then segment real-time extracted glottis images based on the trained convolutional neural network model; a display module, connected with the glottis segmentation module, configured to display the segmentation results on an operation demonstration platform, and draw a glottis area change curve; the image acquisition module comprises: A data set construction unit is configured to acquire glottis images and perform point labeling on the glottis to construct a glottis segmentation data set; A data set updating unit is configured to iteratively generate pseudo-labeling of a glottis boundary from the point labeling based on an automatic seed region growing algorithm, and update the glottis segmentation data set; The glottis segmentation module comprises: A model training unit is configured to construct a convolutional neural network model, design a dynamic structure similarity loss function based on the updated glottis segmentation data set, and train the convolutional neural network model; A glottis segmentation unit is configured to segment a real-time extracted glottis image based on the trained convolutional neural network model; The display module comprises: A display unit is configured to display the segmentation result based on the operation display platform and draw a glottis area change curve; An early warning unit is configured to perform early warning on abnormal conditions based on the glottis area change curve.

Images