An automatic organ segmentation method based on laryngoscope medical images

Abstract

The present invention belongs to the field of computer application technology, and in particular, relates to an automatic organ segmentation method based on laryngoscope medical images. The method of the present invention includes: through a hybrid Transformer encoding module, long-range dependencies and multi-scale contexts can be effectively encoded, so that the model can effectively capture semantic features of different degrees, thereby improving the detail segmentation effect; through an RRM decoding module, in the process of "erasing" from high-level semantic information to underlying semantic information, the inaccurate segmentation area is gradually refined into a complete area; through a convolutional conditional random field module, the segmentation result is optimized using a probability model, and smaller mis-segmented areas are eliminated and the segmentation is refined. The present invention solves the problem of inaccurate laryngeal organ segmentation in the prior art by successively establishing organ context information, distinguishing organs from soft tissue structures, establishing long-distance dependency coding in laryngoscope images, and fusing local features and global features. By comparison, the organ segmentation structure is superior to other current SOTA models, and mIOU is improved by 3.9% and 2.53% respectively compared with the baseline model.

Description

Technical field:

[0001] The invention belongs to the technical field of computer application, and in particular relates to an automatic organ segmentation method based on laryngoscope medical images. Background technology:

[0002] The larynx is a delicate organ with a deep location and complex anatomical structure and physiological functions. It requires electronic laryngoscopes for early detection of small lesions and tumors. However, in clinical diagnosis, electronic laryngoscope image data is huge, manual diagnosis is time-consuming and costly, and prone to missed detection and misdetection. Therefore, computer vision technology can be used to assist doctors in detecting lesions. Organ segmentation, as the research basis of lesion detection and the basic process of medical image reading, helps doctors diagnose diseases based on the lesions and the morphology of the organs in which they are located.

[0003] With the development of DL (deep learning), CNN (Convolutional Neural Networks) has dominated the field of image segmentation. In particular, the fully convolutional network (FCN) has pioneered a new idea for semantic segmentation with an encoding-decoding structure. Inspired by FCN, U-Net combines deep semantic information and spatial information through skip connections, achieving excellent results in various visual tasks. However, U-Net still has shortcomings: first, local constraints limit the model's receptive field, which is crucial for tasks such as segmentation; at the same time, due to the inherent inductive bias of CNN, each convolution kernel can only focus on a sub-region of the entire image, which makes it lose the global context and therefore cannot establish long-term dependencies. Summary of the invention:

[0004] The present invention proposes an automatic organ segmentation method based on laryngoscope medical images. The method solves the problem of inaccurate laryngeal organ segmentation in the prior art by successively establishing organ context information, distinguishing organs from soft tissue structures, establishing long-distance dependency coding in laryngoscope images, and fusing local features with global features.

[0005] In order to achieve the purpose of the present invention, the technical solution provided by the present invention is: an automatic organ segmentation method based on laryngoscope medical images: comprising a hybrid Transformer encoding module, an RRM decoding module and a convolutional conditional random field module, wherein the hybrid Transformer encoding module is composed of a CNN paradigm and a Transformer paradigm, and the RRM decoding module is composed of a reverse attention mechanism and a receptive field enhancement block. The method comprises the following steps

[0006] Step 1: The original image is first processed through a dilated convolution to extract local relations and structural prior information at high resolution, and then the hierarchical Transformer network-PVT is used to capture global context information and generate feature maps of different scales {f 1 ,f 2 ,f 3 ,f 4};

[0007] Step 2: Different scale feature maps {f 1 ,f 2 ,f 3 ,f 4}Use RRM modules to process from top to bottom, and superimpose the upper layer output in a cascade manner to obtain {S 4 ,S 3 ,S 2 ,S 1}, forming a progressive segmentation framework, and the superposition result of the last layer is the preliminary segmentation result.

[0008] Step 3: Optimize the preliminary segmentation result obtained in step 2 through the convolutional conditional random field module, remove smaller mis-segmented areas and refine the segmentation to obtain the final segmentation result.

[0009] Furthermore, in the above hybrid Transformer encoding module

[0010] (1) Generate a feature map f with a larger receptive field through a dilated convolution with a dilated rate of 2 and a convolution kernel of 3*3 or 4*4. 0 , obtain rich semantic information;

[0011] (2) The obtained feature map f 0 Through the four stages of the PVT network, feature maps of different sizes are generated

[0012] Where: H, W represent the image resolution, C represents the number of image channels, f i Represents the feature map generated by the i-th layer.

[0013] Furthermore, the processing of the RRM module in step 2 above is divided into two steps: (1) the output S of the previous layer is i Perform reverse attention processing to obtain A i , then A i And the feature map f i Perform point multiplication fusion to get R i ; (2) Through the receptive field enhancement block, R i Actively mine feature information through different void volumes.

[0014] Furthermore, the above step (1) is specifically expressed as follows:

[0015] R i =f i ⊙A i (1)

[0016] Reverse attention weight A i It has been widely used in salient object detection tasks. It can be expressed as:

[0017] A i =1-σ(UP(S i )) (2)

[0018] Among them, σ(·) represents the Sigmoid activation function, and up(·) represents the upsampling operation using bilinear interpolation.

[0019] By adding reverse attention to suppress non-target areas, the network can be effectively guided to explore missing target areas and details.

[0020] In step (2), R i Mining feature maps with different degrees of semantic features, and finally using 1×1 convolution for semantic fusion to obtain S i .

[0021] Furthermore, the above-mentioned convolutional conditional random field module supplements the conditional independence assumption based on the fully connected conditional random field. If the Manhattan distance d between pixels i and j satisfies d(i,j)>k, it is assumed that i and j are conditionally independent, and k is the filtered distance size.

[0022] Compared with the prior art, the advantages of the present invention are:

[0023] 1. In the hybrid transformer encoding module, the original image is first subjected to a dilated convolution operation to extract local relations and structural prior information at high resolution. Compared with ordinary convolution, it can capture information in a larger area and more context. Secondly, unlike the existing Vision Trasnformer network, due to its fixed patch size, its output resolution is relatively low and fixed, which is not suitable for intensive visual tasks. Based on this, the present invention combines the PVT module on top of the existing Trasnformer network and sets the patch size of each stage to p. i , the number of patches that each stage segments the feature map of the previous stage is Linear Mapping to C i Dimension, this method reduces the length and width of each stage to the input p itimes, so that the size of the feature map at each stage can be flexibly adjusted, and global context information can be captured based on low-level spatial detail features. It effectively encodes long-range dependencies and multi-scale contexts, enabling the model to effectively capture semantic features of different degrees.

[0024] 2. In the RRM module, the reverse attention and receptive field enhancement blocks are combined to transform the feature maps of different scales {f 1 ,f 2 ,f 3 ,f 4}Use RRM modules to process from top to bottom, and superimpose the upper layer output in a cascade manner to obtain {S 4 ,S 3 ,S 2 ,S 1}, in the process of “erasing” from high-level semantic information to low-level semantic information, the inaccurate segmented area is gradually refined into a complete area, thus generating a progressive segmentation framework. The superposition result of the last layer is the preliminary segmentation result. The processing of the RRM module is divided into two parts. The first part first converts the output S of the previous layer into i Perform reverse attention processing to obtain A i , then A i And the feature map f i Perform point multiplication fusion to get R i In the second part, R i The feature information is actively mined through different void volumes. Finally, in the process of "erasing" from high-level semantic information to low-level semantic information, the inaccurate segmentation area is gradually refined into a complete area, making the segmentation accuracy of the framework more accurate and better than the existing methods.

[0025] 3. Taking the problem of pixel islands caused by the network's positioning accuracy on the boundary of the segmentation result and the influence of large mucosal tissue as the starting point, the final segmentation result and the conditional random field of the probability model are combined with the advantage of the probability model. The network segmentation results are optimized using the improved probability model ConvCRF. By eliminating smaller mis-segmented areas and refining the segmentation, the purpose of efficient and accurate segmentation is achieved. Experiments have shown that the present invention is superior to other current SOTA models, and mIOU is improved by 3.9% and 2.53% respectively compared with the baseline model.

[0026] 4. This framework can segment the electronic laryngoscope image in the laryngeal scene and automatically extract the vocal cords, epiglottis, pyriform sinus, lesions and surgical instrument contours. Description of the drawings:

[0027] Figure 1 It is an overall framework diagram of an automatic organ segmentation method based on laryngoscope medical images of the present invention;

[0028] Figure 2 This is the framework diagram of the hybrid Transformer encoder;

[0029] Figure 3 This is the segmentation effect diagram of the RRM module;

[0030] Figure 4 This is the effect diagram after reverse attention, where (a) represents S i , (b) represents the result after σ(·), (c) represents the result after Reverse Attention;

[0031] Figure 5 This is the framework diagram of the convolutional conditional random field module;

[0032] Figure 6 Prediction graph for segmentation optimization of this method on dataset 1;

[0033] Figure 7 Prediction graph for segmentation optimization of this method on dataset 2. Specific implementation method:

[0034] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0035] See also Figure 1 , the embodiments of the present invention use two data sets for experiments. Data set 1 includes routine examination videos of 350 different patients, and each video lasts 2-5 minutes. We screen the video frames, filter out invalid images such as in vitro and blurred images, and crop the invalid borders generated by the equipment in the image. Finally, the data set contains 822 images with a resolution of 768x 768, including vocal cords, epiglottis, pyriform sinus, vocal cord gap, and other tissues. The vocal cord segmentation category here includes two categories: vocal cord open and vocal cord closed, because vocal cord nodules or vocal cord polyps may also make the vocal cords open. This classification facilitates the subsequent inference of the disease based on the vocal cord morphology. The training set and the test set are randomly divided according to 689:133. The labeling process is carried out with the assistance of a doctor and reviewed and confirmed by a senior otolaryngologist.

[0036] Dataset 2 uses a public electronic laryngoscope image dataset from Prof. Giorgio Peretti (Director of Otorhinolaryngology at Ospedale Policlinico San Martino, University of Genova). The images contain 7 categories: vocal cords, other tissues, vocal cord gaps, lesions, surgical instruments, intubations, and blank areas. The dataset consists of five different sequences from two patients, SEQ1-SEQ4 from patient 1, and SEQ5-SEQ8 from patient 2. Each sequence contains the following features:

[0037] SEQ1: There is a clear tumor on the vocal cord before surgery, with translation, rotation, and size changes, without surgical instruments and intubation.

[0038] SEQ2: There is a clear tumor on the vocal cord before surgery, with translation, rotation, and size changes, surgical instruments, and intubation.

[0039] SEQ3-4: Postoperative tumor resection, tissue damage, with translation and size changes, and intubation.

[0040] SEQ5-7: Preoperative surgical instrument operation with translation, size change, and intubation.

[0041] SEQ8: Postoperative vocal cord bleeding, with surgical instruments, surgical dressings, and intubation.

[0042] Embodiment: The present invention provides an automatic organ segmentation method based on laryngoscope medical images, including a hybrid Transformer encoding module, which consists of a CNN paradigm and a Transformer paradigm; an RRM decoding module, which consists of a reverse attention mechanism and a receptive field enhancement block; and a convolutional conditional random field module (ConvCRF).

[0043] The steps of the method include:

[0044] Step 1: See Figure 2 The original image first passes through the hole convolution to extract the local relationship and structural prior information at high resolution, and then captures the global context information through the hierarchical Transformer network-PVT to generate feature maps of different scales {f 1 ,f 2 ,f 3 ,f 4}:

[0045] (1) For a given input C is the number of channels, and H×W represents the image resolution. First, the original image is subjected to a dilated convolution operation through the hybrid Transformer encoding module to obtain f0 The reason why convolution is added before the vision transformer is that the model wants to focus on the local relationships on the initial layers because they contain more high-resolution details. By using convolution, some structural priors can be introduced. While reducing the parameters and maintaining the same receptive field, the image resolution is reduced to reduce the amount of subsequent operations, and the local information of the image is captured on the other hand.

[0046] For large-resolution images, a 4x4 convolution kernel is used, the dilation rate is 2, and the convolution operation of 7x7 is replaced by a convolution with a stride of 2. For relatively small-resolution images, a 3x3 convolution kernel is used, the dilation rate is 2, and the convolution operation of 5x5 is replaced by a convolution with a stride of 1.

[0047] (2) Substitute the f obtained in (1) 0 The feature maps of different scales are generated through the four stages of the PVT (Pyramid vision transformer) network {f i ,i=1,2,3,4}, this method can reduce the length and width of each stage, so that the size of the feature map of each stage can be flexibly adjusted.

[0048] Different from the Vision Transformer network, whose patch size is 16 or 32, resulting in a relatively low and fixed output resolution, which is not suitable for intensive visual tasks. The Pyramid Vision Transformer (PVT) used in this paper sets the patch size p in each stage. i , then the number of patches that each stage segments the feature map of the previous stage is Then through linear mapping to C i Dimension, the size of embedded patches is In this way, the length and width of each stage are reduced to the input P i times, the size of the feature map in each stage can be flexibly adjusted, which enables high-resolution images to be processed under resource-constrained conditions.

[0049] The four different scale feature maps generated by the present invention can be expressed as

[0050]

[0051] The encoding of each stage in PVT consists of an attention layer and a forward propagation layer. According to formula (1), the computational complexity of MHA reaches (hw) 2This requires a huge amount of calculation for high-resolution images. PVT uses the SRA module instead of MHA. The same SRA also takes Q, K and V as input. It can only reduce the spatial scale of K and V through SR(·) before calculating self-attention to reduce the computational overhead. Figure 2 In the figure, the SRA of each stage can be calculated as follows:

[0052]

[0053]

[0054] SR(x)=Norm(Reshape(x,R i )W S ) (4)

[0055] is the linear mapping parameter of Q, K, V. N i represents the number of longs at stage i. SR(·) represents the operation of reducing the spatial scale of K or V. Indicates K or V. R i Reshape(x,R i ) can transform the input x into size. is the transformed linear mapping parameter.

[0056] Finally, self-attention is calculated using formula (5).

[0057]

[0058] By Reshape(x,R i ) operation can reduce the size of K, V space by R i times, according to formula (5), the calculation amount of SRA can be reduced compared with MHA. This enables processing of high-resolution images in resource-constrained situations.

[0059] Step 2: Use the feature maps of different scales {f 1 ,f 2 ,f 3 ,f 4}Use RRM modules to process from top to bottom, and superimpose the upper layer output in a cascade manner to obtain {S 4 ,S 3 ,S 2 ,S 1}, forming a progressive segmentation framework, and the superposition result of the last layer is the preliminary segmentation result.

[0060] The RRM module is composed of a reverse attention mechanism and a receptive field enhancement block to form a top-down cascade segmentation framework, starting from a low-resolution feature map with high semantic information, in a way that erases the foreground. 1 ,f 2 ,f 3 ,f 4}Mining the information and details that are complementary to the foreground to obtain the preliminary segmentation results. The specific steps include:

[0061] (1) First, the output S of the previous layer i Perform reverse attention processing to obtain A i , the multi-scale output {f 1 ,f 2 ,f 3 ,f 4} and the weight A after top-down reverse attention processing i Perform point multiplication to obtain the reverse attention feature R i , as shown in Figure (1)c, can be expressed as:

[0062] R i =f i ⊙A i (6)

[0063] Reverse attention weight A i It has been widely used in salient object detection tasks. It can be expressed as:

[0064] A i =1-σ(UP(S i )) (7)

[0065] σ(·) represents the Sigmoid activation function, and UP(·) represents the upsampling operation using bilinear interpolation. Inverse attention is used to eliminate the target prediction of deep features, effectively allowing the network to explore the exact target area and details.

[0066] (2) The feature map obtained by the reverse attention process is enhanced through the receptive field enhancement block to enhance the network features. The receptive field enhancement block (its structure is as follows Figure 5 The figure shows a multi-branch convolution module, which includes regular convolution layers with different sizes of convolution kernels and dilated convolution layers. The former imitates the receptive fields of humans of different sizes, while the latter reflects the relationship between the size of RFs and the eccentricity in the human visual system. The dilated convolution uses a 3×3 convolution with a dilation rate of 2. Finally, these features are merged through a 3×3 convolution to generate the final prediction result. The specific segmentation effect can be seen in Figure 3 As shown, the reverse attention segmentation effect is as follows Figure 4As shown in the figure, it can be seen that the reverse attention is used to suppress non-target areas and effectively guide the network to explore the missing target areas and details, thus providing effective conditions for efficient segmentation.

[0067] Step 3: To solve the problem of network positioning accuracy of the segmentation result boundary and pixel islands caused by large mucosal tissue, this framework combines the final segmentation result with the probability model conditional random field. Since the training and reasoning speed of the traditional fully connected conditional random field is slow, the present invention adopts ConvCRF, which adds the conditional independence assumption of pixel labels to the fully connected conditional random field framework, so that reasoning can be accelerated on the GPU.

[0068] ConvCRF adds a conditional independence assumption to FullCRF, that is, if the Manhattan distance d between pixel stations i and j satisfies d(i,j)>k, then i and j are assumed to be conditionally independent, and k is the distance size of the filter. The input shape is a tensor P of [b,c,h,w], where b,c,h,w represent the number of batches, the number of categories, and the height and width of the image. Each Gaussian kernel K G With the eigenvector f 1 ,f 2 ,...,f d express.

[0069]

[0070] is a learnable parameter. For a set of Gaussian kernels The combined sum matrix K is:

[0071]

[0072] Then the information transfer Q is defined as:

[0073]

[0074] Formula (10) is similar to the standard 2D convolution operation, so convolution can be used to implement this calculation. By adding the conditional independence assumption of pixel labels in the fully connected conditional random field framework, the reasoning can be accelerated on the GPU based on formulas (8), (9), and (10).

[0075] Finally, the preliminary segmentation result obtained in step 2 is used as the unary potential energy function of the convolutional conditional random field module to optimize the predicted segmentation map. On the one hand, the network improves the accuracy of segmentation boundary positioning and solves the pixel island problem caused by the influence of large mucosal tissues. On the other hand, the segmentation framework achieves efficient and accurate segmentation.

[0076] The optimized prediction graph obtained in the embodiment is evaluated for the throat segmentation effect:

[0077] The evaluation method uses mean Intersection over Union (mIoU) as the key evaluation indicator to measure the similarity between the predicted image and the label. The specific calculation formula is shown in formula (11).

[0078]

[0079] In order to evaluate the effectiveness of the proposed model, the present invention proves the effectiveness of the model from two aspects. (1) By conducting ablation experiments on two data sets, the experimental results are shown in Table 1 and Table 2. (2) The model proposed by the present invention is compared with the SOTA method by conducting experiments on two data sets, and the experimental results are shown in Table 3 and Table 4.

[0080] Table 1. Ablation experiments on dataset I

[0081]

[0082] Table 2 Ablation experiments on dataset II

[0083]

[0084] Table 3 Comparison results of various methods on dataset 1

[0085]

[0086] Table 4 Comparison results of various methods on dataset 2

[0087]

[0088] Compared with other SOTA models, an automatic organ segmentation method based on laryngoscope medical images can make full use of long-range feature dependencies and multi-scale context information to ensure the semantic consistency of feature representation. Figure 3 and Figure 4 ,It can be seen that the best segmentation effect is obtained on both laryngoscope datasets, and the mIoU reaches 76.63% and 88.77% respectively.

[0089] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An automatic organ segmentation method based on laryngoscope medical images: comprising a hybrid Transformer encoding module, an RRM decoding module and a convolutional conditional random field module, wherein the hybrid Transformer encoding module is composed of a CNN paradigm and a Transformer paradigm, and the RRM decoding module is composed of a reverse attention mechanism and a receptive field enhancement block. The method comprises the following steps Step 1: The original image is first processed through a dilated convolution to extract local relations and structural prior information at high resolution, and then the hierarchical Transformer network-PVT is used to capture global context information and generate feature maps {f1, f2, f3, f4} of different scales; Step 2: The feature maps of different scales {f1, f2, f3, f4} are processed from top to bottom using the RRM module, and the upper layer outputs are superimposed in a cascade manner to obtain {S4, S3, S2, S1} respectively, forming a progressive segmentation framework. The superposition result of the last layer is the preliminary segmentation result; Step 3: Optimize the preliminary segmentation result obtained in step 2 through the convolutional conditional random field module, remove smaller mis-segmented areas and refine the segmentation to obtain the final segmentation result.

2. The method for automatic organ segmentation based on laryngoscope medical images according to claim 1, characterized in that: The hybrid Transformer encoding module (1) Generate a feature map f0 with a larger receptive field through a dilated convolution with a dilated rate of 2 and a convolution kernel of 3*3 or 4*4 to obtain rich semantic information; (2) The obtained feature map f0 is passed through the four stages of the PVT network to generate feature maps of different sizes Where: H, W represent the image resolution, C represents the number of image channels, f i Represents the feature map generated by the i-th layer.

3. The method for automatic organ segmentation based on laryngoscope medical images according to claim 1 or 2, characterized in that: The processing of the RRM module in step 2 is divided into two steps: (1) converting the output S of the previous layer i Perform reverse attention processing to obtain A i , then A i And the feature map f i Perform point multiplication fusion to get R i ; (2) Through the receptive field enhancement block, R i Actively mine feature information through different void volumes.

4. The method for automatic organ segmentation based on laryngoscope medical images according to claim 3, characterized in that: Step (1) in step 2 is specifically expressed as: R i =f i ΘA i (1) Reverse attention weight A i It has been widely used in salient object detection tasks and is expressed as: A i =1-σ(UP(S i )) (2) Where σ(.) represents the Sigmoid activation function, and up(.) represents the upsampling operation using bilinear interpolation. By adding reverse attention to suppress non-target areas, the network is effectively guided to explore the missing target areas and details. In step (2), R i Mining feature maps with different levels of semantic features, and finally using 1×1 convolution for semantic fusion to obtain S i .

5. The method for automatic organ segmentation based on laryngoscope medical images according to claim 4, characterized in that: The convolutional conditional random field module supplements the conditional independence assumption based on the fully connected conditional random field. If the Manhattan distance d between pixels i and j satisfies d(i,j)>k, it is assumed that i and j are conditionally independent, and k is the filtered distance size.

Images