Pneumonic medical image segmentation method based on subtitle generation task enhancement

Abstract

The invention relates to the technical field of medical image processing, and discloses a pneumonia medical image segmentation method based on subtitle generation task enhancement. According to the method, an additional text generation module is introduced, medical text description related to a lesion area is generated, and the generated text is used for guiding a segmentation task, so that the accuracy of image segmentation is improved, and meanwhile, additional clinical text diagnosis support is provided; a multi-modal double-flow collaborative feature extraction module is constructed, and differential processing of different features is realized through a double-branch design. A text-guided mask graph reconstruction decoding module is constructed, and a text information area is effectively utilized to guide a segmentation task and fuse the segmentation task. Experimental results show that the method is superior to the prior art in the aspect of segmentation performance, a text generation task is combined for the first time, related lesion description is generated while a segmentation result is obtained, and development of clinical diagnosis is promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a pneumonia medical image segmentation method based on enhanced caption generation tasks. Background Art

[0002] In the six months since the outbreak of COVID-19, more than 1.5 million cases of COVID-19 have been reported worldwide, with more than 92,000 deaths. Computed tomography (CT) has been considered one of the important supplements for early diagnosis of COVID-19 because of its ease of operation and ability to display the three-dimensional structure of the lungs. Therefore, the automatic segmentation of lung CT scan infections using computer vision technology has received widespread attention from clinical researchers. With the continuous development of deep learning technology, medical image segmentation has gradually become a hot research topic in the field of computer vision.

[0003] In recent years, many methods have emerged in the field of medical image segmentation, aiming to assist doctors in improving their work efficiency and have achieved remarkable results in sub-fields such as COVID-19 segmentation. Architectures based on convolutional neural networks (CNNs), such as Unet, Unet++, and AttentionUnet, have shown excellent performance. However, the above traditional automatic segmentation methods based on deep learning pixel-by-pixel supervision ignore the semantic information in medical reports. Medical reports often contain information about the lesion area, such as size, number, etc., which is naturally complementary to image data. Therefore, many studies have begun to explore how to combine the text information in medical reports with the segmentation process to improve the accuracy of the segmentation results. However, the existing technology also has certain limitations—only using text as the input of the model to improve the accuracy of the segmentation results. This does not fully develop the utilization potential of medical text reports. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a pneumonia medical image segmentation method based on subtitle generation task enhancement, which generates a text description of the relevant lesions while segmenting the lesion area, and further uses the generated text information to guide the segmentation, thereby solving the above technical problems.

[0005] The technical solution of the present invention is as follows:

[0006] A pneumonia medical image segmentation method based on caption generation task enhancement specifically includes the following steps:

[0007] S1. Obtain pneumonia lesion image set V∈R H×W×3 , where the superscript H×W×3 indicates that the length, width and number of channels of the image are H, W and 3 respectively, and the resolution of the pneumonia lesion image is unified to 224*224, and the pneumonia lesion image set includes training data and test data;

[0008] S2: Input the training data into the multimodal dual-stream collaborative feature extractor for four feature extractions to obtain four encoding features, and then downsample the four encoding features in turn, denoted as F enc1 、F enc2 、F enc3 、F enc4 , the last obtained coding feature is taken as the final coding feature;

[0009] Among them, F enc1 、F enc2 、F enc3 、F enc4 Different downsampling encoding features for four dimensions;

[0010] Specifically, the feature extraction is to transform the pneumonia lesion image V∈R with a dimension of [8, 3, 224, 224] H×W×3 The local feature extraction flow and global feature extraction flow of the multimodal dual-stream collaborative feature extractor are input in parallel to jointly extract features. After the residual convolution block, batch normalization, and RELU activation in the local feature extraction flow, the local feature (F loci ); at the same time, after the self-attention operation with 4 attention heads and 4 layers in the "global feature extraction flow", the global feature (F gloi ), and then the local features (F loci ) and global features (F gloi ) to obtain the encoding feature (F enci ), the specific steps are as follows:

[0011] S2.1. The training data is used as input and sent to the "local feature extraction flow" based on the convolutional neural network and the "global feature extraction flow" based on the self-attention mechanism to extract local features (F loci ) and global features (F gloi );

[0012] Local features (F loci ) Accumulate spatial information through convolutional neural networks, focusing on capturing the local information of the texture contour of the lesion area;

[0013] Global features (F gloi ) uses the self-attention mechanism to explore the long-range dependencies in the image, focusing on capturing the global information of the image;

[0014] The specific expressions of the global features and local features are as follows:

[0015] F loci =E conv (V)i=1

[0016] Floci =E conv (F enc(i-1) )i=2,3,4

[0017] F gloi =E trans (V)i=1

[0018] F gloi =E trans (F glo(i-1) )i=2,3,4

[0019] In the formula, E conv and E trans They represent the feature extractors of the local feature extraction flow and the global feature extraction flow respectively, i represents the number of layers, V represents the input image, and F enc(i-1) represents the encoding features of the previous layer, F loci and F gloi Respectively represent the local features and global features extracted from the i-th layer;

[0020] S2.2. The global feature (F gloi ) and local features (F loci ) The dimensions are unified and concatenated through the fully connected layer to obtain the encoding feature (F enci ), provides an expressive feature set for multimodal tasks. The encoding feature (F enci ) is as follows:

[0021] F enci =[F loci *F gloi ]

[0022] In the formula, [*] represents the splicing operation, F enci Represents the encoding feature;

[0023] S2.3, repeat S2.1-S2.2, downsample the four coding features respectively, and obtain four downsampled coding features F with different dimensions in turn. enc1 、F enc2 、F enc3 、F enc4 , where F enc1 After downsampling once, the result is [8, 32, 224, 224], F enc2 Based on F enc1 After one downsampling operation, we downsample again and get the result as [8, 64, 112, 112], F enc3 Based on F enc2 After two downsampling operations, we downsample again and get the result as [8, 128, 56, 56], F enc4 Based on F enc3After three downsampling operations, downsample once more and the result is [8, 256, 28, 28];

[0024] S3, the final encoding feature (F enc4 ) is input into the multimodal semantic information enhancement layer, and after visual semantic enhancement and text semantic enhancement, segmentation features suitable for segmentation tasks are obtained (F seg ) and the generated features suitable for text generation tasks (L g );

[0025] The multimodal semantic information enhancement layer includes two branches: the first branch is visual semantic enhancement, including hole convolution with different expansion rates, batch normalization and RELU activation; the second branch is text semantic enhancement including attention mechanism and 1*1 convolution;

[0026] Furthermore, after the visual semantic enhancement branch undergoes four kinds of dilation convolution operations with dilation rates of 1, 3, 6, and 9, the obtained results are concatenated according to the number of channels, and finally a 3*3 convolution is performed to compress the dimension and activate to obtain the segmentation feature. The specific expression is as follows:

[0027]

[0028] In the formula, Represents the intermediate feature, F seg represents the segmentation feature, represents the dilated convolution, d=i represents the dilation rate, Conv 3×3 represents 3*3 convolution, σ represents RELU activation function, BN represents batch normalization, and [*] represents concatenation operation;

[0029] The text semantic enhancement branch is sent to the self-attention layer with 4 attention heads and 2 layers after 1*1 convolution and feature reshaping to capture the extensive context in the image. The specific expression is as follows:

[0030] Lg=Transformer(reshape( conv1×1 (F enc4 )))

[0031] In the formula, Conv 1×1 represents 1*1 convolution, reshape(*) represents feature reshaping operation, L g Represents the generated features;

[0032] S4, generate features (L g ) is input into the long short-term memory text generation module to obtain the medical text description (Texy) related to the lesion area, and the generated text description (Texy) is transformed into four text features (L corresponding to the decoding feature scale) through the linear layer. i);

[0033] Medical text description (Texy) and text features (L i ) is calculated as follows:

[0034] Text=LSTM(Lg)

[0035] L i =Linear(Text)

[0036] In the formula, LSTM represents the long short-term memory network, Text represents the text result generated by the network, and Linear represents the fully connected layer;

[0037] S5, four text features corresponding to the decoding feature scale (L i ) and four downsampled dimensional encoding features (F enci ) is input into the text feature correction module to obtain the corrected text feature (L' i ), avoiding the problem of deviation between the generated text and the true description in a few samples, ensuring the correction of text features (L' i ) is semantically closely related to the image content. The specific steps are as follows:

[0038] S5.1. Four text features corresponding to the decoding feature scale (L i ) and four different down-sampled encoding features (F enci ) After passing through two different mapping layers, we multiply them and get the attention map A i ∈R HW×T , where T represents the number of words, the specific expression is as follows:

[0039] A i =(ω v F enci )(ω l L i )

[0040] Among them, ω v Represents the mapping function of the downsampled encoding features in four dimensions, ω l A mapping function representing four text features corresponding to the decoding feature scale;

[0041] S5.2. Attention map A i The attention matrix of each word t is obtained by summing and normalizing according to the HW pixel dimension The specific expression is as follows:

[0042]

[0043] in, represents the correlation between the k-th pixel and T words, and softmax(·) represents the normalized exponential function and Converted into a probability between 0 and 1;

[0044] S5.3. Attention Matrix According to each word dimension and four text features corresponding to the decoding feature scale (L i ) to obtain the visually guided sentence feature (L si ), the specific expression is as follows:

[0045]

[0046] Among them, ⊙ represents the dot product operation;

[0047] S5.4. Sentence features (L si ) after 1x1 convolution learning and four text features corresponding to the decoding feature scale (L i ) concatenation, the sentence-level information is supplemented to the text features corresponding to the corresponding decoding feature scale. The specific expression is as follows:

[0048]

[0049] S5.5, the concatenated result is subjected to self-attention operation and then combined with four text features corresponding to the decoding feature scale (L i ) are added element by element, and the result of the addition is learned through 1x1 convolution to obtain the corrected text feature (L' i ), the specific expression is as follows:

[0050]

[0051] Among them, Self(·) represents the self-attention operation;

[0052] S6, the obtained segmentation features (F seg ) and corrected text features (L' i ) is input to the text-guided mask reconstruction decoder, and the corrected text features (L' i ) information guides the fusion to obtain the decoding feature with text information (F deci ), and then pass through the segmentation head to obtain the binary segmentation prediction map (P);

[0053] The guided mask image reconstruction decoder will correct the text features (l' i) is fused with the decoding features at each spatial position to obtain decoding features of dimensions [8,256,28,28], [8,128,56,56], [8,64,112,112], [8,32,224,224], where the structure of each decoding layer is the same, including an upsampling operation with an upsampling factor of 2, a fusion operation, a skip connection, a residual block, and a RELU activation. Finally, the last layer of decoding features is convolved and Sigmod activated by the segmentation head to obtain a binary segmentation prediction map (P) with a dimension of [8,1,224,224]. The specific expression is as follows:

[0054] F dec4 =relu(Res(atten(upsmaple(F seg ),L′ i ))) ×4

[0055] P = sigmod( conv1×1 (F dec4 ))

[0056] In the formula, sigmod represents the activation function, atten(*,*) represents the attention fusion operation of text and image features, and conv 1×1 represents 1*1 convolution, relu represents RELU activation, F dec4 Represents the final decoded features, Res represents the residual block, upsmaple represents the upsampling operation, and ×4 represents four-layer decoding;

[0057] The obtained segmentation feature (F seg ) and corrected text features (L' i ) is input to the text-guided mask reconstruction decoder, and the corrected text features (L' i ) information guides the fusion to obtain the decoding feature with text information (F deci ), and then pass through the segmentation head to obtain the binary segmentation prediction map (P). The specific steps are as follows:

[0058] S6.1. The segmentation feature (F seg ) and corrected text features (L' i ) are used as the guiding feature and the feature to be guided, respectively, and after deconvolution, the segmentation feature is upsampled by one time; the specific expression is as follows:

[0059] F Q =UP(F seg )i=1

[0060] F Q =UP(Fdec (i-1) )i=2,3,4

[0061] Among them, UP(·) represents the upsampling operation, F dec(i-1) represents the decoding features of the previous layer, F Q represents the guiding feature;

[0062] S6.2, use the mapping function to encode the spatial context features of the guiding features and the features to be guided, and obtain the query vector Q, key vector K and numerical vector V for attention calculation respectively, and the above sampled segmentation features (F seg ) as the query vector Q, correct the text feature (L' i ) is the key vector K and the numerical vector V, and the specific expression is as follows:

[0063] V Qi =ω qi (F Q )

[0064] L Ki =ω ki (L′ i )

[0065] L Vi =ω vi (L′ i )

[0066] Among them, ω qi represents the mapping function of the query vector Q, ω ki Represents the mapping function of the key vector K, ω vi Represents the mapping function of a numerical vector V, V Qi , L Ki , L Vi Represent the query vector Q, key vector K and value vector V respectively;

[0067] S6.3, perform a dot product operation on the query vector Q generated by the guided feature and the transpose of the key vector K generated by the guided feature to generate a C*C attention map;

[0068] S6.4, multiply the generated C*C attention map by the feature to be guided to obtain the fused feature after guidance compensation (F Ai ), the specific expression is as follows:

[0069]

[0070] Among them, softmax(·) represents the normalized exponential function, d i is the scaling factor;

[0071] S6.5, the obtained fusion feature is then combined with the four different dimensions of the down-sampled encoding features (F enci) splicing along the channel dimension, supplementing the information of the encoding stage to the decoding process, and finally sending the result after the splicing operation to the two-layer residual block for further learning;

[0072] F deci =Res(Res[F Ai ,F enci ])i=1,2,3,4

[0073] S6.6. Repeat S6.1-S6.5, and gradually upsample four times to obtain decoding features of dimensions [8, 256, 28, 28], [8, 128, 56, 56], [8, 64, 112, 112], and [8, 32, 224, 224]. The structure of each decoding layer is the same. Finally, the decoding features of dimension [8, 32, 224, 224] are convolved and Sigmod activated by the segmentation head to obtain a binary segmentation prediction map of dimension [8, 1, 224, 224].

[0074] S7, setting the network model loss function and updating the model parameters according to the final optimization goal;

[0075] The network model loss function includes: segmentation loss of the segmentation task and generation loss of text generation;

[0076] The specific steps are as follows:

[0077] S7.1. The final coding feature (F enc4 ) After passing through the long short-term memory network, the medical text description related to the lesion area is obtained, and the generation loss is calculated on the feature dimension for the generated text and the real text The specific expression is as follows:

[0078]

[0079] Among them, p t (y t |y 1 ,y 2 ,…,y t-1 ; θ) indicates that the model predicts the current word y given the vocabulary and model parameters t The probability of , θ represents the learning parameters of the entire long short-term memory network;

[0080] S7.2, the final coding feature (F enc4 ) After four upsampling decodings of the text-guided mask reconstruction decoder, a binary segmentation prediction map is obtained, and the segmentation loss is calculated for the prediction map and the true mask map The specific expression is as follows:

[0081]

[0082] in, represents the binary cross entropy loss, represents the DICE coefficient loss, represents the semantic consistency loss, λ a Parameter representing the binary cross entropy loss, λ b The parameter representing the DICE coefficient loss, λ c Parameter representing the semantic consistency loss;

[0083] The binary cross entropy loss DICE coefficient loss and semantic consistency loss The specific expression is as follows:

[0084]

[0085] Among them, p j and j Represents the binary segmentation prediction probability and corresponding label classification of the j-th pixel of each input image, N represents the sum of all elements, Represents the corrected text feature L' 1 and L' 2 Pull in the mean square error loss for semantic consistency, Represents the corrected text feature L' 1 and L' 3 Pull in the mean square error loss for semantic consistency, Represents the corrected text feature L' 1 and L' 4 Pull in the mean square error loss for semantic consistency, and Working together, the corrected text features maintain consistent focus on the lesion area;

[0086] Said and The specific expression is as follows:

[0087]

[0088] S7.3. Finally, losses will be generated Segmentation loss Perform weighting and update the model parameters as the final optimization target;

[0089] The overall loss function is expressed as:

[0090]

[0091] In the formula, Represents the total loss function of the entire network, and Represent the segmentation loss and text generation loss respectively, α represents the weighted parameter of the segmentation loss, and β represents the weighted parameter of the text generation loss;

[0092] Beneficial effects of the present invention

[0093] 1. The present invention combines the subtitle generation task to generate relevant medical text descriptions while segmenting the lesion area.

[0094] 2. The method proposed in the present invention extracts multimodal features containing global integrity and local details through a multimodal two-stream collaborative feature extractor, which promotes the two tasks of segmentation and text generation; at the same time, in order to give full play to the utilization potential of medical text reports, the text data in the dataset is used as a supervision signal for the text generator-long short-term memory network instead of input. Once the model is trained, the text will no longer be needed as a supervision signal and the corresponding text information of the lesion area can be automatically inferred; in addition, the generated text further enhances the segmentation quality in the text-guided mask image reconstruction decoding layer, effectively improving the accuracy of medical image segmentation.

[0095] 3. The method of the present invention is superior to the current technical level in terms of visual quality and quantitative evaluation. It generates additional text reports based on segmentation, providing more textual diagnostic references for clinical diagnosis, which has important clinical significance and positive application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1 It is a schematic diagram of the process of the present invention;

[0097] FIG2 is a detailed structural diagram of the present invention, wherein FIG2(a) is a main network framework diagram of the present invention, FIG2(b) is a structural diagram of a multimodal dual-stream collaborative feature extractor, FIG2(c) is a structural diagram of a multimodal semantic information enhancement layer and a long short-term memory text generation layer, FIG2(d) is a structural diagram of a text feature correction layer, and FIG2(e) is a structural diagram of a text-guided mask image reconstruction decoder;

[0098] Figure 3 It is a segmentation visualization display diagram of the present invention;

[0099] Figure 4A visual display diagram is generated for the text of the present invention, wherein part (a) is a display diagram of the text generation result of "bilateral lung infection, 9 infection areas, the infection site is the entire left lung and the entire right lung", part (b) is a display diagram of the text generation result of "unilateral lung infection, 1 infection area, the infection site is the lower left lung", part (c) is a display diagram of the text generation result of "bilateral lung infection, 9 infection areas, the infection site is the entire left lung and the entire right lung", and part (d) is a display diagram of the text generation result of "bilateral lung infection, 4 infection areas, the infection site is the upper left lung and the right middle lung". DETAILED DESCRIPTION

[0100] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0101] Example 1

[0102] See also Figure 1 -2, a pneumonia medical image segmentation method based on caption generation task enhancement, comprising the following steps:

[0103] S1. Obtain pneumonia lesion image set V∈R H×W×3 , where the superscript H×W×3 indicates that the length, width and number of channels of the image are H, W and 3 respectively, and the resolution of the pneumonia lesion image is unified to 224*224, and the pneumonia lesion image set includes training data and test data;

[0104] S2: Input the training data into the multimodal dual-stream collaborative feature extractor for four feature extractions to obtain four encoding features, and downsample the four encoding features respectively, denoted as F enc1 、F enc2 、F enc3 、F enc4 , the downsampled encoding feature obtained for the last time is taken as the final encoding feature;

[0105] Among them, F enc1 、F enc2 、F enc3 、F enc4 Different downsampling encoding features for four dimensions;

[0106] Specifically, the feature extraction is to transform the pneumonia lesion image V∈R with a dimension of [8, 3, 224, 224] H×W×3The local feature extraction flow and global feature extraction flow of the multimodal dual-stream collaborative feature extractor are input in parallel to jointly extract features. After the residual convolution block, batch normalization, and RELU activation in the local feature extraction flow, the local feature (F loci ); at the same time, after the self-attention operation with 4 attention heads and 4 layers in the "global feature extraction flow", the global feature (F gloi ), and then the local features (F loci ) and global features (F gloi ) to obtain the encoding feature (F enci ), the specific steps are as follows:

[0107] S2.1. The training data is used as input and sent to the "local feature extraction flow" based on the convolutional neural network and the "global feature extraction flow" based on the self-attention mechanism to extract local features (F loci ) and global features (F gloi );

[0108] Local features (F loci ) Accumulate spatial information through convolutional neural networks, focusing on capturing the local information of the texture contour of the lesion area;

[0109] Global features (F gloi ) uses the self-attention mechanism to explore the long-range dependencies in the image, focusing on capturing the global information of the image;

[0110] The specific expressions of the global features and local features are as follows:

[0111] F loci =E conv (V)i=1

[0112] F loci =E conv (F enc(i-1) )i=2,3,4

[0113] F gloi =E trans (V)i=1

[0114] F gloi =E trans (F glo(i-1) )i=2,3,4

[0115] In the formula, E conv and E trans They represent the feature extractors of the local feature extraction flow and the global feature extraction flow respectively, i represents the number of layers, V represents the input image, and F enc(i-1) represents the encoding features of the previous layer, F loci and Fgloi Respectively represent the local features and global features extracted from the i-th layer;

[0116] S2.2. The global feature (F gloi ) and local features (F loci ) The dimensions are unified and concatenated through the fully connected layer to obtain the encoding feature (F enci ), provides an expressive feature set for multimodal tasks. The encoding feature (F enci ) is as follows:

[0117] F enci =[F loci *F gloi ]

[0118] In the formula, [*] represents the splicing operation, F enci Represents the encoding feature;

[0119] S2.3, repeat S2.1-S2.2, downsample the four coding features respectively, and obtain four downsampled coding features F with different dimensions in turn. enc1 、F enc2 、F enc3 、F enc4 , where F enc1 After downsampling once, the result is [8, 32, 224, 224], F enc2 Based on F enc1 After one downsampling operation, we downsample again and get the result as [8, 64, 112, 112], F enc3 Based on F enc2 After two downsampling operations, we downsample again and get the result as [8, 128, 56, 56], F enc4 Based on F enc3 After three downsampling operations, downsample once more and the result is [8, 256, 28, 28];

[0120] S3, the final encoding feature (F enc4 ) is input into the multimodal semantic information enhancement layer, and after visual semantic enhancement and text semantic enhancement, segmentation features suitable for segmentation tasks are obtained (F seg ) and the generated features suitable for text generation tasks (L g );

[0121] The multimodal semantic information enhancement layer includes two branches: the first branch is visual semantic enhancement, including hole convolution with different expansion rates, batch normalization and RELU activation; the second branch is text semantic enhancement including attention mechanism and 1*1 convolution;

[0122] Furthermore, after the visual semantic enhancement branch undergoes four kinds of dilation convolution operations with dilation rates of 1, 3, 6, and 9, the obtained results are concatenated according to the number of channels, and finally a 3*3 convolution is performed to compress the dimension and activate to obtain the segmentation feature. The specific expression is as follows:

[0123]

[0124] In the formula, Represents the intermediate feature, F seg represents the segmentation feature, represents the dilated convolution, d=i represents the dilation rate, Conv 3×3 represents 3*3 convolution, σ represents RELU activation function, BN represents batch normalization, and [*] represents concatenation operation;

[0125] The text semantic enhancement branch is sent to the self-attention layer with 4 attention heads and 2 layers after 1*1 convolution and feature reshaping to capture the extensive context in the image. The specific expression is as follows:

[0126] L g = Transformer(reshape( conv1×1 (F enc4 )))

[0127] In the formula, conv 1×1 represents 1*1 convolution, reshape(*) represents feature reshaping operation, L g Represents the generated features;

[0128] S4, generate features (L g ) is input into the long short-term memory text generation module to obtain the medical text description (Text) related to the lesion area, and the generated text description (Text) is transformed into four text features (L corresponding to the decoding feature scale) through the linear layer. i );

[0129] Medical text description (Text) and text features (L i ) is calculated as follows:

[0130] Text=LSTM(L g )

[0131] L i =Linear(Text)

[0132] In the formula, LSTM represents the long short-term memory network, Text represents the text result generated by the network, and Linear represents the fully connected layer;

[0133] S5, four text features corresponding to the decoding feature scale (Li ) and four different downsampling dimension encoding features (F enci ) is input into the text feature correction module to obtain the corrected text feature (L' i ), avoiding the problem of deviation between the generated text and the true description in a few samples, ensuring the correction of text features (L' i ) is semantically closely related to the image content. The specific steps are as follows:

[0134] S5.1. Four text features corresponding to the decoding feature scale (L i ) and four different down-sampled encoding features (F enci ) After passing through two different mapping layers, we multiply them and get the attention map A i ∈R HW×T , where T represents the number of words, the specific expression is as follows:

[0135] A i =(ω v F enci )(ω l L i )

[0136] Among them, ω v Represents the mapping function of the downsampled encoding features in four dimensions, ω l A mapping function representing four text features corresponding to the decoding feature scale;

[0137] S5.2. Attention map A i The attention matrix of each word t is obtained by summing and normalizing according to the HW pixel dimension The specific expression is as follows:

[0138]

[0139] in, represents the correlation between the k-th pixel and T words, and softmax(·) represents the normalized exponential function and Converted into a probability between 0 and 1;

[0140] S5.3. Attention Matrix According to each word dimension and four text features corresponding to the decoding feature scale (L i ) to obtain the visually guided sentence feature (L si ), the specific expression is as follows:

[0141]

[0142] Among them, ⊙ represents the dot product operation;

[0143] S5.4. Sentence features (L si ) after 1x1 convolution learning and four text features corresponding to the decoding feature scale (L i ) concatenation, the sentence-level information is supplemented to the text features corresponding to the corresponding decoding feature scale. The specific expression is as follows:

[0144]

[0145] S5.5, the concatenated result is subjected to self-attention operation and then combined with four text features corresponding to the decoding feature scale (L i ) are added element by element, and the result of the addition is learned through 1x1 convolution to obtain the corrected text feature (L' i ), the specific expression is as follows:

[0146]

[0147] Among them, Self(·) represents the self-attention operation;

[0148] S6, the obtained segmentation features (F seg ) and corrected text features (L' i ) is input to the text-guided mask reconstruction decoder, and the corrected text features (L' i ) information guides the fusion to obtain the decoding feature with text information (F deci ), and then pass through the segmentation head to obtain the binary segmentation prediction map (P);

[0149] The guided mask reconstruction decoder will correct the text features (L' i ) is fused with the decoding features at each spatial position to obtain decoding features of dimensions [8,256,28,28], [8,128,56,56], [8,64,112,112], [8,32,224,224], where the structure of each decoding layer is the same, including an upsampling operation with an upsampling factor of 2, a fusion operation, a skip connection, a residual block, and a RELU activation. Finally, the last layer of decoding features is convolved and Sigmod activated by the segmentation head to obtain a binary segmentation prediction map (P) with a dimension of [8,1,224,224]. The specific expression is as follows:

[0150] F dec4 =relu(Res(atten(upsmaple(F seg ),L′ i ))) ×4

[0151] P = sigmod (conv 1×1 (F dec4 ))

[0152] In the formula, sigmod represents the activation function, atten(*,*) represents the attention fusion operation of text and image features, and conv 1×1 represents 1*1 convolution, relu represents RELU activation, F dec4 Represents the final decoded features, Res represents the residual block, upsmaple represents the upsampling operation, and ×4 represents four-layer decoding;

[0153] The obtained segmentation feature (F seg ) and corrected text features (L' i ) is input to the text-guided mask reconstruction decoder, and the corrected text features (L' i ) information guides the fusion to obtain the decoding feature with text information (F deci ), and then pass through the segmentation head to obtain the binary segmentation prediction map (P). The specific steps are as follows:

[0154] S6.1. The segmentation feature (F seg ) and corrected text features (L' i ) are used as the guiding feature and the feature to be guided, respectively, and after deconvolution, the segmentation feature is upsampled by one time; the specific expression is as follows:

[0155] F Q =UP(F seg )i=1

[0156] F Q =UP(Fdec (i-1) )i=2,3,4

[0157] Among them, UP(·) represents the upsampling operation, F dec(i-1) represents the decoding features of the previous layer, F Q represents the guiding feature;

[0158] S6.2, use the mapping function to encode the spatial context features of the guiding features and the features to be guided, and obtain the query vector Q, key vector K and numerical vector V for attention calculation respectively, and the above sampled segmentation features (F seg ) as the query vector Q, correct the text feature (L' i ) is the key vector K and the numerical vector V, and the specific expression is as follows:

[0159] V Qi =ω qi (F Q )

[0160] L Ki =ω ki (L′ i )

[0161] L Vi =ω vi (L′ i )

[0162] Among them, ω qi represents the mapping function of the query vector Q, ω ki Represents the mapping function of the key vector K, ω vi Represents the mapping function of a numerical vector V, V Qi , L Ki , L Vi Represent the query vector Q, key vector K and value vector V respectively;

[0163] S6.3, perform a dot product operation on the query vector Q generated by the guided feature and the transpose of the key vector K generated by the guided feature to generate a C*C attention map;

[0164] S6.4, multiply the generated C*C attention map by the feature to be guided to obtain the fused feature after guidance compensation (F Ai ), the specific expression is as follows:

[0165]

[0166] Among them, softmax(·) represents the normalized exponential function, d i is the scaling factor;

[0167] S6.5, the obtained fusion feature is then combined with the four different dimensions of the down-sampled encoding features (F enci ) splicing along the channel dimension, supplementing the information of the encoding stage to the decoding process, and finally sending the result after the splicing operation to the two-layer residual block for further learning;

[0168] F deci =Res(Res[F Ai ,F enci ])i=1,2,3,4

[0169] S6.6. Repeat S6.1-S6.5, and gradually upsample four times to obtain decoding features of dimensions [8, 256, 28, 28], [8, 128, 56, 56], [8, 64, 112, 112], and [8, 32, 224, 224]. The structure of each decoding layer is the same. Finally, the decoding features of dimension [8, 32, 224, 224] are convolved and Sigmod activated by the segmentation head to obtain a binary segmentation prediction map of dimension [8, 1, 224, 224].

[0170] S7, setting the network model loss function and updating the model parameters according to the final optimization goal;

[0171] The network model loss function includes: segmentation loss of the segmentation task and generation loss of text generation;

[0172] The specific steps are as follows:

[0173] S7.1. The final coding feature (F enc4 ) After passing through the long short-term memory network, the medical text description related to the lesion area is obtained, and the generation loss is calculated on the feature dimension for the generated text and the real text The specific expression is as follows:

[0174]

[0175] Among them, p t (y t |y 1 ,y 2 ,…,y t-1 ; θ) indicates that the model predicts the current word y given the vocabulary and model parameters t The probability of , θ represents the learning parameters of the entire long short-term memory network;

[0176] S7.2, the final coding feature (F enc4 ) After four upsampling decodings of the text-guided mask reconstruction decoder, a binary segmentation prediction map is obtained, and the segmentation loss is calculated for the prediction map and the true mask map The specific expression is as follows:

[0177]

[0178] in, represents the binary cross entropy loss, represents the DICE coefficient loss, represents the semantic consistency loss, λ a Parameter representing the binary cross entropy loss, λ b The parameter representing the DICE coefficient loss, λ c Parameter representing the semantic consistency loss;

[0179] The binary cross entropy loss DICE coefficient loss and semantic consistency loss The specific expression is as follows:

[0180]

[0181] Among them, p j and jRepresents the binary segmentation prediction probability and corresponding label classification of the j-th pixel of each input image, N represents the sum of all elements, Represents the corrected text feature L' 1 and L' 2 Pull in the mean square error loss for semantic consistency, Represents the corrected text feature L' 1 and L' 3 Pull in the mean square error loss for semantic consistency, Represents the corrected text feature L' 1 and L' 4 Pull in the mean square error loss for semantic consistency, and Working together, the corrected text features maintain consistent focus on the lesion area;

[0182] Said and The specific expression is as follows:

[0183]

[0184] S7.3. Finally, losses will be generated Segmentation loss Perform weighting and update the model parameters as the final optimization target;

[0185] The overall loss function is expressed as:

[0186]

[0187] In the formula, Represents the total loss function of the entire network, and Represent the segmentation loss and text generation loss respectively, α represents the weighted parameter of the segmentation loss, and β represents the weighted parameter of the text generation loss;

[0188] This example conducts experiments on the MosMedData+ public dataset to comprehensively evaluate the effectiveness of the method in this paper. The dataset is divided into training set, validation set, and test set using the commonly used 3:1:1 division method. The number of batches in the network is set to 8, the number of training rounds is set to 200, the learning rate is set to 0.0001, and the early stopping strategy is adopted.

[0189] In this embodiment, the evaluation indicators are: mDice and mIoU, and the larger the mean value, the better the performance.

[0190] This embodiment is compared with the most advanced methods, including: U-Net, Att-Unet, TransUNet, SCOAT-Net, COPLE-Net, TGANet, ConTEXTualNet and LViT-T, and the experimental results are shown in Table 1. It can be seen that the method of the present invention achieves 63.02 in mIoU and 75.87 in Dice. This proves the effectiveness of the method of the present invention in pneumonia segmentation.

[0191] Table 1 shows the objective evaluation comparison of different methods on the MosMedData+ test set

[0192]

[0193]

[0194] The results show that the method of the present invention greatly exceeds the performance of existing methods. In addition, thanks to the novel design of the network structure and the multi-task promotion strategy of the present invention, the accuracy of the segmentation results is further ensured, which has important clinical significance and positive application prospects.

[0195] Furthermore, the present invention compares the segmentation results of U-Net, Att-Unet, TransUNet, SCOAT-Net, COPLE-Net, TGANet, ConTEXTualNet and LViT-T. Figure 4 As shown. In the segmentation visualization results, see Figure 3 In the yellow area of ​​the first row of images, we can see that the segmentation results of CopleNet, TGANet and LViT methods also segment out the extra area, which has the problem of wrong segmentation, while our method can segment out the main lesion area. Figure 3 It can be seen from the yellow area of ​​the third row of pictures that the current methods are still insufficient in identifying the edges and details of lesions. For example, TGANet can only identify a very small number of lesion areas, and CopleNet and ConTEXTualNet do not reflect the edge contours of the lesion areas at all. In contrast, the method of the present invention proposed by us can well identify the lesion area and its edge details, which shows that the present invention has a better ability to capture the lesion area.

[0196] For the evaluation of text generation results, the BLEU index of the text generated by the present invention reached 83.89%. The BLEU score shows the accuracy of the model in generating phrases and longer texts. It can be seen that the accuracy and reliability of the text generated by the present invention method are already high. At the same time, some samples with good and poor performance are randomly selected for visualization, such as Figure 4Part (a) is a display diagram of the text generation results of “bilateral lung infection, 9 infection areas, the infection sites are the entire left lung and the entire right lung”, part (b) is a display diagram of the text generation results of “unilateral lung infection, 1 infection area, the infection site is the lower left lung”, part (c) is a display diagram of the text generation results of “bilateral lung infection, 9 infection areas, the infection sites are the entire left lung and the entire right lung”, and part (d) is a display diagram of the text generation results of “bilateral lung infection, 4 infection areas, the infection sites are the upper left lung and the right middle lung”. Among them, parts (a) and (c) are the display results of different infection conditions in the same infection site.

[0197] In each image, the red dots represent the text labels and the white dots represent the text generated by the model. It can be observed that in a few samples, some words in the generated descriptions are incorrect, such as Figure 4 In (d), the original "left upper lung" was mistakenly generated as "left middle lung". The incorrectly generated text has been marked in red font, but the rest of the words are very close to the correct text label. Figure 4 (a) Figure 4 ((b) and Figure 4 In (c), it can be seen that for lesion areas in various locations and different lesion types, the method of the present invention can better capture the area and location of the lesion and generate reliable medical text annotations.

[0198] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A pneumonia medical image segmentation method based on enhanced caption generation task, Features: The following steps are involved: S1. Acquire a pneumonia lesion image set and unify the resolution of the pneumonia lesion images to 224*224, wherein the pneumonia lesion image set includes training data and test data; S2: Input the training data into the multimodal dual-stream collaborative feature extractor for four feature extractions to obtain four encoding features, and then downsample the four encoding features in turn, denoted as F enc1 、F enc2 、F enc3 、F enc4 , the last obtained down-sampled encoding feature F enc4 As the final encoding feature, F enc1 、F enc2 、F enc3 、F enc4 Different downsampling encoding features for four dimensions; S3, input the final encoded features into the multimodal semantic information enhancement layer, and after visual semantic enhancement and text semantic enhancement, obtain the segmentation features suitable for the segmentation task and the generation features suitable for the text generation task respectively; S4, input the generated features into the long short-term memory text generation module to obtain a medical text description of the lesion area, and the medical text description is transformed into four text features corresponding to the decoding feature scale through a linear layer; S5, inputting four text features corresponding to the decoding feature scale and four downsampled dimensional encoding features with different dimensions into a text feature correction module to obtain corrected text features; S6, input the obtained segmentation features and corrected text features into the text-guided mask image reconstruction decoder to obtain decoding features with text information, and then pass through the segmentation head to obtain a binary segmentation prediction map; S7. Set the network model loss function and update the model.

2. According to claim 1, a pneumonia medical image segmentation method based on subtitle generation task enhancement, Features: The specific steps of inputting the training data into the multimodal dual-stream collaborative feature extractor for four feature extractions to obtain four encoding features, and downsampling the four encoding features in turn, and using the encoding features obtained after the last downsampling as the final encoding features are as follows: S2.

1. The training data is used as input and sent to the "local feature extraction flow" based on the convolutional neural network and the "global feature extraction flow" based on the self-attention mechanism to extract local features and global features respectively. The specific expressions of the global features and local features are as follows: F loci =E conv (V) i=1 F loci =E conv (F enc(i-1) ) i=2,3,4 F gloi =E trans (V) i=1 F gloi =E trans (F glo(i-1) ) i=2,3,4 In the formula, E conv and F trans They represent the feature extractors of the local feature extraction flow and the global feature extraction flow respectively, i represents the number of layers, V represents the input image, and F enc(i-1) represents the encoding feature of the previous layer, F loci and F gloi Respectively represent the local features and global features extracted from the i-th layer; S2.

2. The global features and local features are concatenated through a fully connected layer to unify the dimensions and obtain the coding features. The specific expression of the coding features is as follows: F enci =[F loci *F gloi ] In the formula, [*] represents the splicing operation, F enci Represents the encoding feature; S2.3, repeat S2.1-S2.2, downsample the four coding features respectively, and obtain four downsampled coding features F with different dimensions in turn. enc1 、F enc2 、F enc3 、F enc4 , where F enc1 After downsampling once, the result is [8, 32, 224, 224], F enc2 Based on F enc1 After one downsampling operation, we downsample again and get the result as [8, 64, 112, 112], F enc3 Based on F enc2 After two downsampling operations, we downsample again and get the result as [8, 128, 56, 56], F enc4 Based on F enc3 After three downsampling operations, we downsample once more and get the result as [8, 256, 28, 28].

3. According to claim 1, a pneumonia medical image segmentation method based on subtitle generation task enhancement, Features: The multimodal semantic information enhancement layer includes two branches: the first branch is visual semantic enhancement, which includes hole convolution with different expansion rates, batch normalization and RELU activation; the second branch is text semantic enhancement, which includes attention mechanism and 1*1 convolution.

4. According to claim 3, a pneumonia medical image segmentation method based on subtitle generation task enhancement, Features: After the visual semantic enhancement branch undergoes four kinds of dilation convolution operations with dilation rates of 1, 3, 6, and 9, the obtained results are concatenated according to the number of channels, and finally a 3*3 convolution is performed to compress the dimension and activate it to obtain the segmentation feature. The specific expression is as follows: In the formula, Represents the intermediate feature, F seg represents the segmentation feature, represents the dilated convolution, d=i represents the dilation rate, Conv 3×3 represents 3*3 convolution, σ represents RELU activation function, BN represents batch normalization, and [*] represents concatenation operation; The text semantic enhancement branch is sent to the self-attention layer with 4 attention heads and 2 layers after 1*1 convolution and feature reshaping. The specific expression is as follows: L g =Transformer(reshape(conv 1×1 (F enc4 ))) In the formula, conv 1×1 represents 1*1 convolution, reshape(*) represents feature reshaping operation, L g Represents a generated feature.

5. According to claim 1, a pneumonia medical image segmentation method based on subtitle generation task enhancement, Features: The specific steps of inputting four text features corresponding to the decoding feature scale and four downsampled dimension encoding features into the text feature correction module to obtain the corrected text features are as follows: S5.

1. Multiply the four text features corresponding to the decoding feature scale and the four downsampled encoding features of different dimensions through two different mapping layers to obtain the attention map A. i ∈R HW×T , where T represents the number of words; S5.

2. Attention map A i The attention matrix of each word t is obtained by summing and normalizing according to the HW pixel dimension; S5.3, perform dot multiplication of the attention matrix with four text features corresponding to the decoding feature scale according to each word dimension to obtain the visually guided sentence features; S5.4, concatenate the sentence features after 1x1 convolution learning with the four text features corresponding to the decoding feature scale, and add the sentence-level information to the text features corresponding to the corresponding decoding feature scale; S5.

5. The concatenated result is subjected to self-attention operation and then element-wise added to the four text features corresponding to the decoding feature scale. The result of the addition is subjected to 1x1 convolution learning to obtain the corrected text feature.

6. The pneumonia medical image segmentation method based on the enhanced caption generation task according to claim 1, Features: The specific steps of inputting the obtained segmentation features and corrected text features into the text-guided mask image reconstruction decoder to obtain the decoding features with text information, and then obtaining the binary segmentation prediction map through the segmentation head are as follows: S6.1, the segmentation feature and the corrected text feature are used as the guiding feature and the feature to be guided respectively, and the segmentation feature is upsampled by one time after deconvolution; S6.2, using a mapping function to encode the spatial context features of the guiding features and the features to be guided, and respectively obtaining a query vector Q, a key vector K and a numerical vector V for attention calculation, wherein the above sampled segmentation features are used as the query vector Q, and the corrected text features are the key vector K and the numerical vector V; S6.3, perform a dot product operation on the query vector Q generated by the guided feature and the transpose of the key vector K generated by the guided feature to generate a C*C attention map; S6.4, multiply the generated C*C attention map by the feature to be guided to obtain the fused feature after guidance compensation; S6.5, the obtained fusion features are then concatenated with the four encoding features as skip connections along the channel dimension to supplement the information in the encoding stage into the decoding process. Finally, the result after the concatenation operation is sent to the two-layer residual block for further learning; S6.

6. Repeat S6.1-S6.5, and gradually upsample four times to obtain decoding features of dimensions [8, 256, 28, 28], [8, 128, 56, 56], [8, 64, 112, 112], and [8, 32, 224, 224]. The structure of each decoding layer is the same. Finally, the decoding features of dimension [8, 32, 224, 224] are convolved and Sigmod activated by the segmentation head to obtain a binary segmentation prediction map of dimension [8, 1, 224, 224].

7. The pneumonia medical image segmentation method based on the enhanced caption generation task according to claim 1, Features: The specific steps of setting the network model loss function and updating the model parameters according to the final optimization goal are as follows: S7.

1. After the final encoded features are passed through the long short-term memory network, a medical text description related to the lesion area is obtained, and the generation loss is calculated for the generated text and the real text in the feature dimension; S7.2, after the final encoded features are upsampled and decoded four times by the text-guided mask reconstruction decoder, a binary segmentation prediction map is obtained, and the segmentation loss is calculated for the prediction map and the true mask map; S7.

3. Finally, add the generation loss and segmentation loss to obtain the final optimization target and update the model parameters.

Images