A Pneumonia Medical Image Segmentation Method Enhanced by Caption Generation Task

Abstract

This invention relates to the field of medical image processing technology and discloses a pneumonia medical image segmentation method enhanced by a caption generation task. This method introduces an additional text generation module to generate medical text descriptions related to the lesion region, and uses the generated text to guide the segmentation task, improving the accuracy of image segmentation while providing additional clinical text diagnostic support. A multimodal dual-stream collaborative feature extraction module is constructed, which achieves differentiated processing of different features through a dual-branch design. A text-guided mask image reconstruction and decoding module is constructed to effectively utilize and fuse text information regions to guide the segmentation task. Experimental results show that the method of this invention outperforms existing technologies in segmentation performance and, for the first time, combines a text generation task to generate relevant lesion descriptions while obtaining segmentation results, promoting the development of clinical diagnosis.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically to a pneumonia medical image segmentation method based on caption generation task enhancement. Background Technology

[0002] In the six months since the outbreak of COVID-19, more than 1.5 million cases have been reported globally, with over 92,000 deaths. Computed tomography (CT) scans, due to their ease of operation and ability to display the three-dimensional structure of the lungs, have been considered an important supplement to the early diagnosis of COVID-19. Therefore, the automatic segmentation of lung CT scans using computer vision technology has attracted widespread attention from clinical researchers. With the continuous development of deep learning technology, medical image segmentation has gradually become a hot research topic in the field of computer vision.

[0003] In recent years, numerous methods have emerged in the field of medical image segmentation, aiming to assist doctors in improving their work efficiency and achieving significant results in subfields such as COVID-19 segmentation. Convolutional Neural Network (CNN) architectures, such as Unet, Unet++, and AttentionUnet, have demonstrated superior performance. However, these traditional deep learning-based pixel-by-pixel supervised automatic segmentation methods neglect the semantic information in medical reports. Medical reports often contain information about lesion areas, such as size and quantity, which is naturally complementary to image data. Therefore, many studies have begun to explore how to combine textual information from medical reports with the segmentation process to improve the accuracy of segmentation results. However, existing technologies also have certain limitations—only using text as input to the model to improve the accuracy of segmentation results. This does not fully explore the potential of medical text reports. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a pneumonia medical image segmentation method based on subtitle generation task enhancement. This method generates text descriptions of related lesions while segmenting the lesion region, and further utilizes the generated text information to guide the segmentation, thus solving the aforementioned technical problems.

[0005] The technical solution of the present invention is as follows:

[0006] A medical image segmentation method for pneumonia based on caption generation task enhancement specifically includes the following steps:

[0007] S1. Obtain the pneumonia lesion image set V∈R H×W×3 In this context, the superscript H×W×3 indicates that the length, width, and number of channels of the image are H, W, and 3, respectively, and the resolution of the pneumonia lesion images is unified to 224*224. The pneumonia lesion image set includes training data and test data.

[0008] S2. Input the training data into the multimodal dual-stream collaborative feature extractor for four feature extractions to obtain four encoded features. Then, downsample each of the four encoded features sequentially, denoted as F. enc1 F enc2 F enc3 F enc4 The last encoded feature obtained is used as the final encoded feature;

[0009] Among them, F enc1 F enc2 F enc3 F enc4 The features are downsampled and encoded in four different dimensions;

[0010] Specifically, the feature extraction involves processing pneumonia lesion images V∈R with dimensions [8, 3, 224, 224]. H×W×3 Features are extracted jointly by the "local feature extraction stream" and the "global feature extraction stream" of the multimodal dual-stream collaborative feature extractor, which are input in parallel. After residual convolution blocks, batch normalization, and ReLU activation in the "local feature extraction stream", local features (F) are obtained. loci Simultaneously, after undergoing a self-attention operation with 4 attention heads and 4 layers in the "Global Feature Extraction Stream," the global features (F) are obtained. gloi ), and then the local features (F) loci ) and global features (F gloi The encoded features (F) are obtained by concatenating the components. enci The specific steps are as follows:

[0011] S2.1. The training data is used as input and simultaneously fed into both a "local feature extraction stream" primarily based on convolutional neural networks and a "global feature extraction stream" primarily based on self-attention mechanisms, to extract local features (F) respectively. loci ) and global features (F gloi );

[0012] Local features (F) loci By accumulating spatial information through convolutional neural networks, it focuses on capturing local information of the texture contours of lesion areas;

[0013] Global features (F) gloi Then, it uses a self-attention mechanism to explore long-range dependencies in the image, focusing on capturing global information of the image;

[0014] The specific expressions for the global and local features are as follows:

[0015] F loci =E conv (V)i=1

[0016] Floci =E conv (F enc(i-1) i = 2, 3, 4

[0017] F gloi =E trans (V)i=1

[0018] F gloi =E trans (F glo(i-1) i = 2, 3, 4

[0019] In the formula, E conv and E trans These represent feature extractors for local and global feature extraction flows, respectively, where i represents the layer number, V represents the input image, and F represents the global feature extraction flow. enc(i-1) F represents the encoded features of the previous layer. loci and F gloi These represent the local and global features extracted from the i-th layer, respectively.

[0020] S2.2, global features (F gloi ) and local features (F loci The encoded features (F) are obtained by concatenating the dimensions after unifying them through a fully connected layer. enci This provides an expressive feature set for multimodal tasks, the encoded features (F) enci The specific expression for ) is as follows:

[0021] F enci =[F loci *F gloi ]

[0022] In the formula, [*] represents the splicing operation, F enci Represents encoded features;

[0023] S2.3. Repeat S2.1-S2.2 to downsample the four coding features respectively, thereby obtaining four downsampled coding features F with different dimensions. enc1 F enc2 F enc3 F enc4 , of which F enc1 Performing a downsampling operation yields the result [8,32,224,224], F enc2 For F enc1 Based on the first downsampling operation, a second downsampling operation yields the result [8, 64, 112, 112], F enc3 For F enc2 Based on two downsampling operations, a third downsampling operation yields the result [8,128,56,56], F enc4 For F enc3Based on three downsampling operations, a third downsampling operation yields the result [8,256,28,28].

[0024] S3, final encoded features (F) enc4 The input is fed into a multimodal semantic information enhancement layer, and after visual semantic enhancement and textual semantic enhancement, segmentation features suitable for the segmentation task (F) are obtained. seg ) and generation features suitable for text generation tasks (L g );

[0025] The multimodal semantic information enhancement layer includes two branches: the first branch is for visual semantic enhancement, which includes dilated convolutions with different dilation rates, batch normalization, and ReLU activation; the second branch is for text semantic enhancement, which includes attention mechanisms and 1*1 convolutions.

[0026] Furthermore, after the visual semantic enhancement branch undergoes four dilated convolution operations with dilation rates of 1, 3, 6, and 9, the results are concatenated according to the number of channels, and finally a 3*3 convolution is performed to compress the dimensions and activation to obtain the segmentation features, the specific expression of which is as follows:

[0027]

[0028] In the formula, F represents intermediate characteristics. seg Represents segmentation features, Represents dilated convolution, d = i represents the dilation rate, Conv 3×3 σ represents a 3x3 convolution, σ represents the ReLU activation function, BN represents batch normalization, and [*] represents the concatenation operation.

[0029] The text semantic enhancement branch, after undergoing 1*1 convolution and feature reshaping, is fed into a self-attention layer with 4 attention heads and 2 layers to capture a broad context in the image. The specific expression is as follows:

[0030] Lg = Transformer(reshape( conv1×1 (F enc4 )))

[0031] In the formula, Conv 1×1 Represents a 1x1 convolution, reshape(*) represents the feature reshaping operation, L g Representative generated features;

[0032] S4, Generate features (L) g The input is fed into the Long Short-Term Memory (LSTM) text generation module to obtain a medical text description (Texy) related to the lesion area. The generated text description (Texy) is then processed through a linear transformation layer to obtain four text features (L) corresponding to the decoding feature scale. i);

[0033] Medical text description (Texy) and text features (L) i The calculation expression for ) is as follows:

[0034] Text = LSTM(Lg)

[0035] L i =Linear(Text)

[0036] In the formula, LSTM represents Long Short-Term Memory network, Text represents the text result generated by the network, and Linear represents a fully connected layer;

[0037] S5. The four text features (L) corresponding to the decoding feature scale are... i ) and four downsampled dimensional encoded features (F enci The input is given to the text feature correction module to obtain the corrected text features (L'). i This avoids the problem of discrepancies between the generated text and the true description in a few samples, ensuring the correction of text features (L' i Semantically, it is closely related to the image content. The specific steps are as follows:

[0038] S5.1, Combine the four text features (L) corresponding to the decoding feature scale. i ) and four different dimensions of downsampling coding features (F enci After passing through two different mapping layers and multiplying them, we obtain attention map A. i ∈R HW×T Where T represents the number of words, the specific expression is as follows:

[0039] A i =(ω v F enci )(ω l L i )

[0040] Where, ω v The mapping function ω represents the four different dimensions of downsampled encoded features. l A mapping function representing four text features corresponding to the decoding feature scale;

[0041] S5.2, Attention map A i The attention matrix for each word t is obtained by summing and normalizing along the HW pixel dimension. The specific expression is as follows:

[0042]

[0043] in, This represents the correlation between the k-th pixel and the T-th word, where softmax(·) represents the normalized exponential function and... It is converted into a probability between 0 and 1;

[0044] S5.3, Attention Matrix Based on each word dimension and four text features corresponding to the decoding feature scale (L i Perform a dot product to obtain the visually guided sentence features (L). si The specific expression is as follows:

[0045]

[0046] Where ⊙ represents the dot product operation;

[0047] S5.4, Sentence features (L si After 1x1 convolution learning, it is combined with four text features (L) corresponding to the decoding feature scale. i The concatenation process adds sentence-level information to the text features corresponding to the decoding feature scale. The specific expression is as follows:

[0048]

[0049] S5.5. After performing a self-attention operation on the concatenated result, it is then combined with four text features (L) corresponding to the decoding feature scale. i The elements are added together, and the result is then learned through a 1x1 convolution to obtain the corrected text features (L'). i The specific expression is as follows:

[0050]

[0051] Where Self(·) represents the self-attention operation;

[0052] S6. Obtain the segmentation features (F) seg ) and corrected text features (L' i The input is fed into a text-guided mask map reconstruction decoder, after correcting text features (L' i Information-guided fusion is used to obtain decoding features (F) containing textual information. deci The binary segmentation prediction map (P) is then obtained through the segmentation head.

[0053] The guided mask map reconstruction decoder will correct text features (l' iAt each spatial location, the decoded features are fused to obtain decoded features of dimensions [8,256,28,28], [8,128,56,56], [8,64,112,112], and [8,32,224,224]. Each decoded layer has the same structure, including an upsampling operation with an upsampling factor of 2, a fusion operation, skip connections, residual blocks, and ReLU activation. Finally, the last layer of decoded features is convolved and activated by the segmentation head to obtain a binary segmentation prediction map (P) of dimensions [8,1,224,224]. The specific expression is as follows:

[0054] F dec4 =relu(Res(atten(upsmaple(F) seg ),L′ i ))) ×4

[0055] P = sigmod( conv1×1 (F dec4 ))

[0056] In the formula, sigmoid represents the activation function, atten(*,*) represents the attention fusion operation between text and image features, and conv 1×1 This represents a 1x1 convolution, ReLU represents ReLU activation, and F... dec4 This indicates the final decoded features, Res indicates the residual block, upsmaple indicates the upsampling operation, and ×4 indicates that four layers of decoding are performed.

[0057] The obtained segmentation features (F) seg ) and corrected text features (L' i The input is fed into a text-guided mask map reconstruction decoder, after correcting text features (L' i Information-guided fusion is used to obtain decoding features (F) containing textual information. deci The specific steps for obtaining the binary segmentation prediction map (P) after passing through the segmentation head are as follows:

[0058] S6.1, segmentation features (F seg ) and corrected text features (L' i These are used as guiding features and features to be guided, respectively, and then deconvolutioned to upsample the segmentation features by a factor of one; the specific expression is as follows:

[0059] F Q =UP(F seg i = 1

[0060] F Q =UP(Fdec (i-1) i = 2, 3, 4

[0061] Where UP(·) represents the upsampling operation, F dec(i-1) F represents the decoding feature of the previous layer. Q Represents guiding characteristics;

[0062] S6.2. A mapping function is used to encode the spatial context features of the guiding features and the features to be guided, resulting in the query vector Q, key vector K, and numerical vector V used for attention calculation. The above-sampled segmentation features (F...) seg ) as query vector Q, correcting text features (L' i Let K be the key vector and V be the numerical vector, and their specific expressions are as follows:

[0063] V Qi =ω qi (F Q )

[0064] L Ki =ω ki (L′ i )

[0065] L Vi =ω vi (L′ i )

[0066] Where, ω qi The mapping function ω represents the query vector Q. ki The mapping function representing the key vector K, ω vi The mapping function representing the numerical vector V, V Qi L Ki L Vi These represent the query vector Q, the key vector K, and the numerical vector V, respectively.

[0067] S6.3. Perform a dot product operation between the query vector Q generated by the guiding feature and the transpose of the key vector K generated by the feature to be guided, to generate a C*C attention map.

[0068] S6.4 Multiply the generated C*C attention map with the feature to be guided to obtain the guided compensated fused feature (F). Ai The specific expression is as follows:

[0069]

[0070] Where softmax(·) represents the normalization exponential function, d i This is the scaling factor;

[0071] S6.5. The obtained fused features are then combined with the four different downsampled encoded features (F) that serve as skip connections. enciThe information from the encoding stage is added to the decoding process by splicing along the channel dimension. Finally, the spliced ​​result is fed into two layers of residual blocks for further learning.

[0072] F deci =Res(Res[F Ai ,F enci i = 1, 2, 3, 4

[0073] S6.6 Repeat S6.1-S6.5, upsampling four times step by step to obtain decoding features of dimensions [8,256,28,28], [8,128,56,56], [8,64,112,112], and [8,32,224,224]. The structure of each decoding layer is the same. Finally, the decoding features of dimension [8,32,224,224] are convolved and activated by the segmentation head to obtain a binary segmentation prediction map of dimension [8,1,224,224].

[0074] S7. Set the network model loss function and update the model parameters according to the final optimization objective;

[0075] The network model loss function includes: segmentation loss for the segmentation task and generation loss for text generation;

[0076] The specific steps are as follows:

[0077] S7.1, The final encoded features (F) enc4 After passing through a Long Short-Term Memory (LSTM) network, medical text descriptions related to the lesion area are obtained. The generation loss is calculated on the feature dimension of the generated text and the real text. The specific expression is as follows:

[0078]

[0079] Where, p t (y t |y1,y2,…,y t-1 ;θ) represents the model's prediction of the current word y given the vocabulary and model parameters. t The probability of θ represents the learning parameters of the entire Long Short-Term Memory network;

[0080] S7.2, The final encoded features (F) enc4 After four upsampling decodings by the text-guided mask map reconstruction decoder, a binary segmentation prediction map is obtained. The segmentation loss is then calculated between the prediction map and the ground mask map. The specific expression is as follows:

[0081]

[0082] in, Represents the binary cross-entropy loss. Indicates the DICE coefficient loss. λ represents the semantic consistency loss. a The parameter λ represents the binary cross-entropy loss. b The parameter λ represents the loss of the DICE coefficient. c Parameters representing semantic consistency loss;

[0083] The binary cross-entropy loss DICE coefficient loss and semantic consistency loss The specific expression is as follows:

[0084]

[0085] Where, p j and y j Let represent the binary segmentation prediction probability and corresponding label classification of the j-th pixel in each input image, respectively, and N represent the sum of all elements. This represents the mean squared error loss that improves semantic consistency for corrected text features L'1 and L'2. This represents the mean squared error loss that improves semantic consistency for corrected text features L'1 and L'3. This represents the mean squared error loss that improves semantic consistency for corrected text features L'1 and L'4. and Together, they ensure that the corrected text features maintain a consistent focus on the lesion area;

[0086] The and The specific expression is as follows:

[0087]

[0088] S7.3 Finally, the loss will be generated. Segmentation loss The model parameters are updated by weighting the data and using it as the final optimization objective.

[0089] The overall loss function is expressed as:

[0090]

[0091] In the formula, The total loss function represents the entire network. and These represent the segmentation loss and the text generation loss, respectively. α represents the weighted parameters of the segmentation loss, and β represents the weighted parameters of the text generation loss.

[0092] Beneficial effects of the present invention

[0093] 1. This invention generates relevant medical text descriptions while segmenting the lesion area by combining a subtitle generation task.

[0094] 2. The method proposed in this invention extracts multimodal features containing both global and local details through a multimodal dual-stream collaborative feature extractor, which facilitates both segmentation and text generation tasks. Simultaneously, to fully leverage the potential of medical text reports, the text data in the dataset is used as a supervisory signal for the text generator-Long Short-Term Memory network instead of its input. Once the model is trained, it no longer needs text as a supervisory signal and can automatically infer the corresponding text information for the lesion area. Furthermore, the generated text further enhances the segmentation quality in the text-guided mask image reconstruction decoding layer, effectively improving the accuracy of medical image segmentation.

[0095] 3. The method of the present invention is superior to the current technology in terms of visual quality and quantitative evaluation. It generates additional text reports based on segmentation, providing more text-based diagnostic references for clinical diagnosis, which has important clinical significance and positive application prospects. Attached Figure Description

[0096] Figure 1 This is a schematic diagram of the process of the present invention;

[0097] Figure 2 is a detailed structural diagram of the present invention, wherein Figure 2(a) is the main network framework diagram of the present invention, Figure 2(b) is the structure diagram of the multimodal dual-stream collaborative feature extractor, Figure 2(c) is the structure diagram of the multimodal semantic information enhancement layer and the long short-term memory text generation layer, Figure 2(d) is the structure diagram of the text feature correction layer, and Figure 2(e) is the structure diagram of the text-guided mask map reconstruction decoder.

[0098] Figure 3 This is a visualization illustration of the segmentation process of the present invention;

[0099] Figure 4 The following is a visualization of the text generation of the present invention. Part (a) shows the text generation result of "bilateral lung infection, 9 infection areas, infection sites are the entire left lung and the entire right lung", part (b) shows the text generation result of "unilateral lung infection, 1 infection area, infection site is the lower left lung", part (c) shows the text generation result of "bilateral lung infection, 9 infection areas, infection sites are the entire left lung and the entire right lung", and part (d) shows the text generation result of "bilateral lung infection, 4 infection areas, infection sites are the upper left lung and the middle right lung". Detailed Implementation

[0100] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0101] Example 1

[0102] Please see Figure 1 -2, A pneumonia medical image segmentation method based on caption generation task enhancement, comprising the following steps:

[0103] S1. Obtain the pneumonia lesion image set V∈R H×W×3 In this context, the superscript H×W×3 indicates that the length, width, and number of channels of the image are H, W, and 3, respectively, and the resolution of the pneumonia lesion images is unified to 224*224. The pneumonia lesion image set includes training data and test data.

[0104] S2. Input the training data into the multimodal dual-stream collaborative feature extractor for four feature extractions to obtain four encoded features. Then, downsample each of the four encoded features, denoted as F. enc1 F enc2 F enc3 F enc4 The last downsampled coding feature is used as the final coding feature;

[0105] Among them, F enc1 F enc2 F enc3 F enc4 The features are downsampled and encoded in four different dimensions;

[0106] Specifically, the feature extraction involves processing pneumonia lesion images V∈R with dimensions [8, 3, 224, 224]. H×W×3 Features are extracted jointly by the "local feature extraction stream" and the "global feature extraction stream" of the multimodal dual-stream collaborative feature extractor, which are input in parallel. After residual convolution blocks, batch normalization, and ReLU activation in the "local feature extraction stream", local features (F) are obtained. loci Simultaneously, after undergoing a self-attention operation with 4 attention heads and 4 layers in the "Global Feature Extraction Stream," the global features (F) are obtained. gloi ), and then the local features (F) loci ) and global features (F gloi The encoded features (F) are obtained by concatenating the components. enci The specific steps are as follows:

[0107] S2.1. The training data is used as input and simultaneously fed into both a "local feature extraction stream" primarily based on convolutional neural networks and a "global feature extraction stream" primarily based on self-attention mechanisms, to extract local features (F) respectively. loci ) and global features (F gloi );

[0108] Local features (F) loci By accumulating spatial information through convolutional neural networks, it focuses on capturing local information of the texture contours of lesion areas;

[0109] Global features (F) gloi Then, it uses a self-attention mechanism to explore long-range dependencies in the image, focusing on capturing global information of the image;

[0110] The specific expressions for the global and local features are as follows:

[0111] F loci =E conv (V)i=1

[0112] F loci =E conv (F enc(i-1) i = 2, 3, 4

[0113] F gloi =E trans (V)i=1

[0114] F gloi =E trans (F glo(i-1) i = 2, 3, 4

[0115] In the formula, E conv and E trans These represent feature extractors for local and global feature extraction flows, respectively, where i represents the layer number, V represents the input image, and F represents the global feature extraction flow. enc(i-1) F represents the encoded features of the previous layer. loci and F gloi These represent the local and global features extracted from the i-th layer, respectively.

[0116] S2.2, global features (F gloi ) and local features (F loci The encoded features (F) are obtained by concatenating the dimensions after unifying them through a fully connected layer. enci This provides an expressive feature set for multimodal tasks, the encoded features (F) enci The specific expression for ) is as follows:

[0117] F enci =[F loci *F gloi]

[0118] In the formula, [*] represents the splicing operation, F enci Represents encoded features;

[0119] S2.3. Repeat S2.1-S2.2 to downsample the four coding features respectively, thereby obtaining four downsampled coding features F with different dimensions. enc1 F enc2 F enc3 F enc4 , of which F enc1 Performing a downsampling operation yields the result [8,32,224,224], F enc2 For F enc1 Based on the first downsampling operation, a second downsampling operation yields the result [8, 64, 112, 112], F enc3 For F enc2 Based on two downsampling operations, a third downsampling operation yields the result [8,128,56,56], F enc4 For F enc3 Based on three downsampling operations, a third downsampling operation yields the result [8,256,28,28].

[0120] S3, final encoded features (F) enc4 The input is fed into a multimodal semantic information enhancement layer, and after visual semantic enhancement and textual semantic enhancement, segmentation features suitable for the segmentation task (F) are obtained. seg ) and generation features suitable for text generation tasks (L g );

[0121] The multimodal semantic information enhancement layer includes two branches: the first branch is for visual semantic enhancement, which includes dilated convolutions with different dilation rates, batch normalization, and ReLU activation; the second branch is for text semantic enhancement, which includes attention mechanisms and 1*1 convolutions.

[0122] Furthermore, after the visual semantic enhancement branch undergoes four dilated convolution operations with dilation rates of 1, 3, 6, and 9, the results are concatenated according to the number of channels, and finally a 3*3 convolution is performed to compress the dimensions and activation to obtain the segmentation features, the specific expression of which is as follows:

[0123]

[0124] In the formula, F represents intermediate characteristics. seg Represents segmentation features, Represents dilated convolution, d = i represents the dilation rate, Conv 3×3σ represents a 3x3 convolution, σ represents the ReLU activation function, BN represents batch normalization, and [*] represents the concatenation operation.

[0125] The text semantic enhancement branch, after undergoing 1*1 convolution and feature reshaping, is fed into a self-attention layer with 4 attention heads and 2 layers to capture a broad context in the image. The specific expression is as follows:

[0126] L g =Transformer(reshape( conv1×1 (F enc4 )))

[0127] In the formula, conv 1×1 Represents a 1x1 convolution, reshape(*) represents the feature reshaping operation, L g Representative generated features;

[0128] S4, Generate features (L) g The input is fed into the Long Short-Term Memory (LSTM) text generation module to obtain medical text descriptions (Text) related to the lesion area. The generated text descriptions (Text) are then processed through a linear transformation layer to obtain four text features (L) corresponding to the decoding feature scale. i );

[0129] Medical text description (Text) and text features (L) i The calculation expression for ) is as follows:

[0130] Text = LSTM(L g )

[0131] L i =Linear(Text)

[0132] In the formula, LSTM represents Long Short-Term Memory network, Text represents the text result generated by the network, and Linear represents a fully connected layer;

[0133] S5. The four text features (L) corresponding to the decoding feature scale are... i ) and four different downsampling dimensions encoding features (F enci The input is given to the text feature correction module to obtain the corrected text features (L'). i This avoids the problem of discrepancies between the generated text and the true description in a few samples, ensuring the correction of text features (L' i Semantically, it is closely related to the image content. The specific steps are as follows:

[0134] S5.1, Combine the four text features (L) corresponding to the decoding feature scale. i ) and four different dimensions of downsampling coding features (Fenci After passing through two different mapping layers and multiplying them, we obtain attention map A. i ∈R HW×T Where T represents the number of words, the specific expression is as follows:

[0135] A i =(ω v F enci )(ω l L i )

[0136] Where, ω v The mapping function ω represents the four different dimensions of downsampled encoded features. l A mapping function representing four text features corresponding to the decoding feature scale;

[0137] S5.2, Attention map A i The attention matrix for each word t is obtained by summing and normalizing along the HW pixel dimension. The specific expression is as follows:

[0138]

[0139] in, This represents the correlation between the k-th pixel and the T-th word, where softmax(·) represents the normalized exponential function and... It is converted into a probability between 0 and 1;

[0140] S5.3, Attention Matrix Based on each word dimension and four text features corresponding to the decoding feature scale (L i Perform a dot product to obtain the visually guided sentence features (L). si The specific expression is as follows:

[0141]

[0142] Where ⊙ represents the dot product operation;

[0143] S5.4, Sentence features (L si After 1x1 convolution learning, it is combined with four text features (L) corresponding to the decoding feature scale. i The concatenation process adds sentence-level information to the text features corresponding to the decoding feature scale. The specific expression is as follows:

[0144]

[0145] S5.5. After performing a self-attention operation on the concatenated result, it is then combined with four text features (L) corresponding to the decoding feature scale. iThe elements are added together, and the result is then learned through a 1x1 convolution to obtain the corrected text features (L'). i The specific expression is as follows:

[0146]

[0147] Where Self(·) represents the self-attention operation;

[0148] S6. Obtain the segmentation features (F) seg ) and corrected text features (L' i The input is fed into a text-guided mask map reconstruction decoder, after correcting text features (L' i Information-guided fusion is used to obtain decoding features (F) containing textual information. deci The binary segmentation prediction map (P) is then obtained through the segmentation head.

[0149] The guided mask map reconstruction decoder will correct text features (L' i At each spatial location, the decoded features are fused to obtain decoded features of dimensions [8,256,28,28], [8,128,56,56], [8,64,112,112], and [8,32,224,224]. Each decoded layer has the same structure, including an upsampling operation with an upsampling factor of 2, a fusion operation, skip connections, residual blocks, and ReLU activation. Finally, the last layer of decoded features is convolved and activated by the segmentation head to obtain a binary segmentation prediction map (P) of dimensions [8,1,224,224]. The specific expression is as follows:

[0150] F dec4 =relu(Res(atten(upsmaple(F) seg ),L′ i ))) ×4

[0151] P = sigmod(conv) 1×1 (F dec4 ))

[0152] In the formula, sigmoid represents the activation function, atten(*,*) represents the attention fusion operation between text and image features, and conv 1×1 This represents a 1x1 convolution, ReLU represents ReLU activation, and F... dec4 This indicates the final decoded features, Res indicates the residual block, upsmaple indicates the upsampling operation, and ×4 indicates that four layers of decoding are performed.

[0153] The obtained segmentation features (F) seg ) and corrected text features (L' iThe input is fed into a text-guided mask map reconstruction decoder, after correcting text features (L' i Information-guided fusion is used to obtain decoding features (F) containing textual information. deci The specific steps for obtaining the binary segmentation prediction map (P) after passing through the segmentation head are as follows:

[0154] S6.1, segmentation features (F seg ) and corrected text features (L' i These are used as guiding features and features to be guided, respectively, and then deconvolutioned to upsample the segmentation features by a factor of one; the specific expression is as follows:

[0155] F Q =UP(F seg i = 1

[0156] F Q =UP(Fdec (i-1) i = 2, 3, 4

[0157] Where UP(·) represents the upsampling operation, F dec(i-1) F represents the decoding feature of the previous layer. Q Represents guiding characteristics;

[0158] S6.2. A mapping function is used to encode the spatial context features of the guiding features and the features to be guided, resulting in the query vector Q, key vector K, and numerical vector V used for attention calculation. The above-sampled segmentation features (F...) seg ) as query vector Q, correcting text features (L' i Let K be the key vector and V be the numerical vector, and their specific expressions are as follows:

[0159] V Qi =ω qi (F Q )

[0160] L Ki =ω ki (L′ i )

[0161] L Vi =ω vi (L′ i )

[0162] Where, ω qi The mapping function ω represents the query vector Q. ki The mapping function representing the key vector K, ω vi The mapping function representing the numerical vector V, V Qi L Ki L ViThese represent the query vector Q, the key vector K, and the numerical vector V, respectively.

[0163] S6.3. Perform a dot product operation between the query vector Q generated by the guiding feature and the transpose of the key vector K generated by the feature to be guided, to generate a C*C attention map.

[0164] S6.4 Multiply the generated C*C attention map with the feature to be guided to obtain the guided compensated fused feature (F). Ai The specific expression is as follows:

[0165]

[0166] Where softmax(·) represents the normalization exponential function, d i This is the scaling factor;

[0167] S6.5. The obtained fused features are then combined with the four different downsampled encoded features (F) that serve as skip connections. enci The information from the encoding stage is added to the decoding process by splicing along the channel dimension. Finally, the spliced ​​result is fed into two layers of residual blocks for further learning.

[0168] F deci =Res(Res[F Ai ,F enci i = 1, 2, 3, 4

[0169] S6.6 Repeat S6.1-S6.5, upsampling four times step by step to obtain decoding features of dimensions [8,256,28,28], [8,128,56,56], [8,64,112,112], and [8,32,224,224]. The structure of each decoding layer is the same. Finally, the decoding features of dimension [8,32,224,224] are convolved and activated by the segmentation head to obtain a binary segmentation prediction map of dimension [8,1,224,224].

[0170] S7. Set the network model loss function and update the model parameters according to the final optimization objective;

[0171] The network model loss function includes: segmentation loss for the segmentation task and generation loss for text generation;

[0172] The specific steps are as follows:

[0173] S7.1, The final encoded features (F) enc4 After passing through a Long Short-Term Memory (LSTM) network, medical text descriptions related to the lesion area are obtained. The generation loss is calculated on the feature dimension of the generated text and the real text. The specific expression is as follows:

[0174]

[0175] Where, p t (y t |y1,y2,…,y t-1 ;θ) represents the model's prediction of the current word y given the vocabulary and model parameters. t The probability of θ represents the learning parameters of the entire Long Short-Term Memory network;

[0176] S7.2, The final encoded features (F) enc4 After four upsampling decodings by the text-guided mask map reconstruction decoder, a binary segmentation prediction map is obtained. The segmentation loss is then calculated between the prediction map and the ground mask map. The specific expression is as follows:

[0177]

[0178] in, Represents the binary cross-entropy loss. Indicates the DICE coefficient loss. λ represents the semantic consistency loss. a The parameter λ represents the binary cross-entropy loss. b The parameter λ represents the loss of the DICE coefficient. c Parameters representing semantic consistency loss;

[0179] The binary cross-entropy loss DICE coefficient loss and semantic consistency loss The specific expression is as follows:

[0180]

[0181] Where, p j and y j Let represent the binary segmentation prediction probability and corresponding label classification of the j-th pixel in each input image, respectively, and N represent the sum of all elements. This represents the mean squared error loss that improves semantic consistency for corrected text features L'1 and L'2. This represents the mean squared error loss that improves semantic consistency for corrected text features L'1 and L'3. This represents the mean squared error loss that improves semantic consistency for corrected text features L'1 and L'4. and Together, they ensure that the corrected text features maintain a consistent focus on the lesion area;

[0182] The and The specific expression is as follows:

[0183]

[0184] S7.3 Finally, the loss will be generated. Segmentation loss The model parameters are updated by weighting the data and using it as the final optimization objective.

[0185] The overall loss function is expressed as:

[0186]

[0187] In the formula, The total loss function represents the entire network. and These represent the segmentation loss and the text generation loss, respectively. α represents the weighted parameters of the segmentation loss, and β represents the weighted parameters of the text generation loss.

[0188] This embodiment conducted experiments on the MosMedData+ public dataset to comprehensively evaluate the effectiveness of the proposed method. The dataset was divided into training, validation, and test sets using a common 3:1:1 partitioning method. The number of batches in the network was set to 8, the number of training epochs was set to 200, the learning rate was set to 0.0001, and an early stopping strategy was adopted.

[0189] In this embodiment, the evaluation metrics are mDice and mIoU, and the larger the average value, the better the performance.

[0190] This embodiment was compared with state-of-the-art methods, including U-Net, Att-Unet, TransUNet, SCOAT-Net, COPLE-Net, TGANet, ConTEXTualNet, and LViT-T. The experimental results are shown in Table 1. It can be seen that the method of this invention achieves an mIoU of 63.02 and a Dice of 75.87. This demonstrates the effectiveness of the method of this invention in pneumonia segmentation.

[0191] Table 1 compares the objective evaluations of different methods on the MosMedData+ test set.

[0192]

[0193]

[0194] The results show that the method of this invention significantly outperforms existing methods. Furthermore, thanks to the novel network structure design and multi-task facilitation strategy of this invention, the accuracy of the segmentation results is further ensured, demonstrating significant clinical implications and promising application prospects.

[0195] Furthermore, this invention compares the segmentation results of U-Net, Att-Unet, TransUNet, SCOAT-Net, COPLE-Net, TGANet, ConTEXTualNet, and LViT-T, as follows: Figure 4 As shown. See the segmentation visualization results. Figure 3 In the yellow area of ​​the first row of images, it can be seen that the segmentation results of CopleNet, TGANet, and LViT methods also segmented extra regions, exhibiting segmentation errors. Our method, however, can segment the main lesion area. Figure 3 As can be seen in the yellow area of ​​the third row of images, current methods still have shortcomings in recognizing the edges and details of lesions. For example, TGANet can only identify a very small number of lesion areas, and CopleNet and ConTEXTualNet do not reflect the edge contours of lesion areas at all. In contrast, the method of our invention can identify lesion areas and their edge details very well, which shows that the invention has a better ability to capture lesion areas.

[0196] Regarding the evaluation of the text generation results, the BLEU score of the text generated by this invention reached 83.89%. The BLEU score shows the accuracy of the model in generating phrases and longer texts, indicating that the accuracy and reliability of the text generation method of this invention are already quite high. Meanwhile, some randomly selected samples with good and poor performance were used for visualization, such as... Figure 4 Part (a) shows the text generation result for "bilateral lung infection, 9 infection areas, infection sites are the entire left lung and the entire right lung", Part (b) shows the text generation result for "unilateral lung infection, 1 infection area, infection site is the left lower lung", Part (c) shows the text generation result for "bilateral lung infection, 9 infection areas, infection sites are the entire left lung and the entire right lung", and Part (d) shows the text generation result for "bilateral lung infection, 4 infection areas, infection sites are the left upper lung and the right middle lung". Parts (a) and (c) show the results of different infection situations at the same infection site.

[0197] In each image, red dots represent text labels, while white dots represent text generated by the model. It can be observed that a small number of samples have errors in the generated descriptions, such as... Figure 4In (d), the original 'upper left lung' was incorrectly generated as 'middle left lung'. The incorrectly generated text is marked in red, but the remaining words are very close to the correct text label. Figure 4 (a) Figure 4 (b) and Figure 4 In (c), it can be seen that the method of the present invention can capture the area and location of lesions in various locations and for different types of lesions, and can generate reliable medical text annotations.

[0198] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for segmenting pneumonia medical images based on caption generation task enhancement, characterized in that: Includes the following steps: S1. Obtain a set of pneumonia lesion images and unify the resolution of the pneumonia lesion images to 224*224. The set of pneumonia lesion images includes training data and test data. S2. Input the training data into the multimodal dual-stream collaborative feature extractor for four feature extractions to obtain four encoded features. Then, downsample each of the four encoded features sequentially, and denote them as follows: , , , The last obtained downsampled encoding feature As the final encoded feature, among which, , , , The features are downsampled and encoded in four different dimensions; S3. Input the final encoded features into the multimodal semantic information enhancement layer. After visual semantic enhancement and text semantic enhancement, segmentation features suitable for segmentation tasks and generation features suitable for text generation tasks are obtained respectively. S4. Input the generated features into the long short-term memory text generation module to obtain the medical text description of the lesion area, and then transform the medical text description through a linear layer to obtain four text features corresponding to the scale of the decoded features. S5. Input the four text features corresponding to the decoding feature scale and the four downsampled encoded features with different dimensions into the text feature correction module to obtain the corrected text features; S6. Input the obtained segmentation features and corrected text features into the text-guided mask map reconstruction decoder to obtain the decoded features with text information, and then pass them through the segmentation head to obtain the binary segmentation prediction map. S7. Set the loss function for the network model and update the model.

2. The method for segmenting pneumonia medical images based on caption generation task enhancement according to claim 1, characterized in that: The specific steps for inputting training data into a multimodal dual-stream collaborative feature extractor for four feature extractions to obtain four encoded features, and then downsampling each of the four encoded features sequentially, with the last downsampled encoded feature being used as the final encoded feature, are as follows: S2.

1. The training data is fed into both a "local feature extraction stream" (primarily using a convolutional neural network) and a "global feature extraction stream" (primarily using a self-attention mechanism) to extract local and global features respectively. The specific expressions for the global and local features are as follows: ； ； ； ； In the formula, and These represent feature extractors for local and global feature extraction flows, respectively, where i represents the layer number. Indicates the input image. This represents the encoded features of the previous layer. and These represent the local and global features extracted from the i-th layer, respectively. S2.

2. The global and local features are unified in dimension by passing them through a fully connected layer and then concatenated to obtain the encoded features. The specific expression of the encoded features is as follows: ； In the formula, This represents a splicing operation. Represents encoded features; S2.

3. Repeat S2.1-S2.2 to downsample the four coding features respectively, thereby obtaining four downsampled coding features with different dimensions. , , , ,in, The result obtained from one downsampling is [8,32,224,224]. Based on Based on the first downsampling operation, a second downsampling operation yields the result [8, 64, 112, 112]. Based on Based on the two downsampling operations, a third downsampling operation yields the result [8, 128, 56, 56]. Based on Based on the three downsampling operations, a third downsampling operation yields the result [8,256,28,28].

3. The method for segmenting pneumonia medical images based on subtitle generation task enhancement according to claim 1, characterized in that: The multimodal semantic information enhancement layer includes two branches: the first branch is for visual semantic enhancement, which includes dilated convolutions with different dilation rates, batch normalization, and ReLU activation; the second branch is for text semantic enhancement, which includes attention mechanisms and 1*1 convolutions.

4. The method for segmenting pneumonia medical images based on caption generation task enhancement according to claim 3, characterized in that: The visual semantic enhancement branch undergoes four dilated convolution operations with dilation rates of 1, 3, 6, and 9. The results are then concatenated according to the number of channels, and finally a 3*3 convolution is performed to compress the dimensions and activation to obtain the segmentation features. The specific expression is as follows: ； ； In the formula, F represents intermediate characteristics. seg Represents segmentation features, Represents dilated convolution. Represents the expansion rate. Represents a 3x3 convolution. represent Activation function Represents batch normalization. This represents a splicing operation; The text semantic enhancement branch, after undergoing 1*1 convolution and feature reshaping, is fed into a self-attention layer with 4 attention heads and 2 layers. The specific expression is as follows: ； In the formula, Represents a 1x1 convolution. Represents the characteristic reshaping operation. This indicates the generated features.

5. The method for segmenting pneumonia medical images based on caption generation task enhancement according to claim 1, characterized in that: The specific steps for inputting four text features corresponding to the decoded feature scale and four downsampled dimensional encoded features into the text feature correction module to obtain the corrected text features are as follows: S5.

1. The four text features corresponding to the decoding feature scale and the four downsampled encoded features of different dimensions are multiplied after passing through two different mapping layers to obtain the attention map. ,in Represents the number of words; S5.2, Attention Map according to The attention matrix for each word t is obtained by summing and normalizing along the pixel dimension; S5.

3. Perform dot product of the attention matrix with the four text features corresponding to the decoding feature scale according to each word dimension to obtain the visually guided sentence features; S5.4 After learning the sentence features through 1x1 convolution, concatenate them with four text features corresponding to the decoding feature scale to supplement the sentence-level information into the text features corresponding to the decoding feature scale. S5.

5. After the concatenated result is processed by self-attention, it is added element-wise to the four text features corresponding to the decoding feature scale. The result after addition is learned by 1x1 convolution to obtain the corrected text features.

6. The method for segmenting pneumonia medical images based on caption generation task enhancement according to claim 1, characterized in that: The specific steps for inputting the obtained segmentation features and corrected text features into the text-guided mask map reconstruction decoder to obtain decoded features with text information, and then passing them through the segmentation head to obtain a binary segmentation prediction map, are as follows: S6.

1. The segmentation features and the corrected text features are used as the guiding features and the features to be guided, respectively, and the segmentation features are upsampled by one time after deconvolution. S6.

2. The spatial context features of the guiding features and the features to be guided are encoded using a mapping function to obtain the query vector Q, key vector K and numerical vector V for attention calculation, respectively. The oversampled segmentation features are used as the query vector Q, and the corrected text features are used as the key vector K and numerical vector V. S6.

3. Perform a dot product operation between the query vector Q generated by the guiding feature and the transpose of the key vector K generated by the feature to be guided, to generate a C*C attention map. S6.4 Multiply the generated C*C attention map with the feature to be guided to obtain the fused feature after guidance compensation; S6.5 The obtained fused features are then concatenated with the four encoded features used as skip connections along the channel dimension to supplement the information from the encoding stage into the decoding process. Finally, the result after the concatenation operation is sent to the two-layer residual block for further learning. S6.6 Repeat S6.1-S6.5, upsampling four times step by step to obtain decoding features of dimensions [8,256,28,28], [8,128,56,56], [8,64,112,112], and [8,32,224,224]. The structure of each decoding layer is the same. Finally, the decoding features of dimension [8,32,224,224] are convolved and activated by the segmentation head to obtain a binary segmentation prediction map of dimension [8,1,224,224].

7. The method for segmenting pneumonia medical images based on caption generation task enhancement according to claim 1, characterized in that: The specific steps for setting the network model loss function and updating the model parameters according to the final optimization objective are as follows: S7.1 After passing the final encoded features through a long short-term memory network, a medical text description related to the lesion area is obtained. The generation loss is calculated on the feature dimension for the generated text and the real text. S7.2 After the final encoded features are upsampled and decoded four times by the text-guided mask map reconstruction decoder, a binary segmentation prediction map is obtained. The segmentation loss is calculated on the prediction map and the real mask map. S7.3 Finally, the generation loss and segmentation loss are added together to obtain the final optimization objective and the model parameters are updated.

Images