Pathological image semantic segmentation method based on multi-modal feature fusion and boundary enhancement

Abstract

The invention relates to a pathological image semantic segmentation method based on multi-modal feature fusion and boundary enhancement. The method comprises the following steps: firstly, acquiring an original nephropathy image and an associated pathology report text, and preprocessing the original nephropathy image and the associated pathology report text; then, a multi-modal segmentation network is constructed, and the multi-modal segmentation network comprises an image encoder, a text encoder, an image-text multi-modal fusion and enhancement module, a shared decoder, a segmentation prediction head and an auxiliary boundary detection head; then, training the multi-modal segmentation network by adopting the preprocessed original nephropathy image and the associated pathological report text; and finally, inputting a to-be-segmented pathological image and an associated pathological report text into the trained multi-modal segmentation network, and outputting a target semantic segmentation result. A nephropathy image and a matched pathology report text are effectively and deeply fused and applied to a pixel-level semantic segmentation task, and a more accurate, robust and fine segmentation result is provided.

Description

Technical Field

[0001] The present application relates to the technical field of biomedical image analysis and processing, and in particular to a pathological image semantic segmentation method based on multimodal feature fusion and boundary enhancement. Background Art

[0002] Semantic segmentation of pathological images is a key technology in computational pathology, which is used to automatically identify and distinguish different cells or tissue structures (such as glomeruli, tubules, interstitium, blood vessels, etc.) on tissue sections. Whole Slide Image (WSI) poses memory and computing challenges for deep learning analysis due to its huge size and high resolution. Traditional solutions usually use patch-based processing. For renal pathology, accurate segmentation of different tissue structures is crucial for disease diagnosis and quantitative analysis. Pathological diagnosis does not rely solely on images. Pathologists also combine detailed pathology report text, which includes macroscopic descriptions, key lesions observed under the microscope, diagnostic conclusions, and assessments of the extent of lesions. These textual information can assist doctors in interpreting and judging image details.

[0003] In the prior art, semantic segmentation methods for renal pathology images usually focus on the processing of single-modality images. For example, only H&E or PAS-stained images are used for glomerular or interstitial fibrosis / tubular atrophy IFTA segmentation, and deep learning models such as U-Net, DeepLab series or its variants are used. The input is an image block, and the output is a pixel-level category prediction of the corresponding image block. However, it only relies on the local pixel information of the image and the limited context within the image. Insufficient information utilization leads to decreased accuracy and poor robustness of the segmentation results, which makes it easy to miss or misjudge difficult-to-identify lesion structures. At the same time, some technologies analyze different imaging modalities or images with non-text clinical data at a late stage, but only perform simple splicing or addition at a high level. There is also the problem of insufficient information utilization leading to decreased accuracy and poor robustness of the segmentation results.

[0004] Therefore, in the related technology, there is an urgent need for a method that can improve the accuracy and robustness of semantic segmentation of pathological images. Summary of the Invention

[0005] Based on this, it is necessary to address the above technical problems and provide a pathological image semantic segmentation method based on multimodal feature fusion and boundary enhancement that can improve the accuracy and robustness of pathological image semantic segmentation.

[0006] In a first aspect, the present application provides a pathological image semantic segmentation method based on multimodal feature fusion and boundary enhancement. The method comprises:

[0007] Obtain original renal pathology images and associated pathology report texts and perform preprocessing;

[0008] Constructing a multimodal segmentation network, the multimodal segmentation network including an image encoder, a text encoder, an image-text multimodal fusion and enhancement module, a shared decoder, a segmentation prediction head, and an auxiliary boundary detection head;

[0009] The multimodal segmentation network is trained using the pre-processed original renal pathology image and the associated pathology report text;

[0010] The pathological image to be segmented and the associated pathological report text are input into the trained multimodal segmentation network, and the target semantic segmentation result is output.

[0011] Optionally, in one embodiment of the present application, the preprocessing includes:

[0012] performing color normalization and size adjustment on the original renal pathology image;

[0013] The pathology report text is segmented, stop words are removed, normalized, sentence segmented, and format converted.

[0014] Optionally, in one embodiment of the present application, the image encoder adopts a convolutional neural network architecture, and the text encoder adopts a Transformer-based model.

[0015] Optionally, in one embodiment of the present application, the image-text multimodal fusion and enhancement module includes an image-text attention fusion module and a boundary-aware enhancement module.

[0016] Optionally, in one embodiment of the present application, the training of the multimodal segmentation network using the pre-processed original renal pathology image and the associated pathology report text includes:

[0017] A composite loss function is set, a value of the composite loss function is calculated based on the initial semantic segmentation result, the initial auxiliary boundary prediction result and the true value, and parameters of the multimodal segmentation network are adjusted based on the value of the composite loss function.

[0018] Optionally, in one embodiment of the present application, the composite loss function includes segmentation loss, rare category false negative penalty term and boundary loss.

[0019] In a second aspect, the present application also provides a pathological image semantic segmentation device based on multimodal feature fusion and boundary enhancement. The device comprises:

[0020] Image and text acquisition module, used to obtain original renal pathology images and associated pathology report texts and perform preprocessing;

[0021] A multimodal segmentation network construction module is used to construct a multimodal segmentation network, wherein the multimodal segmentation network includes an image encoder, a text encoder, an image-text multimodal fusion and enhancement module, a shared decoder, a segmentation prediction head, and an auxiliary boundary detection head;

[0022] A multimodal segmentation network training module, configured to train the multimodal segmentation network using the pre-processed original renal pathology images and associated pathology report text;

[0023] The semantic segmentation module is used to input the pathological image to be segmented and the associated pathological report text into the trained multimodal segmentation network and output the target semantic segmentation result.

[0024] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program and the processor executes the steps of the method described in each of the above embodiments.

[0025] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in each of the above embodiments.

[0026] The above-mentioned pathological image semantic segmentation method based on multimodal feature fusion and boundary enhancement first obtains the original renal pathological image and the associated pathological report text and performs preprocessing; then, constructs a multimodal segmentation network, which includes an image encoder, a text encoder, an image-text multimodal fusion and enhancement module, a shared decoder, a segmentation prediction head, and an auxiliary boundary detection head; then, the multimodal segmentation network is trained using the preprocessed original renal pathological image and the associated pathological report text; finally, the pathological image to be segmented and the associated pathological report text are input into the trained multimodal segmentation network, and the target semantic segmentation result is output. Specifically, independent image and text encoders are used to extract spatial features of the image and semantic features of the text in parallel. Cross-attention or learned modulation / gating approaches are then used to achieve deep cross-modal image-text fusion. This approach leverages text context to dynamically modulate and enhance image features, overcoming the gap between macroscopic textual information and microscopic pixel features. A boundary-aware enhancement mechanism is integrated, and the learned boundary information is used to guide the fusion of features or subsequent decoding to improve segmentation boundary accuracy. The boundary-aware process benefits from the more discriminative features provided by text information. Pathology report text information is deeply integrated into the pixel-level segmentation network. Specifically, diagnostic clues and descriptions provided by the text are used to enhance the feature representation of corresponding regions in the image through ITFME, helping the model identify and distinguish atypical, poorly defined, or rare pathological structures in the image. By effectively and deeply fusing renal pathology images and accompanying pathology report text for pixel-level semantic segmentation, this approach delivers more accurate, robust, and refined segmentation results than existing single-modal or simple multimodal approaches. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 FIG2 is an application environment diagram of a pathological image semantic segmentation method based on multimodal feature fusion and boundary enhancement in one embodiment;

[0028] Figure 2 1 is a flow chart of a pathological image semantic segmentation method based on multimodal feature fusion and boundary enhancement in one embodiment;

[0029] Figure 3 Schematic diagram of the structure of a multimodal segmentation network in one embodiment;

[0030] Figure 4 1 is a structural block diagram of a pathological image semantic segmentation device based on multimodal feature fusion and boundary enhancement in one embodiment;

[0031] Figure 5 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0033] The pathological image semantic segmentation method based on multimodal feature fusion and boundary enhancement provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal communicates with the server through the network. The data storage system can store data that the server needs to process. The data storage system can be integrated on the server or placed on the cloud or other network servers. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0034] In one embodiment, Figure 2 As shown in the figure, a semantic segmentation method of pathological images based on multimodal feature fusion and boundary enhancement is provided. Figure 1 The following steps are used as an example to illustrate the server in the example:

[0035] S201: Acquire original renal pathology images and associated pathology report text, and perform preprocessing.

[0036] In the embodiment of the present application, first, the original renal pathology image and the associated pathology report text are obtained, that is, the image patch is extracted from the original renal pathology WSI, and the pathology report text associated with the WSI or the case to which the WSI belongs is obtained, and preprocessed respectively, and each group (corresponding to the same tissue area or WSI) is output as a representation of the preprocessed image patch and its associated text token sequence.

[0037] Specifically, in one embodiment of the present application, the preprocessing includes:

[0038] S301: performing color normalization and size adjustment on the original renal pathology image.

[0039] S303: performing word segmentation, stop word removal, normalization, sentence segmentation, and format conversion on the pathology report text.

[0040] In one embodiment of the present application, standard preprocessing is performed on the original renal pathology image, including color normalization, size adjustment, etc. The pathology report text is preprocessed, including word segmentation using professional medical field word segmentation tools, stop word removal, normalization, sentence segmentation, etc., and the text is converted into a token sequence, an input format acceptable to the model.

[0041] S203: Construct a multimodal segmentation network, where the multimodal segmentation network includes an image encoder, a text encoder, an image-text multimodal fusion and enhancement module, a shared decoder, a segmentation prediction head, and an auxiliary boundary detection head.

[0042] In the embodiment of the present application, an end-to-end deep learning model, a multimodal segmentation network, is constructed. Figure 3 As shown in the figure, it includes an image encoder (Image Encoder), a text encoder (Text Encoder), an image-text multimodal fusion and enhancement module (Image-Text Multimodal Fusion and Enhancement Module, ITFME), a shared decoder (Shared Decoder), a segmentation prediction head (Segmentation Head) and an auxiliary boundary detection head (Auxiliary Boundary Head). Among them, the image encoder and the text encoder respectively encode the features of the image data and the text data. The image-text multimodal fusion and enhancement module is used to fuse the features from the image encoder and the text encoder at different levels and introduce boundary-aware enhancement. The shared decoder is a decoder network for semantic segmentation, such as the decoder part of the U-Net structure, which receives the multi-scale fusion features output by the ITFME module. And through a series of upsampling operations (such as transposed convolution or interpolation followed by convolution), the spatial resolution is gradually restored. At each upsampling step, the fused features of the current resolution can be spliced or added with the features from the corresponding layer of the image encoder (which may have been slightly processed or fused with the text features earlier) through skip connections to integrate high-resolution local information, which is more discriminative after being fused with the text context. The segmentation prediction head is connected to the last upsampled output of the shared decoder. Through one or more convolutional layers, the feature map output by the decoder is mapped to the final multi-category semantic segmentation prediction result. The output is a probability map P with the number of channels equal to the number of categories (including background). seg ∈[0,1] H×W×C , where H, W are the original resolutions of the image block and C is the number of categories. The vector P for each pixel (i, j) is seg(i, j, :) represents the probability distribution of the pixel belonging to each category, which is usually obtained through the softmax activation function. The auxiliary boundary prediction head is obtained from the output of the ITFME module (e.g. or its further processed features) or an intermediate layer of the shared decoder leads to a branch. Through one or more convolutional layers, a single-channel binary boundary prediction result P is output. bnd ∈[0,1] H×W×C The value of each pixel (i, j) represents the probability that the pixel is located on the tissue boundary, which is usually obtained through the sigmoid activation function.

[0043] Specifically, in one embodiment of the present application, the image encoder adopts a convolutional neural network architecture, and the text encoder adopts a Transformer-based model.

[0044] In one embodiment of the present application, the image encoder adopts a convolutional neural network architecture, for example, a pre-trained ResNet (such as ResNet-50 or ResNet-101) can be used as the backbone network, or the encoder part of the U-Net structure can be used. The encoder receives the input image block and gradually extracts multi-scale image feature maps through a series of convolution, activation and pooling layers. Specifically, it generates a series of feature maps with decreasing resolution and increasing number of channels, such as the feature maps output after block 1 to block 4 of ResNet. They correspond to different receptive fields and abstract levels respectively. The text encoder uses a Transformer-based model. For example, BERT (Bidirectional Encoder Representations from Transformers) or its pre-trained version in the medical field (such as BioBERT). The encoder receives the pre-processed text token sequence as input, processes it through the self-attention mechanism and feedforward network, and outputs a text feature representation rich in contextual information. Considering that pathology reports usually provide global diagnostic information, the hidden state corresponding to the [CLS] token output by BERT can be taken as the global feature vector V of the entire text text , or pool the hidden states of all tokens (such as average pooling or maximum pooling) to obtain the text feature vector.

[0045] In one embodiment of the present application, the image-text multimodal fusion and enhancement module includes an image-text attention fusion module and a boundary-aware enhancement module.

[0046] In one embodiment of the present application, the image-text multimodal fusion and enhancement module includes an image-text attention fusion module and a boundary perception enhancement module, wherein the image-text attention fusion module uses a cross-attention-based method to achieve cross-modal fusion. Specifically, a cross-attention unit can be applied at each image feature level (or several selected key levels). (Dimension H k ×W k ×C k ) and text feature vector V text (Dimension C text ), the image feature map can be flattened into a sequence (Dimension H k W k ×C k ), the text feature vector can be copied H k W k A text sequence (Dimension H k W k ×C text ) or directly as part of Query / Key / Value.

[0047] For text query images, the text feature V text Mapped to Flatten the image features Mapped to and Calculate the attention score using the following formula:

[0048]

[0049] The fused text-enhanced image feature sequence is This output is a global vector, and a mechanism needs to be designed to fuse it back to the image feature map in the spatial domain, such as through an additive or gating mechanism with the original image feature map. Combine them, or broadcast them to the spatial dimensions via the learned mapping.

[0050] For image query text, a method more suitable for pixel-level tasks is to flatten the image features. Mapped to The text feature V text Mapped to and (Or use all token feature sequences output by the text encoder as Key and Value.) Calculate the attention score using the following formula:

[0051]

[0052] Fused text-enhanced image feature sequence (Additive Fusion). Reshape back to image feature map This approach enables the features at each location in the image to be modulated and enhanced based on the text content. This cross-attention fusion can be performed in parallel or sequentially at multiple scales.

[0053] Inside the ITFME module, a parallel sub-network or mechanism is also introduced to predict the boundary probability map inside the image block. This sub-network receives the fused image features As input, a single-channel boundary prediction map is output through the convolution layer Using this boundary prediction map as weight, a boundary-guided spatial attention mechanism (e.g. where σ is the sigmoid activation function, or more complex mechanisms utilize As a display guide, the boundary-related representations in the fused features are further enhanced. This enhancement process allows the model to focus more on potential tissue boundary regions in the image after integrating the text information. Text information can sometimes help distinguish structures with ambiguous boundaries (such as renal tubular atrophy and interstitial fibrosis). This information is transferred to the features through fusion, making boundary prediction and enhancement more effective.

[0054] S205: The multimodal segmentation network is trained using the pre-processed original renal pathology image and the associated pathology report text.

[0055] In an embodiment of the present application, the preprocessed image block is input to an image encoder, a multi-scale image feature map is extracted, the preprocessed pathology report text token sequence is input to a text encoder, a text feature vector is extracted, and in the ITFME module, the multi-scale image features output by the image encoder and the text features output by the text encoder are received. At multiple scales, an image-text fusion unit based on a mechanism such as cross attention is used to learn to integrate the contextual information of the text features into the image features to generate fused features. At the same time, a boundary perception mechanism is applied to the fused features, such as predicting a boundary probability map through a sub-network, and using the probability map to guide spatial attention or directly combine with the fused features to enhance the boundaries of the fused features, generate fused and enhanced multi-scale features, and input the fused and enhanced multi-scale features to a shared decoder. The decoder gradually restores the spatial resolution through an upsampling operation, and integrates the multi-scale information in combination with the jump connection from the corresponding layer of the image encoder (or early fused). The segmentation prediction head connected to the output of the decoder is used to process the refined features output by the decoder to obtain the final multi-category semantic segmentation prediction results. The auxiliary boundary prediction head connected to the output of ITFME or the middle layer of the decoder is used to process the corresponding features to obtain auxiliary binary boundary prediction results. The network parameters are continuously adjusted according to the difference between the prediction results and the true values until the prediction results are close to the true values and the model training is completed.

[0056] Specifically, in one embodiment of the present application, the training of the multimodal segmentation network using the pre-processed original renal pathology image and the associated pathology report text includes:

[0057] A composite loss function is set, a value of the composite loss function is calculated based on the initial semantic segmentation result, the initial auxiliary boundary prediction result and the true value, and parameters of the multimodal segmentation network are adjusted based on the value of the composite loss function.

[0058] In one embodiment of the present application, a composite loss function is set, and the value of the composite loss function is calculated based on the initial semantic segmentation result, the initial auxiliary boundary prediction result, and the true value. Based on the value of the composite loss function, the gradient of the network parameters is calculated by the back propagation algorithm, and the parameters of the multimodal segmentation network are updated using an optimizer (such as Adam, SGD, etc.). Wherein, the true value refers to the true value mask and the true value boundary map. The true value boundary map is obtained by performing morphological operations (such as dilation minus corrosion) on the true value segmentation mask.

[0059] In one embodiment of the present application, the composite loss function includes segmentation loss, rare class false negative penalty term and boundary loss.

[0060] In one embodiment of the present application, a composite loss function is set up to take into account multi-class segmentation, class imbalance, rare class identification and boundary prediction, including segmentation loss, rare class false negative penalty and boundary loss. seg ) measure the multi-category segmentation prediction results P seg and the true value mask Y seg The difference between the two is combined with the pixel-level weighted cross-entropy loss (WCE) and Dice loss. Dice loss measures the spatial overlap between the prediction and the true value and is robust to class imbalance. The calculation formula of the segmentation loss is as follows:

[0061] L seg =α·L WCE +β·L Dice

[0062]

[0063] Where N is the total number of pixels, C is the number of categories, is the true value (0 or 1) of pixel i belonging to category c, is the predicted probability that pixel i belongs to category c, ω c is the weight of category c, which is usually calculated based on the pixel ratio of the category in the training set (such as inverse frequency) to alleviate the problem of category imbalance. ∈ is a very small value to prevent division by zero. α and β are weight coefficients.

[0064] The rare category false negative penalty (L FN_Rare ) is a special penalty term added for rare categories (such as glomeruli, IFTA, and blood vessels). It imposes a larger penalty when the model misclassifies pixels of rare categories as other categories. This penalty term can be based on the idea of focal loss or a loss that is sensitive to false negatives of rare categories. The calculation formula is as follows:

[0065]

[0066] or

[0067] Among them, N rare is the total number of pixels of rare categories, ω ′ c >ω c is a higher weight, is a modulation factor (focal loss) that increases the loss when the predicted probability is low, k cThis penalty is much higher than the standard weight. This penalty works in conjunction with textual information. Pathology reports often explicitly mention these rare lesions. ITFME leverages this textual information to enhance the features of rare class regions, making the network more likely to predict the correct class, thereby reducing the loss here.

[0068] Boundary loss (L boundary ) measure the auxiliary boundary prediction result P bnd and the true value boundary map Y bnd The difference between the ground truth boundary map Y bnd You can use Y seg The morphological gradient operator is applied to obtain a binary image (boundary pixels are 1, and all other pixels are 0). Boundary loss forces the network to learn features that distinguish tissue boundaries. Combined with the boundary enhancement mechanism of ITFME, it helps improve the accuracy of segmentation boundaries. Boundary loss is calculated using Binary Cross-Entropy (BCE) and Dice loss. The calculation formula is as follows:

[0069] L boundary =γ·L BCE_bnd +δ·L Dice_bnd

[0070]

[0071] Among them, γ and δ are weight coefficients.

[0072] The calculation formula of the total compound loss function is expressed as follows:

[0073] L total =ω seg ·L seg +ω fn ·L FN_Rare +ω bnd ·L boundary

[0074] Among them, ω seg 、ω fn 、ω bnd It is a weight coefficient that balances the importance of each loss.

[0075] S207: Input the pathological image to be segmented and the associated pathological report text into the trained multimodal segmentation network, and output the target semantic segmentation result.

[0076] In the embodiment of the present application, after the multimodal segmentation network is trained, the pathological image to be segmented and the associated pathological report text are input into it, and the target semantic segmentation result is output.

[0077] In the above-mentioned pathological image semantic segmentation method based on multimodal feature fusion and boundary enhancement, first, the original renal pathological image and the associated pathological report text are obtained and preprocessed; then, a multimodal segmentation network is constructed, which includes an image encoder, a text encoder, an image-text multimodal fusion and enhancement module, a shared decoder, a segmentation prediction head and an auxiliary boundary detection head; then, the multimodal segmentation network is trained using the preprocessed original renal pathological image and the associated pathological report text; finally, the pathological image to be segmented and the associated pathological report text are input into the trained multimodal segmentation network, and the target semantic segmentation result is output. Specifically, independent image and text encoders are used to extract spatial features of the image and semantic features of the text in parallel. Cross-attention or learned modulation / gating approaches are then used to achieve deep cross-modal image-text fusion. This approach leverages text context to dynamically modulate and enhance image features, overcoming the gap between macroscopic textual information and microscopic pixel features. A boundary-aware enhancement mechanism is integrated, and the learned boundary information is used to guide the fusion of features or subsequent decoding to improve segmentation boundary accuracy. The boundary-aware process benefits from the more discriminative features provided by text information. Pathology report text information is deeply integrated into the pixel-level segmentation network. Specifically, diagnostic clues and descriptions provided by the text are used to enhance the feature representation of corresponding regions in the image through ITFME, helping the model identify and distinguish atypical, poorly defined, or rare pathological structures in the image. By effectively and deeply fusing renal pathology images and accompanying pathology report text for pixel-level semantic segmentation, this approach delivers more accurate, robust, and refined segmentation results than existing single-modal or simple multimodal approaches.

[0078] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0079] Based on the same inventive concept, an embodiment of the present application also provides a pathological image semantic segmentation device based on multimodal feature fusion and boundary enhancement for implementing the pathological image semantic segmentation method based on multimodal feature fusion and boundary enhancement involved above. The implementation solution provided by the device is similar to the implementation solution described in the above method. Therefore, the specific limitations of one or more embodiments of the pathological image semantic segmentation device based on multimodal feature fusion and boundary enhancement provided below can be found in the above limitations of the pathological image semantic segmentation method based on multimodal feature fusion and boundary enhancement, and will not be repeated here.

[0080] In one embodiment, Figure 4 As shown, a pathological image semantic segmentation device 400 based on multimodal feature fusion and boundary enhancement is provided, comprising: an image text acquisition module 401, a multimodal segmentation network construction module 403, a multimodal segmentation network training module 405 and a semantic segmentation module 407, wherein:

[0081] The image and text acquisition module 401 is used to acquire the original renal pathology image and the associated pathology report text, and perform pre-processing.

[0082] The multimodal segmentation network construction module 403 is used to construct a multimodal segmentation network, which includes an image encoder, a text encoder, an image-text multimodal fusion and enhancement module, a shared decoder, a segmentation prediction head, and an auxiliary boundary detection head.

[0083] The multimodal segmentation network training module 405 is configured to train the multimodal segmentation network using the pre-processed original renal pathology images and the associated pathology report text.

[0084] The semantic segmentation module 407 is used to input the pathological image to be segmented and the associated pathological report text into the trained multimodal segmentation network and output the target semantic segmentation result.

[0085] In one embodiment of the present application, the image text acquisition module is further configured to:

[0086] performing color normalization and size adjustment on the original renal pathology image;

[0087] The pathology report text is segmented, stop words are removed, normalized, sentence segmented, and format converted.

[0088] In one embodiment of the present application, the image encoder adopts a convolutional neural network architecture, and the text encoder adopts a Transformer-based model.

[0089] In one embodiment of the present application, the image-text multimodal fusion and enhancement module includes an image-text attention fusion module and a boundary-aware enhancement module.

[0090] In one embodiment of the present application, the multimodal segmentation network training module is further used to:

[0091] A composite loss function is set, a value of the composite loss function is calculated based on the initial semantic segmentation result, the initial auxiliary boundary prediction result and the true value, and parameters of the multimodal segmentation network are adjusted based on the value of the composite loss function.

[0092] In one embodiment of the present application, the composite loss function includes segmentation loss, rare class false negative penalty term and boundary loss.

[0093] Each module in the above-mentioned pathological image semantic segmentation device based on multimodal feature fusion and boundary enhancement can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above modules can be embedded in or independent of the processor of the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0094] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a pathological image semantic segmentation method based on multimodal feature fusion and boundary enhancement is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0095] Those skilled in the art will understand that Figure 5The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0096] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0097] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0098] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0099] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0100] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0101] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0102] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A pathological image semantic segmentation method based on multimodal feature fusion and boundary enhancement, characterized in that: The method comprises: Obtain original renal pathology images and associated pathology report texts and perform preprocessing; Constructing a multimodal segmentation network, the multimodal segmentation network including an image encoder, a text encoder, an image-text multimodal fusion and enhancement module, a shared decoder, a segmentation prediction head, and an auxiliary boundary detection head; The multimodal segmentation network is trained using the pre-processed original renal pathology image and the associated pathology report text; The pathological image to be segmented and the associated pathological report text are input into the trained multimodal segmentation network, and the target semantic segmentation result is output.

2. The pathological image semantic segmentation method based on multimodal feature fusion and boundary enhancement according to claim 1, characterized in that: The pretreatment includes: performing color normalization and size adjustment on the original renal pathology image; The pathology report text is segmented, stop words are removed, normalized, sentence segmented, and format converted.

3. The pathological image semantic segmentation method based on multimodal feature fusion and boundary enhancement according to claim 1, characterized in that: The image encoder adopts a convolutional neural network architecture, and the text encoder adopts a Transformer-based model.

4. The pathological image semantic segmentation method based on multimodal feature fusion and boundary enhancement according to claim 1, characterized in that: The image-text multimodal fusion and enhancement module includes an image-text attention fusion module and a boundary perception enhancement module.

5. The pathological image semantic segmentation method based on multimodal feature fusion and boundary enhancement according to claim 1, characterized in that: The training of the multimodal segmentation network using the pre-processed original renal pathology image and the associated pathology report text includes: A composite loss function is set, a value of the composite loss function is calculated based on the initial semantic segmentation result, the initial auxiliary boundary prediction result and the true value, and parameters of the multimodal segmentation network are adjusted based on the value of the composite loss function.

6. The pathological image semantic segmentation method based on multimodal feature fusion and boundary enhancement according to claim 5, characterized in that: The composite loss function includes segmentation loss, rare class false negative penalty term and boundary loss.

7. A pathological image semantic segmentation device based on multimodal feature fusion and boundary enhancement, characterized in that: The device comprises: Image and text acquisition module, used to obtain original renal pathology images and associated pathology report texts and perform preprocessing; A multimodal segmentation network construction module is used to construct a multimodal segmentation network, wherein the multimodal segmentation network includes an image encoder, a text encoder, an image-text multimodal fusion and enhancement module, a shared decoder, a segmentation prediction head, and an auxiliary boundary detection head; A multimodal segmentation network training module, configured to train the multimodal segmentation network using the pre-processed original renal pathology images and associated pathology report text; The semantic segmentation module is used to input the pathological image to be segmented and the associated pathological report text into the trained multimodal segmentation network and output the target semantic segmentation result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images