Multi-modal breast cancer image segmentation method in combination with medical report

By combining the multimodal image segmentation method of medical reports, using cross-attention calculation and DICE coefficient evaluation, and fusing image and text features, the problem of low image classification and segmentation accuracy of single image model in breast cancer is solved, significantly improving breast cancer Classification and segmentation performance of images.

CN118941573APending Publication Date: 2024-11-12ZHUHAI BOZHAO MEDICAL TECHNOLOGY CO LTD
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202410990640.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Existing deep learning methods have low classification accuracy in breast cancer image subcategory and poor identification accuracy in target area, especially due to the low accuracy caused by insufficient information in a single image model.

Method used

Using a multimodal breast cancer image segmentation method combined with medical reports, a cross-attention classification model and multi-head information segmentation model is constructed by obtaining pathological images and text information from public medical databases, and a cross-attention calculation and DICE coefficient evaluation are used to integrate images. and text features, improve the feature extraction process of Mask R-CNN model.

Benefits of technology

It effectively improves the classification accuracy and segmentation accuracy of breast cancer images, significantly improves classification accuracy, recall rate and F1 score, and improves recognition speed and accuracy through multi-task loss function, providing a more comprehensive model performance evaluation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941573A_ABST
    Figure CN118941573A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of medical image recognition, and particularly discloses a multi-modal breast cancer image segmentation method in combination with a medical report, which comprises the following steps: acquiring a pathological image of breast cancer and text information corresponding to the pathological image from a public medical database; dividing the pathological image and the text information into a training set and a test set; constructing a cross attention classification model, and inputting the training set data and the test set data into the cross attention classification model for training; classifying a to-be-classified pathological image by using the trained cross attention classification model to obtain a classified pathological image; and constructing a multi-head information segmentation model, and inputting the classified pathological images into the multi-head information segmentation model to complete detailed segmentation of the pathological images. According to the method, classification and accurate segmentation of the breast cancer image are realized, and the problems of low classification precision of the current image model in the breast cancer image sub-classes and poor target area identification accuracy are solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image recognition, and in particular relates to a design of a multimodal breast cancer image segmentation method combined with medical reports. Background Art

[0002] Breast cancer is the most commonly diagnosed cancer in women worldwide and one of the leading causes of death in women. Histopathological evaluation is a key step in the diagnosis and clinical management of breast cancer. With the development of deep learning technology, it has shown potential in improving the efficiency of pathological evaluation, especially in image recognition and classification. Deep learning technology, especially convolutional neural networks (CNNs), has shown great potential in the recognition and classification of medical images.

[0003] However, current deep learning methods mainly focus on enhancing the capabilities of image models, such as enhancing the representation capabilities of image models to improve the ability of convolutional neural networks to learn early cancer features, and strengthening image models' learning of specific types of cancer to help pathologists perform early cancer screening more effectively. There is a problem of low accuracy due to insufficient information in the classification and recognition process of a single image model. Summary of the invention

[0004] The purpose of the present invention is to solve the problem of low classification accuracy of breast cancer image subcategories and poor recognition accuracy of target areas in current image models, and a multimodal breast cancer image segmentation method combined with medical reports is proposed.

[0005] The technical solution of the present invention is: a multimodal breast cancer image segmentation method combined with medical reports, comprising the following steps:

[0006] S1. Obtain pathological images and text information of breast cancer from a public medical database, and match the pathological images and text information to form data pairs;

[0007] S2. Divide the data pairs into training set and test set;

[0008] S3. construct a cross-attention classification model, and input the training set data and the test set data into the cross-attention classification model for training, to obtain a trained cross-attention classification model;

[0009] S4. Evaluate the trained cross-attention classification model using the evaluation index, and input the pathological image to be classified into the qualified cross-attention classification model for classification to obtain the classified pathological image;

[0010] S5. Construct a multi-head information segmentation model, use the DICE coefficient to evaluate the multi-head information segmentation model, and input the classified pathological image into the multi-head information segmentation model that has passed the evaluation to complete the detailed segmentation of the pathological image.

[0011] The beneficial effects of the present invention are:

[0012] 1. Effectively combines medical report data and pathological image data, giving full play to the role of both types of information to achieve classification and accurate segmentation of breast cancer images.

[0013] 2. The original Mask R-CNN model was improved. The feature extraction process of Mask R-CNN was integrated with the information of medical reports through weight reuse, which effectively improved the performance of Mask R-CNN in target area segmentation.

[0014] Preferably, the pathological images in step S1 include four categories: benign, in situ, infiltrated and normal.

[0015] Preferably, the cross-attention classification model in step S3 includes an image model and a text model; the image model and the text model are subjected to feature fusion through cross-attention calculation.

[0016] Preferably, the calculation formula of the cross attention is:

[0017]

[0018] Among them, r ji represents cross attention; Q represents query; K represents key value; i represents each individual element in query Q; j represents each individual element in key value K; N represents the total number of elements in query Q and key value K respectively, and query Q and key value K have the same total number of elements.

[0019] Preferably, the image model adopts the ResNet50 model; and the text model adopts the BERT model.

[0020] The beneficial effects of the above preferred solution are:

[0021] The ResNet50 model effectively solves the gradient vanishing problem in deep network training. The BERT model can capture rich language features and generate high-quality word embeddings by performing deep bidirectional contextual learning on large amounts of text data, significantly improving the performance of multiple language understanding tasks. It not only enhances the learning ability of the network, but also improves the stability and efficiency of training.

[0022] Preferably, the step S4 specifically comprises the following steps:

[0023] S41. Evaluate the trained cross-attention classification model using evaluation indicators;

[0024] S42. Inputting the pathological image to be classified into the image model to obtain a feature vector of the pathological image, and copying the feature vector to obtain a key value and a feature value;

[0025] S43. Inputting the text information corresponding to the pathological image into the text model to obtain a high-dimensional vector, and treating the high-dimensional vector as a query;

[0026] S44. Performing matrix operations on the key value and the query to obtain a weight matrix, and using the weight matrix to extract the eigenvalues ​​to obtain a high-dimensional vector with multimodal information;

[0027] S45. Obtain a classified pathological image based on the high-dimensional vector with multimodal information.

[0028] The beneficial effects of the above preferred solution are:

[0029] By introducing a multimodal algorithm, combining image features and text features through the calculation of cross-attention, and making full use of the advantages of these two models in feature extraction, the problem of low accuracy caused by insufficient information of a single image model in the classification process can be alleviated.

[0030] Preferably, the evaluation index in step S4 includes classification accuracy, recall rate and F1 score;

[0031] The calculation formula of the classification accuracy is:

[0032]

[0033] The calculation formula of the recall rate Recall is:

[0034]

[0035] The calculation formula of the F1 score is:

[0036]

[0037] Among them, TP represents the number of positive samples correctly predicted by the model as positive samples; TN represents the number of negative samples correctly predicted by the model as negative samples; FP represents the number of negative samples incorrectly predicted by the model as positive samples; FN represents the number of positive samples incorrectly predicted by the model as negative samples; Precision represents the proportion of samples predicted as positive that are actually positive.

[0038] Preferably, the multi-head information segmentation model in step S5 is constructed based on the Mask R-CNN model.

[0039] The beneficial effects of the above preferred solution are:

[0040] The multi-head information segmentation model built based on the Mask R-CNN model greatly improves the recognition speed and accuracy, while decoupling proposal generation and recognition, and performing end-to-end training through a multi-task loss function, allowing the model to perform well when dealing with objects of different sizes and shapes.

[0041] Preferably, the calculation formula of the DICE coefficient in step S5 is:

[0042]

[0043] Among them, TP represents the number of positive samples correctly predicted by the model as positive samples; FP represents the number of negative samples incorrectly predicted by the model as positive samples; FN represents the number of positive samples incorrectly predicted by the model as negative samples.

[0044] The beneficial effects of the above preferred solution are:

[0045] The DICE coefficient takes into account not only true positives but also false positives and false negatives, thus providing a more comprehensive evaluation of model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Shown is a flow chart of a multimodal breast cancer image segmentation method combined with medical reports. DETAILED DESCRIPTION

[0047] Now, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be understood that the embodiments shown and described in the accompanying drawings are only exemplary and are intended to explain the principles and spirit of the present invention, rather than to limit the scope of the present invention.

[0048] like Figure 1 As shown, a multimodal breast cancer image segmentation method combined with medical reports includes the following steps:

[0049] S1. Obtain pathological images and text information of breast cancer from a public medical database, and match the pathological images and text information to form a data pair, wherein the text information is a medical report, and the content of the medical report includes the type details of the breast cancer subclass of each pathological image, and the basic personal information and past medical history of the patient to whom the sample belongs;

[0050] S2. Divide the data pairs into training sets and test sets. The training samples of each category contain 200 pathological images and 200 text information. A total of 800 pathological images and text information matching pairs are used for training, and 50 samples are used to test the classification and recognition effects.

[0051] S3. construct a cross-attention classification model, and input the training set data and the test set data into the cross-attention classification model for training, to obtain a trained cross-attention classification model;

[0052] S4. Evaluate the trained cross-attention classification model using the evaluation index, and input the pathological image to be classified into the qualified cross-attention classification model for classification to obtain the classified pathological image;

[0053] S5. Construct a multi-head information segmentation model, use the DICE coefficient to evaluate the multi-head information segmentation model, and input the classified pathological image into the multi-head information segmentation model that has passed the evaluation to complete the detailed segmentation of the pathological image.

[0054] In this embodiment, the pathological images in step S1 include four categories: benign, in situ, infiltrated, and normal.

[0055] In this embodiment, the cross-attention classification model in step S3 includes an image model and a text model; the image model and the text model are subjected to feature fusion through cross-attention calculation.

[0056] In this embodiment, the calculation formula of the cross attention is:

[0057]

[0058] Among them, r ji represents cross attention; Q represents query, which is a high-dimensional vector output by the text model; K represents key value, which is a feature vector output by the image model; i represents each individual element in query Q; j represents each individual element in key value K; N represents the total number of elements in query Q and key value K respectively, and query Q and key value K have the same total number of elements.

[0059] In this embodiment, the image model adopts the ResNet50 model; the text model adopts the BERT model; ResNet50 is a deep learning architecture that constructs a network by stacking 50 layers of residual blocks, each of which contains two convolutional layers and directly adds the input to the output through skip connections.

[0060] In this embodiment, step S4 specifically includes the following steps:

[0061] S41. Evaluate the trained cross-attention classification model using evaluation indicators;

[0062] S42. Inputting the pathological image to be classified into the image model to obtain a feature vector of the pathological image, and copying the feature vector to obtain a key value and a feature value;

[0063] S43. Inputting the text information corresponding to the pathological image into the text model to obtain a high-dimensional vector, and treating the high-dimensional vector as a query;

[0064] S44. Performing matrix operations on the key value and the query to obtain a weight matrix, and using the weight matrix to extract the eigenvalues ​​to obtain a high-dimensional vector with multimodal information;

[0065] S45. Obtain a classified pathological image based on the high-dimensional vector with multimodal information.

[0066] In this embodiment, the evaluation indicators in step S4 include classification accuracy, recall rate and F1 score;

[0067] The calculation formula of the classification accuracy is:

[0068]

[0069] The calculation formula of the recall rate Recall is:

[0070]

[0071] The calculation formula of the F1 score is:

[0072]

[0073] Among them, TP represents the number of positive samples correctly predicted by the model as positive samples; TN represents the number of negative samples correctly predicted by the model as negative samples; FP represents the number of negative samples incorrectly predicted by the model as positive samples; FN represents the number of positive samples incorrectly predicted by the model as negative samples; Precision represents the proportion of samples predicted as positive that are actually positive.

[0074] In this embodiment, the multi-head information segmentation model described in step S5 is constructed based on the Mask R-CNN model. Mask R-CNN is an advanced target detection and instance segmentation framework that expands on Faster R-CNN and adds a branch for predicting the segmentation mask of the object. Mask R-CNN combines deep convolutional neural networks and region proposal networks to not only propose target regions and accurately regress bounding boxes, but also generate pixel-level masks for each object. This method uses shared convolutional features. First, the region proposal network quickly generates candidate target regions, and then simultaneously classifies these regions, fine-tunes bounding boxes, and predicts pixel-level segmentation masks, thereby significantly improving the accuracy and practicality of recognition.

[0075] In this embodiment, the calculation formula of the DICE coefficient in step S5 is:

[0076]

[0077] Among them, TP represents the number of positive samples correctly predicted by the model; FP represents the number of negative samples incorrectly predicted by the model as positive samples; FN represents the number of positive samples incorrectly predicted by the model as negative samples. The value range of the DICE coefficient is between 0 and 1. The higher the value, the closer the model's prediction result is to the actual situation, and the better the segmentation effect.

[0078] In this embodiment, the performance of the cross-attention classification model in all categories has been significantly improved. In the benign class, the classification accuracy increased from 79% of the baseline model to 85%, the recall rate increased from 74% to 80%, and the F1 score increased from 0.79 to 0.82. In the in situ class, the classification accuracy increased from 78% to 80%, the recall rate increased from 70% to 75%, and the F1 score increased from 0.71 to 0.77. The performance of the infiltrated and normal classes also improved significantly.

[0079] In this embodiment, the DICE coefficients of the pathological image region segmentation of the four categories are between 0.78 and 0.93. Generally speaking, the segmentation effect of the multi-head information segmentation model in each category is relatively ideal.

[0080] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A multimodal breast cancer image segmentation method combined with medical reports, characterized in that: The following steps are involved: S1. Obtain pathological images and text information of breast cancer from a public medical database, and match the pathological images and text information to form data pairs; S2. Divide the data pairs into training set and test set; S3. construct a cross-attention classification model, and input the training set data and the test set data into the cross-attention classification model for training, to obtain a trained cross-attention classification model; S4. Evaluate the trained cross-attention classification model using the evaluation index, and input the pathological image to be classified into the qualified cross-attention classification model for classification to obtain the classified pathological image; S5. Construct a multi-head information segmentation model, use the DICE coefficient to evaluate the multi-head information segmentation model, and input the classified pathological image into the multi-head information segmentation model that has passed the evaluation to complete the detailed segmentation of the pathological image.

2. The multimodal breast cancer image segmentation method combined with medical reports according to claim 1, characterized in that: The pathological images in step S1 include four categories: benign, in situ, infiltrated and normal.

3. The multimodal breast cancer image segmentation method combined with medical reports according to claim 1, characterized in that: The cross-attention classification model in step S3 includes an image model and a text model; the image model and the text model are subjected to feature fusion through cross-attention calculation.

4. The multimodal breast cancer image segmentation method combined with medical reports according to claim 3, characterized in that: The calculation formula of the cross attention is: Among them, r ji represents cross attention; Q represents query; K represents key value; i represents each individual element in query Q; j represents each individual element in key value K; N represents the total number of elements in query Q and key value K respectively, and query Q and key value K have the same total number of elements.

5. The multimodal breast cancer image segmentation method combined with medical reports according to claim 3, characterized in that: The image model adopts the ResNet50 model; the text model adopts the BERT model.

6. The multimodal breast cancer image segmentation method combined with medical reports according to claim 3, characterized in that: The step S4 specifically comprises the following steps: S41. Evaluate the trained cross-attention classification model using evaluation indicators; S42. Inputting the pathological image to be classified into the image model to obtain a feature vector of the pathological image, and copying the feature vector to obtain a key value and a feature value; S43. Inputting the text information corresponding to the pathological image into the text model to obtain a high-dimensional vector, and treating the high-dimensional vector as a query; S44. Performing matrix operations on the key value and the query to obtain a weight matrix, and using the weight matrix to extract the eigenvalues ​​to obtain a high-dimensional vector with multimodal information; S45. Obtain a classified pathological image based on the high-dimensional vector with multimodal information.

7. The multimodal breast cancer image segmentation method combined with medical reports according to claim 1, characterized in that: The evaluation indicators in step S4 include classification accuracy, recall rate and F1 score; The calculation formula of the classification accuracy is: The calculation formula of the recall rate Recall is: The calculation formula of the F1 score is: Among them, TP represents the number of positive samples correctly predicted by the model as positive samples; TN represents the number of negative samples correctly predicted by the model as negative samples; FP represents the number of negative samples incorrectly predicted by the model as positive samples; FN represents the number of positive samples incorrectly predicted by the model as negative samples; Precision represents the proportion of samples predicted as positive that are actually positive.

8. The multimodal breast cancer image segmentation method combined with medical reports according to claim 1, characterized in that: The multi-head information segmentation model described in step S5 is built based on the Mask R-CNN model.

9. The multimodal breast cancer image segmentation method combined with medical reports according to claim 1, characterized in that: The calculation formula of the DICE coefficient in step S5 is: Among them, TP represents the number of positive samples correctly predicted by the model as positive samples; FP represents the number of negative samples incorrectly predicted by the model as positive samples; FN represents the number of positive samples incorrectly predicted by the model as negative samples.

Citation Information

Cited By

  • Endoscopic image description report generation method and device and medium

    CN119724464A

  • Cross-modal breast image segmentation and classification method based on X-ray and clinical medicine text

    CN121259327A

  • X-ray and clinical medical text based cross-modal breast image segmentation and classification method

    CN121259327B