Breast cancer classification method for WSI image and electronic equipment
Through deep learning technology, multimodal feature extraction and fusion of WSI images has been solved, and the problems of low efficiency and insufficient accuracy in breast cancer WSI images diagnosis have been achieved, achieving more efficient breast cancer classification.
Patent Information
- Application Number
- CN202411128995.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-11-08
AI Technical Summary
The prior art in breast cancer WSI image diagnosis is that the localization of small-area tumor areas in large-area images takes a long time and is inefficient, and the traditional methods rely on manual feature design, which affects the classification accuracy.
Deep learning technology is used to analyze WSI images, and the image and text features are extracted to achieve accurate classification of breast cancer through multimodal fusion of image encoding branches, text encoding branches and mutual feature classification branches.
It improves the accuracy and efficiency of breast cancer classification, and by segmenting and splicing large-sized WSI images, the calculation complexity is reduced and the diagnostic efficiency is improved.
Smart Images

Figure CN118918386A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical fields of artificial intelligence, deep learning, and medical image processing, and in particular to a breast cancer classification method and electronic device for WSI images. Background Art
[0002] Breast cancer has become one of the most common malignant tumors in women. The diagnosis of breast cancer mainly relies on breast palpation, breast ultrasound, breast mammography and pathological examination. Among them, pathological examination, as the gold standard for diagnosis, mainly relies on whole-field digital slice (WSI) images. WSI images are large in size. In comparison, the tumor area of breast cancer accounts for a very small proportion. It is necessary to locate the small tumor area from the large WSI image. Therefore, WSI image diagnosis of breast cancer is significantly time-consuming and inefficient.
[0003] Traditional WSI breast cancer tumor automatic classification methods mainly rely on manual features, including the texture, color, directional gradients, superpixel color histograms, etc. of WSI images. The design of manual features directly affects the classification accuracy. Summary of the invention
[0004] In view of the above problems, the present disclosure provides a breast cancer classification method and electronic device for WSI images, which analyze WSI images through deep learning technology to achieve accurate classification of breast cancer.
[0005] According to a first aspect of the present disclosure, a breast cancer classification method for a WSI image is provided. The method is executed by an electronic device, and the method includes: performing regional segmentation on a preprocessed original WSI image to obtain a WSI image, wherein the WSI image includes a plurality of WSI image blocks; inputting the WSI image into an image encoding branch of a classification model, and outputting image features of the WSI image; inputting target metadata into a text encoding branch of the classification model, and outputting text features, wherein the target metadata corresponds to the WSI image; inputting image features and text features into a mutual feature classification branch of the classification model, and outputting a first classification result, wherein the first classification result includes a breast cancer classification result of each WSI image block; and splicing the breast cancer classification results of each WSI image block according to the segmentation position of the WSI image block in the WSI image, to obtain a second classification result.
[0006] According to an embodiment of the present disclosure, an image coding branch includes a prompt word generation module and N stacked adaptive feature weighting modules, where N is a positive integer greater than or equal to 2; a WSI image is input into the image coding branch of a classification model, and image features of the WSI image are output, including: globally pooling the first initial image features output by the Nth adaptive feature weighting module to determine the initial pooling features; inputting the initial pooling features into a prompt word generation module to output prompt word features; fusing the first initial image features and the prompt word features to obtain the second initial image features, and using the second initial image features as the image features output by the image coding branch; wherein the input of the first adaptive feature weighting module is the WSI image, and the input of the remaining N-1 adaptive feature weighting modules is the second initial image features determined based on the first initial image features and the prompt word features output by the previous adaptive feature weighting module.
[0007] According to an embodiment of the present disclosure, the adaptive feature weighting module includes a self-attention branch and an adaptive residual branch; the method also includes: the self-attention branch processes the input features of the adaptive feature weighting module through stacked self-attention modules, a first convolutional layer and a first multi-layer perceptron to obtain attention features; the adaptive residual branch processes the input features through two sets of stacked second convolutional layers to obtain pixel-level weights; and the attention features are screened using the pixel-level weights to obtain the first initial image features.
[0008] According to an embodiment of the present disclosure, a mutual feature classification branch includes a first cross-attention module, a second cross-attention module, a second multi-layer perceptron and an activation layer; image features and text features are input into the mutual feature classification branch of the classification model, and a first classification result is output, including: using image features as the first data source of the first cross-attention module, and using text features as the second data source of the first cross-attention module, and determining a first fusion feature based on a cross-attention mechanism; using text features as the first data source of the second cross-attention module, and using image features as the second data source of the second cross-attention module, and determining a second fusion feature based on a cross-attention mechanism; inputting the first fusion feature and the second fusion feature into the second multi-layer perceptron, and outputting enhanced classification features; inputting the classification features into the activation layer, and outputting the first classification result.
[0009] According to an embodiment of the present disclosure, a text encoding branch includes M stacked metadata feature extraction modules, where M is a positive integer greater than or equal to 2; inputting target metadata into the text encoding branch of a classification model, and outputting text features, including: mining classification features related to breast cancer from the target metadata through the M stacked metadata feature extraction modules; wherein the target metadata includes P attributes, and the attributes are used to describe breast cancer features of an object to which the WSI image belongs.
[0010] According to an embodiment of the present disclosure, the metadata feature extraction module includes a stacked graph convolution layer and a linear layer, the input of the graph convolution layer is the input of the metadata feature extraction module, and the output of the linear layer is residually connected to the input of the graph convolution layer to obtain the output of the metadata feature extraction module; wherein the graph convolution layer includes P nodes corresponding to P attributes, and the node relationship between the P nodes is obtained after training.
[0011] According to an embodiment of the present disclosure, the loss function of the classification model includes cross entropy loss and similarity loss, wherein the similarity loss is used to characterize the semantic similarity between image features and text features.
[0012] According to an embodiment of the present disclosure, the similarity loss is calculated according to the following method: the image features and the text features are respectively input into the activation layer to obtain the first feature of the image modality and the second feature of the text modality respectively; the semantic similarity between the first feature and the second feature is used as the similarity loss.
[0013] According to an embodiment of the present disclosure, performing regional segmentation on the preprocessed original WSI image to obtain the WSI image includes: acquiring the original WSI image of the pathological section by scanning an instrument; unifying the staining distribution of the original WSI image by using a preset staining algorithm to obtain a first WSI image; performing regional segmentation on the first WSI image to obtain a second WSI image, wherein the second WSI image includes a plurality of WSI image blocks of preset sizes; and deleting an image block having a preset image feature from the plurality of image blocks to obtain the WSI image.
[0014] According to a second aspect of the present disclosure, a breast cancer classification device for WSI images is provided in an electronic device, and the device includes: a preprocessing module, which is used to perform region segmentation on the preprocessed original WSI image to obtain a WSI image, wherein the WSI image includes multiple WSI image blocks; an image processing module, which is used to input the WSI image into an image encoding branch of a classification model, and output image features of the WSI image; a text processing module, which is used to input target metadata into a text encoding branch of the classification model, and output text features, wherein the target metadata corresponds to the WSI image; a fusion module, which is used to input image features and text features into a mutual feature classification branch of the classification model, and output a first classification result, wherein the first classification result includes a breast cancer classification result of each WSI image block; and a splicing module, which is used to splice the breast cancer classification results of each WSI image block according to the segmentation position of the WSI image block in the WSI image, to obtain a second classification result.
[0015] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein when the one or more processors execute the one or more computer programs, the steps of the breast cancer classification method for WSI images are implemented.
[0016] A fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instruction stored thereon, which implements the steps of the above-mentioned breast cancer classification method for WSI images when the computer program or instruction is executed by a processor.
[0017] A fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implements the steps of the above-mentioned breast cancer classification method for WSI images when the computer program or instructions are executed by a processor.
[0018] The embodiments of the present disclosure extract features of image modality and text modality through a classification model and perform multimodal fusion, thereby being able to use richer multimodal data for breast cancer classification, thereby improving classification accuracy and classification efficiency. In addition, for WSI images with larger sizes, region segmentation is performed before classification using a classification model, and after classification, the first classification results of multiple image blocks are spliced together to achieve breast cancer classification of large-size WSI images, thereby improving classification efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:
[0020] Figure 1 The application scenario of the breast cancer classification method for WSI images according to an embodiment of the present disclosure is schematically shown;
[0021] Figure 2 A flowchart of a method for classifying breast cancer for WSI images according to an embodiment of the present disclosure is schematically shown;
[0022] Figure 3 The structural diagram of the classification model according to the embodiment of the present disclosure is schematically shown;
[0023] Figure 4 A schematic diagram of a process of training a classification model and using the trained classification model to classify breast cancer according to a specific embodiment of the present disclosure is shown;
[0024] Figure 5 A block diagram of an electronic device suitable for a breast cancer classification method for a WSI image according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION
[0025] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.
[0026] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.
[0027] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.
[0028] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).
[0029] In order to address the technical problem that manual features directly affect the classification accuracy, the embodiments of the present invention use a deep learning method to learn the feature representation of WSI images through a neural network, so that the deep learning-based model can extract features containing richer semantic information, thereby significantly improving the classification effect of breast cancer areas in WSI images.
[0030] Specifically, an embodiment of the present disclosure provides a breast cancer classification method for a WSI image, which is executed by an electronic device, and includes: performing region segmentation on a preprocessed original WSI image to obtain a WSI image, wherein the WSI image includes multiple WSI image blocks; inputting the WSI image into an image encoding branch of a classification model, and outputting image features of the WSI image; inputting target metadata into a text encoding branch of a classification model, and outputting text features, wherein the target metadata corresponds to the WSI image; inputting image features and text features into a mutual feature classification branch of a classification model, and outputting a first classification result, wherein the first classification result includes a breast cancer classification result of each WSI image block; and splicing the breast cancer classification results of each WSI image block according to the segmentation position of the WSI image block in the WSI image to obtain a second classification result.
[0031] It should be noted that the breast cancer classification method, device, and electronic device for WSI images involved in the embodiments of the present disclosure are not intended for the purpose of diagnosing or treating cancer, but only for processing WSI images, target metadata, and other image and text data. The present disclosure is only used for non-medical treatment purposes, including but not limited to medical research and medical auxiliary diagnosis, and is strictly prohibited from being used for other medical treatment related purposes. The use of the present disclosure is strictly restricted, and any attempt to violate this restriction will be deemed an infringement.
[0032] Figure 1 The application scenario of the breast cancer classification method for WSI images according to an embodiment of the present disclosure is schematically shown.
[0033] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.
[0034] The user can use at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, etc.
[0035] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.
[0036] It should be noted that the breast cancer classification method for WSI images provided in the embodiment of the present disclosure is executed by an electronic device, and the electronic device here can be one of the first terminal device 101, the second terminal device 102, and the third terminal device 103, or the server 105. Accordingly, the breast cancer classification device for WSI images provided in the embodiment of the present disclosure can generally be set in the first terminal device 101, the second terminal device 102, the third terminal device 103 or the server 105.
[0037] Figure 2 The flowchart of the breast cancer classification method for WSI images according to an embodiment of the present disclosure is schematically shown.
[0038] like Figure 2 As shown, the method 200 includes operations S210 to S250.
[0039] In operation S210, the pre-processed original WSI image is segmented into regions to obtain a WSI image, wherein the WSI image includes a plurality of WSI image blocks.
[0040] In operation S220, the WSI image is input into an image encoding branch of a classification model, and image features of the WSI image are output.
[0041] In operation S230 , the target metadata is input into a text encoding branch of a classification model, and a text feature is output, wherein the target metadata corresponds to the WSI image.
[0042] In operation S240, the image features and the text features are input into a mutual feature classification branch of the classification model, and a first classification result is output, wherein the first classification result includes a breast cancer classification result of each WSI image block.
[0043] In operation S250, the breast cancer classification result of each WSI image block is spliced according to the segmentation position of the WSI image block in the WSI image to obtain a second classification result.
[0044] According to an embodiment of the present disclosure, the original WSI image refers to a scanned image of a pathological slice of a target object. The preprocessing of the original WSI image includes a variety of image processing techniques to obtain a clearer, denoised, and preset SVS format WSI image.
[0045] Since the size of the original WSI image is relatively large, before using the classification model to perform the classification task, the preprocessed original WSI image is segmented into regions to obtain a plurality of image blocks of smaller sizes, and the plurality of image blocks constitute the WSI image in operation S210.
[0046] The classification model of the embodiment of the present disclosure is constructed based on deep learning technology. For example, the classification model can be a Tissue Feature and Clinical Data based Breast Cancer Classification (TFCD-BCC) model.
[0047] The classification model includes an image encoding branch, a text encoding branch, and a mutual feature classification branch. Both the image encoding branch and the text encoding branch are built based on deep learning technology, for example, based on the Transformer encoder framework; the mutual feature classification branch can be understood as a multimodal classifier.
[0048] According to an embodiment of the present disclosure, the target metadata corresponds to the WSI image, and the target metadata includes multiple texts for describing various information related to breast cancer, such as clinical symptoms, age, gender, disease history, etc. of the object to which the WSI image belongs.
[0049] The target metadata is the metadata after numerical standardization. For example, the Z-score method is used for numerical standardization, and the clinical metadata corresponding to the WSI image is retained, such as gender, age, disease history, and clinical symptoms. The mean and standard deviation of the values of each column are calculated separately, and the value range of each column is scaled to between -1 and 1 to obtain the target metadata.
[0050] The input of the classification model includes WSI images in the image modality and target metadata in the text modality. The image modality and text modality data are processed respectively by the image encoding branch and the text encoding branch. Then, the features of the image modality and text modality are multimodally fused through the mutual feature classification branch, and then breast cancer classification is performed using richer multimodal data.
[0051] According to an embodiment of the present disclosure, since the WSI image includes multiple image blocks, the classification task performed by the classification model can classify breast cancer in the area corresponding to each image block. At this time, the output of the classification model is a first classification result, and the first classification result includes the breast cancer classification result of each image block.
[0052] The embodiments of the present disclosure extract features of image modality and text modality through a classification model and perform multimodal fusion, thereby being able to use richer multimodal data for breast cancer classification, thereby improving classification accuracy and classification efficiency. In addition, for WSI images with larger sizes, region segmentation is performed before classification using a classification model, and after classification, the first classification results of multiple image blocks are spliced together to achieve breast cancer classification of large-size WSI images, thereby improving classification efficiency.
[0053] According to an embodiment of the present disclosure, for operation S210, the preprocessed original WSI image is regionally segmented to obtain a WSI image, including: acquiring the original WSI image of the pathological section by scanning an instrument; unifying the staining distribution of the original WSI image by using a preset staining algorithm to obtain a first WSI image; performing regional segmentation on the first WSI image to obtain a second WSI image, wherein the second WSI image includes multiple WSI image blocks of preset sizes; and deleting image blocks with preset image features from the multiple image blocks to obtain a WSI image.
[0054] In the embodiment of the present disclosure, the original WSI is obtained by scanning the pathological slice of the target object using a digital scanning instrument. Then, the scanned original WSI image is converted into a data format, and the format is uniformly converted into an original WSI image in SVS format.
[0055] The preset staining algorithm is used to unify the color distribution of the original WSI image to avoid the difference in hematoxylin-eosin staining operations affecting the distribution of breast cancer cells in the WSI image. The preset staining algorithm can be a Vahadane algorithm.
[0056] After obtaining the first WSI image with uniform color distribution, the large-size first WSI image is segmented into regions to obtain a WSI image including multiple WSI image blocks. For example, the first WSI image after color normalization is segmented into a grid-like manner without overlap, so as to divide the first WSI image with more than 1 billion pixels into image blocks of 224×224 size to reduce computational complexity.
[0057] The divided image blocks are subjected to tissue region screening, and image blocks with a background close to white and containing no tissue region are deleted to obtain the WSI image in operation S220, thereby avoiding unnecessary calculation.
[0058] In addition, the target metadata of the input text encoding branch is also preprocessed before being input into the text encoding branch. The mean and standard deviation of the values of each column of the target metadata are calculated separately, and the value range of each column of data is scaled to between -1 and 1 to obtain the target metadata. The standardization process is as follows:
[0059] (1)
[0060] (2)
[0061] (3)
[0062] Among them, μ and σ are the mean and standard deviation of a column respectively, N is the number of data in the column (i.e., sample size), is the original value of the column data. The value of this column after normalization.
[0063] According to an embodiment of the present disclosure, the image coding branch includes a prompt word generation module and N stacked adaptive feature weighting modules, where N is a positive integer greater than or equal to 2.
[0064] According to an embodiment of the present disclosure, for operation S220, the WSI image is input into the image encoding branch of the classification model, and the image features of the WSI image are output, including: globally pooling the first initial image features output by the Nth adaptive feature weighting module to determine the initial pooling features; inputting the initial pooling features into the prompt word generation module to output the prompt word features; fusing the first initial image features and the prompt word features to obtain the second initial image features, and using the second initial image features as the image features output by the image encoding branch; wherein the input of the first adaptive feature weighting module is the WSI image, and the input of the remaining N-1 adaptive feature weighting modules is the second initial image features determined based on the first initial image features and the prompt word features output by the previous adaptive feature weighting module.
[0065] In the disclosed embodiment, the N adaptive feature weighting modules of the image coding branch have the same structure, but the input of each adaptive feature weighting module is different. The input of the first adaptive feature weighting module is the input of the image coding branch, that is, the WSI image. The input of the second to Nth adaptive weighting modules not only includes the output of the previous adaptive weighting module, but also reintegrates the prompt word features output by the prompt word generation module to enrich the expressiveness of the features while improving the adaptability of the task.
[0066] For example, each adaptive feature weighting module uses 512 as the feature dimension, and obtains a feature sequence of length 324 by dividing the overlapping image blocks. The input of the prompt word generation module is a feature sequence of length 324, and the output is a feature sequence of length 330, that is, the prompt word generation module provides 6 additional feature blocks after each adaptive feature weighting module, thereby obtaining a feature sequence of length 330. ,in is the length of the feature sequence after integrating the cue word features, which is 330, and D is the feature dimension, which is 512. The image encoding branch extracts WSI image features through self-attention and adaptive weighting, and enriches the feature expression ability and improves the adaptability of the task through additional learnable feature blocks.
[0067] According to the embodiments of the present disclosure, the prompt word generation module is mainly composed of multiple layers of linear layers. The randomly initialized parameters with a feature dimension of 6 become initialized features with a feature dimension of 6 after passing through stacked linear layers, layer normalization layers, and ReLU activation layers. The first initial image feature output by the previous adaptive feature weighting module is compressed into a global mean through the global pooling layer, that is, the initial pooling feature; the mean is added to the above-mentioned initialized feature with a feature dimension of 6 to generate a prompt word feature with a length of 330. The prompt word feature output by the prompt word generation module will be coordinated with the first initial image feature output by the adaptive feature weighting module to enrich the image features of the current adaptive feature weighting module. The process of the prompt word generation module outputting the prompt word feature is as follows:
[0068] (4)
[0069] in is the ReLU activation layer, The corresponding layer is the normalization layer, Corresponding to the linear layer, Corresponding to the global pooling layer, is the first initial image feature output from the previous adaptive feature weighting module, is a randomly initialized learnable parameter, and P is the generated cue word feature. Since the cue word generation module is globally shared, the image encoding branch only adds a small amount of parameter burden.
[0070] According to an embodiment of the present disclosure, the adaptive feature weighting module includes a self-attention branch and an adaptive residual branch; the self-attention branch processes the input features of the adaptive feature weighting module through stacked self-attention modules, a first convolutional layer, and a first multilayer perceptron to obtain attention features; the adaptive residual branch processes the input features through two sets of stacked second convolutional layers to obtain pixel-level weights; the attention features are screened using the pixel-level weights to obtain the first initial image features.
[0071] According to an embodiment of the present disclosure, the self-attention module may be a conventional self-attention weighting process, the process being:
[0072] (5)
[0073] in, , , are the linear layers mapped to features Q, K, and V respectively, and d is the feature dimension. I is the input of the self-attention module, that is, the input of each adaptive feature weighting module.
[0074] The first convolutional layer is a 1×1 convolutional layer, and the first multilayer perceptron is a conventional MLP multilayer perceptron. The 1×1 convolution operation of the first convolutional layer is to replace the position encoding, so that the classification model can learn the relative position information of each feature block through the convolution operation. The attention weighted features after the 1×1 convolution processing are handed over to the first multilayer perceptron for feature enhancement, and finally the output of the self-attention branch is obtained.
[0075] The adaptive residual branch includes a 3×3 convolutional layer and a weight generation module. The disclosed embodiment replaces the commonly used residual connection with an adaptive residual branch, and provides a set of dynamic pixel-level weights through two groups of 3×3 convolutional layers, through which the enhanced features are screened to highlight relatively important feature blocks and reduce the impact of potential noise.
[0076] Specifically, the processing process of the adaptive feature weighting module is shown in the following formula:
[0077] (6)
[0078] in, is a 1×1 convolutional layer, are two sets of consecutive 3×3 convolutional layers, is the input of the current adaptive feature weighting module, is the output of the current adaptive feature weighting module.
[0079] According to an embodiment of the present disclosure, the text encoding branch includes M stacked metadata feature extraction modules, where M is a positive integer greater than or equal to 2. For operation S230, the target metadata is input into the text encoding branch of the classification model, and the text features are output, including: mining classification features related to breast cancer from the target metadata through the M stacked metadata feature extraction modules; wherein the target metadata includes P attributes, and the attributes are used to describe the breast cancer features of the object to which the WSI image belongs.
[0080] The text encoding branch is composed of 5 metadata feature extraction modules stacked together. The feature dimension of each module is also 512, and the number of nodes is the number of metadata attributes, such as 4.
[0081] The text encoding branch mines and extracts the classification clues contained in the target metadata through the metadata feature extraction module. , is the feature length of the target metadata, 4, and D is the feature dimension, 512. The text encoding branch can assist the WSI image encoding branch in completing the breast cancer classification task by extracting text classification clues from the target metadata.
[0082] According to an embodiment of the present disclosure, the metadata feature extraction module includes a stacked graph convolution layer and a linear layer, the input of the graph convolution layer is the input of the metadata feature extraction module, and the output of the linear layer is residually connected to the input of the graph convolution layer to obtain the output of the metadata feature extraction module; wherein the graph convolution layer includes P nodes corresponding to P attributes, and the node relationship between the P nodes is obtained after training.
[0083] Taking the attributes of gender, age, disease history, and clinical symptoms as an example, the number of attributes is 4, that is, the number of nodes is 4, and the relationship between nodes is a graph structure data with randomly initialized learnable parameters. The graph convolution layer learns and aggregates the relationship between nodes. After aggregation, the node features are adjusted through the linear layer, and then through the residual connection, the output features of the metadata feature extraction module are finally obtained.
[0084] (7)
[0085] in, is the input of the current metadata feature extraction module, GCN(·) is the graph convolution layer, It is the output of the current metadata feature extraction module.
[0086] According to an embodiment of the present disclosure, the mutual feature classification branch includes a first cross-attention module, a second cross-attention module, a second multi-layer perceptron and an activation layer. For operation S240, the image feature and the text feature are input into the mutual feature classification branch of the classification model, and the first classification result is output, including: using the image feature as the first data source of the first cross-attention module and the text feature as the second data source of the first cross-attention module, and determining the first fusion feature based on the cross-attention mechanism; using the text feature as the first data source of the second cross-attention module and the image feature as the second data source of the second cross-attention module, and determining the second fusion feature based on the cross-attention mechanism; inputting the first fusion feature and the second fusion feature into the second multi-layer perceptron, and outputting the enhanced classification feature; inputting the classification feature into the activation layer, and outputting the first classification result.
[0087] According to an embodiment of the present disclosure, the structures of the first cross-attention module and the second cross-attention module are the same, but the optimal parameters obtained through training are different. Both the first cross-attention module and the second cross-attention module are constructed based on the standard cross-attention structure.
[0088] The cross-attention module includes two different data sources, namely the first data source and the second data source. The first data source is used to map Q, and the second data source is used to map K and V. The first cross-attention module and the second cross-attention module both use the image features output by the image encoding branch and the text features output by the text encoding branch as two data sources, but the first cross-attention module is used to filter text features using image features, that is, the feature Q in the attention calculation is obtained by mapping the image features, and the K and V in the attention calculation are obtained by mapping the text features, and then the result of the cross-attention (the first fusion feature) is obtained by following the process of self-attention calculation and feature weighting.
[0089] The second cross-attention module is used to filter image features using text features. The feature Q in the attention calculation is obtained from the text feature map, and the K and V in the attention calculation are obtained using the image feature map. Then, the result of the cross-attention (the second fusion feature) is obtained by following the self-attention calculation and feature weighting process.
[0090] The processing process of the first cross-attention module is:
[0091] (8)
[0092] The processing process of the second cross-attention module is:
[0093] (9)
[0094] in, , , They are linear layers mapped to features Q, K, and V respectively, d is the feature dimension, T is the text feature, and I is the image feature.
[0095] The image features based on the WSI image and the text features based on the target metadata are weighted by cross-attention to become fused mutual features; the final classification features are further obtained through the second multi-layer perceptron, and the classification probability of whether it is breast cancer is obtained after the SoftMax activation layer, which is the first classification result. The breast cancer classification results of each image block include breast cancer tumor area and healthy area.
[0096] In summary, in the breast cancer classification task for WSI images, the collected WSI images and the corresponding target metadata are preprocessed according to the five-step process of the preprocessing step, including format conversion, staining normalization, grid segmentation, tissue area screening, and numerical normalization; then the preprocessed WSI images and the corresponding target metadata are input into the classification model loaded with the optimal weights, such as the TFCD-BCC model, so as to obtain the probability that the WSI image contains a breast cancer tumor area. The classification results of all WSI image blocks are reassembled to the original billion-pixel size according to the position before grid division, and the distribution probability map of the breast cancer area in the WSI image can be obtained. The breast cancer distribution probability map can intuitively show the area where the breast cancer tumor is located, which can be used to assist breast cancer screening and improve the efficiency of breast cancer diagnosis in WSI images.
[0097] Figure 3 The structural diagram of the classification model according to the embodiment of the present disclosure is schematically shown.
[0098] like Figure 3 As shown, the image encoding branch includes 16 adaptive feature weighting modules with the same structure and 1 prompt word generation module. The first initial image feature output by each adaptive feature weighting module is used as the input of the prompt word generation module so that the prompt word generation module can adaptively generate optimized prompt word features. Image features are generated based on the first initial image features and prompt word features output by the last adaptive feature weighting module.
[0099] The text encoding branch includes five metadata feature extraction modules. The output of the last metadata feature extraction module is the text feature.
[0100] In the mutual feature classification branch, the image feature and the text feature are respectively used as the first data source and the second data source of the first cross-attention module to obtain the first fusion feature; the image feature and the text feature are respectively used as the second data source and the first data source of the second cross-attention module to obtain the second fusion feature.
[0101] The first fusion feature and the second fusion feature are input into a multilayer perceptron, that is, a second multilayer perceptron, and enhanced classification features are output; finally, the classification features are input into a softmax activation layer to obtain a first classification result.
[0102] Figure 4 The flowchart of training a classification model and using the trained classification model to classify breast cancer according to a specific embodiment of the present disclosure is schematically shown.
[0103] like Figure 4As shown, taking the TFCD-BCC model as an example, the scanned WSI image is preprocessed and divided into a training set and a validation set, and the optimal weight of the classification model is obtained through pre-training + training.
[0104] Specifically, self-supervised pre-training is used to obtain the initial weights of the image encoding branch, and then the MAE model is used to pre-train the TFCD-BCC model. Since the MAE model relies on the image reconstruction task to optimize the model parameters during self-supervised training, only WSI image data is needed, and no corresponding breast cancer category labels are required. In order to complete the image reconstruction task of MAE, the model temporarily introduces an additional 8-layer self-attention model layer with a feature dimension of 512 as a decoder.
[0105] The features extracted by the image encoding branch are input into the decoder composed of 8 layers of self-attention model layers, and the reconstructed WSI image is output; then the MSE loss function is used to calculate the distance between the reconstructed image and the WSI image of the input image encoding branch to constrain the model training. According to the common strategy of MAE pre-training, the input image is subjected to rich data augmentation operations such as random cropping, random flipping, random spatial transformation, and random color transformation to increase the generalization ability of the pre-training parameters. The model parameters of the WSI image encoding branch obtained by MAE pre-training can be used as initialization parameters in the supervised breast cancer classification task training to improve the convergence speed of the model and the final classification performance of the TFCD-BCC model.
[0106] Then, the TFCD-BCC model is trained in a supervised manner to obtain the optimal weight of the TFCD-BCC model, and the TFCD-BCC model with the optimal weight is used as the trained classification model to generate the breast cancer classification results of the WSI image.
[0107] During the training process, the loss function of the classification model includes not only the cross entropy loss between the input WSI image and the output breast cancer classification result, but also the similarity loss. The similarity loss is used to characterize the semantic similarity between the image features output by the image encoding branch and the text features output by the text encoding branch.
[0108] Cross entropy is used as the core constraint of the classification task to ensure the basic classification effect of the classification model, and similarity loss is used as an auxiliary loss to improve the matching degree between WSI image features and metadata features. The loss function is:
[0109] (10)
[0110] in, is the loss function, and Represent the cross entropy loss and similarity loss respectively, and They are the weight coefficients when the two losses are added together, by default and Both are set to 1.
[0111] According to an embodiment of the present disclosure, the similarity loss can be determined by a variety of vector similarity calculation methods, such as inner product, cosine similarity, etc. This similarity is used to describe the semantic similarity between two non-modal features of the same sample.
[0112] In a specific embodiment, the similarity loss is calculated according to the following method: the image features and the text features are respectively input into the activation layer to obtain the first feature of the image modality and the second feature of the text modality respectively; the semantic similarity between the first feature and the second feature is used as the similarity loss.
[0113] It should be noted that the activation layer is the activation layer in the mutual feature classification branch. Before the WSI image features and text features are weighted by the two cross-attention modules, the image features and text features are normalized by the SoftMax activation layer to obtain the first feature and the second feature; then, the cosine similarity between the two is calculated as the similarity loss.
[0114] (11)
[0115] Among them, ρ(⋅) is the SoftMax activation layer, I and T are image features and text features respectively.
[0116] According to an embodiment of the present disclosure, a breast cancer classification device for WSI images is provided in an electronic device, and the device includes: a preprocessing module, which is used to perform region segmentation on the preprocessed original WSI image to obtain a WSI image, wherein the WSI image includes multiple WSI image blocks; an image processing module, which is used to input the WSI image into an image encoding branch of a classification model, and output image features of the WSI image; a text processing module, which is used to input target metadata into a text encoding branch of the classification model, and output text features, wherein the target metadata corresponds to the WSI image; a fusion module, which is used to input image features and text features into a mutual feature classification branch of the classification model, and output a first classification result, wherein the first classification result includes a breast cancer classification result of each WSI image block; and a splicing module, which is used to splice the breast cancer classification result of each WSI image block according to the segmentation position of the WSI image block in the WSI image, to obtain a second classification result.
[0117] It should be noted that the breast cancer classification device for WSI images in the embodiments of the present disclosure corresponds to the breast cancer classification method for WSI images in the embodiments of the present disclosure. The description of the breast cancer classification device for WSI images specifically refers to the breast cancer classification method for WSI images, which will not be repeated here.
[0118] According to an embodiment of the present disclosure, any multiple modules among the preprocessing module, the image processing module, the text processing module, the fusion module and the splicing module can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.
[0119] According to an embodiment of the present disclosure, at least one of the processing module, the image processing module, the text processing module, the fusion module and the splicing module may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware and firmware or by an appropriate combination of any of them. Alternatively, at least one of the processing module, the image processing module, the text processing module, the fusion module and the splicing module may be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function may be executed.
[0120] Figure 5 A block diagram of an electronic device suitable for a breast cancer classification method for a WSI image according to an embodiment of the present disclosure is schematically shown.
[0121] like Figure 5 As shown, the electronic device 500 according to an embodiment of the present disclosure includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 to a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.
[0122] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.
[0123] According to an embodiment of the present disclosure, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output I / O interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage portion 508 as needed.
[0124] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.
[0125] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.
[0126] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the above method provided by the embodiment of the present disclosure.
[0127] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 501. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.
[0128] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0129] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.
[0130] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).
[0131] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0132] It will be appreciated by those skilled in the art that the various embodiments and / or features of the present disclosure may be combined and / or coupled in a variety of ways, even if such combinations and / or couplings are not explicitly described in the present disclosure. In particular, the various embodiments and / or features of the present disclosure may be combined and / or coupled in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or couplings fall within the scope of the present disclosure.
[0133] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present disclosure. It should be understood that the above description is only a specific embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.
Claims
1. A breast cancer classification method for WSI images, characterized in that: The method is performed by an electronic device, and includes: Performing region segmentation on the preprocessed original WSI image to obtain a WSI image, wherein the WSI image includes a plurality of WSI image blocks; Input the WSI image into an image encoding branch of a classification model, and output image features of the WSI image; Inputting target metadata into a text encoding branch of the classification model and outputting text features, wherein the target metadata corresponds to the WSI image; Inputting the image features and the text features into a mutual feature classification branch of the classification model, and outputting a first classification result, wherein the first classification result includes a breast cancer classification result of each of the WSI image blocks; According to the segmentation position of the WSI image block in the WSI image, the breast cancer classification result of each WSI image block is spliced to obtain a second classification result.
2. The method according to claim 1, characterized in that The image encoding branch includes a prompt word generation module and N stacked adaptive feature weighting modules, where N is a positive integer greater than or equal to 2; the image encoding branch inputting the WSI image into the classification model and outputting the image features of the WSI image includes: Globally pooling the first initial image features output by the Nth adaptive feature weighting module to determine an initial pooling feature; Inputting the initial pooling features into the prompt word generation module, and outputting prompt word features; fusing the first initial image feature and the prompt word feature to obtain a second initial image feature, and using the second initial image feature as the image feature output by the image encoding branch; The input of the first adaptive feature weighting module is the WSI image, and the input of the remaining N-1 adaptive feature weighting modules is the second initial image feature determined according to the first initial image feature output by the previous adaptive feature weighting module and the prompt word feature.
3. The method according to claim 2, characterized in that The adaptive feature weighting module includes a self-attention branch and an adaptive residual branch; the method further includes: The self-attention branch processes the input features of the adaptive feature weighting module through a stacked self-attention module, a first convolutional layer, and a first multi-layer perceptron to obtain an attention feature; The adaptive residual branch processes the input features through two sets of stacked second convolutional layers to obtain pixel-level weights; The attention feature is screened using the pixel-level weight to obtain the first initial image feature.
4. The method according to claim 1, characterized in that: The mutual feature classification branch includes a first cross-attention module, a second cross-attention module, a second multi-layer perceptron and an activation layer; Inputting the image feature and the text feature into the mutual feature classification branch of the classification model, and outputting a first classification result, comprising: Using the image feature as a first data source of the first cross-attention module, using the text feature as a second data source of the first cross-attention module, and determining a first fusion feature based on a cross-attention mechanism; Using the text feature as the first data source of the second cross-attention module and the image feature as the second data source of the second cross-attention module, and determining a second fusion feature based on a cross-attention mechanism; Inputting the first fusion feature and the second fusion feature into the second multilayer perceptron, and outputting enhanced classification features; The classification feature is input into the activation layer, and the first classification result is output.
5. The method according to claim 1, characterized in that The text encoding branch includes M stacked metadata feature extraction modules, where M is a positive integer greater than or equal to 2; The step of inputting the target metadata into the text encoding branch of the classification model and outputting text features comprises: Mining classification features related to the breast cancer from the target metadata through M stacked metadata feature extraction modules; The target metadata includes P attributes, and the attributes are used to describe breast cancer characteristics of the object to which the WSI image belongs.
6. The method according to claim 5, characterized in that The metadata feature extraction module includes a stacked graph convolution layer and a linear layer, the input of the graph convolution layer is the input of the metadata feature extraction module, and the output of the linear layer is connected to the input residual of the graph convolution layer to obtain the output of the metadata feature extraction module; The graph convolution layer includes P nodes corresponding to the P attributes, and the node relationships between the P nodes are obtained after training.
7. The method according to claim 4, characterized in that The loss function of the classification model includes cross entropy loss and similarity loss, wherein the similarity loss is used to characterize the semantic similarity between the image feature and the text feature.
8. The method according to claim 7, characterized in that The similarity loss is calculated according to the following method: Inputting the image feature and the text feature into the activation layer respectively, to obtain a first feature of the image modality and a second feature of the text modality respectively; The semantic similarity between the first feature and the second feature is used as the similarity loss.
9. The method according to claim 1, characterized in that: The performing region segmentation on the preprocessed original WSI image to obtain the WSI image includes: Acquire the original WSI image of the pathological section by a scanning instrument; Unifying the coloring distribution of the original WSI image by using a preset coloring algorithm to obtain a first WSI image; Performing region segmentation on the first WSI image to obtain a second WSI image, wherein the second WSI image includes a plurality of WSI image blocks of preset sizes; The image blocks having preset image features are deleted from the multiple image blocks to obtain the WSI image.
10. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 9.
Citation Information
Patent Citations
Automatic analysis system and method for full-slice digital pathological image
CN112435243A
Methods for training generative large language models and for processing image tasks
CN117114063A
Breast cancer sentinel lymph node metastasis prediction method based on multi-modal fusion characterization
CN117524479A
Cited By
Breast cancer HER2 prediction method based on multiple prompt learning
CN119993468A
A multi-cue learning method for HER2 prediction in breast cancer
CN119993468B
Cross-modal breast image segmentation and classification method based on X-ray and clinical medicine text
CN121259327A
Multi-modal deep learning fusion model construction method for breast cancer
CN121439245A
