Pathology image segmentation prediction method, system, device, and medium

Abstract

The application discloses a pathological image segmentation prediction method, system, device and medium, and belongs to the cross technical field of digital pathological image analysis and computer vision technology. The technical problem to be solved by the application is how to inhibit context mixing, enhance target tissue causal discrimination features, effectively model multi-scale tissue structure relationships and relieve over-smoothing in graph reasoning process under weak annotation conditions. The technical scheme adopted is as follows: preprocessing a pathological image; encoding the pathological image; causal counterfactual projection: mapping image-level global semantic features into a query vector and mapping patch-level features of local spatial information into a key matrix and a value matrix; calculating a fact response based on cross attention according to the query vector, the key matrix and the value matrix; calculating a counterfactual response through the value matrix; taking the difference between the fact response and the counterfactual response as a causal effect representation of a target tissue region; re-weighting causal features; constructing a multi-scale topological graph; performing multi-scale spectral graph reasoning; and performing segmentation prediction.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of digital pathological image analysis and computer vision technology, specifically a method, system, device and medium for pathological image segmentation and prediction. Background Technology

[0002] Pathological image segmentation is one of the fundamental tasks in digital pathology analysis. Its goal is to locate and segment different tissue regions or lesion-related regions in pathological slide images. Accurate segmentation results can provide key support for quantitative statistics of tissue structure, automatic localization of lesion regions, auxiliary judgment of disease grading, and subsequent clinical research. It is a core link in promoting the intelligent implementation of pathological diagnosis.

[0003] Current mainstream fully supervised segmentation methods heavily rely on pixel-level fine-grained annotations. These annotations must be manually drawn region by region by professional pathologists, which is not only time-consuming and extremely costly, but also prone to subjective differences in judgment among annotators, making it difficult to guarantee annotation consistency. This severely restricts the construction of large-scale pathological datasets and the clinical application of intelligent models. Therefore, segmentation schemes that rely solely on low-cost, weakly supervised information such as image-level labels, bounding box annotations, and point annotations for training have become a key research direction in the field.

[0004] Existing weakly supervised pathological image methods are mainly divided into four categories: class activation map-based, pseudo-label iteration, Transformer global modeling, and graph neural network structure modeling. Although some progress has been made, there are still obvious shortcomings: ① Pathological images exhibit significant tissue heterogeneity and contextual confusion. The target tissue region is often highly spatially correlated with surrounding stroma, necrosis, lymphocyte infiltration, and other areas. Under the constraint of only weak image-level annotation, the model is prone to learning background contextual features that are related to the category but are not true evidence of the target tissue, leading to unstable localization. ②Although Transformer can model long-distance dependencies, the global attention mechanism may amplify background context bias, further coupling target organization features with non-target organization features; ③Graph neural networks can explicitly model the spatial structural relationships between image patches, but conventional graph convolution usually uses repeated neighborhood aggregation. As the number of graph propagation layers increases, the problem of over-smoothing of features can easily occur, which weakens the organizational boundaries and local morphological differences, affecting the accuracy of segmentation boundaries.

[0005] Therefore, how to suppress contextual confusion under weak labeling conditions, enhance the causal discriminative features of target organizations, effectively model multi-scale organizational structure relationships, and alleviate oversmoothing in the graph reasoning process are the technical problems that urgently need to be solved. Summary of the Invention

[0006] The technical objective of this invention is to provide a method, system, device, and medium for pathological image segmentation and prediction, in order to address how to suppress contextual confusion under weakly labeled conditions, enhance the causal discriminative features of target tissues, effectively model multi-scale tissue structural relationships, and alleviate the problem of oversmoothing in the graph reasoning process.

[0007] The technical objective of this invention is achieved as follows: a method for segmenting and predicting pathological images, the specific method of which is as follows: Preprocessing pathological images: Converting the two-dimensional spatial structure of the pathological image to be segmented into a sequence form to obtain the preprocessed pathological image to be segmented; Encoding pathological images: Using the preprocessed pathological images to be segmented, the patch-level features of local spatial information and the global semantic features of the image are extracted by the Vision Transformer (ViT) image encoder. Causal counterfactual projection: Image-level global semantic features are mapped to query vectors, and patch-level features of local spatial information are mapped to key and value matrices. Based on the query vector, key and value matrices, the factual response is calculated using cross-attention. The counterfactual response is calculated using the value matrix, and the difference between the factual and counterfactual responses is used as the causal effect representation of the target organizational region. Causal feature reweighting: The channel reweighting signal is generated by using the causal effect representation, and the local image patch features are reweighted according to the channel reweighting signal to obtain causally enhanced local image patch features; Constructing a multi-scale topology graph: Using causal enhancement of local image patch features as graph nodes and the correlation between local image patch regions as edges, a multi-scale topology graph is constructed based on the spatial location of local image patches in the preprocessed pathological image to be segmented. Multi-scale spectral inference: Spectral Chebyshev graph convolution is used to perform multi-scale spectral inference on multi-scale topological graphs to obtain graph enhancement features; Segmentation prediction: The segmentation decoder performs segmentation prediction on the image enhancement features to generate image block-level segmentation prediction results. The image block-level segmentation prediction results are then restored to pixel-level segmentation prediction results through interpolation and upsampling. The pixel-level segmentation prediction results are used as the segmentation prediction results corresponding to the pathological image to be segmented.

[0008] As a preferred embodiment, encoding the pathological image specifically involves: using a visual Transformer image encoder to fully capture the global contextual information between different tissue regions in the preprocessed pathological image to be segmented, thereby obtaining image-level global semantic features; specifically as follows: The Patch Embedding mechanism in the visual Transformer image encoder is used to segment the preprocessed pathological images. Divided into A total of three image patches of the same size that do not overlap; among them... This indicates the height of the preprocessed pathological image to be segmented. This indicates the width of the preprocessed pathological image to be segmented; This indicates the number of RGB channels in the preprocessed pathological image to be segmented; Represents the real number field; Each image patch is linearly projected and mapped into a high-dimensional feature space to form a corresponding patch token sequence. The patch token sequence is processed by a visual Transformer image encoder to obtain patch-level features of local spatial information. ;in, Indicates the first The high-dimensional semantic feature vector obtained by encoding each image patch using a visual Transformer image encoder; This represents the feature dimension of each patch token sequence; Patch-level feature representation of all local spatial information is achieved by using either category-based tokens or global average pooling of patch tokens. Perform aggregation processing to obtain image-level global semantic features. .

[0009] More preferably, the causal counterfactual projection is as follows: The formula for mapping image-level global semantic features to query vectors is as follows: ;in, Represents the query vector; Represents the query mapping matrix; The patch-level features of local spatial information are mapped to a key matrix and a value matrix, as shown in the formula: ; ;in, Represents the key matrix; Represents a value matrix; Represents the key mapping matrix; Represents a value mapping matrix; The formula for calculating factual responses based on cross-attention is as follows: ;in, Indicate a factual response; Indicated by For query vectors, For the key matrix, The attention calculation function is defined as follows: ;in, express The transpose of the matrix; Indicates the dimension of attention space; represents the square root of the attention space dimension; softmax represents the normalized exponential function; value matrix The global context prototype is obtained by averaging the local value vectors in the vectors. ;in, Represents the global context prototype; Value matrix The corresponding number in the middle Value vectors of local image patches; global context prototype copy Next, obtain the counterfactual value matrix. : Simulate the context response after removing local organizational structure differences by using a counterfactual value matrix composed of global context prototypes; Calculate the counterfactual response based on the query vector, key matrix, and counterfactual matrix. : ; Respond with facts and counterfactual response The difference is used as a representation of the causal effect of the target organizational region. : Among them, causal effect means It can retain discriminative information that cannot be interpreted by the global context prototype alone, thereby enhancing the target organization-related features.

[0010] More preferably, the causal features are reweighted as follows: Using causal effect to represent Generate channel reweighted signal: ;in, Indicates the channel reweighting signal; This represents the Sigmoid activation function; This represents the channel reweighting mapping matrix; Indicates the bias term; By reweighting the local image patch features X, and enhancing the features related to the target tissue while suppressing background interference, causal enhanced local image patch features are obtained. ;in, ⊙ indicates causal enhancement of local image patch features; ⊙ indicates element-wise multiplication.

[0011] More preferably, the construction of a multi-scale topology graph is as follows: For the Construct a topological graph using multiple topological scales: ;in, Indicates the first Topological graphs at various topological scales , This represents the total number of topological scales, which determines the extent of relationships established between image patches. This represents a set of graph nodes, where each graph node corresponds to a causal augmentation image patch feature. Indicates the first The set of edges at each scale; For the Construct an adjacency matrix from a topological graph of several topological scales: ;in, Indicates the first An adjacency matrix of topological scale; adjacency matrix elements in Defined as: if ,but ;like ,but ;in, Indicates the first Each graph node; Indicates the first Each graph node; Indicates the first Nodes at various scales The spatial neighborhood set; Representing the adjacency matrix The Middle Line 1 Column elements; For adjacency matrix Add self-connection, obtain Then, normalization is performed to obtain the normalized adjacency matrix. : ;in, Indicates the first Normalized adjacency matrices of various scales; This represents the adjacency matrix after adding self-connections; express The corresponding degree matrix; Degree matrix The inverse matrix; the degree matrix diagonal elements in Defined as: ;in, Degree matrix The Middle Line 1 Column elements; Represents the adjacency matrix after adding self-connections. The Middle Line 1 The elements of the column.

[0012] More specifically, multi-scale spectral inference is as follows: Suppose causal enhancement of local image patch features The input feature matrix of the spectral Chebyshev plot convolutional layer ,Right now For the first Using topological scales, we construct zero-order, first-order, and second-order Chebyshevs, as follows: ; ; ; in, Indicates the first Zero-order Chebyshevsky at a topological scale; Indicates the first First-order Chebyshevky at a topological scale; Indicates the first Second-order Chebyshevsky at a topological scale; Indicates the first Normalized adjacency matrices of topological scale; For the Each topological scale, spectral response Represented as: ;in, They represent the first Learnable parameter matrices for zero-order, first-order, and second-order Chebyshev bases at each topological scale; The spectral responses at multiple topological scales are fused with the self-projection results of the input features to obtain multi-scale fused features. : ;in, Represents the self-projection parameter matrix; Indicates the total number of topological scales; The output of the spectral Chebyshev plot convolutional layer is obtained through residual connections: ;in, This indicates that the spectral Chebyshev plot convolutional layer is paired with the input feature matrix. The output; Will A multi-scale spectral inference module is constructed by stacking convolutional layers with layered Chebyshev plots, represented as follows: ;in, This represents the index of the multi-scale spectral inference module. This indicates the total number of layers in the spectral Chebyshev plot convolutional layer; , This represents the input features of the multi-scale spectral inference module; This indicates causal enhancement of local image patch features; Indicates the first The node feature matrix input to the layered graph inference module describes the process before... Semantic representation of each pathological image block after layer graph convolution propagation; The topological enhancement node features output by the spectral Chebyshev graph convolutional layer are represented by the fusion of local organizational semantics and global topological relationships by combining neighborhood node information in the aggregated graph structure with residual connections. go through After multi-scale spectral inference of the layered Chebyshev diagram convolutional layer, the data is obtained. Layer-level Chebyshev plot convolutional layer pairs input feature matrix The output of the input feature matrix is ​​used to... The output is normalized to obtain graph enhancement features. : ;in, Presentation layer normalization operation; The segmentation prediction is as follows: Graph enhancement features via a segmentation decoder (Mask Transformer Decoder) Perform segmentation prediction to generate image block-level segmentation prediction results. : ;in, Indicates the segmentation decoder; The image block-level segmentation prediction results are restored to pixel-level segmentation prediction results through interpolation and upsampling, and these pixel-level segmentation prediction results are used as the segmentation prediction results corresponding to the pathological image to be segmented; wherein, the pixel-level segmentation prediction results ;in, This indicates an interpolation upsampling operation.

[0013] More preferably, the method further includes optimizing the segmentation prediction results: a weakly supervised segmentation loss and a causal orthogonal constraint loss are jointly used to optimize the segmentation prediction results corresponding to the pathological image to be segmented; specifically as follows: Introducing causal orthogonal constraint loss By minimizing the causal orthogonality constraint loss causal effect indicates Counterfactual response Pushed to the complementary feature subspace, the information overlap between the causal effect representation and the context representation is reduced, thereby further reducing contextual clutter, as shown in the following formula: ; in, This indicates the number of samples in the training batch. Indicates the first in the batch One sample; Indicates the first The causal effect is represented for each sample; Indicates the first Counterfactual responses of a sample; Represents the dot product of vectors; Represents the L2 norm, Represents absolute value; and Let L2 norms of the causal effect representation and counterfactual response be represented respectively. These are used to normalize the dot product of the causal effect representation and the counterfactual response, measure the cosine similarity between the causal effect representation and the counterfactual response, and thus constrain the correlation between the causal representation and the counterfactual representation. Overall loss function The formula is as follows: ; in, This indicates the loss from weakly supervised segmentation. This represents the loss due to causal orthogonal constraints. This represents the weighting coefficient used to balance the weakly supervised segmentation loss and the causal orthogonal constraint loss.

[0014] A pathological image segmentation prediction system, which implements the pathological image segmentation prediction method described above; the system includes: The pathological image preprocessing unit is used to convert the two-dimensional spatial structure of the pathological image to be segmented into a sequence form and obtain the preprocessed pathological image to be segmented. The pathological image coding unit is used to extract patch-level features of local spatial information and image-level global semantic features from the preprocessed pathological image to be segmented by a visual Transformer image encoder. The causal counterfactual projection unit is used to map image-level global semantic features into query vectors and local spatial information patch-level features into key and value matrices. Based on the query vector, key and value matrices, it calculates the factual response using cross-attention and calculates the counterfactual response using the value matrix. The difference between the factual response and the counterfactual response is used as the causal effect representation of the target organization region. The causal feature reweighting unit is used to generate channel reweighted signals using causal effect representation, and to reweight local image patch features based on the channel reweighted signals to obtain causally enhanced local image patch features; The multi-scale topology graph construction unit is used to construct a multi-scale topology graph based on the spatial location of local image blocks in the preprocessed pathological image to be segmented, using causal enhanced local image block features as graph nodes and the correlation between local image block regions as edges. The multi-scale spectral inference unit is used to perform multi-scale spectral inference on multi-scale topological graphs using spectral Chebyshev graph convolution to obtain graph-enhanced features. The segmentation prediction unit is used to perform segmentation prediction on the image enhancement features through the segmentation decoder, generate image block-level segmentation prediction results, and restore the image block-level segmentation prediction results to pixel-level segmentation prediction results through interpolation upsampling. The pixel-level segmentation prediction results are used as the segmentation prediction results corresponding to the pathological image to be segmented.

[0015] An electronic device includes: a memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the pathological image segmentation and prediction method as described above.

[0016] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the pathological image segmentation and prediction method described above.

[0017] The pathological image segmentation and prediction method, system, device, and medium of the present invention have the following advantages: (i) This invention first acquires the pathological image to be segmented, and extracts patch-level features of local spatial information and image-level global semantic features through a visual Transformer image encoder; then, it uses a causal counterfactual projection module to estimate the causal effect representation of the target tissue (tumor region) region based on the difference between the factual response and the counterfactual response, so as to suppress contextual confounding factors such as background tissue, staining differences and scanning conditions; then, it performs channel reweighting on the image patch features according to the causal effect representation to obtain causal enhanced image patch features; then, it uses the causal enhanced image patch features as graph nodes to construct a spatial topology graph at multiple scales; then, it uses spectral Chebyshev graph convolution on the topology graph at multiple scales to perform multi-scale spectral graph inference to obtain graph enhanced features; finally, it generates the pathological image segmentation result through a segmentation decoder, which can improve the localization accuracy of pathological tissue regions and improve the ability to preserve tissue boundaries under weak supervision. (ii) This invention estimates the causal effect of the target tissue region by the difference between the factual response and the counterfactual response, which can suppress contextual confounding factors such as background tissue, staining differences and scanning style under weak supervision, thereby improving the reliability of target region localization; (iii) The present invention employs a causal reweighting mechanism to perform channel-level modulation on image patch features, thereby enhancing the discriminative channels related to the target tissue and suppressing the context-dominated non-target channels; (iv) This invention constructs pathological image blocks into multi-scale topological maps, which can simultaneously model local cell and tissue structures and spatial dependencies of distant tissues; (v) This invention uses spectral Chebyshev graph convolution for graph reasoning. Compared with the first-order neighborhood aggregation of ordinary graph convolution, it can capture rich frequency responses using high-order spectral basis, which helps to alleviate the over-smoothing of graph features and maintain the organization boundary. (vi) This invention is applicable to pathological image segmentation scenarios with only weak or coarse image-level annotations, which can reduce the reliance on pixel-level manual annotation. Attached Figure Description

[0018] The invention will be further described below with reference to the accompanying drawings.

[0019] Appendix Figure 1 The flowchart is for a pathological image segmentation prediction method. Appendix Figure 2 A flowchart for causal counterfactual projection. Detailed Implementation

[0020] The pathological image segmentation and prediction method, system, device, and medium of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] Example 1:

[0022] As attached Figure 1 As shown, this embodiment provides a pathological image segmentation and prediction method. This method can be used for the automatic segmentation of target tissue regions in pathological images, such as the segmentation of tissue categories like tumor epithelium, stroma, lymphocyte infiltration, and necrotic areas. Furthermore, this method is used for image processing and auxiliary analysis, and does not directly provide a disease diagnosis conclusion; details are as follows: S1. Preprocessing pathological images: Converting the two-dimensional spatial structure of the pathological image to be segmented into a sequence form to obtain the preprocessed pathological image to be segmented. S2. Encoding pathological images: Using the preprocessed pathological images to be segmented, the patch-level features of local spatial information and the global semantic features of the image are extracted by the Vision Transformer (ViT) image encoder. S3. Causal Counterfactual Projection: Maps image-level global semantic features to query vectors and local spatial information patch-level features to key and value matrices. Based on the query vector, key and value matrices, the factual response is calculated using cross-attention. The counterfactual response is calculated using the value matrix, and the difference between the factual and counterfactual responses is used as the causal effect representation of the target organizational region. S4. Causal Feature Reweighting: Generate channel reweighted signals using causal effect representation, and reweight local image patch features based on channel reweighted signals to obtain causally enhanced local image patch features; S5. Construct a multi-scale topology graph: Using causal enhanced local image patch features as graph nodes and the correlation between local image patch regions as edges, construct a multi-scale topology graph based on the spatial location of local image patches in the preprocessed pathological image to be segmented. S6. Multi-scale spectral inference: Multi-scale spectral inference is performed on the multi-scale topological graph using spectral Chebyshev graph convolution to obtain graph enhancement features; S7. Segmentation Prediction: The segmentation decoder performs segmentation prediction on the image enhancement features to generate image block-level segmentation prediction results. The image block-level segmentation prediction results are then restored to pixel-level segmentation prediction results through interpolation upsampling. The pixel-level segmentation prediction results are used as the segmentation prediction results corresponding to the pathological image to be segmented. S8. Optimize segmentation prediction results: The segmentation prediction results corresponding to the pathological images to be segmented are jointly optimized by using weakly supervised segmentation loss and causal orthogonal constraint loss.

[0023] In this embodiment, the encoding of the pathological image in step S2 specifically involves: using a visual Transformer image encoder to fully capture the global contextual information between different tissue regions in the preprocessed pathological image to be segmented, thereby obtaining image-level global semantic features; specifically as follows: S201. Using the Patch Embedding mechanism in the visual Transformer image encoder, the preprocessed pathological image to be segmented... Divided into A total of three image patches of the same size that do not overlap; among them... This indicates the height of the preprocessed pathological image to be segmented. This indicates the width of the preprocessed pathological image to be segmented; This indicates the number of RGB channels in the preprocessed pathological image to be segmented; Represents the real number field; S202. Map each image patch to a high-dimensional feature space through linear projection to form a corresponding patch token sequence; S203. Process the patch token sequence using a visual Transformer image encoder to obtain patch-level features of local spatial information. ;in, Indicates the first The high-dimensional semantic feature vector obtained by encoding each image patch using a visual Transformer image encoder; This represents the feature dimension of each patch token sequence; S204. Use category-labeled tokens or global average pooling of patch tokens to represent patch-level features of all local spatial information. Perform aggregation processing to obtain image-level global semantic features. .

[0024] The visual Transformer image encoder can adopt the ViT-S / 16 structure, with a feature dimension d=384 and 12 Transformer layers. To enable the Transformer to process images, the two-dimensional spatial structure needs to be converted into a sequence. ViT employs a concept similar to "word tokenization" in NLP, dividing the image into multiple non-overlapping regions, each region serving as a visual token. ViT-S / 16 uses this approach. The visual model is pre-trained using a patch size, and an image is converted into 196 image patches.

[0025] In supervised pathological image segmentation, models tend to learn target tissue features mixed with background context features. For example, some non-target tissue regions may frequently co-occur with the target category in the training set, causing the model to mistakenly treat them as evidence of the target region. To mitigate this problem, this embodiment introduces causal counterfactual projection to eliminate the confounding relationship between target tissue features and background context features during weakly supervised pathological image segmentation. By using a causal intervention mechanism, true discriminative features related to the target tissue are extracted, and the model's ability to identify target regions is enhanced.

[0026] As attached Figure 2 As shown, the causal counterfactual projection in step S3 of this embodiment is as follows: S301. Map the image-level global semantic features to the query vector, using the following formula: ;in, Represents the query vector; Represents the query mapping matrix; S302. Map the patch-level features of local spatial information into a key matrix and a value matrix, using the following formula: ; ;in, Represents the key matrix; Represents a value matrix; Represents the key mapping matrix; Represents a value mapping matrix; S303. Calculate the factual response based on cross-attention, using the following formula: ;in, Indicate a factual response; Indicated by For query vectors, For the key matrix, The attention calculation function is defined as follows: ;in, express The transpose of the matrix; Indicates the dimension of attention space; represents the square root of the attention space dimension; softmax represents the normalized exponential function; This embodiment employs a fact-counterfact response modeling method based on cross-attention, which associates the patch-level features of local spatial information extracted by ViT with the image-level global semantic features to calculate the fact response under the original image conditions; the fact response This represents the association response between image-level global semantic features and image patches under the original observation conditions. Since the original image patch features contain target organization information and contextual background information, the factual response... It may also contain confounding factors; in order to estimate the response generated by contextual information alone, this embodiment constructs a counterfactual value matrix.

[0027] S304, Logarithmic Matrix The global context prototype is obtained by averaging the local value vectors in the vectors. ;in, Represents the global context prototype; Value matrix The corresponding number in the middle Value vectors of local image patches; S305, Set the global context prototype copy Next, obtain the counterfactual value matrix. : ; Simulates the context response after removing local organizational structure differences using a counterfactual value matrix composed of global context prototypes; due to It eliminates the structural differences between local image patches, so it mainly reflects global contextual information, while weakening the differences in local target organization morphology; S306. Calculate the counterfactual response based on the query vector, key matrix, and counterfactual matrix. : ; S307, Respond with Facts and counterfactual response The difference is used as a representation of the causal effect of the target organizational region. : Among them, causal effect means It can retain discriminative information that cannot be interpreted by the global context prototype alone, thereby enhancing the target organization's relevant features; it can obtain causal effect representation by utilizing the difference between factual and counterfactual responses, and further enhance causal relevant features through a causal feature reweighting mechanism to achieve accurate representation of the target organization region.

[0028] The specific causal feature reweighting in step S4 of this embodiment is as follows: S401, Using causal effects to represent Generate channel reweighted signal: ;in, Indicates the channel reweighting signal; This represents the Sigmoid activation function; This represents the channel reweighting mapping matrix; Indicates the bias term; S402. Reweight the local image patch features X to enhance the features related to the target tissue and suppress background interference, thereby obtaining causally enhanced local image patch features: ;in, ⊙ indicates causal enhancement of local image patch features; ⊙ indicates element-wise multiplication.

[0029] Through the aforementioned residual reweighting mechanism, feature channels related to causal evidence of the target organization are enhanced, while context-driven non-target channels are suppressed.

[0030] Tissue structures in pathological images exhibit significant multi-scale features; for example, local cell aggregation, glandular structure, stroma distribution, and larger-scale tissue morphology can all influence segmentation results. Therefore, this embodiment utilizes causal enhancement of image patch features. As graph nodes, a topology graph is constructed at multiple scales based on the spatial location of image patches in the original image. A topology graph is a data structure used to describe the spatial relationships between image patches. It is not the image itself, but rather an abstraction of local regions in a pathological image into "nodes," with "edges" representing the relationships between these regions.

[0031] The specific steps for constructing the multi-scale topology graph in step S5 of this embodiment are as follows: S501, Regarding the first Construct a topological graph using multiple topological scales: ;in, Indicates the first Topological graphs at various topological scales , This represents the total number of topological scales, which determines the extent of relationships established between image patches. This represents a set of graph nodes, where each graph node corresponds to a causal augmentation image patch feature. Indicates the first The set of edges at each scale; S502, for the first Construct an adjacency matrix from a topological graph of several topological scales: ;in, Indicates the first An adjacency matrix of topological scale; adjacency matrix elements in Defined as: if ,but ;like ,but ;in, Indicates the first Each graph node; Indicates the first Each graph node; Indicates the first Nodes at various scales The spatial neighborhood set; Representing the adjacency matrix The Middle Line 1 Column elements; S503, Adjacency Matrix Add self-connection, obtain Then, normalization is performed to obtain the normalized adjacency matrix. : ;in, Indicates the first Normalized adjacency matrices of various scales; This represents the adjacency matrix after adding self-connections; express The corresponding degree matrix; Degree matrix The inverse matrix; the degree matrix diagonal elements in Defined as: ;in, Degree matrix The Middle Line 1 Column elements; Represents the adjacency matrix after adding self-connections. The Middle Line 1 The elements of the column.

[0032] The multi-scale spectral inference in step S6 of this embodiment is as follows: S601, Assume causal enhancement of local image patch features The input feature matrix of the spectral Chebyshev plot convolutional layer ,Right now For the first Using topological scales, we construct zero-order, first-order, and second-order Chebyshevs, as follows: ; ; ; in, Indicates the first Zero-order Chebyshevsky at a topological scale; Indicates the first First-order Chebyshevky at a topological scale; Indicates the first Second-order Chebyshevsky at a topological scale; Indicates the first Normalized adjacency matrices of topological scale; S602, for the first Each topological scale, spectral response Represented as: ;in, They represent the first Learnable parameter matrices for zero-order, first-order, and second-order Chebyshev bases at each topological scale; S603. Fuse the spectral responses at multiple topological scales with the self-projection results of the input features to obtain multi-scale fused features. : ;in, Represents the self-projection parameter matrix; Indicates the total number of topological scales; S604. Obtain the output of the spectral Chebyshev plot convolutional layer through residual connections: ;in, This indicates that the spectral Chebyshev plot convolutional layer is paired with the input feature matrix. The output; S605, will A multi-scale spectral inference module is constructed by stacking convolutional layers with layered Chebyshev plots, represented as follows: ;in, This represents the index of the multi-scale spectral inference module. This indicates the total number of layers in the spectral Chebyshev plot convolutional layer; , This represents the input features of the multi-scale spectral inference module; This indicates causal enhancement of local image patch features; Indicates the first The node feature matrix input to the layered graph inference module describes the process before... Semantic representation of each pathological image block after layer graph convolution propagation; The topological enhancement node features output by the spectral Chebyshev graph convolutional layer are represented by the fusion of local organizational semantics and global topological relationships by combining neighborhood node information in the aggregated graph structure with residual connections. S606, passing through After multi-scale spectral inference of the layered Chebyshev diagram convolutional layer, the data is obtained. Layer-level Chebyshev plot convolutional layer pairs input feature matrix The output of the input feature matrix is ​​used to further analyze the input feature matrix. The output is normalized to obtain graph enhancement features. : ;in, Presentation layer normalization operation.

[0033] In this embodiment A value of 8 is recommended to achieve a better balance between remote organizational dependency modeling and boundary preservation.

[0034] The segmentation prediction in step S7 of this embodiment is as follows: S701, Enhance graph features using a segmentation decoder (Mask Transformer Decoder). Perform segmentation prediction to generate image block-level segmentation prediction results. : ;in, Indicates the segmentation decoder; S702. The image block-level segmentation prediction result is restored to the pixel-level segmentation prediction result through interpolation and upsampling, and the pixel-level segmentation prediction result is used as the segmentation prediction result corresponding to the pathological image to be segmented; wherein, the pixel-level segmentation prediction result ;in, This indicates an interpolation upsampling operation.

[0035] The optimized segmentation prediction result in step S8 of this embodiment is as follows: S801, Introducing Causal Orthogonal Constraint Loss By minimizing the causal orthogonality constraint loss causal effect indicates Counterfactual response Pushed to the complementary feature subspace, the information overlap between the causal effect representation and the context representation is reduced, thereby further reducing contextual clutter, as shown in the following formula: ; in, This indicates the number of samples in the training batch. Indicates the first in the batch One sample; Indicates the first The causal effect is represented for each sample; Indicates the first Counterfactual responses of a sample; Represents the dot product of vectors; Represents the L2 norm, Represents absolute value; and Let L2 norms of the causal effect representation and counterfactual response be represented respectively. These are used to normalize the dot product of the causal effect representation and the counterfactual response, measure the cosine similarity between the causal effect representation and the counterfactual response, and thus constrain the correlation between the causal representation and the counterfactual representation. S802, Overall Loss Function The formula is as follows: ; in, This indicates the loss from weakly supervised segmentation. This represents the loss due to causal orthogonal constraints. This represents the weighting coefficient used to balance the weakly supervised segmentation loss and the causal orthogonal constraint loss.

[0036] Example 2: This embodiment provides a pathological image segmentation prediction system, which is used to implement the pathological image segmentation prediction method as described in Embodiment 1; the system includes: The pathological image preprocessing unit is used to convert the two-dimensional spatial structure of the pathological image to be segmented into a sequence form and obtain the preprocessed pathological image to be segmented. The pathological image coding unit is used to extract patch-level features of local spatial information and image-level global semantic features from the preprocessed pathological image to be segmented by a visual Transformer image encoder. The causal counterfactual projection unit is used to map image-level global semantic features into query vectors and local spatial information patch-level features into key and value matrices. Based on the query vector, key and value matrices, it calculates the factual response using cross-attention and calculates the counterfactual response using the value matrix. The difference between the factual response and the counterfactual response is used as the causal effect representation of the target organization region. The causal feature reweighting unit is used to generate channel reweighted signals using causal effect representation, and to reweight local image patch features based on the channel reweighted signals to obtain causally enhanced local image patch features; The multi-scale topology graph construction unit is used to construct a multi-scale topology graph based on the spatial location of local image blocks in the preprocessed pathological image to be segmented, using causal enhanced local image block features as graph nodes and the correlation between local image block regions as edges. The multi-scale spectral inference unit is used to perform multi-scale spectral inference on multi-scale topological graphs using spectral Chebyshev graph convolution to obtain graph-enhanced features. The segmentation prediction unit is used to perform segmentation prediction on the image enhancement features through the segmentation decoder, generate image block-level segmentation prediction results, and restore the image block-level segmentation prediction results to pixel-level segmentation prediction results through interpolation upsampling. The pixel-level segmentation prediction results are used as the segmentation prediction results corresponding to the pathological image to be segmented.

[0037] Example 3:

[0038] This embodiment also provides an electronic device, including: a memory and a processor; The memory stores the instructions executed by the computer. The processor executes computer execution instructions stored in the memory, causing the processor to perform the pathological image segmentation and prediction method in any embodiment of the present invention.

[0039] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0040] Memory can be used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0041] Example 4:

[0042] This embodiment also provides a computer-readable storage medium storing multiple instructions, which are loaded by a processor to cause the processor to execute the pathological image segmentation and prediction method in any embodiment of the present invention. Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0043] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0044] Examples of storage media used to provide program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0045] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby achieving the function of any of the embodiments described above.

[0046] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0047] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for segmenting and predicting pathological images, characterized in that, The method is as follows: Preprocessing pathological images: Converting the two-dimensional spatial structure of the pathological image to be segmented into a sequence form to obtain the preprocessed pathological image to be segmented; Encoding pathological images: Using the preprocessed pathological images to be segmented, the visual Transformer image encoder extracts patch-level features of local spatial information and image-level global semantic features; Causal counterfactual projection: Image-level global semantic features are mapped to query vectors, and patch-level features of local spatial information are mapped to key and value matrices. Based on the query vector, key and value matrices, the factual response is calculated using cross-attention. The counterfactual response is calculated using the value matrix, and the difference between the factual and counterfactual responses is used as the causal effect representation of the target organizational region. Causal feature reweighting: The channel reweighting signal is generated by using the causal effect representation, and the local image patch features are reweighted according to the channel reweighting signal to obtain causally enhanced local image patch features; Constructing a multi-scale topology graph: Using causal enhancement of local image patch features as graph nodes and the correlation between local image patch regions as edges, a multi-scale topology graph is constructed based on the spatial location of local image patches in the preprocessed pathological image to be segmented. Multi-scale spectral inference: Spectral Chebyshev graph convolution is used to perform multi-scale spectral inference on multi-scale topological graphs to obtain graph enhancement features; Segmentation prediction: The segmentation decoder performs segmentation prediction on the image enhancement features to generate image block-level segmentation prediction results. The image block-level segmentation prediction results are then restored to pixel-level segmentation prediction results through interpolation and upsampling. The pixel-level segmentation prediction results are used as the segmentation prediction results corresponding to the pathological image to be segmented.

2. The pathological image segmentation and prediction method according to claim 1, characterized in that, Encoding pathological images specifically involves: using a visual Transformer image encoder to fully capture the global contextual information between different tissue regions in the preprocessed pathological image to be segmented, and obtaining image-level global semantic features; Specifically as follows: The Patch Embedding mechanism in the visual Transformer image encoder is used to segment the preprocessed pathological images. Divided into A total of three image blocks of the same size that do not overlap; among which... This indicates the height of the preprocessed pathological image to be segmented. This indicates the width of the preprocessed pathological image to be segmented; This indicates the number of RGB channels in the preprocessed pathological image to be segmented; Represents the real number field; Each image patch is linearly projected and mapped into a high-dimensional feature space to form a corresponding patch token sequence. The patch token sequence is processed by a visual Transformer image encoder to obtain patch-level features of local spatial information. ;in, Indicates the first The high-dimensional semantic feature vector obtained by encoding each image patch using a visual Transformer image encoder; This represents the feature dimension of each patch token sequence; Patch-level feature representation of all local spatial information is achieved by using either category-based tokens or global average pooling of patch tokens. Perform aggregation processing to obtain image-level global semantic features. .

3. The pathological image segmentation and prediction method according to claim 1 or 2, characterized in that, The counterfactual projection of causality is as follows: The formula for mapping image-level global semantic features to query vectors is as follows: ;in, Represents the query vector; Represents the query mapping matrix; The patch-level features of local spatial information are mapped to a key matrix and a value matrix, as shown in the formula: ; ;in, Represents the key matrix; Represents a value matrix; Represents the key mapping matrix; Represents a value mapping matrix; The formula for calculating factual responses based on cross-attention is as follows: ;in, Indicate a factual response; Indicated by For query vectors, For the key matrix, The attention calculation function is defined as follows: ;in, express The transpose of the matrix; Indicates the dimension of attention space; represents the square root of the attention space dimension; softmax represents the normalized exponential function; value matrix The global context prototype is obtained by averaging the local value vectors in the vectors. ;in, Represents the global context prototype; Value matrix The corresponding number in the middle Value vectors of local image patches; global context prototype copy Next, obtain the counterfactual value matrix. : Simulate the context response after removing local organizational structure differences by using a counterfactual value matrix composed of global context prototypes; Calculate the counterfactual response based on the query vector, key matrix, and counterfactual matrix. : ; Respond with facts and counterfactual response The difference is used as a representation of the causal effect of the target organizational region. : Among them, causal effect means It can retain discriminative information that cannot be interpreted by the global context prototype alone, thereby enhancing the target organization-related features.

4. The pathological image segmentation and prediction method according to claim 3, characterized in that, The specific details of the causal feature reweighting are as follows: Using causal effect to represent Generate channel reweighted signal: ;in, Indicates the channel reweighting signal; This represents the Sigmoid activation function; This represents the channel reweighting mapping matrix; Indicates the bias term; By reweighting the local image patch features X, and enhancing the features related to the target tissue while suppressing background interference, causal enhanced local image patch features are obtained. ;in, ⊙ indicates causal enhancement of local image patch features; ⊙ indicates element-wise multiplication.

5. The pathological image segmentation and prediction method according to claim 4, characterized in that, The construction of the multi-scale topology graph is as follows: For the Construct a topological graph using multiple topological scales: ;in, Indicates the first Topological graphs at various topological scales , This represents the total number of topological scales, which determines the extent of relationships established between image patches. This represents a set of graph nodes, where each graph node corresponds to a causal augmentation image patch feature. Indicates the first The set of edges at each scale; For the Construct an adjacency matrix from a topological graph of several topological scales: ;in, Indicates the first An adjacency matrix of topological scale; adjacency matrix elements in Defined as: if ,but ;like ,but ;in, Indicates the first Each graph node; Indicates the first Each graph node; Indicates the first Nodes at various scales The spatial neighborhood set; Representing the adjacency matrix The Middle Line number Column elements; For adjacency matrix Add self-connection, obtain Then, normalization is performed to obtain the normalized adjacency matrix. : ;in, Indicates the first Normalized adjacency matrices of various scales; This represents the adjacency matrix after adding self-connections; express The corresponding degree matrix; Degree matrix The inverse matrix; the degree matrix diagonal elements in Defined as: ;in, Degree matrix The Middle Line number Column elements; Represents the adjacency matrix after adding self-connections. The Middle Line number The elements of the column.

6. The pathological image segmentation and prediction method according to claim 5, characterized in that, The multi-scale spectral inference is as follows: Suppose causal enhancement of local image patch features The input feature matrix of the spectral Chebyshev plot convolutional layer ,Right now For the first Using topological scales, we construct zero-order, first-order, and second-order Chebyshevs, as follows: ； ； ； in, Indicates the first Zero-order Chebyshevsky at a topological scale; Indicates the first First-order Chebyshevky at a topological scale; Indicates the first Second-order Chebyshevsky at a topological scale; Indicates the first Normalized adjacency matrices of topological scale; For the Each topological scale, spectral response Represented as: ;in, They represent the first Learnable parameter matrices for zero-order, first-order, and second-order Chebyshev bases at each topological scale; The spectral responses at multiple topological scales are fused with the self-projection results of the input features to obtain multi-scale fused features. : ;in, Represents the self-projection parameter matrix; Indicates the total number of topological scales; The output of the spectral Chebyshev plot convolutional layer is obtained through residual connections: ;in, This indicates that the spectral Chebyshev plot convolutional layer is paired with the input feature matrix. The output; Will A multi-scale spectral inference module is constructed by stacking convolutional layers with layered Chebyshev plots, represented as follows: ;in, This represents the index of the multi-scale spectral inference module. This indicates the total number of layers in the spectral Chebyshev plot convolutional layer; , This represents the input features of the multi-scale spectral inference module; This indicates causal enhancement of local image patch features; Indicates the first The node feature matrix input to the layered graph inference module describes the process before... Semantic representation of each pathological image block after layer graph convolution propagation; The topological enhancement node features output by the spectral Chebyshev graph convolutional layer are represented by the fusion of local organizational semantics and global topological relationships by combining neighborhood node information in the aggregated graph structure with residual connections. go through After multi-scale spectral inference of the layered Chebyshev diagram convolutional layer, the data is obtained. Layer-level Chebyshev plot convolutional layer pairs input feature matrix The output of the input feature matrix is ​​used to further analyze the input feature matrix. The output is normalized to obtain graph enhancement features. : ;in, Presentation layer normalization operation; The segmentation prediction is as follows: Graph enhancement features are obtained through a segmentation decoder. Perform segmentation prediction to generate image block-level segmentation prediction results. : ;in, Indicates the segmentation decoder; The image block-level segmentation prediction results are restored to pixel-level segmentation prediction results through interpolation and upsampling, and these pixel-level segmentation prediction results are used as the segmentation prediction results corresponding to the pathological image to be segmented; wherein, the pixel-level segmentation prediction results ;in, This indicates an interpolation upsampling operation.

7. The pathological image segmentation and prediction method according to claim 1, characterized in that, This method also includes optimizing the segmentation prediction results: a weakly supervised segmentation loss and a causal orthogonal constraint loss are jointly used to optimize the segmentation prediction results corresponding to the pathological image to be segmented; specifically as follows: Introducing causal orthogonal constraint loss By minimizing the causal orthogonality constraint loss causal effect indicates Counterfactual response Pushed to the complementary feature subspace, the information overlap between the causal effect representation and the context representation is reduced, thereby further reducing contextual clutter, as shown in the following formula: ； in, This indicates the number of samples in the training batch. Indicates the first in the batch One sample; Indicates the first The causal effect is represented for each sample; Indicates the first Counterfactual responses of a sample; Represents the dot product of vectors; Represents the L2 norm, Represents absolute value; and Let L2 norms of the causal effect representation and counterfactual response be represented respectively. These are used to normalize the dot product of the causal effect representation and the counterfactual response, measure the cosine similarity between the causal effect representation and the counterfactual response, and thus constrain the correlation between the causal representation and the counterfactual representation. Overall loss function The formula is as follows: ； in, This indicates the loss from weakly supervised segmentation. This represents the loss due to causal orthogonal constraints. This represents the weighting coefficient used to balance the weakly supervised segmentation loss and the causal orthogonal constraint loss.

8. A pathological image segmentation and prediction system, characterized in that, This system is used to implement the pathological image segmentation and prediction method according to any one of claims 1 to 7; the system includes: The pathological image preprocessing unit is used to convert the two-dimensional spatial structure of the pathological image to be segmented into a sequence form and obtain the preprocessed pathological image to be segmented. The pathological image coding unit is used to extract patch-level features of local spatial information and image-level global semantic features from the preprocessed pathological image to be segmented by a visual Transformer image encoder. The causal counterfactual projection unit is used to map image-level global semantic features into query vectors and local spatial information patch-level features into key and value matrices. Based on the query vector, key and value matrices, it calculates the factual response using cross-attention and calculates the counterfactual response using the value matrix. The difference between the factual response and the counterfactual response is used as the causal effect representation of the target organization region. The causal feature reweighting unit is used to generate channel reweighted signals using causal effect representation, and to reweight local image patch features based on the channel reweighted signals to obtain causally enhanced local image patch features; The multi-scale topology graph construction unit is used to construct a multi-scale topology graph based on the spatial location of local image blocks in the preprocessed pathological image to be segmented, using causal enhanced local image block features as graph nodes and the correlation between local image block regions as edges. The multi-scale spectral inference unit is used to perform multi-scale spectral inference on multi-scale topological graphs using spectral Chebyshev graph convolution to obtain graph-enhanced features. The segmentation prediction unit is used to perform segmentation prediction on the image enhancement features through the segmentation decoder, generate image block-level segmentation prediction results, and restore the image block-level segmentation prediction results to pixel-level segmentation prediction results through interpolation upsampling. The pixel-level segmentation prediction results are used as the segmentation prediction results corresponding to the pathological image to be segmented.

9. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the pathological image segmentation and prediction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the pathological image segmentation and prediction method as described in any one of claims 1 to 7.

Images