A pathological image analysis method based on cross-scale space constraint fusion and local perception

Abstract

The application provides a pathological image analysis method based on cross-scale space constraint fusion and local perception, comprising the following steps: step 1, data preprocessing and feature extraction; step 2, performing a bidirectional fusion strategy based on space constraint to obtain fused enhanced features; step 3, using a Hadamard product-based bilinear fusion mode to capture high-order semantic information between scales; step 4, connecting the enhanced features and the fused features in residual to serve as the input of a 10-fold branch and a 20-fold branch; step 5, remodeling the features connected in residual into a two-dimensional shape and inputting the features into three convolution layers arranged in layers and expanded layer by layer; and step 6, further performing gated adaptive fusion on the global semantic features obtained after step 5 of the 10-fold branch and the 20-fold branch. On the basis of easy deployment, the application obtains a feature representation with high expression capacity, fully utilizes the information between different scale branches, and achieves better performance in downstream tasks.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a pathological image analysis method based on cross-scale spatial constraint fusion and local perception. Background Art

[0002] Pathological examination is widely considered the "gold standard" for clinical tumor diagnosis. Advances in whole-slide imaging technology have enabled pathologists to manually inspect tumors using whole-slide images (WSIs). However, because each WSI typically contains billions of pixels (e.g., 50,000 x 50,000), manual inspection is labor-intensive, time-consuming, costly, and error-prone. With the rapid development of whole-slide imaging and deep learning technologies, the potential for computer-assisted clinical diagnosis and digital pathology is rapidly growing. This development has made digital pathology a promising research field. However, using deep learning for WSI biopsy diagnosis faces numerous challenges, primarily due to the large amount of data and the lack of pixel-level annotations. To address this challenge, WSI analysis is formulated as a weakly supervised learning problem. However, existing methods typically focus solely on WSIs at a single resolution, neglecting important multi-resolution information. Multi-resolution WSI analysis has garnered increasing attention in recent years, but these methods fail to fully exploit the inherent connections between instances at different spatial scales, suffer from insufficient directionality, and fail to fully utilize local information in feature aggregation. Summary of the Invention

[0003] Purpose of the invention: The technical problem to be solved by the present invention is to address the deficiencies of the existing technology and provide a pathological image analysis method based on cross-scale spatial constraint fusion and local perception, which fully integrates the features between different scales and further introduces local information at the aggregation level for enhancement, thereby obtaining a more expressive feature representation and improving the stability, accuracy and efficiency of pathological image processing.

[0004] The method comprises the following steps:

[0005] Step 1: Data preprocessing and feature extraction: A universal self-supervised pathology basic model is used to perform non-overlapping sliding window feature extraction on pathology images at low and high resolutions, obtaining features and corresponding spatial indices. The area of ​​the high-resolution image unit is 1 / 4 of the low-resolution unit, satisfying the spatial mapping relationship. The low resolution refers to 10x magnification, approximately 1.0 μm / pixel, for evaluating the range of tumor infiltration; the high resolution refers to 20x magnification, approximately 0.5 μm / pixel, for evaluating cellular heterogeneity.

[0006] Step 2, a bidirectional fusion strategy based on spatial constraints is performed, and feature groups are constructed according to a spatial index mapping relationship between 10 times and 20 times, each feature group containing 10 times original features and corresponding 20 times original features; the 10 times original features and the corresponding 20 times original features after spatial constraints are subjected to bidirectional asymmetric fusion, in each feature group, bidirectional cross attention is performed to simulate top-down and bottom-up feature interaction, first, 10 times feature attention operation to 20 times feature, 10 times feature as query, 20 times feature as key and value, to obtain the fused enhanced feature X S* ; then, 20 times feature attention operation to 10 times feature, 20 times feature as query, 10 times feature as key and value, to obtain the fused enhanced feature X L* , since the number of 20 times features in the feature group is 4 times that of 10 times features, in order to ensure the consistency of feature dimensions in subsequent calculation, the fused enhanced feature X L* is subjected to average pooling;

[0007] Step 3, for the fused enhanced features X S* and X L* , a bilinear fusion method based on Hadamard product is used to further capture high-order semantic information between scales, first, the fused enhanced features X S* and X L* are respectively subjected to linear projection using a learnable linear layer, then, the two linearly projected features are multiplied by Hadamard product to obtain 10 times and 20 times fusion features;

[0008] Step 4, in order to fully fuse multi-resolution semantic information while maintaining the unique semantic information of single-resolution, the 10 times enhanced feature and the 20 times enhanced feature are respectively connected with the 10 times and 20 times fusion features, and the two residual connected features are respectively taken as the input of the 10 times branch and the 20 times branch for subsequent feature aggregation;

[0009] Step 5, the two residual connected features obtained in step 4 are reshaped from one-dimensional shape to two-dimensional shape, and input into three hierarchically arranged convolution layers, the first layer is pointwise convolution, the second layer and the third layer are depthwise separable convolution, and have gradually expanding receptive fields due to the increase of the convolution kernel size, the output of each layer of convolution comes from different receptive fields, then the query Q, the key K and the value V output by each layer are rearranged to form the final query Q, the key K and the value V containing multi-scale local prior knowledge, so as to introduce local information into the global information obtained by self-attention;

[0010] In step 6, the global semantic features obtained by the 10x branch and the 20x branch in step 5 are further gated and adaptively fused, which further enhances the semantic features of the WSI at different scales at the global level.

[0011] In step 1, pathology images are obtained from the public dataset CAMELYON 16, the BRACS dataset, and the private dataset CBCGA. The public dataset CAMELYON16 is constructed based on in-depth analysis of a large number of digital pathology slides. This dataset was jointly developed by the European Organization for Research and Treatment of Cancer (EORTC) and the Netherlands Cancer Institute (NKI) to provide a standardized training and testing platform for computer-aided diagnosis systems.

[0012] Each pathology image is annotated with the tumor area to ensure high data quality and accuracy;

[0013] The BRACS dataset contains annotated hematoxylin and eosin (H&E)-stained histopathology images. The dataset was created under an agreement between the Institute for High Performance Computing and Networking (ICAR) of the Italian National Research Council (CNR) and IBM Research Zurich. Each full-length slide and its corresponding region of interest were annotated by three expert pathologists.

[0014] The private dataset CBCGA was collected by the Shanghai Cancer Center of Fudan University and annotated by professional pathologists.

[0015] Adaptive threshold segmentation (Otsu method) is used to obtain the initial binary mask of the pathological image, and morphological post-processing such as inversion, hole filling, closing operation, and area threshold filtering is performed in sequence to achieve effective area segmentation of the pathological tissue image. Subsequently, a sliding window is used to extract non-overlapping fixed-size image blocks in the effective area, and the image blocks are input into the public pre-trained pathology basic model UNI for feature extraction to obtain low-resolution feature X S and high-resolution features X L , save the features and the spatial index of the features;

[0016] For the low-resolution feature X S Spatial index C S and high-resolution features X L Spatial index C L , the feature group is constructed by the following mapping function:

[0017] ,

[0018] Among them, Group x Represents the feature group after mapping. Mapping is the mapping function. By LScaled by the scaling factor γ, it is similar to the low-resolution spatial index C S Align, so that each set of low-resolution features X S Strictly aligned to a specific high-resolution feature X L .

[0019] In step 2, a bidirectional cross attention (CA) operation is used within the feature group to simulate top-down and bottom-up feature interactions;

[0020] Low-resolution features present coarse-grained semantic information at the tissue scale (such as blood vessels and glands), and find the high-resolution features it focuses on to integrate fine-grained spatial semantic information.

[0021] The low-resolution feature is used as the query Q, the high-resolution feature is used as the key K and the value V to perform the cross-attention operation, which is expressed as:

[0022] ,

[0023] Where softmax is a normalized exponential function; T represents transposition, X S represents the original 10-fold features, K Li Indicates that the ith 20-fold feature is used as a key, V Li represents the i-th 20-fold feature as the value; Q S Indicates that 10 times the features are used as queries, and d represents the dimension of each attention head of the query vector and key vector;

[0024] High-resolution features present fine-grained features at the cellular scale (e.g., nucleus and microenvironment). Within a feature group, it only focuses on the corresponding low-resolution features and uses high-resolution features as anchors to integrate coarse-grained spatial semantic information.

[0025] The high-resolution feature is used as the query Q, and the low-resolution feature is used as the key K and value V to perform the cross-attention operation. Due to the spatial index relationship between low-resolution and high-resolution features, the low-resolution features and high-resolution features have a one-to-many relationship. Specifically, a 10x feature corresponds to four 20x features.

[0026] In each feature group, the following operation is repeated four times: after each low-resolution feature is fused with the corresponding high-resolution feature, it is residually connected with the original feature to preserve the semantic information of the original feature. Then, the four enhanced features obtained are average pooled to obtain the final high-resolution enhanced feature in the group, which is expressed as:

[0027] ,

[0028] ,

[0029] where X Li represents the i-th 20-fold feature, the value range of i is {1,2,3,4}, Q Li Indicates that the 20th feature of i is used as the query, K S Indicates that 10 times the feature is used as the key, V S Indicates that 10 times the features are used as values.

[0030] In step 3, the obtained low-resolution enhanced feature X S* and high-resolution enhancement features X L* , processed using two learnable linear layers:

[0031] ,

[0032] ,

[0033] Among them, and are two learnable linear functions, and the low-resolution enhanced features X are obtained by learning two linear layers. S* The projection feature h S and high-resolution enhancement features X L* The projection feature h L ;

[0034] The Hadamard product between projected features is calculated using the following formula:

[0035] ,

[0036] in represents an element-wise multiplication operation, Indicates fusion features.

[0037] In step 4, in order to retain the semantic information of the enhanced features of each scale on the basis of introducing the fusion feature, the residual connection is introduced to With 10 times enhanced feature X S* and 20 times enhanced feature X L* Add together, the formula is:

[0038] ,

[0039] ,

[0040] in represents 10-fold branch features, Indicates 20-fold branch features.

[0041] In step 5, the 10-fold branch feature Z obtained in step 4 isS and 20-fold branch feature Z L Perform Transformer-based aggregation and establish a local perception enhancement module, which is used to explicitly model local spatial prior knowledge before global aggregation. The local perception enhancement module uses three layered convolution kernels. The first layer uses point-by-point convolution with minimum receptive field, and the second and third layers use depthwise separable convolution. The second layer consists of depthwise convolution with a kernel of 3×3 and point-by-point convolution, and the third layer consists of depthwise convolution with a kernel of 5×5 and point-by-point convolution. The output features of each layer represent local information collected from different receptive fields. The features obtained in each layer are then rearranged to obtain the final query Q, key K, and value V.

[0042] Due to the variation in slide and tissue area, the number of patch tokens in the corresponding sequence is X p_token The number will also change, and the absolute position information is provided to the convolution by zero padding; then the patch token X in the sequence p_token ∈R N×D Rearrange into , where N represents the sequence length, represents the side length of the rearranged two-dimensional rectangle, D represents the feature dimension, X p represents the rearranged patch tokens; R represents the real number space;

[0043] X p Through three layers of convolutional layers with gradually increasing receptive fields:

[0044] ,

[0045] , ,

[0046] , ,

[0047] Among them, PWConv1, PWConv2, and PWConv3 represent point-by-point convolutions in three layers respectively, DWConv2 represents a depth convolution with a kernel of 3×3, and DWConv3 represents a depth convolution with a kernel of 5×5. Represents the features after the first layer of point-by-point convolution, Indicates F p1 Features after the second layer of deep convolution;

[0048] Indicates F p2 After the second layer of point-by-point convolution, the features represents F* p2 After the third layer of deep convolution, the features Indicates F p3 Features after the third layer of point-by-point convolution; F p1 、F* p2 and F* p3 Represents the local features of three different scales obtained. The features are divided into three parts in the feature dimension and rearranged to form the query Q p , key K p Sum :

[0049] ,

[0050] ,

[0051] ,

[0052] ,

[0053] in represents the output feature of the i-th convolutional layer, The query Q represents the features output by the i-th convolutional layer, The key K represents the feature output of the i-th convolutional layer, Represents the value V of the feature output of the i-th convolutional layer; Concat represents the connection operation;

[0054] Then, query Q p , key K p Sum Expand in the spatial dimension to form the final Q p ,K p ,V p ∈R N×D ;

[0055] The local perception enhancement module can process various auxiliary tokens, including class tokens or distilled tokens, and convert class tokens X c ∈R 1×D After three layers of point-by-point convolution, local information at multiple scales is obtained. The formula is:

[0056] ,

[0057] ,

[0058] ,

[0059] Among them F c1 ,F c2 ,F c3 ∈R 1×D Represents X cAfter the linear projection of the output of the first layer of point-by-point convolution layer, the linear projection of the output of the second layer of point-by-point convolution layer, and the linear projection of the output of the third layer of point-by-point convolution layer, the output F c1 ,F c2 ,F c3 The feature dimension is divided into three parts and rearranged. The formula is:

[0060] ,

[0061] ,

[0062] ,

[0063] ,

[0064] in represents the linear projection of the output of the i-th point-by-point convolutional layer, The query Q represents the features output by the i-th point-wise convolutional layer, The key K representing the feature output of the i-th point-by-point convolutional layer, The value V representing the feature output by the i-th point-by-point convolutional layer;

[0065] Then, rearrange to form query Q c , key K c Sum V c ∈R 1×D .

[0066] Step 5 also includes: Q P ,K P ,V P ∈R N×D and Q C ,K C V C ∈R 1×D By making connections and constructing the complete input of the attention mechanism, including local features from different receptive fields, the final Q, K, V becomes:

[0067] , , ,

[0068] where Q,K,V∈R (N+1)×D ; 10-fold branch and 20-fold branch respectively obtained highly expressive class token C 10 and C 20 , C 10 ∈R 1×D , C 20 ∈R 1×D .

[0069] Step 6 includes: using gated adaptive fusion mechanism to combine C 10 、C 20 Integrate into the final decision representation: First, C 10 、C 20 Connect into a joint representation C concat ∈R 1×2D , will jointly represent C concat Feed it to a linear gating function GatedNet to generate a fusion weight vector g=[g1,g2], where g1 and g2 are the gating weights output by the linear gating function GatedNet, which are used to adjust the 10-fold branch features in the final fusion feature C final The ratio and 20 times branch features in the final fusion feature C final The final fusion feature is obtained by weighted summation, and then the prediction result is obtained by multi-layer perceptron MLP and softmax activation function. The formula is:

[0070] ,

[0071] ,

[0072] ,

[0073] ,

[0074] Using the cross entropy loss function L, the formula is:

[0075] ,

[0076] where Y ij It is a label at the pathological tissue section level. is the corresponding prediction result, M is the number of patients, and C is the number of categories.

[0077] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the method.

[0078] The present invention also provides a storage medium storing a computer program or instruction, which executes the steps of the method when the computer program or instruction is run on a computer.

[0079] Compared with the existing technology, the present invention has the following beneficial effects: (1) The present invention utilizes a spatially constrained bidirectional fusion strategy to enhance the directionality and sensitivity of feature fusion, avoids spatial confusion of features, effectively fuses features at different scales, and removes interference from other features.

[0080] (2) By further performing bilinear Hadamard product fusion on the bidirectionally fused features, high-order semantic information at two scales is further captured to guide the aggregation of subsequent features.

[0081] (3) By introducing the local perception enhancement module, a multi-layer lightweight convolutional structure is used to obtain local information under different receptive fields, and the global aggregation of the Transformer is integrated to obtain a feature representation that combines local and global information.

[0082] (4) The proposed method achieves competitive performance on camelyon16, BRACS, and CBCGA. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 This is the overall framework diagram of the method of the present invention.

[0084] Figure 2 This is a diagram of the spatially constrained bidirectional cross-attention structure of the method of the present invention.

[0085] Figure 3 This is a structural diagram of the local perception enhancement module of the method of the present invention.

[0086] Figure 4 Visualization of attention between multi-scale features in the spatially constrained bidirectional cross-attention of our method.

[0087] Figure 5 This is the prediction heat map of the method of the present invention. DETAILED DESCRIPTION

[0088] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more apparent.

[0089] An embodiment of the present invention provides a pathological image analysis method based on cross-scale spatial constraint fusion and local perception, comprising the following steps:

[0090] Step 1: Data preprocessing and feature extraction: A universal self-supervised pathology basic model is used to perform non-overlapping sliding window feature extraction on pathology images at low and high resolutions, obtaining features and corresponding spatial indices. The area of ​​the high-resolution image unit is 1 / 4 of the low-resolution unit, satisfying the spatial mapping relationship. The low resolution refers to 10x magnification, approximately 1.0 μm / pixel, for evaluating the range of tumor infiltration; the high resolution refers to 20x magnification, approximately 0.5 μm / pixel, for evaluating cellular heterogeneity.

[0091] Step 2: Execute a bidirectional fusion strategy based on spatial constraints. Construct feature groups based on the spatial index mapping relationship between 10x and 20x. Each feature group contains 10x original features and corresponding 20x original features. Perform bidirectional asymmetric fusion on the 10x original features and corresponding 20x original features after spatial constraints. In each feature group, perform bidirectional cross attention to simulate top-down and bottom-up feature interactions. First, perform an attention operation from 10x features to 20x features. The 10x features are used as queries and the 20x features are used as keys and values ​​to obtain the fused enhanced features X. S* ; Then the attention operation from 20 times the feature to 10 times the feature is performed, the 20 times the feature is used as the query, and the 10 times the feature is used as the key and value to obtain the fused enhanced feature X L* , since the number of 20-fold features in the feature group is 4 times that of 10-fold features, in order to ensure the consistency of feature dimensions in subsequent calculations, the fused enhanced features X L* Perform average pooling;

[0092] Step 3: For the fused enhanced feature X S* and X L* , a bilinear fusion method based on Hadamard product is used to further capture the high-order semantic information between scales. First, the fused enhanced features X S* and X L* Use a learnable linear layer to perform linear projection, and then multiply the two linearly projected features by Hadamard product to obtain 10 times and 20 times fused features;

[0093] Step 4: In order to fully fuse the multi-resolution semantic information while maintaining the unique semantic information of a single resolution, the 10x enhanced features and 20x enhanced features are residually connected with the 10x and 20x fused features respectively. The two residually connected features are used as the input of the 10x branch and the 20x branch respectively for subsequent feature aggregation;

[0094] Step 5: Reshape the features after the two residual connections obtained in step 4 from a one-dimensional shape to a two-dimensional shape, and input three hierarchically arranged convolution layers. The first layer is point-by-point convolution, and the second and third layers are depth-separable convolutions. Due to the increase in the size of the convolution kernel, the receptive field is gradually expanded. Each layer of convolution outputs information from a different receptive field. Then, the query Q, key K, and value V output by each layer are rearranged to form the final query Q, key K, and value V containing multi-scale local prior knowledge, thereby introducing local information into the global information obtained by self-attention.

[0095] Step 6. The global semantic features obtained after step 5 for 10x branch and 20x branch are further fused by gated adaptive fusion, which again enhances the semantic features of WSI after different scale fusion at the global level.

[0096] In step 1, pathological images are obtained from public datasets CAMELYON 16, BRACS dataset and private dataset CBCGA. The public dataset CAMELYON 16 is constructed based on deep analysis of a large number of digitized pathological sections, which is jointly developed by the European Organization for Research and Treatment of Cancer (EORTC) and the Netherlands Cancer Institute (NKI), aiming to provide a standardized training and testing platform for computer-aided diagnosis systems.

[0097] Each pathological image is labeled with tumor regions, ensuring high quality and accuracy of the data.

[0098] The BRACS dataset contains annotated hematoxylin and eosin (H&E) stained histopathological images. The BRACS dataset is established based on the agreement between the High Performance Computing and Networking Institute (ICAR) under the Italian National Research Council (CNR) and IBM Zurich Research Institute. Each whole section and its corresponding region of interest is annotated by three professional pathologists.

[0099] The private dataset CBCGA is collected by Fudan University Shanghai Cancer Center and annotated by professional pathologists.

[0100] Adaptive threshold segmentation (Otsu method) is used to obtain the initial binary mask of the pathological image, and morphological post-processing such as inversion, hole filling, closing operation and area threshold filtering is sequentially performed to realize effective region segmentation of the pathological tissue image. Then, non-overlapping fixed-size image blocks are extracted by sliding window in the effective region. The image blocks are input into the public pre-trained pathological base model UNI for feature extraction to obtain low-resolution features X S and high-resolution features X L The features and spatial indexes of the features are saved.

[0101] For the spatial index C S of the low-resolution features X S and the spatial index C L of the high-resolution features X L , the feature group is constructed by the following mapping function:

[0102] ,

[0103] where Group x represents the mapped feature group, Mapping is the mapping function, and the spatial index C Lscaled by a scale factor γ, thus and low resolution spatial index C S aligned, thus ensuring that each group of low resolution features X S strictly aligned to a specific high resolution feature X L .

[0104] In step 2, within each feature group, a bidirectional cross attention (CA) operation is used to simulate top-down and bottom-up feature interactions.

[0105] Low resolution features present coarse-grained semantic information of organizational scales (e.g., blood vessels and glands), find the high resolution features it is interested in to integrate fine-grained spatial semantic information.

[0106] Perform cross attention operation with low resolution features as query Q, high resolution features as key K and value V, denoted as:

[0107] ,

[0108] where softmax is the normalized exponential function; T represents transpose, X S represents the original 10x feature, K Li represents the i-th 20x feature as key, V Li represents the i-th 20x feature as value; Q S represents the 10x feature as query, d represents the dimension of each attention head of the query vector and the key vector;

[0109] High resolution features present fine-grained features of cellular scales (e.g., cell nuclei and microenvironments), within a feature group it only focuses on the corresponding low resolution feature, take high resolution features as anchor points to integrate coarse-grained spatial semantic information.

[0110] Perform cross attention operation with high resolution features as query Q, low resolution features as key K and value V, due to the spatial index relationship between low resolution and high resolution features, low resolution features and high resolution features are one-to-many relationship, which is specifically manifested as: 10x feature corresponds to 4 20x features.

[0111] Within each feature group, repeat the following operation four times: after each low resolution feature is fused with the corresponding high resolution feature, it will be connected with the original feature in residual connection to maintain the semantic information of the original feature, then the obtained four enhanced features are averaged and pooled to obtain the final high resolution enhanced feature within the group, denoted as:

[0112] ,

[0113] ,

[0114] where X Li represents the i-th 20-fold feature, the value range of i is {1,2,3,4}, Q Li Indicates that the 20th feature of i is used as the query, K S Indicates that 10 times the feature is used as the key, V S Indicates that 10 times the features are used as values.

[0115] In step 3, the obtained low-resolution enhanced feature X S* and high-resolution enhancement features X L* , processed using two learnable linear layers:

[0116] ,

[0117] ,

[0118] Among them, and are two learnable linear functions, and the low-resolution enhanced features X are obtained by learning two linear layers. S* The projection feature h S and high-resolution enhancement features X L* The projection feature h L ;

[0119] The Hadamard product between projected features is calculated using the following formula:

[0120] ,

[0121] in represents an element-wise multiplication operation, Indicates fusion features.

[0122] In step 4, in order to retain the semantic information of the enhanced features of each scale on the basis of introducing the fusion feature, the residual connection is introduced to With 10 times enhanced feature X S* and 20 times enhanced feature X L* Add together, the formula is:

[0123] ,

[0124] ,

[0125] in represents 10-fold branch features, Indicates 20-fold branch features.

[0126] In step 5, the 10-fold branch feature Z obtained in step 4 isS and 20-fold branch feature Z L Perform Transformer-based aggregation and establish a local perception enhancement module, which is used to explicitly model local spatial prior knowledge before global aggregation. The local perception enhancement module uses three layered convolution kernels. The first layer uses point-by-point convolution with minimum receptive field, and the second and third layers use depthwise separable convolution. The second layer consists of depthwise convolution with a kernel of 3×3 and point-by-point convolution, and the third layer consists of depthwise convolution with a kernel of 5×5 and point-by-point convolution. The output features of each layer represent local information collected from different receptive fields. The features obtained in each layer are then rearranged to obtain the final query Q, key K, and value V.

[0127] Due to the variation in slide and tissue area, the number of patch tokens in the corresponding sequence is X p_token The number will also change, and the absolute position information is provided to the convolution by zero padding; then the patch token X in the sequence p_token ∈R N×D Rearrange into , where N represents the sequence length, represents the side length of the rearranged two-dimensional rectangle, D represents the feature dimension, X p represents the rearranged patch tokens; R represents the real number space;

[0128] X p Through three layers of convolutional layers with gradually increasing receptive fields:

[0129] ,

[0130] , ,

[0131] , ,

[0132] Among them, PWConv1, PWConv2, and PWConv3 represent point-by-point convolutions in three layers respectively, DWConv2 represents a depth convolution with a kernel of 3×3, and DWConv3 represents a depth convolution with a kernel of 5×5. Represents the features after the first layer of point-by-point convolution, Indicates F p1 Features after the second layer of deep convolution;

[0133] Indicates F p2 After the second layer of point-by-point convolution, the features represents F* p2 After the third layer of deep convolution, the features Indicates F p3 Features after the third layer of point-by-point convolution; F p1 、F* p2 and F* p3 Represents the local features of three different scales obtained. The features are divided into three parts in the feature dimension and rearranged to form the query Q p , key K p Sum :

[0134] ,

[0135] ,

[0136] ,

[0137] ,

[0138] in represents the output feature of the i-th convolutional layer, The query Q represents the features output by the i-th convolutional layer, The key K represents the feature output of the i-th convolutional layer, Represents the value V of the feature output of the i-th convolutional layer; Concat represents the connection operation;

[0139] Then, query Q p , key K p Sum Expand in the spatial dimension to form the final Q p ,K p ,V p ∈R N×D ;

[0140] The local perception enhancement module can process various auxiliary tokens, including class tokens or distilled tokens, and convert class tokens X c ∈R 1×D After three layers of point-by-point convolution, local information at multiple scales is obtained. The formula is:

[0141] ,

[0142] ,

[0143] ,

[0144] Among them F c1 ,F c2 ,F c3 ∈R 1×D Represents X cAfter the linear projection of the output of the first layer of point-by-point convolution layer, the linear projection of the output of the second layer of point-by-point convolution layer, and the linear projection of the output of the third layer of point-by-point convolution layer, the output F c1 ,F c2 ,F c3 The feature dimension is divided into three parts and rearranged. The formula is:

[0145] ,

[0146] ,

[0147] ,

[0148] ,

[0149] in represents the linear projection of the output of the i-th point-by-point convolutional layer, The query Q represents the features output by the i-th point-wise convolutional layer, The key K representing the feature output of the i-th point-by-point convolutional layer, The value V representing the feature output by the i-th point-by-point convolutional layer;

[0150] Then, rearrange to form query Q c , key K c Sum V c ∈R 1×D .

[0151] Step 5 also includes: Q P ,K P ,V P ∈R N×D and Q C ,K C V C ∈R 1×D By making connections and constructing the complete input of the attention mechanism, including local features from different receptive fields, the final Q, K, V becomes:

[0152] , , ,

[0153] where Q,K,V∈R (N+1)×D ; 10-fold branch and 20-fold branch respectively obtained highly expressive class token C 10 and C 20 , C 10 ∈R 1×D , C 20 ∈R 1×D .

[0154] Step 6 includes: using a gated adaptive fusion mechanism to integrate C 10 , C 20 into the final decision representation: first, C 10 , C 20 are connected into joint representation C concat ∈R 1×2D , the joint representation C concat is fed into a linear gating function GatedNet to generate a fusion weight vector g = [g1, g2], where g1 and g2 are gating weights output by the linear gating function GatedNet, respectively used to adjust the proportion of 10 times branch features in the final fusion feature C final and the proportion of 20 times branch features in the final fusion feature C final , and the final fusion feature is obtained by weighted summation, and then passes through a multi-layer perception MLP and a softmax activation function to obtain a prediction result, the formula is:

[0155] ,

[0156] ,

[0157] ,

[0158] ,

[0159] The cross-entropy loss function L is used, and the formula is:

[0160] ,

[0161] Where Y ij is the label of the pathological tissue section level, is the corresponding prediction result, M is the number of patients, and C is the number of categories.

[0162] In one specific embodiment of the application, the method comprises the following steps:

[0163] Step 1: This invention utilizes three datasets: the public dataset CAMELYON 16, the BRACS dataset, and the private dataset CBCGA. The CAMELYON 16 dataset, jointly developed by the European Organization for Research and Treatment of Cancer (EORTC) and the Netherlands Cancer Institute (NKI), is constructed based on an in-depth analysis of a large number of digitized pathology slides. The dataset aims to provide a standardized training and testing platform for computer-aided diagnosis systems. Tumor regions are annotated in each image to ensure high data quality and accuracy. The BRACS dataset contains annotated hematoxylin and eosin (H&E)-stained histopathology images. BRACS was developed under an agreement between the Institute for High Performance Computing and Networking (ICAR) under the Italian National Research Council (CNR) and IBM Research Zurich. Each full-length slide and its corresponding region of interest were annotated by three professional pathologists. The CBCGA dataset was collected by the Shanghai Cancer Center of Fudan University and annotated by professional pathologists. CAMELYON 16 contains 399 slides, BRACS contains 547 slides, and CBCGA contains 682 slides.

[0164] like Figure 1 As shown in the data preprocessing and feature extraction in

[15] , the data is preprocessed. First, adaptive threshold segmentation (Otsu method) is used to obtain the initial binary mask. Then, morphological post-processing is performed through inversion, hole filling, closing operation, and area threshold filtering to achieve effective area segmentation of the pathological tissue image. Subsequently, non-overlapping image blocks of a fixed size of 512×512 pixels are extracted in the effective area using 10x (approximately 1.0μm per pixel) and 20x (approximately 0.5μm per pixel) sliding windows, respectively. At the same time, corresponding indexes are formulated to meet the spatial constraints between 10x and 20x. The image blocks are input into the pre-trained pathology basic model UNI to extract deep features. The features and corresponding indexes are saved, and the index relationship is used for feature alignment for subsequent feature fusion. The spatial index relationship is as follows:

[0165] ,

[0166] Among them, X S Indicates 10-fold features, C S Represents the spatial index of the corresponding feature, X L Indicates 20 times the feature, C L Indicates the spatial index of the corresponding feature, Group x It represents the feature group after index relationship mapping, and Mapping represents the mapping function.

[0167] Step 2: The present invention designs a bidirectional fusion strategy based on spatial constraints, such as Figure 1In each feature group, spatially constrained bidirectional cross attention is used for feature fusion. The specific fusion process is as follows: Figure 2 As shown. The 10-fold feature f S and 20 times the feature f L1 ,f L2 ,f L3 ,f L4 As query q, another scale feature as key k and query v, the fused 10x enhanced feature f is obtained L->S and 20 times enhanced feature f S->L1 ,f S->L2 ,f S->L3 ,f S->L4 .

[0168] Step 3: For the enhanced features obtained in step 2, Figure 1 As shown in FIG, the 20 times enhanced features are average pooled to align them with the 10 times enhanced features in dimension for subsequent further fusion operations.

[0169] Step 4: For the 10-fold and 20-fold enhanced features obtained in step 3, two learnable linear layers con1×1 are used for projection respectively, and the projections are further subjected to high-order semantic interaction using Hadamard product to obtain high-order fusion features f fusion In order to retain the semantic information of 10 times and 20 times, the high-order fusion features and enhanced features are residually connected to obtain Z S and Z L , as features for further aggregation.

[0170] Step 5: For the 10-fold branch feature Z obtained in step 4 S and 20-fold branch feature Z L ,like Figure 1 As shown in FIG, a local perception enhancement module LPGM proposed in the present invention is used to perform aggregation based on the Transformer model. Figure 3As shown in the figure, a global learning parameter classification token cls is introduced to reshape the features from one-dimensional features into two-dimensional features. Three consecutive convolutional layers are used on the obtained two-dimensional features. conv1 is the first convolutional layer, which contains a 1×1 point-by-point convolution and outputs the first-layer query q1, key k1, and value v1; conv2 is the second convolutional layer, which is a depth-wise separable convolution and contains a 3×3 depth-wise convolution and a 1×1 point-by-point convolution. The output is the first-layer query q2, key k2, and value v2; conv3 is the third convolutional layer, also a depth-wise separable convolution, which contains a 5×5 depth-wise convolution and a 1×1 point-by-point convolution. The output is the first-layer query q3, key k3, and value v3. The query, key, and value obtained at each layer are rearranged to obtain the final query Q, key K, and value V. Then, the global features are obtained through multi-head attention MSA and multi-layer perceptron MLP. The global features of the 10-fold branch and the global features of the 20-fold branch are gated and adaptively fused to obtain the final representation.

[0171] The learning rate used during training is 2×10 −4 Adam optimizer with weight decay of 1×10 −5 The cosine annealing scheduler is used. The cross entropy loss function is used to evaluate the model and calculate the model loss. The model network parameters are optimized to obtain the trained case image processing model.

[0172] In order to evaluate the robustness of the model, 5-fold cross validation was used to evaluate the model effect. The present invention uses the following models as baselines to compare the model performance.

[0173] (1) ABMIL: This baseline uses an attention mechanism to calculate the attention score of each image patch and uses a simple weighted pooling operation to generate bag-level representations. (2) CLAM_SB: It enhances the local important areas of the image by introducing a class activation map (CAM) and strengthens the information interaction between image patches of different resolutions through a selective branching mechanism. (3) CLAM_MB: Based on CLAM_SB, it adds a multi-branch mechanism to process image patches from different resolutions in parallel, and optimizes the fusion of cross-scale features through a multi-branch approach. (4) Trans-MIL: It uses the self-attention mechanism of the Transformer architecture to capture the long-range dependencies between image patches. In this way, it can effectively model complex spatial relationships and adapt to the diversity and heterogeneity in pathological images. (5) DTFT-MIL: It extracts features from image patches of different resolutions through a two-layer feature distillation mechanism and fuses high- and low-level features to optimize the discriminative ability of the model. (6) IBMIL: A multi-instance learning framework based on causal intervention aims to eliminate contextual bias in pathological images through backdoor adjustment and achieve more robust and de-obfuscated bag-level classification. (7) MHIM-MIL: A multi-instance learning method that improves model robustness through a multi-head instance masking mechanism. (8) DS-MIL: A two-stream architecture is used, in which one stream uses maximum pooling to identify key instances, and the other stream calculates the distance between each instance and the key instance to evaluate the attention score.

[0174] Table 1

[0175]

[0176] As shown in Table 1, the model performance is evaluated using three indicators: accuracy (acc), area under the receiver operating characteristic curve (auc), and F1 score. The method proposed in this invention outperforms all the compared baseline models on the three datasets, thus verifying the effectiveness of the present invention.

[0177] Step 6, in order to provide interpretability of the spatially constrained bidirectional cross-attention module, as Figure 4 As shown, two image patches were selected from the boundary region between tumor and non-tumor areas, as well as two image patches from the interior of the tumor. For the boundary region, the attention distribution between the 10x and 20x magnification image patches is more uneven. Attention values ​​are higher for larger tumor areas, and lower for larger tumor areas. This indicates that the fusion module prioritizes tumor areas within the boundary region, effectively capturing key information. For the interior region, the attention values ​​between the 10x and 20x magnifications are more evenly distributed, indicating that each 10x magnification feature fully utilizes the 20x magnification feature information. This demonstrates that the fusion module effectively learns information between the two scales.

[0178] Step 7, in order to provide the interpretability of the overall model, such as Figure 5 Shown are two tumor histopathology images annotated by pathologists, along with their corresponding heatmaps and high-attention patches. The heatmaps show a high degree of consistency between the model's focus and the pathologists' detailed annotations, consistently focusing on key areas and enhancing its diagnostic capabilities.

[0179] The present invention provides a pathological image analysis method based on cross-scale spatial constraint fusion and local perception. There are many methods and approaches to implement this technical solution. The above is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention. Such improvements and modifications should also be considered within the scope of protection of the present invention. Any components not specified in this embodiment may be implemented using existing technologies.

Claims

1. A pathological image analysis method based on cross-scale spatial constraint fusion and local perception, characterized in that: The following steps are involved: Step 1: Data preprocessing and feature extraction: Using a self-supervised pathology foundation model, non-overlapping sliding window feature extraction is performed on pathology images at both low and high resolutions to obtain features and corresponding spatial indices. The area of ​​a high-resolution image unit is 1 / 4 of that of a low-resolution unit, satisfying the spatial mapping relationship. The low resolution refers to a 10x magnification for assessing the extent of tumor infiltration; the high resolution refers to a 20x magnification for assessing cellular heterogeneity. Step 2: Execute a bidirectional fusion strategy based on spatial constraints. Construct feature groups based on the spatial index mapping relationship between 10x and 20x. Each feature group contains 10x original features and corresponding 20x original features. Perform bidirectional asymmetric fusion on the 10x original features and corresponding 20x original features after spatial constraints. In each feature group, perform bidirectional cross attention to simulate top-down and bottom-up feature interactions. First, perform an attention operation from 10x features to 20x features. The 10x features are used as queries and the 20x features are used as keys and values ​​to obtain the fused enhanced features X. S* ; Then the attention operation from 20 times the feature to 10 times the feature is performed, the 20 times the feature is used as the query, and the 10 times the feature is used as the key and value to obtain the fused enhanced feature X L* , the fused enhanced features X L* Perform average pooling; Step 3: For the fused enhanced feature X S* and X L* , a bilinear fusion method based on Hadamard product is used to further capture the high-order semantic information between scales. First, the fused enhanced features X S* and X L* Use a learnable linear layer to perform linear projection, and then multiply the two linearly projected features by Hadamard product to obtain 10 times and 20 times fused features; Step 4: Perform residual connections on the 10-fold enhanced features and the 20-fold enhanced features with the 10-fold and 20-fold fusion features, respectively. The two residual-connected features are used as the input of the 10-fold branch and the 20-fold branch, respectively, for subsequent feature aggregation. Step 5: Reshape the features after the two residual connections obtained in step 4 from a one-dimensional shape to a two-dimensional shape, and input three hierarchical convolution layers. The first layer is point-by-point convolution, and the second and third layers are depth-separable convolution. Each convolution layer outputs information from different receptive fields. Then, the query Q, key K, and value V output by each layer are rearranged to form the final query Q, key K, and value V containing multi-scale local prior knowledge, thereby introducing local information into the global information obtained through self-attention. In step 6, the global semantic features obtained by the 10-fold branch and the 20-fold branch in step 5 are further gated and adaptively fused.

2. The method according to claim 1, characterized in that In step 1, pathological images are obtained from the public dataset CAMELYON 16, BRACS dataset, and private dataset CBCGA; Each pathology image was annotated with the tumor area; Adaptive threshold segmentation is used to obtain the initial binary mask of the pathological image, and morphological post-processing such as inversion, hole filling, closing operation, and area threshold filtering is performed in sequence to achieve effective area segmentation of the pathological tissue image. Subsequently, a sliding window is used to extract non-overlapping fixed-size image blocks in the effective area, and the image blocks are input into the pre-trained pathology basic model UNI for feature extraction to obtain low-resolution feature X S and high-resolution features X L , save the features and the spatial index of the features; For the low-resolution feature X S Spatial index C S and high-resolution features X L Spatial index C L , the feature group is constructed by the following mapping function: ， Among them, Group x Represents the feature group after mapping. Mapping is the mapping function. By L Scaled by the scaling factor γ, it is similar to the low-resolution spatial index C S Align.

3. The method according to claim 2, characterized in that In step 2, a bidirectional cross attention operation is used within the feature group to simulate top-down and bottom-up feature interactions; The low-resolution feature is used as the query Q, the high-resolution feature is used as the key K and the value V to perform the cross-attention operation, which is expressed as: ， Where softmax is a normalized exponential function; T represents transposition, X S represents the original 10-fold features, K Li Indicates that the ith 20-fold feature is used as a key, V Li represents the i-th 20-fold feature as the value; Q S Indicates that 10 times the features are used as queries, and d represents the dimension of each attention head of the query vector and key vector; Use high-resolution features as query Q, low-resolution features as key K and value V to perform cross-attention operation. The low-resolution features and high-resolution features have a one-to-many relationship, specifically: 10x features correspond to 4 20x features; In each feature group, the following operation is repeated four times: after each low-resolution feature is fused with the corresponding high-resolution feature, it is residually connected with the original feature to preserve the semantic information of the original feature. Then, the four enhanced features obtained are average pooled to obtain the final high-resolution enhanced feature in the group, which is expressed as: ， ， where X Li represents the i-th 20-fold feature, the value range of i is {1,2,3,4}, Q Li Indicates that the 20th feature of i is used as the query, K S Indicates that 10 times the feature is used as the key, V S Indicates that 10 times the features are used as values.

4. The method according to claim 3, characterized in that In step 3, the obtained low-resolution enhanced feature X S* and high-resolution enhancement features X L* , processed using two learnable linear layers: ， ， Among them, and are two learnable linear functions, and the low-resolution enhanced features X are obtained by learning two linear layers. S* The projection feature h S and high-resolution enhancement features X L* The projection feature h L ; The Hadamard product between projected features is calculated using the following formula: ， in represents the element-wise multiplication operation, Indicates fusion features.

5. The method according to claim 4, characterized in that In step 4, residual connection is introduced to fuse features With 10 times enhanced feature X S* and 20 times enhanced feature X L* Add together, the formula is: ， ， in represents 10-fold branch features, Indicates 20-fold branch features.

6. The method according to claim 5, characterized in that In step 5, the 10-fold branch feature Z obtained in step 4 is S and 20-fold branch feature Z L Perform Transformer-based aggregation and establish a local perception enhancement module, which is used to explicitly model local spatial prior knowledge before global aggregation. The local perception enhancement module uses three layered convolution kernels. The first layer uses point-by-point convolution with minimum receptive field, and the second and third layers use depthwise separable convolution. The second layer consists of depthwise convolution with a kernel of 3×3 and point-by-point convolution, and the third layer consists of depthwise convolution with a kernel of 5×5 and point-by-point convolution. The output features of each layer represent local information collected from different receptive fields. The features obtained in each layer are then rearranged to obtain the final query Q, key K, and value V. The patch token X in the sequence p_token ∈R N×D Rearrange into , where N represents the sequence length, represents the side length of the rearranged two-dimensional rectangle, D represents the feature dimension, X p represents the rearranged patch tokens; R represents the real number space; X p Through three layers of convolutional layers with gradually increasing receptive fields: ， ， ， ， ， Among them, PWConv1, PWConv2, and PWConv3 represent point-by-point convolutions in three layers respectively, DWConv2 represents a depth convolution with a kernel of 3×3, and DWConv3 represents a depth convolution with a kernel of 5×5. Represents the features after the first layer of point-by-point convolution, Indicates F p1 Features after the second layer of deep convolution; Indicates F p2 After the second layer of point-by-point convolution, the features represents F* p2 After the third layer of deep convolution, the features Indicates F p3 Features after the third layer of point-by-point convolution; F p1 、F* p2 and F* p3 Represents the local features of three different scales obtained. The features are divided into three parts in the feature dimension and rearranged to form the query Q p , key K p Sum : ， ， ， ， in represents the output feature of the i-th convolutional layer, The query Q represents the features output by the i-th convolutional layer, The key K represents the feature output of the i-th convolutional layer, Represents the value V of the feature output of the i-th convolutional layer; Concat represents the connection operation; Then, query Q p , key K p Sum Expand in the spatial dimension to form the final Q p ,K p ,V p ∈R N ×D ; The local perception enhancement module can process various auxiliary tokens, including class tokens or distilled tokens, and convert class tokens X c ∈R 1×D After three layers of point-by-point convolution, local information at multiple scales is obtained. The formula is: ， ， ， Among them F c1 ,F c2 ,F c3 ∈R 1×D Represents X c After the linear projection of the output of the first layer of point-by-point convolution layer, the linear projection of the output of the second layer of point-by-point convolution layer, and the linear projection of the output of the third layer of point-by-point convolution layer, the output F c1 ,F c2 ,F c3 The feature dimension is divided into three parts and rearranged. The formula is: ， ， ， ， in represents the linear projection of the output of the i-th point-by-point convolutional layer, The query Q represents the features output by the i-th point-wise convolutional layer, The key K representing the feature output of the i-th point-by-point convolutional layer, The value V representing the feature output by the i-th point-by-point convolutional layer; Then, rearrange to form query Q c , key K c Sum V c ∈R 1×D .

7. The method according to claim 6, characterized in that Step 5 also includes: P ,K P ,V P ∈R N×D and Q C ,K C V C ∈R 1×D By making connections and constructing the complete input of the attention mechanism, including local features from different receptive fields, the final Q, K, V becomes: ， ， ， where Q,K,V∈R (N+1)×D ; 10-fold branch and 20-fold branch respectively obtained highly expressive class token C 10 and C 20 , C 10 ∈R 1×D , C 20 ∈R 1×D .

8. The method according to claim 7, characterized in that Step 6 includes: using gated adaptive fusion mechanism to combine C 10 、C 20 Integrate into the final decision representation: First, C 10 、C 20 Connect into a joint representation C concat ∈R 1×2D , will jointly represent C concat Feed it to a linear gating function GatedNet to generate a fusion weight vector g=[g1,g2], where g1 and g2 are the gating weights output by the linear gating function GatedNet, which are used to adjust the 10-fold branch features in the final fusion feature C final The ratio and 20 times branch features in the final fusion feature C final The final fusion feature is obtained by weighted summation, and then the prediction result is obtained by multi-layer perceptron MLP and softmax activation function. The formula is: ， ， ， ， Using the cross entropy loss function L, the formula is: ， where Y ij It is a label at the pathological tissue section level. is the corresponding prediction result, M is the number of patients, and C is the number of categories.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 8.

10. A storage medium, characterized in that: A computer program or instruction is stored, and when the computer program or instruction is run on a computer, the steps of the method according to any one of claims 1 to 8 are executed.

Images