Multi-scale cross-regional lymph node metastasis pathological image recognition method

Abstract

The invention provides a multi-scale cross-regional lymph node metastasis pathological image recognition method, which comprises the following steps of: cutting a region of interest from a full-slice pathological image, dividing standard image blocks under an amplification factor, and extracting a feature sequence by utilizing a pre-training model; the multi-scale module performs intra-scale and inter-scale self-attention calculation and fusion on each amplification factor feature to construct unified multi-scale feature representation; the cross-region module recombines the features into a two-dimensional grid and divides regions, and local feature optimization and global semantic association are realized through multi-head self-attention in the regions and among the regions; and outputting the diagnosis result and probability of lymph node metastasis through the classifier, and generating a multi-scale attention heat map to integrate into a visual diagnosis report. According to the method, the accuracy and the interpretability of lymph node metastasis detection can be remarkably improved through collaborative modeling of the multi-scale information and the cross-regional dependency relationship of the pathological image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent medical treatment, in particular to a multi-scale cross-region lymph node metastasis pathological image recognition method. BACKGROUND

[0002] In the current conventional diagnostic practice, pathologists mainly rely on morphological observation of whole slide images to determine lymph node metastasis. However, the identification of lymph node metastasis not only needs to focus on the tumor region itself, but also needs to comprehensively analyze the complex spatial relationship between the tumor and the surrounding normal tissue. Simple reliance on artificial visual inspection is prone to miss small or atypical metastatic lesions. If further confirmed by immunohistochemistry, although it can improve the detection rate of metastatic lesions, it will significantly increase the diagnostic time and economic cost, and is difficult to be widely promoted in the conventional pathological workflow.

[0003] Existing tumor recognition methods mostly focus on single-dimensional feature modeling, which is difficult to fully meet the comprehensive information needs of lymph node metastasis recognition. Some methods attempt to integrate multi-scale image information at different magnifications, but usually only aggregate features at the image block level, failing to effectively model the long-range spatial dependency relationship across regions within the whole slide. Some other methods, although achieving spatial perception at the image block level through attention mechanisms or graph structure networks at a single scale, ignore the complementarity and joint modeling value between morphological features at different scales. This leads to limited ability of general tumor recognition models when migrating to the specific diagnostic task of lymph node metastasis, making it difficult to fully capture the necessary multi-scale morphological features and cross-region spatial relationship information for diagnosis.

[0004] Existing tumor recognition methods mostly focus on single-dimensional feature modeling, which is difficult to fully meet the comprehensive information needs of lymph node metastasis recognition. Some methods attempt to integrate multi-scale image information at different magnifications, but usually only aggregate features at the image block level, failing to effectively model the long-range spatial dependency relationship across regions within the whole slide. Some other methods, although achieving spatial perception at the image block level through attention mechanisms or graph structure networks at a single scale, ignore the complementarity and joint modeling value between morphological features at different scales. This leads to limited ability of general tumor recognition models when migrating to the specific diagnostic task of lymph node metastasis, making it difficult to fully capture the necessary multi-scale morphological features and cross-region spatial relationship information for diagnosis. SUMMARY

[0005] The present application provides a multi-scale cross-region lymph node metastasis pathological image recognition method, characterized in that the method comprises:

[0006] Step 1, cropping out regions of interest from the whole slide pathological image, in At consecutive magnifications, each region will be magnified at each magnification. Subdivision into A total of standard image patches were extracted. Image blocks;

[0007] Step 2: Extract using a pre-trained deep neural network Dimensional features, for each region The features are connected in order of magnitude to form a multi-scale feature sequence. The characteristic matrix is ​​obtained. ;

[0008] Step 3, within each multiplier, Each image patch feature is concatenated with a category token, through... Layer relative position self-attention optimization to extract scale features ;

[0009] Step 4: Between adjacent magnifications, concatenate each block feature at the low magnification with the corresponding four block features at the high magnification. After multi-head self-attention fusion, weighted aggregate features at each scale are then performed. To form unified multi-scale features ;

[0010] Step 5, The regional features are rearranged as follows: Mesh, divided into zero-fill indivual In local regions, features are optimized through multi-head self-attention within each region;

[0011] Step 6: Map the features of each region to Representative features After self-attention enhancement, based on similarity weights , , Distribute back to each region and update the feature grid;

[0012] Step 7: Calculate weights using a dual attention mechanism. Weighted aggregation into global features Output after linear classifier Class of diagnostic results and probability ;

[0013] Step 8, Extract Attention maps at various scales Generate based on coarse-grained heatmap and based on fusion attention Fine-grained heatmap A visual report is generated based on the diagnostic results.

[0014] Specifically, in step 1, the step of cropping from the whole-section pathological image... Region of interest, in At consecutive magnifications, each region will be magnified at each magnification. Subdivision into A total of standard image patches were extracted. A number of image patches, specifically: the size of a standard image patch is defined as follows: Pixels, for a full slice image At the base magnification Cut out from the bottom. Regions of interest, each region of interest The corresponding image size is Pixel, in the Magnification Below, its magnification relative to the base magnification The resolution ratio is That is, the same physical region The image size at this magnification becomes Pixels, denoted as the region of interest image at that magnification. Each Can be evenly divided into A standard image patch, for a region of interest In all The total number of standard image patches extracted at each magnification level indivual;

[0015] Specifically, in step 2, the extraction using a pre-trained deep neural network... Dimensional features, for each region The features are connected in order of magnitude to form a multi-scale feature sequence. The characteristic matrix is ​​obtained. Specifically, this involves using a pre-trained deep neural network. As a feature extractor, this extractor can be a general model pre-trained on natural images (such as ResNet50), a specialized model pre-trained on medical images under self-supervised conditions (such as CTransPath), or a general model based on large-scale medical visual-language alignment (such as Conch), etc., extracting features from each standard image patch to obtain a... 3D feature vector Each region of interest In all Magnification On Features of a standard image patch, categorized by maximum magnification. To minimum magnification Sequential connections form multi-scale feature sequences. For the entire slice image In I have regions of interest, and finally I get a shape of... Feature matrix This feature matrix will serve as the basic input for subsequent multi-scale visual representation learning;

[0016] Specifically, in step 3, within each magnification, the following will be... Each image patch feature is concatenated with a category token, through... Layer relative position self-attention optimization to extract scale features Specifically, for each region of interest , input it The feature vectors are grouped according to magnification, with each group corresponding to a magnification factor. Below A standard image patch feature Multi-scale feature fusion is performed using a pyramid visual Transformer, starting from the maximum magnification. Start by gradually increasing the magnification to the minimum. Proceeding, at each magnification Internally, the features are first mapped to a uniform dimension through a linear projection layer. , ,in , For learnable parameters, a learnable category token is then added. It is concatenated with the projected features and a learnable positional encoding is added. This forms the complete input representation at that scale. Next, through Layer relative position self-attention Transformer encoder pair The update process involves updating each encoder layer, which includes a multi-head self-attention (MHA) and a feedforward network (FNN) module, both employing residual connections and layer normalization. , The self-attention mechanism is relative position self-attention, which introduces a bias term based on the relative position of image patch space when calculating attention weights. , ,in Each by Obtained through linear transformation, Scaling factor The two-dimensional relative positional relationships between image patches are encoded through a learnable relative position embedding table. After layer processing, feature representations at that scale are extracted from the updated category token locations. ;

[0017] Specifically, in step 4, between adjacent magnifications, each block feature at the low magnification is concatenated with the corresponding four block features at the high magnification, and after multi-head self-attention fusion, the features at each scale are weighted and aggregated. To form unified multi-scale features Specifically, this means: at adjacent magnifications and between Utilizing spatial correspondence, low magnification Each standard image patch, at high magnification The following corresponds to four spatially aligned standard image patches, which will Features of each standard image patch without intra-scale self-attention optimization ,and The following are the features of the four standard image patches corresponding to this block, which have undergone in-scale self-attention optimization. The features are concatenated to form a fused feature vector. For all blocks, the fused feature matrix is ​​obtained. Cross-scale feature fusion is achieved through a multi-head self-attention mechanism. After multi-head self-attention processing, features are extracted from the first position of each fused sequence as low magnification. Preliminary fusion characteristics Subsequently, these fusion features It will be concatenated with the category token and magnified at a low magnification. Intra-scale self-attention optimization is performed by repeating the inter-scale self-attention fusion and intra-scale self-attention optimization processes described above, ultimately optimizing for each magnification. Representative features are extracted from the category token positions after self-attention processing within the scale. ,Will Representative features are weighted and fused to form a unified multi-scale feature table. The weight It can be predefined or obtained through learnable parameters;

[0018] Specifically, in step 5, the step of... The regional features are rearranged as follows: Mesh, divided into zero-fill indivual In local regions, features are optimized through multi-head self-attention within each region. Specifically, this involves processing a full-slice image... All A set of multi-scale feature sequences corresponding to each region of interest Rearranged into a two-dimensional feature grid Each grid cell corresponds to a region of interest. eigenvectors Based on the number of regions of interest Calculate the grid size for an approximate square. and , making Fill length Expanding the feature set into a neat grid through zero-padding ,in For a matrix consisting entirely of zeros, the feature grid is... Evenly divided into Local area Each local region consists of a set of spatially adjacent feature vectors, and the region size is... For each local area flatten it into Furthermore, a multi-head self-attention mechanism is employed for feature interaction and optimization, firstly through layer normalization. Then calculate the region multi-head self-attention, for the th Height, attention score calculated as ,in , , Depend on The enhanced positional encoding, obtained through linear transformation, modulates the attention score using one-dimensional or two-dimensional separable convolution. The outputs of all heads are spliced ​​together and then linearly projected. ,in Finally, the optimized local region features are obtained through residual connections and a feedforward network. ;

[0019] Specifically, in step 6, the mapping of each region's features to... Representative features After self-attention enhancement, based on similarity weights , , Distribute back to each region and update the feature mesh, specifically by: transforming each region using a learnable transformation. This is mapped to a fixed number of representative feature vectors, s. The mapping is achieved through a learnable projection matrix. ,in For the learnable parameter matrix, based on the similarity matrix Simultaneously, three weights are generated: feature aggregation weights. Feature distribution weights Normalized feature distribution weights ,in , , For minimal positive numbers, based on aggregate weights Aggregate local region features into representative features The representative feature matrix is ​​obtained. For representative feature vectors By applying a self-attention mechanism, representative features after global augmentation are obtained. Based on the feature distribution weights, the globally enhanced representative features are... Backhaul to the corresponding local area The distribution process is divided into two stages. The first stage uses normalized distribution weights. , The second phase uses the original distribution weights. Perform weighted fusion. Update the feature vector within the region to obtain the updated local region features. Then, through a region departitioning operation, all local regions are reorganized into a complete feature mesh. Finally, the zero-filled elements are removed, restoring the updated feature sequence that corresponds one-to-one with each region of interest. ;

[0020] Specifically, in step 7, the weights are calculated using a dual attention mechanism. Weighted aggregation into global features Output after linear classifier Class of diagnostic results and probability Specifically, this means: updating the feature sequence We obtain the global feature representation by weighted aggregation through a dual attention mechanism. The aggregation process includes two independent attention branches. Branch one is based on feature similarity attention, which calculates the query vector. Dot product similarity with feature sequences Branch two is based on feature importance attention, which is computed through a learnable gating mechanism. ,in , , The attention weights are obtained by weighted fusion, using the sigmoid function. ,in The mean of the characteristic sequence. The global feature is represented as a weighted sum. ,Will Input linear classifier Diagnosis of lymph node metastasis ,in The number of categories is used, and the corresponding probability estimates are calculated using the Softmax function. , ;

[0021] Specifically, in step 8, the extraction Attention maps at various scales Generate based on coarse-grained heatmap and based on fusion attention Fine-grained heatmap The diagnostic results are combined to generate a visual report, specifically by extracting data from different levels of the model. Attention map at each magnification ,in Corresponding to the lowest magnification, Corresponding to the highest magnification, two different strategies are used to generate attention heatmaps: a coarse-grained heatmap and a fine-grained heatmap. Attention map using only the lowest magnification For each region of interest Assign a uniform attention value Fine-grained heatmap Through the Attention map at each magnification A weighted average fusion is performed to obtain a uniform attention distribution. Then select each region of interest Subdivided into Each sub-region grid, and according to Assign independent attention values ​​to each sub-region ,in express The Sub-regions The generated heatmap is mapped to a minimum-maximum normalization map. scope, Gaussian smoothing filter is applied to enhance the visualization effect. ,in For Gaussian kernel, Controlling the smoothness of the results will affect the diagnosis of lymph node metastasis. Probability estimation Coarse-grained heatmap and fine-grained heatmap The data is integrated to generate a structured auxiliary diagnostic report. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0023] Figure 1 Flowchart of a multi-scale, cross-regional lymph node metastasis pathological image recognition method;

[0024] Figure 2 Flowchart of a multi-scale, cross-regional pathological image recognition method for identifying lymph node metastases in pancreatic ductal carcinoma (PDAC); Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Current methods for automatic lymph node metastasis identification have limitations in comprehensive information utilization: most methods focus on a single scale or only perform multi-scale feature aggregation at the local image block level, making it difficult to effectively model long-range spatial dependencies across regions in whole-slice images; while some single-scale methods based on attention or graph networks can perceive local spatial relationships, they fail to fully combine and model the complementarity of multi-scale morphological features. This makes it difficult for existing general tumor identification models to fully capture the multi-scale morphological information and global spatial structure required for diagnosis when transferred to lymph node metastasis tasks, thus limiting their clinical applicability. To address these issues, this invention proposes a method that integrates multi-scale features and cross-regional feature modeling, fully utilizing the multi-scale morphological information and long-range tissue structure of whole-slice images to improve the accuracy of lymph node metastasis identification without increasing additional detection costs. The generated heatmap can focus on clinically significant metastatic areas, has good interpretability, and can provide effective assistance for pathological diagnosis and treatment decisions. The specific steps are as follows:

[0027] S101: Cropped from whole-section pathological images Region of interest, in At consecutive magnifications, each region will be magnified at each magnification. Subdivision into A total of standard image patches were extracted. Image blocks.

[0028] Specifically, the size of the standard image patch is defined as Pixels, for a full slice image At the base magnification Cut out from the bottom. Regions of interest, each region of interest The corresponding image size is Pixel, in the Magnification Below, its magnification relative to the base magnification The resolution ratio is That is, the same physical region The image size at this magnification becomes Pixels, denoted as the region of interest image at that magnification. Each Can be evenly divided into A standard image patch, for a region of interest In all The total number of standard image patches extracted at each magnification level indivual.

[0029] S102: Extracting using a pre-trained deep neural network Dimensional features, for each region The features are connected in order of magnitude to form a multi-scale feature sequence. The characteristic matrix is ​​obtained. .

[0030] Specifically, a pre-trained deep neural network is used. As a feature extractor, this extractor can be a general model pre-trained on natural images (such as ResNet50), a specialized model pre-trained on medical images under self-supervised conditions (such as CTransPath), or a general model based on large-scale medical visual-language alignment (such as Conch), etc., extracting features from each standard image patch to obtain a... 3D feature vector Each region of interest In all Magnification On Features of a standard image patch, categorized by maximum magnification. To minimum magnification Sequential connections form multi-scale feature sequences. For the entire slice image In I have regions of interest, and finally I get a shape of... Feature matrix This feature matrix will serve as the basic input for subsequent multi-scale visual representation learning.

[0031] S103: Within each multiplier, Each image patch feature is concatenated with a category token, through... Layer relative position self-attention optimization to extract scale features .

[0032] Specifically, for each region of interest , input it The feature vectors are grouped according to magnification, with each group corresponding to a magnification factor. Below A standard image patch feature Multi-scale feature fusion is performed using a pyramid visual Transformer, starting from the maximum magnification. Start by gradually increasing the magnification to the minimum. Proceeding, at each magnification Internally, the features are first mapped to a uniform dimension through a linear projection layer. , ,in , For learnable parameters, a learnable category token is then added. It is concatenated with the projected features and a learnable positional encoding is added. This forms the complete input representation at that scale. Next, through Layer relative position self-attention Transformer encoder pair The update process involves updating each encoder layer, which includes a multi-head self-attention (MHA) and a feedforward network (FNN) module, both employing residual connections and layer normalization. , The self-attention mechanism is relative position self-attention, which introduces a bias term based on the relative position of image patch space when calculating attention weights. , ,in Each by Obtained through linear transformation, Scaling factor The two-dimensional relative positional relationships between image patches are encoded through a learnable relative position embedding table. After layer processing, feature representations at that scale are extracted from the updated category token locations. .

[0033] S104: Between adjacent magnifications, each block feature at the low magnification is concatenated with the corresponding four block features at the high magnification. After multi-head self-attention fusion, the features at each scale are weighted and aggregated. To form unified multi-scale features .

[0034] Specifically, at adjacent magnifications and between Utilizing spatial correspondence, low magnification Each standard image patch, at high magnification The following corresponds to four spatially aligned standard image patches, which will Features of each standard image patch without intra-scale self-attention optimization ,and The following are the features of the four standard image patches corresponding to this block, which have undergone in-scale self-attention optimization. The features are concatenated to form a fused feature vector. For all blocks, the fused feature matrix is ​​obtained. Cross-scale feature fusion is achieved through a multi-head self-attention mechanism. After multi-head self-attention processing, features are extracted from the first position of each fused sequence as low magnification. Preliminary fusion characteristics Subsequently, these fusion features It will be concatenated with the category token and magnified at a low magnification. Intra-scale self-attention optimization is performed by repeating the inter-scale self-attention fusion and intra-scale self-attention optimization processes described above, ultimately optimizing for each magnification. Representative features are extracted from the category token positions after self-attention processing within the scale. ,Will Representative features are weighted and fused to form a unified multi-scale feature table. The weight It can be predefined or obtained through learnable parameters.

[0035] S105: Will The regional features are rearranged as follows: Mesh, divided into zero-fill indivual In local regions, features are optimized through multi-head self-attention within each region.

[0036] Specifically, a full-slice image All A set of multi-scale feature sequences corresponding to each region of interest Rearranged into a two-dimensional feature grid Each grid cell corresponds to a region of interest. eigenvectors Based on the number of regions of interest Calculate the grid size for an approximate square. and , making Fill length Expanding the feature set into a neat grid through zero-padding ,in For a matrix consisting entirely of zeros, the feature grid is... Evenly divided into Local area Each local region consists of a set of spatially adjacent feature vectors, and the region size is... For each local area flatten it into Furthermore, a multi-head self-attention mechanism is employed for feature interaction and optimization, firstly through layer normalization. Then calculate the region multi-head self-attention, for the th Height, attention score calculated as ,in , , Depend on The enhanced positional encoding, obtained through linear transformation, modulates the attention score using one-dimensional or two-dimensional separable convolution. The outputs of all heads are spliced ​​together and then linearly projected. ,in Finally, the optimized local region features are obtained through residual connections and a feedforward network. .

[0037] S106: Map the features of each region to Representative features After self-attention enhancement, based on similarity weights , , Distribute back to each region and update the feature grid.

[0038] Specifically, through learnable transformations, each This is mapped to a fixed number of representative feature vectors, s. The mapping is achieved through a learnable projection matrix. ,in For the learnable parameter matrix, based on the similarity matrix Simultaneously, three weights are generated: feature aggregation weights. Feature distribution weights Normalized feature distribution weights ,in , , For minimal positive numbers, based on aggregate weights Aggregate local region features into representative features The representative feature matrix is ​​obtained. For representative feature vectors By applying a self-attention mechanism, representative features after global augmentation are obtained. Based on the feature distribution weights, the globally enhanced representative features are... Backhaul to the corresponding local area The distribution process is divided into two stages. The first stage uses normalized distribution weights. , The second phase uses the original distribution weights. Perform weighted fusion. Update the feature vector within the region to obtain the updated local region features. Then, through a region departitioning operation, all local regions are reorganized into a complete feature mesh. Finally, the zero-filled elements are removed, restoring the updated feature sequence that corresponds one-to-one with each region of interest. .

[0039] S107: Calculate weights using a dual attention mechanism Weighted aggregation into global features Output after linear classifier Class of diagnostic results and probability .

[0040] Specifically, for the updated feature sequence We obtain the global feature representation by weighted aggregation through a dual attention mechanism. The aggregation process includes two independent attention branches. Branch one is based on feature similarity attention, which calculates the query vector. Dot product similarity with feature sequences Branch two is based on feature importance attention, which is computed through a learnable gating mechanism. ,in , , The attention weights are obtained by weighted fusion, using the sigmoid function. ,in The mean of the characteristic sequence. The global feature is represented as a weighted sum. ,Will Input linear classifier Diagnosis of lymph node metastasis ,in The number of categories is used, and the corresponding probability estimates are calculated using the Softmax function. , .

[0041] S108: Extraction Attention maps at various scales Generate based on coarse-grained heatmap and based on fusion attention Fine-grained heatmap A visual report is generated based on the diagnostic results.

[0042] Specifically, extract from different levels of the model. Attention map at each magnification ,in Corresponding to the lowest magnification, Corresponding to the highest magnification, two different strategies are used to generate attention heatmaps: a coarse-grained heatmap and a fine-grained heatmap. Attention map using only the lowest magnification For each region of interest Assign a uniform attention value Fine-grained heatmap Through the Attention map at each magnification A weighted average fusion is performed to obtain a uniform attention distribution. Then select each region of interest Subdivided into Each sub-region grid, and according to Assign independent attention values ​​to each sub-region ,in express The Sub-regions The generated heatmap is mapped to a minimum-maximum normalization map. scope, Gaussian smoothing filter is applied to enhance the visualization effect. ,in For Gaussian kernel, Controlling the smoothness of the results will affect the diagnosis of lymph node metastasis. Probability estimation Coarse-grained heatmap and fine-grained heatmap The data is integrated to generate a structured auxiliary diagnostic report.

[0043] like Figure 2The diagram shows the workflow of a multi-scale, cross-regional pathological image recognition method for identifying lymph node metastases in pancreatic ductal carcinoma (PDAC), specifically including:

[0044] S201: At three consecutive magnifications , , Below, 21 regions from 1000 whole-section pathological images of PDAC were extracted. Pixel image blocks, generated Feature matrix. Specifically, the size of the standard image patch is defined as... Pixels, for a whole slice image of pancreatic ductal carcinoma, at a base magnification. Cut out A region of interest, For example, the image size corresponding to each region of interest is Pixel, relative to the base magnification resolution ratio ,exist Resolution ratio at magnification The image size becomes Pixels can be evenly divided into four standard image blocks. Resolution ratio at magnification The image size becomes Pixels can be evenly divided into 16 standard image blocks. At each magnification level, a total of standard image patches were extracted. One method uses a pre-trained deep neural network, ResNet50, as the feature extractor. Features are extracted from each standard image patch to obtain a The feature vector is a 3D feature vector that combines the features of 21 standard image patches for each region of interest at three magnifications, sorted by the maximum magnification. To minimum magnification Sequential connections form a feature sequence, which is relevant to a whole slice image of pancreatic ductal carcinoma. I have regions of interest, and finally I get a shape of... The characteristic matrix of .

[0045] S202: with arrive The sequence involves performing two layers of relative position self-attention optimization on features at each scale, followed by cross-scale fusion using multi-head self-attention with four heads, and finally aggregating the three features with a weighted average of 0.3333. Specifically, for each region of interest, its 21 input feature vectors are divided into three groups according to magnification, and a pyramid visual Transformer is used for multi-scale feature fusion, starting from the maximum magnification. At the beginning, for The features of 16 standard image patches at magnification are mapped to a uniform dimension through a linear projection layer. A learnable category token is concatenated and a learnable positional code is added to form a complete input representation for that scale. The Transformer encoder with relative layer position self-attention performs intra-scale self-attention computation and optimization, and finally extracts the feature representation of that scale from the updated class token position, followed by... and Inter-scale feature fusion between them Features of each standard image patch below, and The features of the four standard image patches corresponding to this block, after being optimized by in-scale self-attention, are concatenated to form a... The fused feature vectors of dimension are fused across scales using a multi-head self-attention mechanism with 4 heads. Features are extracted from the first position of each fused sequence as... Preliminary fusion characteristics of magnification, then... The magnification factor is used to perform intra-scale self-attention calculations. The four preliminary fused features obtained in the previous step are concatenated with a new category token and then incorporated with positional encoding. This is then optimized using a relative positional self-attention Transformer encoder, and finally, features are extracted from the category token. and Inter-scale feature fusion between them Standard image patch features, and The features of four standard image patches, optimized by intra-scale self-attention, are concatenated and then fused across scales using a multi-head self-attention mechanism to obtain the desired result. The initial fusion characteristics, and finally the Intra-scale self-attention calculations are performed at magnification levels. The preliminary fused features are concatenated with the category token and added to the positional encoding. This is then processed by a relative positional self-attention Transformer encoder, and features are extracted from the category token. Finally, features are derived from... , , Representative features are extracted from the category token positions after self-attention processing across these three magnification scales. These three representative features are then weighted and fused to form a unified multi-scale feature representation, with the weights being... , , .

[0046] S203: Rearrange the 1000 regional features into The image was divided into 64 local regions with zero padding and a mesh. Self-attention optimization with 8 heads was used, mapping the mesh to 5 representative features for global interactive updates back to each region. Specifically, the multi-scale feature sequences corresponding to all regions of interest in the whole slice image of pancreatic ductal carcinoma were rearranged into a two-dimensional feature mesh, based on the number of regions of interest. Calculate the size of an approximately square mesh , and by adjusting with zero fill To ensure neat mesh division, the feature mesh is evenly divided into... For each local region, a multi-head self-attention mechanism with 8 heads is used for feature interaction and optimization to obtain optimized local features. Then, through a learnable transformation, it is mapped to a fixed number of... Given the representative feature vectors, calculate the similarity matrix between the two. And based on this matrix, feature aggregation weights are generated. Distribution weight and normalized distribution weights A self-attention mechanism with a layer of 1 is applied to representative feature vectors to obtain globally enhanced features. And based on the distribution weight, it is sent back to each local region to update the region features. Finally remove Zero padding, recovery and Updated feature sequences corresponding to each region of interest .

[0047] S204: Uses a dual-attention mechanism to aggregate global features, the classifier outputs two types of diagnostic results and probabilities, and then fuses them. , , Attention maps and their use , The convolutional kernel generates smooth coarse- and fine-grained heatmaps. Specifically, a dual-attention mechanism is used to aggregate global features, and the classifier outputs two types of diagnostic results and probabilities, which are then fused. , , Attention maps and their use , The convolution kernel generates smooth coarse and fine-grained heatmaps.

[0048] This invention proposes a multi-scale, cross-regional pathological image recognition method for lymph node metastasis. By simultaneously performing multi-scale feature fusion and cross-regional long-range dependency modeling, this method can simultaneously capture the complementarity of morphological features at different magnifications and effectively analyze global tissue structure information within the entire slice. This overcomes the limitations of existing methods in single-dimensional feature modeling, local multi-scale aggregation, or single-scale spatial perception. Without increasing additional detection costs, this method significantly improves the recognition accuracy for the specific diagnostic task of lymph node metastasis. The generated heatmap can accurately focus on clinically significant metastatic areas and has high interpretability, thus providing reliable auxiliary support for clinical pathological diagnosis and treatment decisions.

Claims

1. A method for identifying pathological images of multi-scale, cross-regional lymph node metastasis, characterized in that, The method includes the following steps: Step 1: Crop out from the whole-section pathological image Region of interest, in At consecutive magnifications, each region will be magnified at each magnification. Subdivision into A total of standard image patches were extracted. Image blocks; Step 2: Extract using a pre-trained deep neural network Dimensional features, for each region The features are connected in order of magnitude to form a multi-scale feature sequence. The characteristic matrix is ​​obtained. ; Step 3, within each multiplier, will Each image patch feature is concatenated with a category token, through... Layer relative position self-attention optimization to extract scale features ; Step 4: Between adjacent magnifications, each block feature at the low magnification is concatenated with the corresponding four block features at the high magnification. After multi-head self-attention fusion, the features at each scale are weighted and aggregated. To form unified multi-scale features ; Step 5, The regional features are rearranged as follows: Mesh, divided into zero-fill indivual In local regions, features are optimized through multi-head self-attention within each region; Step 6: Map the features of each region to Representative features After self-attention enhancement, based on similarity weights , , Distribute back to each region and update the feature grid; Step 7: Calculate weights using a dual attention mechanism. Weighted aggregation into global features Output after linear classifier Diagnostic results and probability ; Step 8, Extract Attention maps at various scales Generate based on coarse-grained heatmap and based on fusion attention Fine-grained heatmap A visual report is generated based on the diagnostic results.

2. The method for identifying pathological images of multi-scale, cross-regional lymph node metastasis as described in claim 1, characterized in that, In step 1, the section cropped from the whole-section pathological image Region of interest, in At consecutive magnifications, each region will be magnified at each magnification. Subdivision into A total of standard image patches were extracted. A number of image blocks, specifically: the size of a standard image block is defined as follows: Pixels, for a full slice image At the base magnification Cut out from the bottom. Regions of interest, each region of interest The corresponding image size is Pixel, in the Magnification Below, its magnification relative to the base magnification The resolution ratio is That is, the same physical region The image size at this magnification becomes Pixels, denoted as the region of interest image at that magnification. each Can be evenly divided into A standard image patch, for a region of interest In all Total number of standard image patches extracted at each magnification level indivual.

3. The method for identifying pathological images of multi-scale, cross-regional lymph node metastasis as described in claim 1, characterized in that, In step 2, the extraction using a pre-trained deep neural network... Dimensional features, for each region The features are connected in order of magnitude to form a multi-scale feature sequence. The characteristic matrix is ​​obtained. Specifically, this involves using a pre-trained deep neural network. As a feature extractor, this extractor can be a general model pre-trained on natural images (such as ResNet50), a specialized model pre-trained on medical images under self-supervised conditions (such as CTransPath), or a general model based on large-scale medical visual-language alignment (such as Conch), etc., extracting features from each standard image patch to obtain a... 3D feature vector , for each region of interest In all Magnification On Features of a standard image patch, at maximum magnification To minimum magnification Sequential connections form multi-scale feature sequences. For the entire slice image In I have regions of interest, and finally I get a shape of... Feature matrix This feature matrix will serve as the basic input for subsequent multi-scale visual representation learning.

4. The method for identifying pathological images of multi-scale, cross-regional lymph node metastasis as described in claim 1, characterized in that, In step 3, within each magnification, the following will be... Each image patch feature is concatenated with a category token, through... Layer relative position self-attention optimization to extract scale features Specifically, for each region of interest , input it The feature vectors are grouped according to magnification, with each group corresponding to a magnification factor. Below A standard image patch feature Multi-scale feature fusion is performed using a pyramid visual Transformer, starting from the maximum magnification. Start by gradually increasing the magnification to the minimum. Proceeding, at each magnification Internally, the features are first mapped to a uniform dimension through a linear projection layer. , ,in , For learnable parameters, a learnable category token is then added. It is concatenated with the projected features and a learnable positional encoding is added. This forms the complete input representation at that scale. Next, through Layer relative position self-attention Transformer encoder pair The update process involves updating each encoder layer, which includes a multi-head self-attention (MHA) and a feedforward network (FNN) module, both employing residual connections and layer normalization. , The self-attention mechanism is relative position self-attention, which introduces a bias term based on the relative position of image patch space when calculating attention weights. , ,in Each by Obtained through linear transformation, Scaling factor The two-dimensional relative positional relationships between image patches are encoded through a learnable relative position embedding table. After layer processing, feature representations at that scale are extracted from the updated category token locations. .

5. The method for identifying pathological images of multi-scale, cross-regional lymph node metastasis as described in claim 1, characterized in that, In step 4, between adjacent magnifications, each block feature at the low magnification is concatenated with the corresponding four block features at the high magnification. After multi-head self-attention fusion, the features at each scale are weighted and aggregated. To form unified multi-scale features Specifically, this means: at adjacent magnifications and between Utilizing spatial correspondence, low magnification Each standard image patch, at high magnification The following corresponds to four spatially aligned standard image patches, which will Features of each standard image patch without intra-scale self-attention optimization ,and The following are the features of the four standard image patches corresponding to this block, which have undergone in-scale self-attention optimization. The features are concatenated to form a fused feature vector. For all blocks, the fused feature matrix is ​​obtained. Cross-scale feature fusion is achieved through a multi-head self-attention mechanism. After multi-head self-attention processing, features are extracted from the first position of each fused sequence as low magnification. Preliminary fusion characteristics Subsequently, these fusion features It will be concatenated with the category token and magnified at a low magnification. Intra-scale self-attention optimization is performed by repeating the inter-scale self-attention fusion and intra-scale self-attention optimization processes described above, ultimately optimizing for each magnification. Representative features are extracted from the category token positions after self-attention processing within the scale. ,Will Representative features are weighted and fused to form a unified multi-scale feature table. The weight It can be predefined or obtained through learnable parameters.

6. The method for identifying pathological images of multi-scale, cross-regional lymph node metastasis as described in claim 1, characterized in that, In step 5, the... The regional features are rearranged as follows: Mesh, divided into zero-fill indivual In local regions, features are optimized through multi-head self-attention within each region. Specifically, this involves processing a full-slice image... All A set of multi-scale feature sequences corresponding to each region of interest Rearranged into a two-dimensional feature grid Each grid cell corresponds to a region of interest. eigenvectors Based on the number of regions of interest Calculate the grid size for an approximate square. and , making Fill length Expanding the feature set into a neat grid through zero-padding ,in For a matrix consisting entirely of zeros, the feature grid is... Evenly divided into Local area Each local region consists of a set of spatially adjacent feature vectors, and the region size is... For each local area flatten it into Furthermore, a multi-head self-attention mechanism is employed for feature interaction and optimization, firstly through layer normalization. Then calculate the region multi-head self-attention, for the th Height, attention score calculated as ,in , , Depend on The enhanced positional encoding, obtained through linear transformation, modulates the attention score using one-dimensional or two-dimensional separable convolution. The outputs of all heads are spliced ​​together and then linearly projected. ,in Finally, the optimized local region features are obtained through residual connections and a feedforward network. .

7. The method for identifying pathological images of multi-scale, cross-regional lymph node metastasis as described in claim 1, characterized in that, In step 6, the feature mapping of each region is... Representative features After self-attention enhancement, based on similarity weights , , Distribute back to each region and update the feature mesh, specifically by: transforming each region using a learnable transformation. This is mapped to a fixed number of representative feature vectors, s. The mapping is achieved through a learnable projection matrix. ,in For the learnable parameter matrix, based on the similarity matrix Simultaneously, three weights are generated: feature aggregation weights. Feature distribution weights Normalized feature distribution weights ,in , , For minimal positive numbers, based on aggregate weights Aggregate local region features into representative features The representative feature matrix is ​​obtained. For representative feature vectors By applying a self-attention mechanism, representative features after global augmentation are obtained. Based on the feature distribution weights, the globally enhanced representative features are... Backhaul to the corresponding local area The distribution process is divided into two stages. The first stage uses normalized distribution weights. , The second phase uses the original distribution weights. Perform weighted fusion. Update the feature vector within the region to obtain the updated local region features. Then, through a region departitioning operation, all local regions are reorganized into a complete feature mesh. Finally, the zero-filled elements are removed, restoring the updated feature sequence that corresponds one-to-one with each region of interest. .

8. The method for identifying pathological images of multi-scale, cross-regional lymph node metastasis as described in claim 1, characterized in that, In step 7, the weights are calculated using a dual attention mechanism. Weighted aggregation into global features Output after linear classifier Diagnostic results and probability Specifically, this means: updating the feature sequence We obtain the global feature representation by weighted aggregation through a dual attention mechanism. The aggregation process includes two independent attention branches. Branch one is based on feature similarity attention, which calculates the query vector. Dot product similarity with feature sequences Branch two is based on feature importance attention, which is computed through a learnable gating mechanism. ,in , , The attention weights are obtained by weighted fusion, using the sigmoid function. ,in The mean of the characteristic sequence. The global feature is represented as a weighted sum. ,Will Input linear classifier Diagnosis of lymph node metastasis ,in The number of categories is used, and the corresponding probability estimates are calculated using the Softmax function. , .

9. The method for identifying pathological images of multi-scale, cross-regional lymph node metastasis as described in claim 1, characterized in that, In step 8, the extraction Attention maps at various scales Generate based on coarse-grained heatmap and based on fusion attention Fine-grained heatmap The diagnostic results are combined to generate a visual report, specifically by extracting data from different levels of the model. Attention map at each magnification ,in Corresponding to the lowest magnification, Corresponding to the highest magnification, two different strategies are used to generate attention heatmaps: a coarse-grained heatmap and a fine-grained heatmap. Attention map using only the lowest magnification For each region of interest Assign a uniform attention value Fine-grained heatmap Through the Attention map at each magnification A weighted average fusion is performed to obtain a uniform attention distribution. Then select each region of interest Subdivided into Each sub-region grid, and according to Assign independent attention values ​​to each sub-region ,in express The Sub-regions The generated heatmap is mapped to a minimum-maximum normalization map. scope, Gaussian smoothing filter is applied to enhance the visualization effect. ,in For Gaussian kernel, Controlling the smoothness of the results will affect the diagnosis of lymph node metastasis. Probability estimation Coarse-grained heatmap and fine-grained heatmap The data is integrated to generate a structured auxiliary diagnostic report.

Images