Method and device for detecting pan-tumor lymph node metastasis based on bidirectional reciprocity cross-scale attention fusion

Abstract

The invention discloses a pan-tumor lymph node metastasis detection method and device based on bidirectional reciprocity cross-scale attention fusion. The method comprises the following steps: acquiring a high-scale image of a region of interest and a low-scale image of the region of interest; obtaining a high-scale depth feature flow; obtaining a low-scale depth feature flow; performing bidirectional reciprocal attention processing on the high-scale depth feature flow and the low-scale depth feature flow to obtain enhanced high-scale features and enhanced low-scale features; fusing the enhanced high-scale features and the enhanced low-scale features to obtain multi-scale fusion features; and obtaining a final classification result according to the multi-scale fusion features. According to the universal tumor lymph node metastasis detection method based on bidirectional reciprocal cross-scale attention fusion, the multi-scale information of the pathological image is utilized, and an innovative bidirectional reciprocal cross-scale attention fusion mechanism is combined, so that the classification accuracy and credibility can be remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a pan-tumor lymph node metastasis detection method and device based on bidirectional reciprocal cross-scale attention fusion. BACKGROUND

[0002] In the process of cancer diagnosis and treatment, the status of lymph nodes is a crucial factor. The detection and accurate staging of lymph node metastasis are of great significance for the prognosis of patients and the development of treatment plans. However, there are significant differences in the characteristics and patterns of lymph node metastasis of tumors in different organs, which makes the identification of tumor lymph nodes across multiple organs particularly complex and challenging.

[0003] Traditionally, the detection of lymph node metastasis mainly relies on visual inspection and manual analysis by pathologists. This method is not only time-consuming and labor-intensive, but also limited by the experience and subjective judgment of pathologists, with certain risks of misdiagnosis and missed diagnosis. With the development of medical imaging technology, imaging has been increasingly widely used in tumor diagnosis. However, relying solely on imaging methods for the detection of lymph node metastasis also has certain limitations, especially for small metastatic lesions and lesions that are difficult to distinguish.

[0004] In recent years, artificial intelligence (AI) technology, especially deep learning and neural networks, has shown great potential in medical image analysis. AI-based algorithms can learn from a large amount of medical image data to automatically identify and analyze features in images, thereby playing an important role in tumor detection and staging. Studies have shown that AI technology has made significant achievements in the detection of lymph node metastasis of single-organ tumors such as breast cancer and lung cancer, with significantly better accuracy and efficiency than traditional methods.

[0005] However, current algorithms for lymph node identification of multiple organ tumors are still in the exploratory stage. The pathological characteristics and metastatic patterns of tumors in different organs are different, which requires algorithms to have higher generalization ability and precision to cope with complex clinical situations. Therefore, developing an algorithm that can effectively identify tumor lymph node metastasis across multiple organs can not only improve the accuracy of clinical diagnosis, but also significantly reduce the workload of pathologists, and has important clinical application value and broad prospects. Although there have been attempts to develop Pan-cancer detection models, the reported model performance (AUC only 0.925) is far from meeting the clinical requirements for high precision, which confirms that cross-organ accurate identification is still a highly challenging technical problem.

[0006] Although digital pathology scanning has been gradually applied, existing tumor lymph node metastasis detection algorithms still expose significant technical bottlenecks in dealing with complex clinical needs, mainly in the following aspects:

[0007] Lack of algorithm generalization and performance limitation: Existing computational methods are usually "organ-specific", i.e., designed for single organ tumor, lacking adaptability across multiple organs. Due to the significant differences in metastasis characteristics of tumors in different organs, such algorithms are difficult to effectively apply in multi-organ scenarios, greatly limiting their clinical promotion value. The performance (AUC of 0.925) of the multi-organ detection model reported in the existing research is far from meeting the clinical needs, confirming that this is still a challenging problem.

[0008] In-depth analysis shows that the above performance bottleneck is due to the methodological defects of existing algorithms in handling the multi-scale characteristics of pathological images:

[0009] Dependence on single scale analysis leads to information loss: Most algorithms rely on a single fixed magnification for analysis. This method has inherent information bias: high magnification scale can capture cytological details, but it is easy to lose macroscopic contextual information such as tissue structure; conversely, low magnification scale can provide a global view, but it cannot observe the fine cell features necessary for diagnosis. Since different organ tumors have different metastasis patterns, a single scale cannot cover the diagnostic needs of pan-tumor lymph node.

[0010] Lack of effective multi-scale feature fusion mechanism: Some methods attempt to use multi-scale information, but their feature fusion mechanism has serious defects. First, many methods use simple concatenation, addition, and other "hard fusion" methods. These methods do not solve a core problem: how to absorb external information while effectively maintaining and strengthening their own key features? This leads to the dilution or loss of key local or global information during the fusion process, limiting the upper limit of model performance. Second, even if some techniques attempt to use cross-attention mechanisms, these applications mostly follow the conventional, one-way information query mode, i.e., one feature stream queries and integrates the information of another feature stream. This one-way fusion cannot achieve equal and bidirectional information exchange between scales, so it still cannot allow macroscopic contextual information to guide microscopic detail interpretation, while allowing microscopic features to verify macroscopic judgments, with limited fusion effect and performance improvement.

[0011] Therefore, it is desirable to have a technical solution to overcome or at least alleviate at least one of the above-mentioned deficiencies of the prior art.

[0012] Summary

[0013] The purpose of the present application is to provide a pan-tumor lymph node metastasis detection method based on bidirectional reciprocal cross-scale attention fusion to overcome or at least alleviate at least one of the above-mentioned deficiencies of the prior art.

[0014] To achieve the above object, the application provides a pan-tumor lymph node metastasis detection method based on bidirectional reciprocal cross-scale attention fusion, which comprises the following steps:

[0015] obtaining a high-scale image of a region of interest and a low-scale image of the region of interest;

[0016] extracting features from the high-scale image of the region of interest to obtain a high-scale deep feature flow;

[0017] extracting features from the low-scale image of the region of interest to obtain a low-scale deep feature flow;

[0018] performing bidirectional reciprocal attention processing on the high-scale deep feature flow and the low-scale deep feature flow to obtain enhanced high-scale features and enhanced low-scale features;

[0019] fusing the enhanced high-scale features and the enhanced low-scale features to obtain a multi-scale fusion feature;

[0020] obtaining a final classification result according to the multi-scale fusion feature.

[0021] Optionally, the bidirectional reciprocal attention processing on the high-scale deep feature flow and the low-scale deep feature flow to obtain enhanced high-scale features and enhanced low-scale features comprises the following steps:

[0022] performing linear projection on the high-scale deep feature flow to obtain a high-scale query vector, a high-scale key vector and a high-scale value vector;

[0023] performing linear projection on the low-scale deep feature flow to obtain a low-scale query vector, a low-scale key vector and a low-scale value vector;

[0024] calculating high-scale self-attention according to the high-scale query vector and the high-scale key vector;

[0025] calculating cross-attention of high scale to low scale according to the high-scale query vector and the low-scale key vector;

[0026] calculating cross-attention of low scale to high scale according to the low-scale query vector and the high-scale key vector;

[0027] calculating low-scale self-attention according to the low-scale query vector and the low-scale key vector;

[0028] The enhanced high-scale feature and the enhanced low-scale feature are generated according to the high-scale self-attention, the high-scale-to-low-scale cross-attention, the low-scale-to-high-scale cross-attention, the low-scale self-attention, the high-scale value vector, and the low-scale value vector.

[0029] Optionally, the generating the enhanced high-scale feature and the enhanced low-scale feature according to the high-scale self-attention, the high-scale-to-low-scale cross-attention, the low-scale-to-high-scale cross-attention, the low-scale self-attention, the high-scale value vector, and the low-scale value vector comprises:

[0030] generating a high-scale output vector according to the high-scale self-attention and the high-scale value vector;

[0031] generating a low-scale-to-high-scale output vector according to the low-scale-to-high-scale cross-attention and the high-scale value vector;

[0032] generating a high-scale-to-low-scale output vector according to the high-scale-to-low-scale cross-attention and the low-scale value vector;

[0033] generating a low-scale output vector according to the low-scale self-attention and the low-scale value vector;

[0034] generating the enhanced high-scale feature according to the high-scale output vector and the low-scale-to-high-scale output vector;

[0035] generating the enhanced low-scale feature according to the low-scale output vector and the high-scale-to-low-scale output vector.

[0036] Optionally, the high-scale self-attention is calculated according to the high-scale query vector and the high-scale key vector by the following formula:

[0037]

[0038] wherein, HH attn is the high-scale self-attention; H q is the high-scale query vector; H k is the high-scale key vector.

[0039] Optionally, the high-scale-to-low-scale cross-attention is calculated according to the high-scale query vector and the low-scale key vector by the following formula:

[0040]

[0041] wherein, HL attn is the high-scale-to-low-scale cross-attention; H q is the high-scale query vector; L k is the low-scale key vector.

[0042] Optionally, the low-scale-to-high-scale cross attention is calculated according to the low-scale query vector and the high-scale key vector, and is obtained by the following formula:

[0043]

[0044] wherein, LH attn is the low-scale-to-high-scale cross attention; L q is the low-scale query vector; H k is the high-scale key vector.

[0045] Optionally, the low-scale self-attention is calculated according to the low-scale query vector and the low-scale key vector, and includes:

[0046]

[0047] wherein, LL attn is the low-scale self-attention; L q is the low-scale query vector; L k is the low-scale key vector.

[0048] Optionally, the enhanced high-scale feature is generated according to the high-scale output vector and the low-scale-to-high-scale output vector, and is obtained by the following formula:

[0049] H out =(HH out +LH out ) / 2;

[0050] wherein, H out is the enhanced high-scale feature; HH out is the high-scale output vector; LH out is the low-scale-to-high-scale output vector.

[0051] Optionally, the enhanced low-scale feature is generated according to the low-scale output vector and the high-scale-to-low-scale output vector, and is obtained by the following formula:

[0052] L out =(LL out +HL out ) / 2;

[0053] wherein, L out is the enhanced low-scale feature; LL out is the low-scale output vector; HL out is the high-scale-to-low-scale output vector.

[0054] The application further provides a device for identifying lymph nodes of tumors of multiple organs, which comprises:

[0055] a scale image acquisition module, configured to acquire a high-scale image of a region of interest and a low-scale image of the region of interest;

[0056] a high-scale deep feature flow acquisition module, configured to perform feature extraction on the high-scale image of the region of interest to acquire a high-scale deep feature flow;

[0057] a low-scale deep feature flow acquisition module, configured to perform feature extraction on the low-scale image of the region of interest to acquire a low-scale deep feature flow;

[0058] a bidirectional reciprocal attention enhancement module, configured to perform bidirectional reciprocal attention processing on the high-scale deep feature flow and the low-scale deep feature flow to acquire enhanced high-scale features and enhanced low-scale features;

[0059] a multi-scale fusion feature acquisition module, configured to fuse the enhanced high-scale features and the enhanced low-scale features to acquire multi-scale fusion features;

[0060] a classification module, configured to acquire a final classification result according to the multi-scale fusion features.

[0061] The pan-tumor lymph node metastasis detection method based on bidirectional reciprocal cross-scale attention fusion provided in the present application utilizes multi-scale information of pathological images and combines an innovative bidirectional reciprocal cross-scale attention fusion mechanism, and thus can significantly improve the accuracy and reliability of classification. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 is a flowchart of the pan-tumor lymph node metastasis detection method based on bidirectional reciprocal cross-scale attention fusion according to an embodiment of the present application.

[0063] Figure 2 is an inference result diagram (blue represents a normal region and red represents a tumor region) according to an embodiment of the present application.

[0064] Figure 3 is a patch-level ROC curve diagram (the blue curve is the result of the model trained by the method proposed in the present patent, and the auc is 0.9841; the purple curve is the result of the model obtained by replacing the attention module in the present solution with a conventional attention module (i.e., only LH out and HL out , and the auc is 0.9362; and the orange curve is the result of the model obtained by replacing the multi-scale training in the present solution with single-scale training, and the auc is 0.9278). DETAILED DESCRIPTION

[0065] For the purpose of the embodiments of the present application, the technical solutions and advantages, the technical solutions of the embodiments of the present application will be described in more detail below with reference to the drawings in the embodiments of the present application. In the drawings, the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The described embodiments are part of the embodiments of the present application, not all embodiments. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application. The embodiments of the present application will be described in detail below with reference to the drawings.

[0066] In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the scope of protection of the present application.

[0067] As shown in the method for detecting pan-tumor lymph node metastasis based on bidirectional reciprocal cross-scale attention fusion, the method comprises: Figure 1

[0068] obtaining a high-scale image of a region of interest and a low-scale image of the region of interest;

[0069] extracting features from the high-scale image of the region of interest, thereby obtaining a high-scale deep feature stream;

[0070] extracting features from the low-scale image of the region of interest, thereby obtaining a low-scale deep feature stream;

[0071] performing bidirectional reciprocal attention processing on the high-scale deep feature stream and the low-scale deep feature stream, thereby obtaining enhanced high-scale features and enhanced low-scale features;

[0072] fusing the enhanced high-scale features and the enhanced low-scale features, thereby obtaining multi-scale fusion features;

[0073] obtaining a final classification result according to the multi-scale fusion features.

[0074] In the present embodiment, the high-scale image of the region of interest and the low-scale image of the region of interest can be obtained by the following method:​

[0075] According to the mask map, the pathological tiff image is cut into images with a size of 256*256 under 10x magnification, and the center coordinates of the high-scale image are taken as the center coordinates of the low-scale image. The pathological tiff image is cut into images with a size of 256*256 under 4x magnification, and the images are taken as low-scale images. Under the guidance of the mask map, a tiff image is cut to obtain a pair of high-scale and low-scale images of the region of interest.

[0076] In this embodiment, the feature extraction part of the application is based on a Vision Transformer (ViT-L) basic model pre-trained unsupervised on a large number of lymph node pathological images.

[0077] In this embodiment, the application is processed in parallel for multi-scale input images to extract features with different pathological semantics:

[0078] For high-scale images (such as 10x magnification): the module focuses on analyzing fine cytological features such as nuclear atypia, size, morphology, and local texture details.

[0079] For low-scale images (such as 4x magnification): the module focuses on capturing macroscopic histological structures such as tumor infiltration boundaries, lymphoid follicle distribution, and overall tissue context information.

[0080] Through this module, the original image information is converted into two parallel high-scale deep feature streams H in and low-scale deep feature streams L in .

[0081] In this embodiment, bidirectional reciprocal attention processing is performed on the high-scale deep feature stream and the low-scale deep feature stream to obtain enhanced high-scale features and enhanced low-scale features, including:

[0082] Linearly projecting the high-scale deep feature stream to obtain a high-scale query vector, a high-scale key vector, and a high-scale value vector;

[0083] Linearly projecting the low-scale deep feature stream to obtain a low-scale query vector, a low-scale key vector, and a low-scale value vector;

[0084] Calculating high-scale self-attention according to the high-scale query vector and the high-scale key vector;

[0085] Calculating cross-attention from high-scale to low-scale according to the high-scale query vector and the low-scale key vector;

[0086] According to the low-scale query vector, the low-scale key vector, low-scale self-attention is calculated.

[0087] According to the low-scale query vector, the low-scale key vector, low-scale self-attention is calculated.

[0088] According to the high-scale self-attention, the high-scale to low-scale cross-attention, the low-scale to high-scale cross-attention, the low-scale self-attention, the high-scale value vector and the low-scale value vector, enhanced high-scale features and enhanced low-scale features are generated.

[0089] In the embodiment, first, respective query (Q), key (K), value (V) vectors H q ,H k ,H v and L q ,L k ,L v are obtained by linear projection.

[0090] H q = Linear(H in );

[0091] H k = Linear(H in );

[0092] H v = Linear(H in );

[0093] L q ,L k ,L v are obtained by linear projection.

[0094] In the embodiment, four groups of attention score matrices containing self-attention and cross-attention are calculated:

[0095] The high-scale self-attention according to the high-scale query vector and the high-scale key vector is obtained by the following formula:

[0096]

[0097] wherein, HH attn is high-scale self-attention; H q is high-scale query vector; H k is high-scale key vector.

[0098] The high-scale to low-scale cross-attention according to the high-scale query vector and the low-scale key vector is obtained by the following formula:

[0099]

[0100] wherein, HL attn is high-to-low cross-attention; H q is a high-scale query vector; L k is a low-scale key vector.

[0101] The low-to-high cross-attention is calculated according to the low-scale query vector and the high-scale key vector, and is obtained by the following formula:

[0102]

[0103] wherein, LH attn is low-to-high cross-attention; L q is a low-scale query vector; H k is a high-scale key vector.

[0104] The low-scale self-attention is calculated according to the low-scale query vector and the low-scale key vector, and includes:

[0105]

[0106] wherein, LL attn is low-scale self-attention; L q is a low-scale query vector; L k is a low-scale key vector.

[0107] In the embodiment, the enhanced high-scale feature and the enhanced low-scale feature are generated according to the high-scale self-attention, the high-to-low cross-attention, the low-to-high cross-attention, the low-scale self-attention, the high-scale value vector and the low-scale value vector, and include:

[0108] The high-scale output vector is generated according to the high-scale self-attention and the high-scale value vector;

[0109] The low-scale-to-high-scale output vector is generated according to the low-to-high cross-attention and the high-scale value vector;

[0110] The high-scale-to-low-scale output vector is generated according to the high-to-low cross-attention and the low-scale value vector;

[0111] The low-scale output vector is generated according to the low-scale self-attention and the low-scale value vector;

[0112] The enhanced high-scale feature is generated according to the high-scale output vector and the low-scale-to-high-scale output vector;

[0113] According to the low-scale output vector and the high-scale-to-low-scale output vector, an enhanced low-scale feature is generated.

[0114] In this embodiment, the high-scale output vector is generated according to the high-scale self-attention and the high-scale value vector, which is obtained by the following formula:

[0115] HH out =HH attn *H v ;

[0116] wherein, HH out is the high-scale output vector; HH attn is the high-scale self-attention; H v is the high-scale value vector.

[0117] In this embodiment, the low-scale-to-high-scale output vector is generated according to the low-scale-to-high-scale cross-attention and the high-scale value vector, which is obtained by the following formula:

[0118] LH out =LH attn *H v ;

[0119] wherein, LH attn is the low-scale-to-high-scale cross-attention; H v is the high-scale value vector; LH out is the low-scale-to-high-scale output vector.

[0120] In this embodiment, the high-scale-to-low-scale output vector is generated according to the high-scale-to-low-scale cross-attention and the low-scale value vector, which is obtained by the following formula:

[0121] HL out =HL attn *L v ;

[0122] wherein, HL out is the high-scale-to-low-scale output vector; HL attn is the high-scale-to-low-scale cross-attention; L v is the low-scale value vector.

[0123] In this embodiment, the low-scale output vector is generated according to the low-scale self-attention and the low-scale value vector, which is obtained by the following formula:

[0124] LL out =LL attn *L v ;

[0125] wherein, LL out is the low-scale output vector; LLattn is low-scale self-attention; L v is a low-scale value vector.

[0126] The conventional method usually only uses the result of querying the low-scale after the high-scale feature is updated (i.e., HL out ). This way is easy to cause the key detail information inherent in the high-scale feature itself to be diluted.

[0127] In the present embodiment, the self-attention output of the feature itself is fused with the cross-attention output from another scale perspective:

[0128] The enhanced high-scale feature according to the high-scale output vector and the low-scale-to-high-scale output vector is obtained by the following formula:

[0129] H out = (HH out + LH out ) / 2;

[0130] wherein H out is the enhanced high-scale feature; HH out is the high-scale output vector; and LH out is the low-scale-to-high-scale output vector.

[0131] In the present embodiment, the enhanced low-scale feature according to the low-scale output vector and the high-scale-to-low-scale output vector is obtained by the following formula:

[0132] L out = (LL out + HL out ) / 2;

[0133] wherein L out is the enhanced low-scale feature; LL out is the low-scale output vector; and HL out is the high-scale-to-low-scale output vector.

[0134] The present application takes the calculation of H out as an example, H out represents the core internal information extracted after the high-scale feature "examines itself", and LH out represents the information obtained after the high-scale feature is "observed" from the low-scale global perspective. The average of the two is essentially a modification and synergistic enhancement of the refined local detail information with global context information, thereby introducing complementary information while maintaining the integrity of the key features to the greatest extent, effectively avoiding feature degradation. By stacking two such bidirectional reciprocal attention layers, the features can be deeply interacted and fused at different abstraction levels.

[0135] In the present embodiment, obtaining the final classification result according to the multi-scale fusion feature comprises:

[0136] The high-scale and low-scale features H out and L out enhanced by the two layers of bidirectional reciprocal attention layers are concatenated to form a unified and comprehensive multi-scale fusion feature F:

[0137] F = Concatenate(H out , L out , dim = 1).

[0138] The fusion feature F is sent to two standard self-attention layers. The role of this step is to perform final global relationship modeling on the already deeply fused feature map, further focusing on the most classification discriminative dependent relationship in the entire feature space, so as to obtain a highly refined feature vector F out .

[0139] The specific operation of the two standard self-attention layers of the fusion feature F is as follows:

[0140] First, linear projection is performed to obtain F q , F k , and F v

[0141] F q = Linear(F);

[0142] F k = Linear(F);

[0143] F v = Linear(F);

[0144] Calculate attention:

[0145]

[0146] Get the output of the attention part:

[0147] F out = F attn * F v .

[0148] Finally, after passing through a layer of NormLayer, the scores of predicting normal and tumor are obtained by sending them to the classification head.

[0149] In the training stage of the present application, after the tiff image is cut, a series of data augmentation is performed, and the model is iteratively optimized using the adamw optimizer.

[0150] Data preprocessing:

[0151] For the cut images, data augmentation including stain normalization, random rotation, random crop, vertical and horizontal flip, Gaussian blur, etc. is applied.

[0152] The classifier is trained, and the loss function selected for training the classifier is cross-entropy loss.

[0153]

[0154] After the model training is completed, the tiff file is inferred. At this time, the cutting process saves the coordinate information of the cut image, so that it can be mapped back to the original image when the category is obtained.

[0155] Referring to Figure 3 , the AUC on the test set is significantly improved from the unimproved 0.9278 to 0.9841, and the performance of the model is greatly improved. The great leap in performance is directly due to the unique bidirectional reciprocal cross-scale attention fusion architecture described above. It is this aggregation strategy of "self-attention + cross-attention" that enables the model to more effectively integrate multi-scale information, thereby making more accurate judgments in complex general tumor pathological images, which is a technical effect that cannot be achieved by conventional fusion methods.

[0156] The application also provides a device for identifying lymph nodes of tumors of multiple organs, which comprises a scale image acquisition module, a high-scale deep feature flow acquisition module, a low-scale deep feature flow acquisition module, a bidirectional reciprocal attention enhancement module, a multi-scale fusion feature acquisition module, and a classification module, wherein

[0157] The scale image acquisition module is used to acquire a high-scale image of a region of interest and a low-scale image of the region of interest;

[0158] The high-scale deep feature flow acquisition module is used to extract features from the high-scale image of the region of interest, thereby acquiring a high-scale deep feature flow;

[0159] The low-scale deep feature flow acquisition module is used to extract features from the low-scale image of the region of interest, thereby acquiring a low-scale deep feature flow;

[0160] The bidirectional reciprocal attention enhancement module is used to perform bidirectional reciprocal attention processing on the high-scale deep feature flow and the low-scale deep feature flow, thereby acquiring enhanced high-scale features and enhanced low-scale features;

[0161] The multi-scale fusion feature acquisition module is used to fuse the enhanced high-scale features and the enhanced low-scale features, thereby acquiring multi-scale fusion features;

[0162] The classification module is configured to obtain a final classification result according to the multi-scale fusion feature.

[0163] The present application is based on a deep neural network, and its core task is to process pathological sections from different magnifications (scales) and extract pathological semantic information with clear hierarchical relationship. Specifically, it captures global contextual features such as tissue structure and tumor infiltration boundary from low-scale images, and analyzes local detailed features such as cell nucleus atypia and morphology from high-scale images. This process aims to provide high-quality, information-complementary feature input for the subsequent fusion module, and is the basis for ensuring effective discrimination of the model.

[0164] As the technical core of the present application, the bidirectional reciprocal attention enhancement module performs deep interaction and adaptive fusion on the hierarchical features extracted by the previous module through a novel attention mechanism. It realizes bidirectional information enhancement between high and low scale features: the high-scale feature stream can dynamically query and integrate the global context provided by the low-scale feature to better understand the meaning of local lesions; conversely, the low-scale feature stream can also use the key details in the high-scale feature to correct and sharpen its representation. This design enables the final output fusion feature to have both macroscopic vision and microscopic insight, thereby enabling the classifier to make accurate judgments far beyond a single scale or simple information merging. The cross-attention layer functions to realize the synergistic enhancement and fusion of high and low scale features. It enables bidirectional information query between the features of the two scales, generating a new feature representation that combines macroscopic context and microscopic details. Then, the fused feature is sent to the self-attention layer, which refines the global context of the fused information, further focuses on the most diagnostic areas in the entire feature map, and finally forms a highly refined feature vector for classification.

[0165] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them. Although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing examples, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting pan-tumor lymph node metastasis based on bidirectional reciprocal cross-scale attention fusion, characterized in that, The method comprises the following steps: obtaining a high-scale image of a region of interest and a low-scale image of the region of interest; extracting features from the high-scale image of the region of interest to obtain a high-scale deep feature flow; extracting features from the low-scale image of the region of interest to obtain a low-scale deep feature flow; performing bidirectional reciprocal attention processing on the high-scale deep feature flow and the low-scale deep feature flow to obtain enhanced high-scale features and enhanced low-scale features; fusing the enhanced high-scale features and the enhanced low-scale features to obtain multi-scale fusion features; obtaining a final classification result according to the multi-scale fusion features.

2. The method for detecting pan-tumor lymph node metastasis based on bidirectional reciprocal cross-scale attention fusion according to claim 1, wherein, The bidirectional reciprocal attention processing on the high-scale deep feature flow and the low-scale deep feature flow to obtain enhanced high-scale features and enhanced low-scale features comprises the following steps: performing linear projection on the high-scale deep feature flow to obtain a high-scale query vector, a high-scale key vector, and a high-scale value vector; performing linear projection on the low-scale deep feature flow to obtain a low-scale query vector, a low-scale key vector, and a low-scale value vector; calculating high-scale self-attention according to the high-scale query vector and the high-scale key vector; calculating high-scale cross-attention on the low-scale according to the high-scale query vector and the low-scale key vector; calculating low-scale cross-attention on the high-scale according to the low-scale query vector and the high-scale key vector; calculating low-scale self-attention according to the low-scale query vector and the low-scale key vector; generating enhanced high-scale features and enhanced low-scale features according to the high-scale self-attention, the high-scale cross-attention on the low-scale, the low-scale cross-attention on the high-scale, the low-scale self-attention, the high-scale value vector, and the low-scale value vector.

3. The method for detecting pan-tumor lymph node metastasis based on bidirectional reciprocal cross-scale attention fusion according to claim 2, wherein, The generating enhanced high-scale features and enhanced low-scale features according to the high-scale self-attention, the high-scale cross-attention on the low-scale, the low-scale cross-attention on the high-scale, the low-scale self-attention, the high-scale value vector, and the low-scale value vector comprises the following steps: generating a high-scale output vector according to the high-scale self-attention and the high-scale value vector; generating a low-scale output vector according to the low-scale self-attention and the low-scale value vector; generating a high-scale output vector according to the high-scale self-attention and the high-scale value vector; generating a low-scale output vector according to the low-scale self-attention and the low-scale value vector; generating enhanced high-scale features according to the high-scale output vector and the low-scale output vector on the high-scale; generating enhanced low-scale features according to the low-scale output vector and the high-scale output vector on the low-scale.

4. The method for detecting pan-tumor lymph node metastasis based on bidirectional reciprocal cross-scale attention fusion according to claim 2, wherein, The high-scale self-attention is calculated according to the high-scale query vector and the high-scale key vector by the following formula: where HH attn is a high-scale self-attention; H q is a high-scale query vector; H k is a high-scale key vector.

5. The method for detecting pan-tumor lymph node metastasis based on bidirectional reciprocal cross-scale attention fusion according to claim 2, wherein, The high-scale cross-attention on the low-scale is calculated according to the high-scale query vector and the low-scale key vector by the following formula: wherein, HL attn is a high-to-low cross attention; H q is a high-scale query vector; L k is a low-scale key vector.

6. The method for detecting pan-tumor lymph node metastasis based on bidirectional reciprocal cross-scale attention fusion according to claim 2, wherein, The low-scale-to-high-scale cross-attention is calculated according to the low-scale query vector and the high-scale key vector, and is obtained by the following formula: wherein LH attn is a low-to-high cross-attention; L q is a low-scale query vector; H k is a high-scale key vector.

7. The method for detecting pan-tumor lymph node metastasis based on bidirectional reciprocal cross-scale attention fusion according to claim 2, wherein, The low-scale self-attention is calculated according to the low-scale query vector and the low-scale key vector, and includes the following formula: where LL attn is a low-scale self-attention; L q is a low-scale query vector; L k is a low-scale key vector.

8. The method for detecting pan-tumor lymph node metastasis based on bidirectional reciprocal cross-scale attention fusion of claim 3, wherein, The enhanced high-scale feature is generated according to the high-scale output vector and the low-scale-to-high-scale output vector, and is obtained by the following formula: H out = (HH out + LH out ) / 2; where H out is the enhanced high scale feature; HH out is the high scale output vector; LH out is the low scale to high scale output vector.

9. The method for detecting pan-tumor lymph node metastasis based on bidirectional reciprocal cross-scale attention fusion according to claim 3, wherein, The enhanced low-scale feature is generated according to the low-scale output vector and the high-scale-to-low-scale output vector, and is obtained by the following formula: L out = (LL out + HL out ) / 2; where L out is the enhanced low scale feature; LL out is the low scale output vector; HL out is the high scale to low scale output vector.

10. A device for detecting pan-tumor lymph node metastasis based on bidirectional reciprocal cross-scale attention fusion, characterized in that, The device for detecting lymph node metastasis of tumors based on bidirectional reciprocal cross-scale attention fusion includes: A scale image acquisition module is configured to acquire a high-scale image of a region of interest and a low-scale image of the region of interest; A high-scale deep feature stream acquisition module is configured to extract features from the high-scale image of the region of interest to obtain a high-scale deep feature stream; A low-scale deep feature stream acquisition module is configured to extract features from the low-scale image of the region of interest to obtain a low-scale deep feature stream; A bidirectional reciprocal attention enhancement module is configured to perform bidirectional reciprocal attention processing on the high-scale deep feature stream and the low-scale deep feature stream to obtain an enhanced high-scale feature and an enhanced low-scale feature; A multi-scale fusion feature acquisition module is configured to fuse the enhanced high-scale feature and the enhanced low-scale feature to obtain a multi-scale fusion feature; A classification module is configured to obtain a final classification result according to the multi-scale fusion feature.

Images