Breast cancer axillary lymph node metastasis prediction method based on multi-instance learning

Abstract

The invention discloses a breast cancer axillary lymph node metastasis prediction method based on multi-instance learning. The breast cancer axillary lymph node metastasis prediction method is used for predicting the metastasis state of axillary lymph nodes through breast cancer pathological sections. Firstly, a data enhancement function containing various disturbance strategies is designed in a data enhancement part, and the adaptability of a model to small samples and variation data is improved; secondly, a ConvNeXt network is adopted to extract instance features in the image blocks, and a feature optimization structure based on a global attention mechanism, a feedforward enhancement module, multi-head attention pooling and a residual connection mechanism is designed on the basis, so that the global context modeling capability is enhanced, and the consistency and discrimination of feature representation are improved; in addition, an MS-AttnFusion architecture based on fusion of a Scattering2D two-dimensional wavelet scattering network and an attention mechanism is designed, and through multi-scale feature decomposition and key area adaptive enhancement, a packet-level feature vector with higher discrimination capability is generated; and finally, the classifier receives the packet-level feature vector as input, and outputs a prediction result of the axillary lymph node metastasis state.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and digital image processing methods, and particularly to a method for predicting breast cancer axillary lymph node metastasis based on multi-instance learning. Background Art

[0002] Breast cancer is one of the major malignant tumors threatening the health of women globally, and the formulation of its treatment plan highly depends on the accurate assessment of the axillary lymph node (ALN) status. Whether the axillary lymph nodes are metastasized is an important basis for judging tumor staging, prognosis, and postoperative adjuvant treatment strategies. Traditional assessment methods mainly rely on the manual observation and diagnosis of pathological section images, but this process has problems such as complex operation, low diagnostic efficiency, and strong subjectivity, and there is an urgent need to improve the discrimination efficiency and stability with the help of intelligent algorithms.

[0003] With the development of artificial intelligence, especially deep learning, pathological image analysis has gradually shifted from traditional methods based on feature engineering to an end-to-end model training framework, significantly promoting the performance improvement of tasks such as image classification, detection, and segmentation. In the task of breast cancer auxiliary diagnosis, due to the characteristics of pathological images such as high resolution, high structural complexity, and strong regional differences, directly modeling and training the entire image is costly and not practically feasible. Therefore, multiple-instance learning (MIL), as a typical weakly supervised learning method, has gradually become the mainstream strategy in pathological image analysis. MIL regards the entire image as a set (bag) composed of several image patches (instances), and the model is trained according to the image-level label, avoiding the need for precise annotation of each image patch, significantly reducing the difficulty of data preparation, and improving the transferability and training efficiency of the model.

[0004] However, existing MIL methods still face many challenges in practical applications: First, traditional feature extraction networks usually rely on the convolutional neural network (CNN) architecture. Although they can extract local structural features, when dealing with regions with complex morphology and texture changes in pathological images, it is often difficult to fully capture global dependencies and context relationships, resulting in limited classification performance; Second, most feature aggregation modules adopt simple average pooling or linear weighting mechanisms, lacking the ability to perform fine-grained modeling of multi-scale and key regions, and it is difficult to achieve effective instance selection and discriminative feature enhancement; Third, the contributions of different instances to the final classification result are significantly different, and traditional methods are difficult to dynamically model the weight distribution of each instance and difficult to highlight the importance of key image patches.

[0005] To further improve the prediction performance of the axillary lymph node metastasis status in breast cancer pathological section images, it is urgent to construct a feature expression and fusion framework with high discriminative ability, which can not only maintain the integrity of local structure expression, but also adapt to the global dependence modeling requirements, and has good scale sensitivity and feature robustness, so as to achieve accurate analysis and intelligent classification of complex pathological images. Summary of the Invention

[0006] In view of this, the embodiments of the present invention provide a method for predicting axillary lymph node metastasis in breast cancer based on multi-instance learning, which is used to solve problems such as low classification accuracy, weak key region extraction ability, and poor model generalization performance in traditional methods.

[0007] The embodiments of the present invention provide a method for predicting axillary lymph node metastasis in breast cancer based on multi-instance learning, and the method includes the following steps:

[0008] S1. Obtain the BCNB breast cancer pathology open dataset, preprocess its images, and generate an image patch dataset for training;

[0009] S2. Use the pre-trained ConvNeXt neural network to extract the instance features F in the image patches;

[0010] S3. Based on the extracted instance features, design a feature optimization structure to enhance the global feature modeling ability while maintaining the integrity of local feature expression;

[0011] S4. Design a multi-scale attention fusion MS-AttnFusion architecture in the feature aggregation part, which integrates the Scattering2D two-dimensional wavelet scattering network and the multi-head attention mechanism to achieve multi-scale information decomposition and adaptive enhancement of key region features, so as to optimize the collaborative modeling of global and local information and generate a bag-level feature vector for classification prediction;

[0012] S5. Use a two-layer fully connected classifier containing the ReLU activation function to classify the bag-level feature vector and generate a prediction label for the axillary lymph node metastasis status.

[0013] Further, the preprocessing of the target image in step S1 is specifically as follows:

[0014] S11. Convert the format of the BCNB breast cancer pathology data in S1 to RGB and load it into the Tensor data structure to support subsequent image enhancement and calculation;

[0015] S12. Based on the BCNB breast cancer pathology image data after format conversion, design a data augmentation function that integrates multiple perturbation strategies Used to simulate different acquisition conditions and pathological variability, thereby enhancing the model's adaptability to complex image inputs. The function consists of multiple image transformation operations in series, and its formula is as follows:

[0016]

[0017] Where * represents the product operation, ε represents the random erasing operation of the image with probability p = 0.5, simulating the contamination of the pathological image. The size range of the erasing area is set to a rectangular area with an area of 5% to 33% of the original image and an aspect ratio range of 0.6 to 3.3, which is used to randomly select an area in the image for pixel replacement. N P represents the vertical flipping of the image (directional perturbation) with a probability of p = 0.5, τ represents the color dithering operation, which is used to simulate the interference of different lighting and imaging conditions. The brightness, contrast, and saturation are all set to be perturbed within a range of 12%, and the hue perturbation range is set to 6%. R θ Represents random rotation of the image within the angle range, with a maximum rotation of ±16° for the input image, F P represents the image horizontal flip (direction perturbation) with probability p = 0.5, C s Represents random cropping and scaling of the image (field of view change). The cropped area is 0.8 to 0.95 of the original image area, and the cropped area is scaled to a fixed size of 224×224 to unify the model input size.

[0018] This enhancement module can effectively simulate real interference factors such as different shooting angles, staining conditions, and imaging noise, thereby generating a diverse and perturbed image sample set, improving the model's adaptability to small samples and variant data.

[0019] S13. The data augmented BCNB breast cancer pathology image data is normalized to a standard range to match the input distribution of the deep learning model. Specifically, the standard deviation and mean of the image dataset are calculated and normalized according to the following formula:

[0020] I * =(I-Mean) / Std

[0021] Where I represents the pixel value of the pathological image dataset (range [0,1], converted to a tensor), Mean represents the mean of the pathological image dataset, Std represents the standard deviation of the pathological image dataset, and I * Represents the normalized pixel value, which is used to optimize feature distribution and improve the learning stability of the model.

[0022] Furthermore, based on the extracted instance features described in S3, a feature optimization architecture is designed to enhance the global context modeling capability, as follows:

[0023] S31. Perform layer normalization (LayerNorm) on the instance-level feature vector F extracted in S2 to eliminate the difference in feature distribution among different samples and improve the stability of model training;

[0024] S32. To adapt to the input dimension of the subsequent modeling structure and enhance the expressive ability of the feature dimension, linearly map the standardized features to obtain the transformed feature F e ;

[0025] S33. To model the long-range dependence relationship between image patches, introduce a global attention mechanism to calculate the contribution degree of different image patches to the global representation. This mechanism dynamically adjusts the contribution of each patch to the global feature by calculating the attention weights between the features of different image patches, thereby enhancing the discriminative ability of the model. The calculation process is as follows:

[0026] Q = W q F e ,K = W k F e ,V = W v F e ,

[0027] F T = AV

[0028] where W q ,W k ,W v are the transformation matrices of the query Q (Query), key K (Key), and value V (Value) respectively, A is the attention distribution matrix representing the relationship between different image patches, K T is the transpose of the K matrix, D is the feature dimension, is used as a scaling factor to prevent gradient disappearance or explosion, and F T is the weighted sum of the Value by the attention matrix A to obtain the final globally optimized feature, which contains global context information;

[0029] S34. To retain the original local structure features while enhancing the global expression, construct a feed-forward enhancement module FFN, and the feature F T after being calculated by the global attention mechanism enters the FFN layer to further improve the feature learning ability of the model. The calculation process is as follows:

[0030] F1 = Relu(W1F T + b1) F2 = W2F1 + b2

[0031] F′ T = LayerNorm(F2 + F T )

[0032] Among them, W1, b1, W2, and b2 are trainable parameters. W1 and W2 represent the weight matrices of the first and second fully connected layers respectively, and b1 and b2 represent the bias terms of the first and second fully connected layers respectively. The first fully connected transformation is mainly used to expand the feature dimension and enhance the model's learning ability for complex patterns. The second fully connected transformation restores the original feature dimension to ensure information integrity, F′ T As the feature vector processed by the FFN, it is input into the subsequent module to further optimize the discriminative ability of classification prediction;

[0033] S35. To enhance the global modeling ability of the key region, for the optimized image patch feature F′ T perform multi-head attention pooling processing. This mechanism calculates the attention distribution from different dimensions through multiple parallel attention heads respectively, learns the importance of each image patch for the overall discriminative task, and comprehensively constructs a more discriminative global representation, finally obtaining the pooled global feature representation F P ;

[0034] S36. To maintain the integrity of local information and utilize the optimized expression ability of global features at the same time, adopt a residual connection mechanism to fuse the initial instance-level features and the optimized global features:

[0035] F final = F + F P

[0036] Finally, F final is the optimized feature vector, which is used for subsequent feature aggregation and classification tasks.

[0037] Furthermore, in S4, a multi-scale attention fusion MS-AttnFusion architecture is designed in the feature aggregation part to achieve more effective image patch-level feature aggregation, specifically as follows:

[0038] S41. To capture discriminative features at different frequencies and spatial scales, the wavelet two-dimensional scattering network Scattering2D is introduced as a feature transformation operator to perform multi-scale transformation processing on each instance feature vector in F final Scattering2D is a predefined network structure based on stable wavelet transform and non-linear activation, which can extract stable and discriminative multi-scale feature representations without relying on a large number of training parameters. Its calculation process is as follows:

[0039] First, let the set of input instance feature vectors be: F final = {f i |f i ∈R C×H×W , i = 1, 2,..., N}

[0040] where f i represents the i-th image instance feature, C is the number of channels, H and W are the spatial dimensions of the image instance feature, and N is the total number of instances;

[0041] In the present invention, Scattering2D is composed of a set of fixed Morlet-based wavelet filter banks, which includes an omnidirectional low-pass filter and high-pass filters in multiple directions At the same time, Scattering2D is designed to construct a two-layer scattering network structure to extract the input image block features layer by layer.

[0042] In the first layer, the low-pass filter transforms the image block features at three scale levels (j = 1, 2, 3) to extract the stable low-frequency features at different spatial resolutions; at the same time, the high-pass filters in multiple directions (the number of directions θ = 8) perform directional detail modeling within the multi-scale levels (j = 1, 2, 3) on the image block features to extract the high-frequency detail information of the local structure. These responses together constitute the multi-scale and multi-directional feature representation of the instance features at the first order. For the feature f i The calculation expression for the first-layer two-dimensional scattering wavelet decomposition is:

[0043]

[0044] where, c A (j, m, n) are the approximate subband coefficients (low-frequency information) obtained after low-pass filtering, are the detail subband coefficients (high-frequency information) obtained after high-pass filtering, including 8 directions, represents the two-dimensional low-pass filter; represents the two-dimensional high-pass filter bank, (m, n) represents the spatial position index of the subband coefficients at the corresponding scale, k and l are the convolution kernel indices of the filter in the vertical and horizontal directions respectively, used to traverse the local area of the feature map to implement the convolution operation, and j represents the current scale level;

[0045] In the second layer, Scattering2D further performs the same convolution and downsampling operations on the detail subband coefficients (high-frequency information) obtained by the first-layer high-pass filter (processed by the same two-dimensional high-pass filter bank ) to extract deeper structural features. Finally, the feature representation after being processed by the two-dimensional wavelet scattering network Scattering2D is denoted as:

[0046]

[0047] where, W(f i ) represents the input instance feature fi Multi-scale feature representation processed by a two-dimensional discrete wavelet transform operator; c A (j) represents the low-frequency features at each final scale; Represents the high-frequency features in each direction at each final scale;

[0048] S42. After being processed by the above-mentioned wavelet scattering network Scattering2D, each instance feature vector f i can be represented as a multi-scale feature representation composed of low-frequency sub-band features (approximate components) at multiple different scales and high-frequency sub-band features (detail components) at multiple different scales and directions. This multi-scale and multi-direction wavelet scattering feature not only preserves the spatial hierarchical structure of the image but also has good translational invariance and stable response ability to small perturbations. However, simply using the sub-band coefficient features at all scales directly for subsequent classification or prediction tasks may have problems such as feature redundancy, information imbalance, or difficulty in highlighting effective features. Therefore, the present invention further designs a feature fusion structure based on a multi-head attention mechanism to automatically learn and highlight the key discriminant information in the multi-scale sub-band features, thereby optimizing the feature expression ability and improving the accuracy and stability of subsequent prediction tasks. The calculation process is as follows:

[0049] First, a flattening operation is performed on the multi-scale feature W(f i ) obtained by wavelet transform processing, denoted as: x i .

[0050] Secondly, to reduce feature dimension redundancy and facilitate subsequent attention mechanism processing, a fully connected (FC) layer is designed to compress the dimension of the flattened feature, denoted as: x' i .

[0051] Finally, to enable the model to adaptively capture the long-term dependence relationship between features at different scales and directions, automatically learn the importance and relevance within the features, and improve the model's ability to express key information, the present invention uses a multi-head attention mechanism to model the global correlation within the feature vector for the compressed feature x' i . Let the input of the attention mechanism be X' = [x'1, x'2,..., x' N , where N is the number of instances. The calculation formula of the multi-head self-attention mechanism is:

[0052] X attn = MHA(X') = Concat(head1, head2,..., head8)W

[0053] where head is the attention head (8 heads are used), W is the parameter to be learned, and X attn Output feature of the multi-head attention mechanism;

[0054] S43. To further enhance the ability to represent the importance of different instance features, adaptively highlight the key instances in multiple instance feature sets, and effectively reduce the interference of irrelevant features on the final prediction task, the present invention also designs an additional feature weight attention mechanism to automatically generate the attention weights of each instance feature vector. The calculation process is as follows:

[0055] u i = tanh(W u X attn + b u )

[0056]

[0057] where W u , b u , W a , b a are learnable parameters, and a i is the normalized attention weight, representing the importance of the i-th instance feature;

[0058] S44. Use the attention weight a i to perform weighted aggregation on the attention features of all instances to obtain the final fused feature representation:

[0059]

[0060] where M is the final fused feature representation (bag-level feature vector), which comprehensively integrates the key instance information and multi-scale feature expressions and serves as the input to the subsequent classifier for predicting the metastasis status of axillary lymph nodes.

[0061] In summary, the beneficial effects of the present invention are as follows:

[0062] 1. Since the prediction of axillary lymph node metastasis status relies on high-quality annotations and it is difficult for traditional methods to fully exploit the key region information in pathological images, the present invention proposes a method for predicting axillary lymph node metastasis in breast cancer based on multi-instance learning. Without the need for pixel-level annotations of images, it can directly use the image patch-level feature information to efficiently discriminate the metastasis status of axillary lymph nodes, effectively alleviating the label dependence problem, reducing the training cost, and improving the practicality and adaptability of the model.

[0063] 2. To further improve the robustness and generalization ability of the model, the present invention designs a data augmentation function that integrates multiple perturbation strategies. These data augmentation strategies effectively enhance the adaptability of the model to different image perturbations, avoid overfitting, and improve the adaptability of the model to different actual scenarios.

[0064] 3. Aiming at the problems that traditional backbone networks such as ConvNeXt have insufficient global feature modeling ability and insufficient expression of the correlation relationship between image patches, the present invention designs a feature optimization architecture to enhance the feature extraction ability of the network.

[0065] 4. Aiming at the problem that it is difficult to completely express complex tissue morphology with single spatial scale features, the present invention constructs a feature aggregation module (MS-AttnFusion) that combines multi-scale modeling and attention mechanism. This module extracts discriminant information of image patches at different frequencies and scales by designing multi-level wavelet transform operations, thereby enhancing the sensitivity of the model to regions with small structural changes and improving the expression diversity of structural features.

[0066] 5. On this basis, MS-AttnFusion effectively alleviates the problem that the information in key regions is averaged and diluted in feature aggregation by further introducing an attention weighting mechanism. By dynamically allocating the aggregation weights of image patches, it focuses on strengthening the contribution of diagnosis-related regions, improves the discriminant ability and expression integrity of the final bag-level feature vector, and provides input for subsequent classification tasks.

[0067] 6. The finally generated bag-level features not only fuse multi-scale structural features and regional saliency information, but also have better discriminability and generalization ability. As the input of the axillary lymph node metastasis status classifier, it effectively improves the prediction accuracy and has good clinical practical value and promotion potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required to be used in the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to these drawings, and these are all within the protection scope of the present invention.

[0069] Figure 1 is a framework diagram of a method for predicting axillary lymph node metastasis of breast cancer based on multi-instance learning provided by an embodiment of the present invention.

[0070] Figure 2 is a structural diagram of the MS-AttnFusion feature aggregation provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0071] The features and exemplary embodiments of various aspects of the present invention will be described in detail below. To make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only configured to explain the present invention and are not configured to limit the present invention. For those skilled in the art, the present invention can be implemented without some of these specific details. The following description of the embodiments is only provided to provide a better understanding of the present invention by showing examples of the present invention.

[0072] Please refer to Figure 1 , an embodiment of the present invention provides a method for predicting breast cancer axillary lymph node metastasis based on multi-instance learning. The method is specifically implemented according to the following steps:

[0073] Step 1, obtain the BCNB breast cancer pathology open dataset, and preprocess the target image, including the following specific steps:

[0074] (1) Convert the data format of the BCNB breast cancer pathology image to RGB and load it into the Tensor data structure to support subsequent image enhancement and calculation.

[0075] (2) Based on the BCNB breast cancer pathology image data after format conversion, a data enhancement function integrating multiple perturbation strategies is designed to simulate different acquisition conditions and pathological variabilities, thereby enhancing the model's adaptability to complex image inputs. This function is composed of multiple image transformation operations in series, and its formula is as follows:

[0076]

[0077] where * represents the product operation, ε represents performing the image random erasing operation with a probability p = 0.5 to simulate the soiling situation of the pathological image. The size range of the erased area is set to a rectangular area of 5% - 33% of the original image area, and the aspect ratio range is 0.6 - 3.3, which is used to randomly select areas in the image for pixel replacement, N P represents performing the image vertical flipping (direction perturbation) with a probability p = 0.5, τ represents the color jitter operation, which is used to simulate the interference of different lighting and imaging conditions. The perturbations of brightness, contrast, and saturation are all set within a 12% range, and the hue perturbation range is set to 6%, R θ represents randomly rotating the image within an angle range, and the input image is rotated by a maximum of ±16°, F P represents performing the image horizontal flipping (direction perturbation) with a probability p = 0.5, C sIt represents randomly cropping and scaling the image (field of view change). The cropping area is 0.8 - 0.95 of the original image area, and the cropped area will be scaled to a fixed size of 224×224 for unifying the model input size.

[0078] This enhancement module can effectively simulate real interference factors such as different shooting angles, staining conditions, and imaging noise, thereby generating an image sample set with diversity and perturbation, and improving the model's adaptability to small samples and variant data.

[0079] (3) Normalize the BCNB breast cancer pathological image data after data enhancement. The data is normalized to the standard range to match the input distribution of the deep learning model. Specifically, calculate the standard deviation and mean of the image dataset, and perform normalization according to the following formula:

[0080] I * =(I - Mean) / Std

[0081] Where I represents the pixel values of the pathological image dataset (range [0,1], converted to a tensor), Mean represents the mean of the pathological image dataset, which are 0.485, 0.456, and 0.406 respectively (corresponding to the R, G, B three channels), Std represents the standard deviation of the pathological image dataset, which are 0.229, 0.224, and 0.225 respectively (corresponding to the R, G, B three channels), and I * represents the pixel values after normalization, which is used to optimize the feature distribution and improve the learning stability of the model;

[0082] Step 2: Use the pre-trained ConvNeXt neural network to extract instance features in the image patch. The ConvNeXt network consists of four stages, each stage contains multiple ConvNeXt Blocks (composed of depthwise separable convolutions), and combines the SE Block (channel attention mechanism) and the normalization layer to enhance the stability and robustness of feature expression. Finally, perform dimensionality reduction on the extracted features through global average pooling (GAP) to obtain the instance-level feature vector F as the input for subsequent feature optimization;

[0083] Step 3: Based on the extracted instance features, construct a feature optimization architecture to enhance the global context modeling ability, including the following specific steps:

[0084] (1) Perform layer normalization (LayerNorm) on the extracted instance-level feature vector F to eliminate the feature distribution differences between different samples and improve the stability of model training.

[0085] (2) To adapt to the input dimension of the subsequent modeling structure and improve the expression ability of the feature dimension, linearly map the standardized features to obtain the transformed feature Fe 。

[0086] (3) To model the long - distance dependencies between image patches, a global attention mechanism is introduced to calculate the contribution degrees of different image patches to the global representation. This mechanism dynamically adjusts the contribution of each patch to the global feature by calculating the attention weights between the features of different image patches, thereby enhancing the discriminative ability of the model. The calculation process is as follows:

[0087] Q = W q F e ,K = W k F e ,V = W v F e ,

[0088] F T = AV

[0089] where W q 、W k 、W v are the transformation matrices of the query Q (Query), key K (Key), and value V (Value) respectively. A is the attention distribution matrix, representing the relationship between different image patches. K T is the transpose of the K matrix, d is the feature dimension, is used as a scaling factor to prevent gradient vanishing or explosion. F T is the weighted sum of the Value by the attention matrix A, obtaining the final globally optimized feature, which contains global context information.

[0090] (4) To retain the original local structural features while enhancing the global representation, a feed - forward enhancement module FFN is constructed. The feature F T calculated by the global attention mechanism enters the FFN layer to further improve the feature learning ability of the model. The calculation process is as follows:

[0091] F1 = Relu(W1F T +b1) F2 = W2F1 + b2

[0092] F′ T = LayerNorm(F2 + F T )

[0093] where W1, b1, W2, b2 are trainable parameters. W1 and W2 represent the weight matrices of the first - layer and second - layer fully - connected layers respectively, and b1 and b2 represent the bias terms of the first - layer and second - layer fully - connected layers respectively. The first - layer fully - connected transformation is mainly used to expand the feature dimension and enhance the model's learning ability for complex patterns. The second - layer fully - connected transformation restores the original feature dimension to ensure information integrity. F′ TAs the feature vector after FFN processing, it is input into the subsequent module to further optimize the discriminative ability of classification prediction.

[0094] (5) To enhance the global modeling ability of the key region, the optimized image patch feature F′ T is subjected to multi-head attention pooling. This mechanism calculates the attention distribution from different dimensions through multiple parallel attention heads respectively, learns the importance of each image patch for the overall discriminative task, and comprehensively constructs a more discriminative global representation, finally obtaining the pooled global feature representation F P .

[0095] (6) To maintain the integrity of local information and utilize the optimized global feature expression ability at the same time, a residual connection mechanism is adopted to fuse the initial instance-level feature and the optimized global feature. The calculation process is as follows:

[0096] F final = F + F P

[0097] Finally, F final is the optimized feature vector, which is used for subsequent feature aggregation and classification tasks;

[0098] Step 4, based on the optimized feature vector, a multi-scale attention fusion MS-AttnFusion architecture is designed and constructed to achieve more effective image patch-level feature aggregation, as Figure 2 shown specifically as follows:

[0099] (1) To capture discriminative features at different frequencies and spatial scales, a two-dimensional wavelet scattering network Scattering2D is introduced as a feature transformation operator to perform multi-scale transformation on each instance feature vector in F final . Scattering2D is a predefined network structure based on stable wavelet transform and non-linear activation, which can extract stable and discriminative multi-scale feature representations without relying on a large number of training parameters. The calculation process is as follows:

[0100] Let the input set of instance feature vectors be: F final = {f i |f i ∈ R C×H×W , i = 1, 2,..., N}.

[0101] Among them, f i represents the i-th image instance feature, C is the number of channels, H and W are the spatial dimensions of the image instance feature, and N is the total number of instances.

[0102] In the present invention, Scattering2D is composed of a set of fixed Morlet-based wavelet filter banks, which includes an undirectional low-pass filter and high-pass filters in multiple directions At the same time, Scattering2D is designed to construct a two-layer scattering network structure to extract the features of the input image block layer by layer.

[0103] In the first layer, the low-pass filter transforms the features of the image block at three scale levels (j = 1, 2, 3) to extract its stable low-frequency features at different spatial resolutions; at the same time, within the three scale levels (j = 1, 2, 3), the high-pass filters in multiple directions (the number of directions θ = 8) perform directional detail modeling on the features of the image block to extract the high-frequency detail information of the local structure. These responses together constitute the multi-scale and multi-directional feature representation of the instance feature at the first order. For the feature f i The calculation expression for the first-layer two-dimensional scattering wavelet decomposition is:

[0104]

[0105] where c A (j + 1, m, n) is the approximate subband coefficient (low-frequency information) obtained after low-pass filtering, is the detail subband coefficient (high-frequency information) obtained after high-pass filtering, including 8 directions, represents the two-dimensional low-pass filter; represents the two-dimensional high-pass filter bank, (m, n) represents the spatial position index of the subband coefficient at the corresponding scale, k and l are the convolution kernel indices of the filter in the vertical and horizontal directions respectively, used to traverse the local area of the feature map to implement the convolution operation, and j represents the current scale level.

[0106] In the second layer, Scattering2D further performs the same convolution and downsampling operations on the detail subband coefficients (high-frequency information) obtained by the first-layer high-pass filter (processed by the same two-dimensional high-pass filter bank ) to extract deeper structural features. Finally, the feature representation after being processed by the two-dimensional wavelet scattering network Scattering2D is denoted as:

[0107]

[0108] where W(f i ) represents the multi-scale feature representation of the input instance feature f i after being processed by the two-dimensional discrete wavelet transform operator; c A (j) represents the low-frequency features at the final scales; represents the high-frequency features at the final scales and in all directions;

[0109] (2) After being processed by the above wavelet scattering network Scattering2D, each instance feature vector f i They can all be represented as multi-scale feature representations consisting of multiple low-frequency sub-band features (approximate components) of different scales and multiple high-frequency sub-band features (detail components) of different scales and directions. This multi-scale and multi-directional wavelet scattering feature not only maintains the spatial hierarchical structure of the image, but also has good translation invariance and stable response to small disturbances. However, simply using sub-band coefficient features of all scales directly for subsequent classification or prediction tasks may have problems such as feature redundancy, information imbalance, or difficulty in highlighting effective features. Therefore, the present invention further designs a feature fusion structure based on a multi-head attention mechanism to automatically learn and highlight the key discriminant information in the multi-scale sub-band features, thereby optimizing the feature expression capability and improving the accuracy and stability of subsequent prediction tasks. The calculation process is as follows:

[0110] First, the multi-scale features W(f i ) is flattened and recorded as x i .

[0111] Secondly, in order to reduce the redundancy of feature dimensions and facilitate the subsequent attention mechanism processing, a fully connected (FC) layer is designed to compress the dimension of the flattened features, which is recorded as: i .

[0112] Finally, in order to enable the model to adaptively capture the long-term dependencies between features of different scales and directions, automatically learn the importance and correlation within the features, and improve the model's ability to express key information, the present invention performs the following operations on the compressed features x′: i Use the multi-head attention mechanism to model the global correlation within the feature vector, and assume that the attention mechanism input X′=[x′1,x′2,…,x′ N ], where N is the number of instances. The calculation formula of the multi-head self-attention mechanism is:

[0113] X attn =MHA(X′)=Concat(head1,head2,…,head h )W∈R N×L

[0114] Where head is the attention head (8 heads are used), W is the parameter to be learned, X attn Multi-head attention mechanism output features.

[0115] (3) To further enhance the ability to represent the importance of different instance features, adaptively highlight the key instances in multiple instance feature sets, and effectively reduce the interference of irrelevant features on the final prediction task, the present invention also designs an additional feature weight attention mechanism to automatically generate the attention weights of each instance feature vector. The calculation process is as follows:

[0116] u i =tanh(W u X attn +b u )

[0117]

[0118] where W u and b u and W a and b a are learnable parameters, and a i is the normalized attention weight, representing the importance degree of the i-th instance feature.

[0119] (4) Use the attention weight a i to perform weighted aggregation on the attention features of all instances to obtain the final fused feature representation:

[0120]

[0121] where M is the final fused feature representation (bag-level feature vector), which comprehensively integrates the key instance information and multi-scale feature expressions, and serves as the input to the subsequent classifier for predicting the metastasis status of axillary lymph nodes;

[0122] Step 5, based on the final fused feature, use a two-layer fully connected classifier containing the ReLU activation function to classify the bag-level feature vector. The specific process is as follows:

[0123] First, input the bag-level feature vector M into the first-layer fully connected layer for non-linear feature transformation to obtain the hidden layer feature h to improve the discriminative ability of the features; subsequently, input the hidden layer feature h into the second-layer fully connected layer (output layer), and map it to the range of [0,1] through the Sigmoid activation function to generate the prediction probability y of the instance belonging to the class; finally, according to the prediction probability y and the preset threshold, obtain the prediction label of the axillary lymph node metastasis status corresponding to the breast cancer pathological slice instance. The calculation process is as follows:

[0124] h = ReLU(W1M + b1)

[0125] y = σ(W2h + b2)

[0126]

[0127] Among them, ReLU is a non-linear activation function, σ is the Sigmoid activation function, W1 and W2 are the weight parameters of the fully connected layer, and b1 and b2 are the bias terms of the fully connected layer; in the lable, a label of 1 indicates that the predicted axillary lymph node metastasis status corresponding to the breast cancer pathological section instance is positive (metastasis occurred), and a label of 0 indicates negative (no metastasis occurred).

[0128] To verify that the method proposed in the present invention significantly improves the performance in the axillary lymph node metastasis status prediction task, the present invention conducts experimental evaluations based on the real breast cancer pathological section image dataset BCNB, and uses different pre-trained neural networks to extract instance features in the image patches. The results are shown in Table 1.

[0129] Table 1 Experimental comparison of different feature extractors (backbone)

[0130] backbone AUC ACC(%) SENS(%) SPEC(%) PPV(%) NPV(%) AlexNet 0.780 73.39 83.33 67.16 61.40 86.54 ResNet50 0.607 58.72 66.67 53.73 47.46 72.00 DenseNet121 0.739 69.27 85.71 58.96 56.69 86.81 Inception-v3 0.762 71.10 85.71 61.94 58.54 87.37 VGG-11BN 0.830 73.91 60.76 82.19 68.55 76.92 VGG-13BN 0.837 73.95 54.84 85.86 70.82 75.63 VGG-16BN 0.831 75.69 89.29 67.16 63.03 90.91 ConvNeXt (Ours) 0.856 79.43 66.79 87.39 76.75 80.76

[0131] Among multiple mainstream network structures, the ConvNeXt structure adopted by the present method performs excellently in multiple evaluation indicators such as AUC, ACC, SPEC, PPV, and NPV. The overall performance is better than other comparison networks. Especially in terms of accuracy (ACC 79.43%) and specificity (SPEC 87.39%), it shows significant advantages, verifying that the selected backbone has stronger feature representation ability and effectively supports the subsequent classification task.

[0132] Furthermore, to evaluate the prediction performance of the overall method of the present invention, a final complete axillary lymph node status prediction model is constructed, and the performance of the model is tested on multiple core evaluation indicators. The results show that the model reaches 0.871, 81.70%, 78.62%, 83.67%, 75.13%, and 86.27% in AUC, ACC, SENS, SPEC, PPV, and NPV respectively. The overall performance is excellent, and it has high prediction accuracy and clinical application potential.

[0133] In summary, the present invention provides an axillary lymph node metastasis prediction method for breast cancer based on multi-instance learning.

[0134] It should be clear that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, the detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order between steps after understanding the spirit of the present invention.

[0135] It should also be noted that in the exemplary embodiments mentioned in the present invention, some methods or systems are described based on a series of steps or devices. However, the present invention is not limited to the order of the above steps. That is to say, the steps can be executed in the order mentioned in the embodiments, or different from the order in the embodiments, or several steps can be executed simultaneously.

[0136] As described above, the above is only the specific implementation manner of the present invention. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described systems, modules, and units can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present invention.

Claims

1. A method for predicting breast cancer axillary lymph node metastasis based on multi-instance learning, characterized in that, The method includes the following steps: S1. Obtain the BCNB breast cancer pathology open dataset, preprocess its images, and generate an image patch dataset for training; S2. Use the pre-trained ConvNeXt neural network to extract the instance features F in the image patches; S3. Based on the extracted instance features, design a feature optimization structure to enhance the global feature modeling ability while maintaining the integrity of local feature expression; S4. Design a multi-scale attention fusion MS-AttnFusion architecture in the feature aggregation part. This architecture fuses the Scattering2D two-dimensional wavelet scattering network and the multi-head attention mechanism to achieve multi-scale information decomposition and adaptive enhancement of key region features, thereby optimizing the collaborative modeling of global and local information and generating a bag-level feature vector for classification prediction; S5. Use a two-layer fully connected classifier containing the ReLU activation function to classify the bag-level feature vector and generate a prediction label for the axillary lymph node metastasis status.

2. The breast cancer axillary lymph node metastasis prediction method based on multi-instance learning according to claim 1, wherein The preprocessing of the target image described in S1 is specifically as follows: S11. Convert the format of the BCNB breast cancer pathology data described in S1 to RGB and load it into the Tensor data structure to support subsequent image enhancement and calculation; S12. Based on the BCNB breast cancer pathological image data after format conversion, a data augmentation function θ that integrates multiple perturbation strategies is designed aug , which is used to simulate different acquisition conditions and pathological variabilities, so as to enhance the model's adaptability to complex image inputs. The function θ aug is composed of multiple image transformation operations in series, and its formula is as follows: θ aug (I) = ε * N P * τ * R θ * F P * C s (I) where * represents the multiplication operation, ε represents performing the image random erasing operation with probability p = 0.5 to simulate the soiling condition of the pathological image. The size range of the erased area is set as a rectangular area with 5% - 33% of the original image area and an aspect ratio range of 0.6 - 3.3, which is used to randomly select an area in the image for pixel replacement, N P represents performing the vertical flipping of the image (direction perturbation) with probability p = 0.5, and τ represents the color jitter operation, which is used to simulate the interference of different lighting and imaging conditions. The perturbations of brightness, contrast, and saturation are all set within a range of 12%, and the perturbation range of hue is set as 6%, R θ represents randomly rotating the image within an angle range, and the input image is rotated by a maximum of ±16°, F P represents performing the horizontal flipping of the image (direction perturbation) with probability p = 0.5, C s represents randomly cropping and scaling the image (field of view change). The cropped area is 0.8 - 0.95 of the original image area, and the cropped area will be scaled to a fixed size of 224×224 for unifying the model input size; This enhancement module can effectively simulate real interference factors such as different shooting angles, staining conditions, and imaging noise, thereby generating a diverse and perturbed image sample set and improving the model's adaptability to small samples and variant data; S13. Normalize the BCNB breast cancer pathology image data after data enhancement, and normalize the data to the standard range to match the input distribution of the deep learning model; specifically, calculate the standard deviation and mean of the image dataset and perform normalization processing according to the following formula: Where I represents the pixel values of the pathological image dataset (range [0, 1], converted to a tensor), Mean represents the mean of the pathological image dataset, Std represents the standard deviation of the pathological image dataset, and I * represents the pixel values after normalization, which are used to optimize the feature distribution and improve the learning stability of the model.

3. A method for predicting breast cancer axillary lymph node metastasis based on multi-instance learning according to claim 1, wherein The design of the feature optimization architecture based on the extracted instance features described in S3 to enhance the global context modeling ability is specifically as follows: S31. Perform layer normalization (LayerNorm) on the instance-level feature vector F extracted in S2 to eliminate the feature distribution differences between different samples and improve the stability of model training; S32. To adapt to the input dimension of the subsequent modeling structure and enhance the expression ability of the feature dimension, linearly map the standardized features to obtain transformed feature F e ; S33. To model the long-range dependence relationship between image patches, introduce a global attention mechanism to calculate the contribution degree of different image patches to the global representation. This mechanism dynamically adjusts the contribution of each patch to the global feature by calculating the attention weights between different image patch features, thereby enhancing the discriminative ability of the model. The calculation process is as follows: F T = aV Among them, W q , W k , W v are the transformation matrices of query Q (Query), key K (Key), and value V (Value) respectively. A is the attention distribution matrix, representing the relationship between different image patches. K T is the transpose of the K matrix, D is the feature dimension, is used as a scaling factor to prevent gradient vanishing or explosion. F T [[ID= S34. To retain the original local structural features while enhancing the global representation, a feed-forward enhancement module FFN is constructed, and the feature F after the global attention mechanism calculation T enters the FFN layer to further improve the feature learning ability of the model, and its calculation process is as follows: F1 = Relu(W1F T + b1) F2 = W2F1 + b2 F′ T = LayerNorm(F2 + F T ) Among them, W1, b1, W2, and b2 are trainable parameters. W1 and W2 represent the weight matrices of the first and second fully connected layers respectively, and b1 and b2 represent the bias terms of the first and second fully connected layers respectively. The first fully connected transformation is mainly used to expand the feature dimension and enhance the model's learning ability for complex patterns. The second fully connected transformation restores the original feature dimension to ensure information integrity, F ′ As the feature vector processed by the FFN, it is input into the subsequent module to further optimize the discriminant ability of classification prediction; S35. To enhance the global modeling ability of the key region, the optimized image patch feature F' T is subjected to multi-head attention pooling. This mechanism calculates the attention distribution from different dimensions in parallel through multiple attention heads, learns the importance of each image patch for the overall discrimination task, and comprehensively constructs a more discriminative global representation, finally obtaining the pooled global feature representation F P ; S36. To maintain the integrity of local information and utilize the global feature optimization expression ability at the same time, adopt a residual connection mechanism to fuse the initial instance-level feature and the optimized global feature: F final = F + F P Finally, F final is the optimized feature vector for subsequent feature aggregation and classification tasks.

4. A method for predicting breast cancer axillary lymph node metastasis based on multi-instance learning according to claim 1, characterized in that, The design of a multi-scale attention fusion MS-AttnFusion architecture in the feature aggregation part described in S4 to achieve more effective image patch-level feature aggregation is specifically as follows: S41. To capture discriminative features at different frequencies and spatial scales, a two-dimensional wavelet scattering network Scattering2D is introduced as a feature transformation operator to perform multi-scale transformation processing on each instance feature vector in F. final Scattering2D is a predefined network structure based on the stationary wavelet transform and non-linear activation, which can extract stable and discriminative multi-scale feature representations without relying on a large number of training parameters. Its calculation process is as follows: Let the set of input instance feature vectors be: F final ={f i |f i ∈R C×H×W , i = 1, 2, …, N}; where f i represents the i-th image instance feature, C is the number of channels, H and W are the spatial dimensions of the image instance feature, and N is the total number of instances; In the present invention, Scattering2D is composed of a set of fixed Morlet-based wavelet filter banks, which includes an isotropic low-pass filter and high-pass filters in multiple directions At the same time, Scattering2D is designed to construct a two-layer scattering network structure to extract the features of the input image block layer by layer; In the first layer, the low-pass filter transforms the image patch features at three scale levels (j = 1, 2, 3) to extract the stable low-frequency features at different spatial resolutions; meanwhile, the high-pass filters in multiple directions (number of directions θ = 8) perform directional detail modeling within the multi-scale levels (j = 1, 2, 3) on the image patch features to extract the high-frequency detail information of the local structure. These responses together constitute the multi-scale and multi-directional feature representation of the instance feature at the first order, for the feature f i The computational expression for performing the two-dimensional scattering wavelet decomposition in the first layer is as follows: Among them, c A (j, m, n) are the approximate subband coefficients (low-frequency information) obtained after low-pass filtering, are the detail subband coefficients (high-frequency information) obtained after high-pass filtering, including 8 directions, represents a two-dimensional low-pass filter, represents a two-dimensional high-pass filter bank, (m, n) represents the spatial position index of the subband coefficients at the corresponding scale, k and l are the convolution kernel indices of the filter in the vertical and horizontal directions respectively, used to traverse the local area of the feature map to implement the convolution operation, and j represents the current scale level; In the second layer, Scattering2D further performs the same convolution and downsampling operations (the same two-dimensional high-pass filter bank processing) on the detail subband coefficients (high-frequency information) obtained by the first-layer high-pass filter to extract deeper structural features. Finally, the feature representation after being processed by the two-dimensional wavelet scattering network Scattering2D is denoted as: Among them, W(f i ) represents the multi-scale feature representation of the input instance feature f i after being processed by the two-dimensional discrete wavelet transform operator; c A (j) represents the low-frequency feature at each final scale; represents the high-frequency feature in each direction at each final scale; S42. After being processed by the above-mentioned wavelet scattering network Scattering2D, each instance feature vector f i can be represented as a multi-scale feature representation composed of low-frequency sub-band features (approximate components) of multiple different scales and high-frequency sub-band features (detail components) of multiple different scales and directions. This multi-scale and multi-directional wavelet scattering feature not only preserves the spatial hierarchical structure of the image, but also has good translation invariance and stable response ability to small perturbations. However, simply using the sub-band coefficient features of all scales directly for subsequent classification or prediction tasks may have problems such as feature redundancy, information imbalance, or difficulty in highlighting effective features. Therefore, the present invention further designs a feature fusion structure based on the multi-head attention mechanism to automatically learn and highlight the key discriminant information in the multi-scale sub-band features, thereby optimizing the feature expression ability and improving the accuracy and stability of subsequent prediction tasks. The calculation process is as follows: First, perform a flattening operation on the multi-scale feature W(f i ) obtained by wavelet transform processing, denoted as: x i ; Secondly, to reduce the redundancy of feature dimensions and facilitate subsequent attention mechanism processing, a fully connected (FC) layer is designed to compress the flattened features, denoted as: x' i ; Finally, to enable the model to adaptively capture the long-term dependencies between features of different scales and directions, automatically learn the importance and correlation within the features, and improve the model's ability to express key information, the present invention processes the compressed feature x′ i uses the multi-head attention mechanism to model the global correlation within the feature vector. Let the input of the attention mechanism be X′ = [x′1, x′2, …, x′ N , where N is the number of instances. The calculation formula of the multi-head self-attention mechanism is as follows: X attn = MHA(X′) = Concat(head1, head2, …, head8)W where head is the attention head (with 8 heads), W is the parameter to be learned, and X attn Output features of the multi-head attention mechanism; S43. To further enhance the ability to represent the importance of different instance features, adaptively highlight the key instances in multiple instance feature sets, and effectively reduce the interference of irrelevant features on the final prediction task, the present invention also designs an additional feature weight attention mechanism to automatically generate the attention weights of each instance feature vector. The calculation process is as follows: u i =tanh(W u X attn +b u ) Among them, W u , b u , W a , b a are learnable parameters, and a i is the normalized attention weight, indicating the importance of the i-th instance feature; S44. Adopt the attention weight a i Perform weighted aggregation on the attention features of all instances to obtain the final fused feature representation: Among them, M is the final fused feature representation (bag-level feature vector), which comprehensively integrates the key instance information and multi-scale feature expressions, and serves as the input for the subsequent classifier to predict the metastasis status of axillary lymph nodes.

Images