Breast cancer metastasis risk assessment method based on neural network fusion mechanism

Abstract

The present invention discloses a breast cancer metastasis risk assessment method based on a neural network fusion mechanism, and belongs to the technical field of medical image processing. The present invention comprises: constructing a sentinel lymph node SLN ultrasound image dataset, including a SLN two-dimensional ultrasound dataset TDUS and a SLN ultrasound contrast imaging dataset CEUS; constructing a breast cancer metastasis risk assessment model, the assessment model comprising: a first sub-network based on a residual network, a second sub-network based on a self-attention neural network, and a fuzzy integral fusion module based on Choquet fuzzy integral; wherein the first and second sub-networks are respectively used to predict the binary assessment results of the breast cancer metastasis risk of the current input image data; the assessment results output by the sub-networks are entered into the result fusion processing through the fuzzy integral fusion module based on Choquet fuzzy integral to output the final assessment result, thereby realizing a breast cancer metastasis risk auxiliary assessor for machine reading of SLN ultrasound images. The breast cancer metastasis risk assessment model provided by the method of the present invention can efficiently and accurately assist physicians in predicting the risk of breast cancer metastasis, improve the efficiency of doctors, and reduce the workload of doctors.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a method for assessing breast cancer metastasis risk based on a neural network fusion mechanism. Background Art

[0002] Breast cancer is the malignant tumor with the highest incidence rate among women, and it is endangering the health of women around the world. Assessing whether breast cancer has metastasized is crucial for the formulation of patient treatment plans. Breast cancer tumor cells mainly spread and metastasize to various parts of the patient's body through the axillary lymph nodes, so the status of the axillary lymph nodes is a key factor in assessing whether breast cancer has metastasized. The initial method for assessing the status of axillary lymph nodes is axillary lymph node dissection (ALND). Because ALND removes local lymphatic vessels, lymph nodes, and blood vessels, patients are prone to sequelae such as upper limb edema, local pain, and limited mobility. At the same time, all removed axillary lymph nodes must be tested, which is time-consuming, labor-intensive, and expensive. In order to address the shortcomings of ALND, the sentinel lymph node biopsy technology (SLNB) has been proposed in the field of breast cancer diagnosis and treatment. The sentinel lymph node (SLN) is the first lymph node to identify the lymphatic flow direction of malignant lesions. The literature "N. Beechey-Newman. Sentinel node biopsy: a revolution in the surgical management of breast cancer [J]. Cancer Treat ReV 1998, 24: 185-203" shows that the status of the sentinel lymph node can represent the status of the entire axillary lymph node, and SLNB can replace ALND as an important technical means to determine breast cancer metastasis and staging. SLNB uses invasive surgery to remove the sentinel lymph node for biopsy. This method is very accurate in determining whether breast cancer has metastasized and is called the "gold standard" by the industry. Because SLNB uses invasive surgery, it still causes damage to the patient's upper limbs, causing local bleeding, infection, pain and other sequelae. In the latest medical practice, doctors usually use SLN ultrasound images to assess whether breast cancer has metastasized. This method has the advantages of being non-invasive, convenient, and inexpensive. However, traditional manual film reading is affected by the professional level of doctors and has a high misdiagnosis rate. Therefore, a fast, accurate, and universal method for predicting the risk of breast cancer metastasis is urgently needed. Summary of the invention

[0003] The present invention proposes a breast cancer metastasis risk assessment method based on a neural network fusion mechanism to solve the problem that the current manual reading of SLN ultrasound images is affected by the professional level of doctors and has a high misdiagnosis rate, thereby realizing intelligent auxiliary diagnosis based on neural networks.

[0004] The technical solution adopted by the present invention is:

[0005] A method for assessing breast cancer metastasis risk based on a neural network fusion mechanism, the method comprising the following steps:

[0006] Step 1, constructing a sentinel lymph node SLN ultrasound image dataset to tune the network parameters of the constructed breast cancer metastasis risk assessment model on the dataset (i.e., constructing a training set and a validation set for the assessment model); the SLN ultrasound image dataset includes a SLN two-dimensional ultrasound dataset TDUS and a SLN contrast-enhanced ultrasound dataset CEUS;

[0007] Step 2, constructing a breast cancer metastasis risk assessment model, which includes: a first sub-network (DE_ResNet sub-network) based on the residual network ResNet, a second sub-network (Fold_ViT sub-network) based on the self-attention neural network Vision Transformer, and a fuzzy integral fusion module based on Choquet fuzzy integral; wherein the DE_ResNet sub-network and the Fold_ViT sub-network are respectively used to predict the binary classification assessment results of the breast cancer metastasis risk of the current input image data, including the positive probability and the negative probability;

[0008] The DE_ResNet sub-network and the Fold_ViT sub-network are trained and verified on the TDUS dataset respectively. When the preset training end conditions are met, the trained TDUS-based DE_ResNet sub-network and TDUS-based Fold_ViT sub-network are obtained.

[0009] The DE_ResNet subnetwork and the Fold_ViT subnetwork are trained and verified on the CEUS dataset respectively. When the preset training end conditions are met, the trained CEUS-based DE_ResNet subnetwork and CEUS-based Fold_ViT subnetwork are obtained; wherein the training end conditions can be set to the number of training times, the convergence of the loss function value, or the prediction performance reaching the expected value, etc.

[0010] The evaluation results output by the four sub-networks are fused through a fuzzy integral fusion module based on Choquet fuzzy integral to obtain the fuzzy integral positive probability and the fuzzy integral negative probability, and then the final result of the breast cancer metastasis risk is obtained based on the maximum principle. Thus, based on the constructed breast cancer metastasis risk assessment model (four sub-networks and a fuzzy integral fusion module based on Choquet fuzzy integral), an auxiliary evaluator for breast cancer metastasis risk that can realize machine reading of SLN ultrasound images is obtained.

[0011] Furthermore, step 1 specifically includes:

[0012] Organize the collected SLN ultrasound image data set and divide it into training set and validation set according to the specified ratio;

[0013] Manual annotation and expert review of sentinel lymph nodes were performed on the image data of the SLN ultrasound image dataset;

[0014] According to the annotation results, the sentinel lymph node image is cropped and the image size is normalized.

[0015] Furthermore, in step 2, the DE_ResNet subnetwork includes, in sequence: a convolutional layer, a first convolutional module, a second convolutional module, a third convolutional module, a fourth convolutional module, an average pooling layer, a fully connected layer, and a softmax function layer;

[0016] Among them, the first convolution module includes a stacked structure consisting of a maximum pooling layer and several ELU_ResNet residual modules in sequence; the second convolution module, the third convolution module and the fourth convolution module are stacked structures consisting of several residual modules respectively, and the first residual module of the second convolution module, the third convolution module and the fourth convolution module is a residual module with downsampling, and the remaining residual modules are ELU_ResNet residual modules; wherein, the main branch of the ELU_ResNet module includes three convolution layers in sequence, and a jump connection is set between the input of the ELU_ResNet module and its last convolution layer; the residual module with downsampling includes a main branch and a secondary branch, the main branch includes three convolution layers, and the secondary branch includes a convolution layer and an average pooling layer in sequence. The inputs of the main and secondary branches are the same, and finally the output feature maps of the main and secondary branches are added to obtain the output feature map of the residual module with downsampling.

[0017] Furthermore, in step 2, the structure of the Fold_ViT subnetwork is as follows: the input is a three-channel color image, the input image data is dimensionally upgraded through the convolution layer, and then the output feature map of the convolution layer is flattened into a vector through the flattening operation of height and width, and then the flattened vector is concatenated with the classification vector and added to the position code, and then passed through the Dropout layer; then the output feature map of the Dropout layer is decomposed into multiple data blocks through the Unfold operation to form a block sequence; each block sequence is sent to the Transformers module for encoding processing, and the output of the Transformers module is restored to the block sequence shape through the Fold operation, and then the class tokens in the sequence are extracted after normalization, and finally the classification result is output through the prediction head based on the multi-layer perceptron.

[0018] Furthermore, in step 1, the data labels of the constructed sentinel lymph node SLN ultrasound image dataset are set based on the sentinel lymph node biopsy results.

[0019] The technical solution provided by the present invention brings at least the following beneficial effects:

[0020] The breast cancer metastasis risk assessment method based on the neural network fusion mechanism provided by the present invention has excellent assessment performance. The breast cancer metastasis risk assessment model provided by the method of the present invention can efficiently and accurately assist physicians in predicting the risk of breast cancer metastasis, thereby improving physician efficiency and reducing physician workload. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A schematic diagram of the process of a method for assessing breast cancer metastasis risk based on a neural network fusion mechanism provided by an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of the structure of the ELU_ResNet Block used in an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of the structure of the DDsampling Block used in an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of the structure of the Fold_ViT network used in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] To make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described in detail and completely in conjunction with the drawings in the implementation of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present invention.

[0027] The embodiment of the present invention provides a breast cancer metastasis risk assessment method based on a neural network fusion mechanism, which mainly includes: constructing a sentinel lymph node SLN ultrasound image data set (including a SLN two-dimensional ultrasound data set and a SLN ultrasound angiography data set), dividing it into a training set, a validation set and a test set; then constructing a network model for risk assessment, which includes a DE_ResNet subnetwork, a Fold_ViT subnetwork and a fusion network ChoquetFusionNet, training the constructed DE_ResNet subnetwork and Fold_ViT subnetwork based on the training set and the validation set, and using a hierarchical analysis method to calculate the fuzzy density corresponding to each subnetwork to obtain the corresponding fuzzy measure between the subnetworks, and then using Choquet fuzzy integral to measure the importance of each subnetwork, and then fusing the judgment results of the subnetwork on the dual-modality ultrasound image. Choquet fuzzy integral is a comprehensive evaluation method based on fuzzy measurement to process the correlation problem between attributes, and intuitively reflects the mutual influence between multiple attributes in the form of addition and product. This integral form can separate the size of the fuzzy measure from the importance of the attribute and the subjective judgment of the expert, so that the evaluation result is more objective and accurate. That is, in the embodiment of the present invention, a fuzzy integral fusion module based on Choquet fuzzy integral is used to fuse the output evaluation results of each sub-network to obtain the final risk assessment.

[0028] In an embodiment of the present invention, the constructed data sets include three types of data sets, namely, a SLN two-dimensional ultrasound data set TDUS (Two-dimensional Ultrasound) data set, a SLN ultrasound contrast imaging data set CEUS (Contrast-enhanced Ultrasound) data set, and a dual-modality data set DMUS (Dual Modality Ultrasound) having both patient two-dimensional ultrasound and ultrasound contrast imaging images.

[0029] In one embodiment, the steps for constructing the TDUS and CEUS datasets of the embodiment of the present invention are as follows:

[0030] 1) Organize hospital data: The naming method of ultrasound images can be set as: "Expert judgment result + gold standard + patient number". The expert judgment result is the judgment result of the relevant medical research team, which is used in the subsequent model evaluation stage and compared with the model judgment result. Negative is n and positive is p. The gold standard is the sentinel lymph node biopsy result, which is used as the annotation label during model training and is also used to calculate the model accuracy, false negative rate and other indicators in the test set. Negative is n and positive is p. The patient number can be a 5-digit number. For example, the patient folder is named "nn23157", which means that the doctor judged that the patient was negative, the lymph node biopsy result of the patient was negative, and the patient number was 23157.

[0031] 2) Dataset division: First, the hospital's ultrasound images were classified according to two-dimensional ultrasound and ultrasound contrast. Then, the dataset was divided into training set, validation set, and test set according to the ratio of 8:1:1. Finally, the training set, validation set, and test set were divided into negative and positive categories according to the gold standard test results.

[0032] 3) Labeling sentinel lymph nodes: Since the sentinel lymph nodes in the axilla are relatively small, and the ultrasound image is collected from the entire axilla, it is necessary to manually label and select the sentinel lymph node image in the ultrasound image. After communicating with medical experts, we learned that the state of the tissue surrounding the lymph node (such as whether the edge is smooth, whether the echo is clear, etc.) is important information for judgment, so during the selection process, the image should be larger than the size of the sentinel lymph node to include the surrounding tissue.

[0033] 4) Hospital expert review: The annotated and framed images are returned to the hospital team to double-check whether the frame is correct.

[0034] 5) Cropping the sentinel lymph node image: Cropping out a separate sentinel lymph node image based on the frame selection result.

[0035] 6) Adjust the image size: Since the input image size of the neural network of this system is 224×224, the image size needs to be adjusted. The long side of the image is adjusted to 224 and the short side of the image is filled with black.

[0036] 7) Storing in a corresponding data set: storing the obtained sentinel lymph node ultrasound image in a corresponding data set.

[0037] In one embodiment, the steps of constructing the DMUS dataset of the embodiment of the present invention are as follows:

[0038] 1) Organize hospital data. Create folders based on patients. The folder naming method is "expert judgment result + gold standard + patient number".

[0039] 2) Determine whether both 2D and angiography are available: In the actual diagnosis and treatment process, the most basic 2D ultrasound image acquisition is performed first. If the doctor cannot make a judgment based on 2D ultrasound, the more expensive and accurate ultrasound angiography will be used. Some patients may cancel the acquisition of ultrasound angiography because 2D ultrasound is sufficient for judgment, so some patients lack angiography data. The embodiment of the present invention is based on comprehensive judgment based on both 2D and angiography. Therefore, if one of 2D and angiography is missing, the patient data set needs to be discarded.

[0040] 3) Label the sentinel lymph nodes.

[0041] 4) Review by hospital experts.

[0042] 5) Crop the sentinel lymph node image.

[0043] 6) Resize the image.

[0044] 7) Save in patient folder: The patient folder has two sub-files, erwei (two-dimensional ultrasound) and zaoying (ultrasound contrast imaging). Save the two-dimensional and contrast images of the sentinel lymph node in the files respectively. The image naming method is "expert judgment result + gold standard + patient number + image category".

[0045] In one embodiment, the network model constructed by the embodiment of the present invention includes: a DE_ResNet sub-network based on a convolutional neural network ResNet, a Fold_ViT sub-network based on a self-attention neural network Vision Transformer, and a neural network fusion network ChoquetFusionNet based on Choquet fuzzy integral, and the specific settings are as follows:

[0046] (1) Based on the convolutional neural network ResNet, it adopts optimization measures such as delayed downsampling and ELU (Exponential Linear Units) activation function to obtain the DE_ResNet sub-network, and trains and verifies the model on the training set and validation set of the TDUS dataset and the training set and validation set of the CEUS dataset respectively; records the training weights to obtain the TDUS-based DE_ResNet sub-network and the CEUS-based DE_ResNet sub-network;

[0047] (2) Based on the self-attention neural network Vision Transformer, a lightweight improvement was made, and the Unfold_Fold optimization measure was adopted to obtain the Fold_ViT sub-network. The model was trained and verified on the training set and validation set of the TDUS dataset and the training set and validation set of the CEUS dataset respectively; the training weights were recorded to obtain the Fold_ViT sub-network based on TDUS and the Fold_ViT sub-network based on CEUS;

[0048] (3) A neural network fusion network based on Choquet fuzzy integral, ChoquetFusionNet, was constructed. The fuzzy density corresponding to each sub-network (DE_ResNet sub-network and Fold_ViT sub-network) and the corresponding fuzzy measure between sub-networks were calculated using the hierarchical analysis method. The judgment results of the sub-networks on dual-modal ultrasound images were fused using the Choquet fuzzy integral.

[0049] In one embodiment, the test steps of the constructed network model in the embodiment of the present invention include:

[0050] T1, input the ultrasound images in the test set of the TDUS dataset into the TDUS-based DE_ResNet subnetwork and the TDUS-based Fold_ViT subnetwork, and calculate the accuracy, false negative rate, AUC (Area Under Curve) and other indicators based on the judgment results output by them and the gold standard;

[0051] T2, input the ultrasound images in the test set of the CEUS dataset into the CEUS-based DE_ResNet subnetwork and the CEUS-based Fold_ViT subnetwork, and calculate the accuracy, false negative rate, AUC and other indicators based on their output judgment results and the gold standard;

[0052] T3, the two-dimensional and contrast ultrasound images of the patients in the DMUS data set are simultaneously input into the breast cancer metastasis risk assessment model constructed by the embodiment of the present invention, which includes a DE_ResNet subnetwork and a Fold_ViT subnetwork based on TDUS, a DE_ResNet subnetwork and a Fold_ViT subnetwork based on CEUS, and a fuzzy integral fusion module based on Choquet fuzzy integral; according to the judgment results output by the model and the gold standard, its accuracy, false negative rate, AUC judgment and other indicators are calculated.

[0053] Among them, the evaluation indicators of the judgment results output by the DE_ResNet subnetwork and the Fold_ViT subnetwork are used as reference objects to compare the performance of the comprehensive results of the judgment results of the two types of subnetworks. In step T3, the two-dimensional and contrast ultrasound images of the patient in the DMUS data set are simultaneously input to ChoquetFusionNet, specifically, the two-dimensional and contrast ultrasound images of the same patient are respectively input into the corresponding two subnetworks to obtain the judgment results output by each subnetwork: positive probability and negative probability; then, based on the fuzzy integral fusion module, the positive probability and negative probability output by each subnetwork are fused respectively to obtain the fused fuzzy integral positive probability and the fused negative probability, and then the final risk assessment result is obtained based on the larger of the two.

[0054] Example

[0055] See also Figure 1In the breast cancer metastasis risk assessment method based on the neural network fusion mechanism provided in this embodiment, the ultrasound images of the input model include: LN two-dimensional ultrasound image and SLN ultrasound contrast image, that is, a patient has both SLN two-dimensional ultrasound image and SLN ultrasound contrast image; the positive probability and negative probability are obtained by DE_ResNet and Fold_ViT respectively; the fuzzy density corresponding to each subnetwork and the fuzzy measure corresponding to the subnetwork are calculated by using the hierarchical analysis method, and the positive probability and negative probability of the four subnetwork analyses are fused to obtain the final positive probability and negative probability; the positive probability and negative probability are compared according to the maximum value principle to obtain the metastasis risk assessment result of the patient.

[0056] Environment construction and language selection: This implementation will deploy the constructed network model on a small workstation equipped with Ubuntu18.05linux operating system. The basic configuration of the workstation is as follows. CPU model: Intel i9-10000K; GPU model: Nvidia GeForce RTX 2080Ti (12GB video memory); Memory: 64GB; and other hardware and software conditions to maintain the normal operation of the workstation. At the same time, the workstation is configured with the dependencies and environment required for deep learning as follows. Anaconda deep learning management tool, Python 3.8 version, Pytorch deep learning framework version 1.6, CUDA10.3 version, and python dependency libraries for image processing neural networks - torchvision, OpenCV, etc.

[0057] In this embodiment, the DE_ResNet subnetwork structure is shown in Table 1. DE_ResNet consists of five layers, which are respectively defined as Conv1, Conv2_x, Conv3_x, Conv_4x, and Conv5_x. Among them, Conv1 uses a convolution layer with a convolution kernel size of 7×7, 64 channels, and a stride of 2; Conv2_x uses a maximum pooling layer with a convolution kernel size of 3×3 and a stride of 2, and then connects 3 ELU_ResNet blocks; Conv3_x, Conv_4x, and Conv5_x are stacked structures of 4, 6, and 3 ELU_ResNet blocks respectively; finally, the classification result is obtained through average pooling, fully connected layers, and softmax operations.

[0058] Table 1

[0059]

[0060] Among them, the specific structure of ELU_ResNet block is as follows Figure 2As shown in the figure, the main branch includes three convolutional layers with kernel sizes of 1×1, 3×3 and 1×1, and a jump connection is set between the input and the last convolutional layer, that is, the input is added to the output of the third convolutional layer to obtain the output of the ELU_ResNet block (output feature map). The figure takes the ELU_ResNet block in Conv_2x as an example. The ELU_ResNet blocks in other layers only have different numbers of convolutional layer channels.

[0061] In addition, the first residual structure of Conv3_x, Conv_4x, and Conv5_x also has a downsampling function, so that the size and number of channels of the output feature map of the previous layer are adapted to the size and number of channels of the input feature map of the next layer. Due to the delayed downsampling operation adopted, it is named DDsampling Block (delay down sampling block). The detailed structure of DDsampling Block is as follows Figure 3 As shown in the figure, the main branch includes three convolution layers with kernel sizes of 1×1, 3×3 and 1×1, and the secondary branch includes a convolution layer with a kernel size of 1×1 and an average pooling layer (AvgPool) with a kernel size of 2×2. The inputs of the main and secondary branches are the same. Finally, the output feature maps of the main and secondary branches are added to obtain the output feature map of the DDsampling Block. Figure 3 Taking the ELU_ResNet block in Conv_3x as an example, the DDsampling Block in other layers only has the number of convolutional layer channels that differ.

[0062] In this embodiment, the Fold_ViT sub-network is based on the ViT-B / 16 model and optimized through the Unfold_Fold mechanism. Its network structure is as follows: Figure 4As shown in the figure, the input of the network structure is a three-channel color image of size 224×224. The dimension is increased by a convolution layer with a convolution kernel size of 16×16, a stride of 16, and a number of convolution kernels of 768. The feature map is flattened into a vector by a Flatten operation of height and width. The flattened vector is then concatenated with the classification vector (Class Token) and added to the position encoding (Position Rmbedding). The robustness of the model is improved by a Dropout layer. Then, the Unfold operation (used to split the input data into small blocks (or patches), which can be used for subsequent processing, such as convolution operations, etc.) is used to decompose it into multiple patch sequences. Each patch sequence enters Transformers Bolck for calculation. Transformers Bolck is composed of several Encoder Blocks. In this embodiment, 12 Encoder Blocks are included. The Fold operation is used to restore the original sequence shape, and then the Layer Normalization operation is used to normalize the input to stabilize and accelerate the convergence of the neural network. Finally, the ClassToken (which contains classification information) in the sequence is captured, and the final classification result is obtained through the MLP Head (prediction head based on multi-layer perceptron).

[0063] The breast cancer metastasis risk assessment method based on the neural network fusion mechanism provided by the embodiment of the present invention can efficiently and accurately help doctors predict the risk of breast cancer metastasis, improve doctors' efficiency, and reduce doctors' work intensity. At the same time, the experiment shows that the method proposed in the embodiment of the present invention can obtain a higher accuracy rate, false negative rate, and AUC value. In this embodiment, the final ChoquetFusionNet algorithm has an accuracy rate of 89.57%, a false negative rate of 9.05%, and an AUC of 93.25%, which are all better than the judgment results of the expert team of the ultrasound inspection department of a provincial people's hospital.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0065] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A method for assessing breast cancer metastasis risk based on a neural network fusion mechanism, characterized in that: The following steps are involved: Step 1, constructing a sentinel lymph node SLN ultrasound image dataset to tune the network parameters of the constructed breast cancer metastasis risk assessment model on the dataset; the SLN ultrasound image dataset includes a SLN two-dimensional ultrasound dataset TDUS and a SLN ultrasound contrast imaging dataset CEUS; Step 2, constructing a breast cancer metastasis risk assessment model, the assessment model comprising: a first sub-network based on a residual network, a second sub-network based on a self-attention neural network, and a fuzzy integral fusion module based on Choquet fuzzy integral; wherein the first and second sub-networks are respectively used to predict the binary assessment results of the breast cancer metastasis risk of the current input image data, including a positive probability and a negative probability; The first and second sub-networks are trained and verified on the data set TDUS, respectively, and when the preset training end condition is met, the trained first sub-network based on TDUS and the trained second sub-network based on TDUS are obtained; The first and second sub-networks are trained and verified on the data set CEUS, respectively, and when a preset training end condition is met, a trained first sub-network based on CEUS and a trained second sub-network based on CEUS are obtained; The evaluation results of the four sub-networks output are fused through the fuzzy integral fusion module based on Choquet fuzzy integral to obtain the fuzzy integral positive probability and fuzzy integral negative probability, and then the final result of breast cancer metastasis risk is obtained based on the maximum principle.

2. The method according to claim 1, characterized in that Step 1 specifically includes: Organize the collected SLN ultrasound image data set and divide it into training set and validation set according to the specified ratio; Manual annotation and expert review of sentinel lymph nodes were performed on the image data of the SLN ultrasound image dataset; According to the annotation results, the sentinel lymph node image is cropped and the image size is normalized.

3. The method according to claim 2, characterized in that The image size after normalization is set to 224×224.

4. The method according to claim 1, characterized in that In step 2, the first sub-network includes in sequence: a convolution layer, a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, an average pooling layer, a fully connected layer and a softmax function layer; Among them, the first convolution module includes a maximum pooling layer and a stacked structure consisting of several first residual modules in sequence; the second convolution module, the third convolution module and the fourth convolution module are respectively stacked structures consisting of several residual modules, and the first residual module of the second convolution module, the third convolution module and the fourth convolution module is a residual module with downsampling, and the remaining residual modules are first residual modules; wherein, the main branch of the first residual module includes three convolution layers in sequence, and a jump connection is set between the input of the first residual module and its last convolution layer; the residual module with downsampling includes a main branch and a secondary branch, the main branch includes three convolution layers, and the secondary branch includes a convolution layer and an average pooling layer in sequence. The inputs of the main and secondary branches are the same, and finally the output feature maps of the main and secondary branches are added to obtain the output feature map of the residual module with downsampling.

5. The method according to claim 4, characterized in that The first convolutional layer of the first subnetwork uses a convolutional layer with a convolution kernel size of 7×7 and a stride of 2; The kernel size adopted by the maximum pooling layer of the second convolution module is 3×3, and the second convolution module includes 3 first residual modules; the number of first residual modules included in the second convolution module, the third convolution module, and the fourth convolution module is set to 4, 6, and 3 respectively; The convolution kernel sizes of the three convolutional layers of the main branch of the first residual module are set to 1×1, 3×3, and 1×1 respectively; The convolution kernel sizes of the three convolution layers on the main branch of the residual module with downsampling are set to 1×1, 3×3 and 1×1 respectively, the convolution kernel size of the convolution layer on the secondary branch is 1×1, and the kernel size of the average pooling layer is 2×2.

6. The method according to claim 1, characterized in that In step 2, the structure of the second subnetwork is as follows: the input is a three-channel color image, the input image data is dimensionally upgraded through the convolution layer, and then the output feature map of the convolution layer is flattened into a vector through the flattening operation of height and width, and then the flattened vector is concatenated with the classification vector and added with the position code, and then passed through the Dropout layer; then the output feature map of the Dropout layer is decomposed into multiple data blocks through the Unfold operation to form a block sequence; each block sequence is sent to the Transformers module for encoding processing, and the output of the Transformers module is restored to the block sequence shape through the Fold operation, and then the class tokens in the sequence are extracted after normalization, and finally the classification result is output through the prediction head based on the multi-layer perceptron.

7. The method according to claim 6, characterized in that The first convolutional layer of the second sub-network uses a convolution kernel size of 16×16 and a stride of 16.

8. The method according to any one of claims 1 to 7, characterized in that In step 1, the data labels of the constructed sentinel lymph node SLN ultrasound image dataset are set based on the sentinel lymph node biopsy results.

Images