Superficial lymph node classification method, system and terminal based on multi-feature fusion

Abstract

The invention discloses a superficial lymph node classification method, system and terminal based on multi-feature fusion, and the method comprises the steps: constructing a multi-modal fusion network model, and training the multi-modal fusion network model to obtain a superficial lymph node classification model; the method comprises the following steps: acquiring a US image and a CDFI image of a to-be-predicted target, and preprocessing the US image and the CDFI image to obtain a target US image and a target CDFI image; inputting the target US image and the target CDFI image into a superficial lymph node classification model for feature extraction and fusion, and outputting a target superficial lymph node classification result; a multi-feature multi-mode fusion network is applied to superficial lymph node classification, the network integrates a convolutional neural network branch and a visual branch, multi-scale features are extracted from US and CDFI images, cross-scale feature interaction and fusion are promoted through a CBA feature fusion module and a CMA feature fusion module, marginal region detection is enhanced by using blood flow information, and the classification accuracy of superficial lymph nodes is improved. And the accuracy of superficial lymph node classification is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a superficial lymph node classification method, system, terminal and computer-readable storage medium based on multi-feature fusion. Background Art

[0002] Lymph nodes are important immune organs that collect lymph fluid from various tissues and organs of the body and play a core role in the human immune system. Superficial lymph nodes are mainly distributed in the neck, armpits and groin. The enlargement of these superficial lymph nodes is easy to touch, and the determination of the type of lymph node enlargement is of great help in the subsequent assessment of human health.

[0003] In recent years, with the continuous development of artificial intelligence, deep learning has also been applied in the field of lymph node classification. In view of the importance of accurately judging lymph node classification, some people have proposed deep learning methods for lymph node classification. For example, multiple imaging modalities are integrated, and then CNN is used for feature extraction, followed by Transformer for feature classification, and contrast learning is used to coordinate global and local features, and reinforcement learning is used to automatically adjust the weight of each modality loss function to finally achieve classification.

[0004] However, most of the above technologies are aimed at processing lymph nodes in a single region, without considering the lymph nodes throughout the body, and without considering that the characteristics of lymph nodes in different regions are diverse and interrelated, resulting in inaccurate analysis and classification of lymph nodes.

[0005] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0006] The main purpose of the present invention is to provide a superficial lymph node classification method, system, terminal and computer-readable storage medium based on multi-feature fusion, aiming to solve the problem that the existing technology does not take into account the lymph nodes of the whole body when classifying superficial lymph nodes, and does not take into account the diversity and correlation between the characteristics of lymph nodes in different regions, resulting in the current inaccurate analysis and classification of lymph nodes.

[0007] To achieve the above object, the present invention provides a superficial lymph node classification method based on multi-feature fusion, the superficial lymph node classification method based on multi-feature fusion comprising the following steps:

[0008] constructing a multimodal fusion network model, and training the multimodal fusion network model to obtain a superficial lymph node classification model;

[0009] Acquire a US image and a CDFI image of a target to be predicted, and preprocess the US image and the CDFI image to obtain a target US image and a target CDFI image;

[0010] The target US image and the target CDFI image are input into the superficial lymph node classification model for feature extraction and fusion, and the target superficial lymph node classification result is output.

[0011] Optionally, the superficial lymph node classification method based on multi-feature fusion, wherein the training of the multimodal fusion network model further includes:

[0012] Acquiring historical data of a lymph node region, the historical data including historical US images, historical CDFI images, and superficial lymph node types corresponding to the historical US images and historical CDFI images;

[0013] Using the historical US images and the historical CDFI images as features, using the superficial lymph node types corresponding to the historical US images and the historical CDFI images as labels, and constructing a data set according to the features and the labels;

[0014] The data set is divided into a training set, a test set and a validation set according to a preset ratio. The training set is used to train the multimodal fusion network model, the test set is used to evaluate the multimodal fusion network model in each round of training, and the validation set is used to evaluate the multimodal fusion network model after training.

[0015] Optionally, the superficial lymph node classification method based on multi-feature fusion, wherein the superficial lymph node classification model includes: a CNN branch, a ViT branch, a CBA feature fusion module, an FFM feature fusion module and a CMA feature fusion module;

[0016] The data dimension of the feature map of the CNN branch is (C, H, W), where c represents the number of channels, H represents the height, and W represents the width;

[0017] The data dimension of the feature map of the ViT branch is (K+1, E), where K+1 represents the number of patches, 1 represents an additional separate feature for determining the patch position, and E represents the length of the patch.

[0018] Optionally, in the superficial lymph node classification method based on multi-feature fusion, the CNN branch includes a first CNN branch and a second CNN branch, and the ViT branch includes a first ViT branch and a second ViT branch;

[0019] The step of inputting the target US image and the target CDFI image into the superficial lymph node classification model for feature extraction and fusion, and outputting the target superficial lymph node classification result, specifically includes:

[0020] Input the target US image into the first CNN branch for feature extraction to obtain local US image features, and input the target CDFI image into the second CNN branch for feature extraction to obtain local CDFI image features;

[0021] Input the target US image into the first ViT branch for feature extraction to obtain global US image features, and input the target CDFI image into the second ViT branch for feature extraction to obtain global CDFI image features;

[0022] Aligning and fusing the global US image features and the local US image features through the FFM feature fusion module to obtain a first target global US image feature and a target local US image feature;

[0023] The target local US image feature and the local CDFI image feature are fused by the CBA feature fusion module to obtain the target local CDFI image feature that provides additional blood flow information, and the target local CDFI image feature and the target local US image feature are used as the input of the next round of CNN respectively, and the final local feature is output after repeated iterations;

[0024] The target global US image feature and the global CDFI image feature are cross-learned by the CMA feature fusion module to obtain a second target global US image feature and a target global CDFI image feature, and the second target global US image feature and the target global CDFI image feature are used as inputs of the next round of ViT, respectively, and the final global feature is output after repeated iterations;

[0025] The final local features and the final global features are fused to generate a target superficial lymph node classification result.

[0026] Optionally, in the superficial lymph node classification method based on multi-feature fusion, the FFM feature fusion module includes an upsampling unit and a downsampling unit;

[0027] The aligning and fusing the global US image features and the local US image features by the FFM feature fusion module to obtain a first target global US image feature and a target local US image feature specifically includes:

[0028] Input the local US image features into the downsampling unit, use Conv1x1 to align the local US image features with the embedding length of the patch to obtain an aligned local US image, and use Flatten to planarize the aligned local US image to obtain a planarized local US image;

[0029] Normalizing the planarized local US image by LayerNorm and GeLU to obtain a two-dimensional local US image, and concatenating the two-dimensional local US image with the global US image feature to obtain a first target global US image feature;

[0030] Inputting the patch of the first target global US image feature into the upsampling unit for reshape operation to obtain a reshaped patch, and using Conv1x1 to align the number of the reshaped patches with the number of channels of the feature map to obtain an aligned patch;

[0031] The aligned patch is normalized, and the local US image feature is concatenated with the normalized patch to obtain the target local US image feature.

[0032] Optionally, the superficial lymph node classification method based on multi-feature fusion, wherein the target local US image feature and the local CDFI image feature are fused by the CBA feature fusion module to obtain the target local CDFI image feature that provides additional blood flow information, specifically includes:

[0033] The target local US image features are input into the CBA feature fusion module to calculate the channel attention weight matrix:

[0034] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F)));

[0035] Among them, M c (F) represents the channel attention weight matrix, F represents the target local US image feature, σ represents the Sigmoid activation function, MLP represents the multi-layer perceptron, AvgPool() represents the global average pooling operation, and MaxPool() represents the maximum pooling operation;

[0036] The target local US image feature is multiplied by the channel attention weight matrix to obtain the channel local US image feature F′, and the spatial attention weight matrix is ​​calculated according to the channel local US image feature F′:

[0037] M s (F′)=σ(f 7×7 ([AvgPool(F′);MaxPool(F′)]));

[0038] Among them, M s (F′) represents the spatial attention weight matrix, f 7×7Represents a 7x7 convolution operation, [AvgPool(F′); MaxPool(F′)] means concatenating the average pooling and maximum pooling results along the channel axis;

[0039] The spatial attention weight matrix M s (F′) is multiplied with the channel local US image feature F′ to obtain a spatial local US image feature, and the spatial local US image feature is fused with the local CDFI image feature to obtain a target local CDFI image feature that provides additional blood flow information.

[0040] Optionally, the superficial lymph node classification method based on multi-feature fusion, wherein the cross-learning of the target global US image feature and the global CDFI image feature by the CMA feature fusion module to obtain a second target global US image feature and a target global CDFI image feature, specifically includes:

[0041] The target global US image features and the global CDFI image features are input into the CMA feature fusion module for calculation to obtain US cross attention and CDFI cross attention:

[0042]

[0043] Among them, F US represents US cross attention, F CDFI represents CDFI cross attention, softmax() represents the normalized exponential function, Q CDFI The query matrix K represents the global CDFI image feature. US The key matrix representing the target global US image features, V US Represents the value matrix of the target global US image feature, Q US The query matrix K represents the target global US image features. CDFI The key matrix representing the global CDFI image features, V CDFI The value matrix representing the global CDFI image features, d k Indicates the dimension of the key matrix;

[0044] A second target global US image feature and a target global CDFI image feature are generated according to the US cross-attention and CDFI cross-attention.

[0045] In addition, to achieve the above-mentioned purpose, the present invention also provides a superficial lymph node classification system based on multi-feature fusion, wherein the superficial lymph node classification system based on multi-feature fusion includes:

[0046] A model training module, used for constructing a multimodal fusion network model, training the multimodal fusion network model, and obtaining a superficial lymph node classification model;

[0047] A data acquisition module, used for acquiring a US image and a CDFI image of a target to be predicted, and preprocessing the US image and the CDFI image to obtain a target US image and a target CDFI image;

[0048] The lymph node classification module is used to input the target US image and the target CDFI image into the superficial lymph node classification model for feature extraction and fusion, and output the target superficial lymph node classification result.

[0049] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a superficial lymph node classification program based on multi-feature fusion stored in the memory and executable on the processor, wherein the superficial lymph node classification program based on multi-feature fusion, when executed by the processor, implements the steps of the superficial lymph node classification method based on multi-feature fusion as described above.

[0050] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a superficial lymph node classification program based on multi-feature fusion, and when the superficial lymph node classification program based on multi-feature fusion is executed by a processor, the steps of the superficial lymph node classification method based on multi-feature fusion as described above are implemented.

[0051] In the present invention, a multimodal fusion network model is constructed, and the multimodal fusion network model is trained to obtain a superficial lymph node classification model; US images and CDFI images of the target to be predicted are obtained, and the US images and CDFI images are preprocessed to obtain target US images and target CDFI images; the target US images and target CDFI images are input into the superficial lymph node classification model for feature extraction and fusion, and the target superficial lymph node classification results are output; the present invention applies a multi-feature multi-mode fusion network to superficial lymph node classification, and the network integrates a convolutional neural network branch and a visual branch, extracts multi-scale features from US and CDFI images, and promotes cross-scale feature interaction and fusion through a CBA feature fusion module and a CMA feature fusion module, and uses blood flow information to enhance edge area detection, thereby improving the accuracy of superficial lymph node classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of a preferred embodiment of the superficial lymph node classification method based on multi-feature fusion of the present invention;

[0053] Figure 2It is an overall architecture diagram of the superficial lymph node classification model in the superficial lymph node classification method based on multi-feature fusion of the present invention;

[0054] Figure 3 It is a specific schematic diagram of the FFM feature fusion module in the superficial lymph node classification method based on multi-feature fusion of the present invention;

[0055] Figure 4 It is a specific schematic diagram of the CBA feature fusion module in the superficial lymph node classification method based on multi-feature fusion of the present invention;

[0056] Figure 5 It is a specific schematic diagram of the CMA feature fusion module in the superficial lymph node classification method based on multi-feature fusion of the present invention;

[0057] Figure 6 It is a structural diagram of a preferred embodiment of the superficial lymph node classification system based on multi-feature fusion of the present invention;

[0058] Figure 7 Schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0059] The present application provides a superficial lymph node classification method, system and terminal based on multi-feature fusion. In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0061] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0062] The superficial lymph node classification method based on multi-feature fusion described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the superficial lymph node classification method based on multi-feature fusion includes the following steps:

[0063] Step S10: construct a multimodal fusion network model, train the multimodal fusion network model, and obtain a superficial lymph node classification model.

[0064] Specifically, a multimodal fusion network model (MMFN) is constructed. The multimodal fusion network model is based on a CNN branch, a ViT branch and three feature fusion modules. The multimodal fusion network model has two modes of input images: ultrasound image (US) and Doppler blood flow imaging (CDFI). The images are sent to CNN-branch and vitc-branch to obtain local features and global features respectively; then, the feature map is fed into different feature fusion modules to obtain additional multi-scale fusion features at each stage; finally, the multimodal and multi-scale features are fused to obtain the fused classification results.

[0065] The step of training the multimodal fusion network model also includes:

[0066] Acquiring historical data of a lymph node region, the historical data including historical US images, historical CDFI images, and superficial lymph node types corresponding to the historical US images and historical CDFI images;

[0067] Using the historical US images and the historical CDFI images as features, using the superficial lymph node types corresponding to the historical US images and the historical CDFI images as labels, and constructing a data set according to the features and the labels;

[0068] The data set is divided into a training set, a test set and a validation set according to a preset ratio. The training set is used to train the multimodal fusion network model, the test set is used to evaluate the multimodal fusion network model in each round of training, and the validation set is used to evaluate the multimodal fusion network model after training.

[0069] It is understandable that the data set used in the present invention includes various ultrasonic medical images provided by a hospital, and the data set contains 1,365 cases of dual-modal superficial lymph node data, including various lymph node regions, such as image data of axillary, breast, cervical and abdominal lymph nodes (historical US images, historical CDFI images), and classifications corresponding to various lymph node regions (the superficial lymph node types corresponding to the historical US images and historical CDFI images). After being sorted, the data set provides a solid foundation for the training and evaluation of the multimodal fusion network model.

[0070] The historical US images and the historical CDFI images are used as features (features can also be understood as independent variables, which are independent variables used to train the model, usually a matrix, each row represents a sample, and each column represents a feature), and the superficial lymph node types corresponding to the historical US images and historical CDFI images are used as labels (labels can be understood as target variables, which are dependent variables that the model needs to predict, usually a vector. For classification problems, it represents category labels; for regression problems, it represents continuous target values), a data set is constructed according to the features and the labels, and the data set is divided into a training set, a test set, and a validation set according to a preset ratio (for example, the data ratio of the training set, the test set, and the validation set is 6:2:2), the training set is used to train the multimodal fusion network model, the test set is used to evaluate the multimodal fusion network model in each round of training, and the validation set is used to evaluate the trained multimodal fusion network model.

[0071] Furthermore, the training set is used to train the multimodal fusion network model, the test set is used to evaluate the multimodal fusion network model in each round of training, and the validation set is used to evaluate the trained multimodal fusion network model.

[0072] In this example, all training was performed using the PyTorch framework, executed on an Nvidia GeForce RTXA5000 GPU, with a training regimen involving 16 batches, and the model undergoing 50 training epochs, supplemented by a 5-fold cross-validation strategy to ensure the generalizability of the findings. The Adam optimizer was used for training, with the initial learning rate set to 0.00001, and the learning rate was adaptively adjusted using a cosine annealing scheduler. In order to enhance the dataset and enhance the robustness of the model, a series of enhancement techniques were performed on the images before they were fed into the network, including random scaling, flipping, and rotation.

[0073] Finally, the present invention uses accuracy, F1 score, Jaccard score, sensitivity, specificity, youden and AUC to quantitatively compare the performance of different classification methods. The technical indicators used in the present invention include: Accuracy, Specificity, and Sensitivity, AUC (area under the ROC curve), Prec (Precision), Bac / Uar (Balanced Accuracy / Unweighted Average Recall) and F1-score in the classification indicators. Through testing on private data sets, the method of the present invention shows better performance than existing methods, achieving an accuracy of 87.53% on unregistered and unbalanced multimodal ultrasound data.

[0074] Step S20: Acquire the US image and CDFI image of the target to be predicted, and pre-process the US image and the CDFI image to obtain the target US image and the target CDFI image.

[0075] Specifically, for the target to be predicted, its US image and CDFI image are obtained. Before the US image and the CDFI image are input into the superficial lymph node classification model for feature prediction, the US image and the CDFI image need to be preprocessed to improve the accuracy of the model. The preprocessing is a series of enhancement techniques, including random scaling: the main purpose of random scaling is to adapt the model to objects of different scales, thereby improving the generalization ability and robustness of the model; flipping: in machine learning and deep learning, flipping gradients can be used to optimize the performance of the model. By changing the input of the transfer function, the output results of the algorithm can be adjusted to selectively increase or decrease the weight of the feature; rotation: the main purpose of rotation is to increase the diversity of training data, thereby improving the generalization ability and recall rate of the model.

[0076] Step S30: input the target US image and the target CDFI image into the superficial lymph node classification model for feature extraction and fusion, and output the target superficial lymph node classification result.

[0077] like Figure 2As shown, the superficial lymph node classification model includes: CNN branch, ViT branch, CBA feature fusion module, FFM feature fusion module and CMA feature fusion module. The data dimension of the feature map of the CNN branch is (C, H, W), where C represents the number of channels, H represents the height, and W represents the width; the data dimension of the feature map of the ViT branch is (K+1, E), where K+1 represents the number of patches, 1 represents an additional separate feature for determining the patch position, and E represents the length of the patch.

[0078] Furthermore, the CNN branch includes a first CNN branch and a second CNN branch, and the ViT branch includes a first ViT branch and a second ViT branch.

[0079] The step of inputting the target US image and the target CDFI image into the superficial lymph node classification model for feature extraction and fusion, and outputting the target superficial lymph node classification result, specifically includes:

[0080] Input the target US image into the first CNN branch for feature extraction to obtain local US image features, and input the target CDFI image into the second CNN branch for feature extraction to obtain local CDFI image features;

[0081] Input the target US image into the first ViT branch for feature extraction to obtain global US image features, and input the target CDFI image into the second ViT branch for feature extraction to obtain global CDFI image features;

[0082] Aligning and fusing the global US image features and the local US image features through the FFM feature fusion module to obtain a first target global US image feature and a target local US image feature;

[0083] The target local US image feature and the local CDFI image feature are fused by the CBA feature fusion module to obtain the target local CDFI image feature that provides additional blood flow information, and the target local CDFI image feature and the target local US image feature are used as the input of the next round of CNN respectively, and the final local feature is output after repeated iterations;

[0084] The target global US image feature and the global CDFI image feature are cross-learned by the CMA feature fusion module to obtain a second target global US image feature and a target global CDFI image feature, and the second target global US image feature and the target global CDFI image feature are used as inputs of the next round of ViT, respectively, and the final global feature is output after repeated iterations;

[0085] The final local features and the final global features are fused to generate a target superficial lymph node classification result.

[0086] like Figure 3 As shown, further, the FFM feature fusion module includes an upsampling unit and a downsampling unit;

[0087] The aligning and fusing the global US image features and the local US image features by the FFM feature fusion module to obtain a first target global US image feature and a target local US image feature specifically includes:

[0088] Input the local US image features into the downsampling unit, use Conv1x1 to align the local US image features with the embedding length of the patch to obtain an aligned local US image, and use Flatten to planarize the aligned local US image to obtain a planarized local US image;

[0089] Normalizing the planarized local US image by LayerNorm and GeLU to obtain a two-dimensional local US image, and concatenating the two-dimensional local US image with the global US image feature to obtain a first target global US image feature;

[0090] Inputting the patch of the first target global US image feature into the upsampling unit for reshape operation to obtain a reshaped patch, and using Conv1x1 to align the number of the reshaped patches with the number of channels of the feature map to obtain an aligned patch;

[0091] The aligned patch is normalized, and the local US image feature is concatenated with the normalized patch to obtain the target local US image feature.

[0092] It is understandable that CNN and ViT extract features from different perspectives. CNN can discover the depth information in the image, while ViT divides the image into different patches and highlights the relationship between these patches. Using information from both architectures helps to make a better diagnosis, but the information from the two architectures is inconsistent. To address this problem, the present invention proposes a feature fusion module (FFM). FFM uses a series of upsampling and downsampling operations to align the spatial dimensions of the features extracted by the CNN branch and the ViT branch, aiming to promote interactive learning between branches, thereby enriching the representation ability of the network.

[0093] Specifically, the feature map of the CNN branch uses Conv1x1 to align the number of channels in the feature map with the length of the patch embedding, and then uses Flatten to perform a planarization operation, converting H and W to the length of the patch, and then normalizing through LayerNorm (LN) and GeLU to convert the three-dimensional feature map format to a two-dimensional patch format. The final converted data is concatenated with the original data of the ViT branch to fuse the two features, thereby realizing the conversion from feature map to patch. Before feature fusion, the feature map will obtain the local features of the image through a series of convolution operations such as Conv1x1 and dwConv3x3, and fuse the local features with the global features of the ViT branch to obtain highly robust features.

[0094] The patch of the ViT branch uses the reshape operation to divide the length of the patch into two dimensions, H×W. Then, Conv1x1 is used to align the number of patches with the number of channels of the feature map, and normalization is also performed. Finally, the converted data is spliced ​​with the original data of the CNN branch to achieve the conversion from patch to feature map. Before feature fusion, the patch will obtain the global features of the image through a series of attention calculations such as Attention, and fuse the global features with the local features of the CNN branch to obtain more comprehensive features.

[0095] like Figure 4 As shown, further, the target local US image feature and the local CDFI image feature are fused by the CBA feature fusion module to obtain the target local CDFI image feature that provides additional blood flow information, specifically including:

[0096] The target local US image features are input into the CBA feature fusion module to calculate the channel attention weight matrix:

[0097] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F)));

[0098] Among them, M c (F) represents the channel attention weight matrix, F represents the target local US image feature, σ represents the Sigmoid activation function, MLP represents the multi-layer perceptron, AvgPool() represents the global average pooling operation, and MaxPool() represents the maximum pooling operation;

[0099] The target local US image feature is multiplied by the channel attention weight matrix to obtain the channel local US image feature F′, and the spatial attention weight matrix is ​​calculated according to the channel local US image feature F′:

[0100] M s (F′)=σ(f 7×7 ([AvgPool(F′);MaxPool(F′)]));

[0101] Among them, M s (F′) represents the spatial attention weight matrix, f 7×7 Represents a 7x7 convolution operation, [AvgPool(F′); MaxPool(F′)] means concatenating the average pooling and maximum pooling results along the channel axis;

[0102] The spatial attention weight matrix M s (F′) is multiplied with the channel local US image feature F′ to obtain a spatial local US image feature, and the spatial local US image feature is fused with the local CDFI image feature to obtain a target local CDFI image feature that provides additional blood flow information.

[0103] It can be understood that in order to enhance the network's ability to focus on relevant features, the present invention introduces a CBA module, which uses a channel attention mechanism to prioritize salient features and a spatial attention mechanism to emphasize relevant areas in the feature map.

[0104] The CBA module achieves dual refinement of input features by combining channel attention and spatial attention. In the model, the feature map of the US image is input into the module using the CBA module and fused with the features of the CDFI image, so that the features of the US image guide the CDFI image to focus on the lymph node area, enabling the CDFI image to provide additional blood flow information.

[0105] like Figure 5 As shown, further, the target global US image feature and the global CDFI image feature are cross-learned by the CMA feature fusion module to obtain a second target global US image feature and a target global CDFI image feature, specifically including:

[0106] The target global US image features and the global CDFI image features are input into the CMA feature fusion module for calculation to obtain US cross attention and CDFI cross attention:

[0107]

[0108] Among them, FUS represents US cross attention, F CDFI represents CDFI cross attention, softmax() represents the normalized exponential function, Q CdFI The query matrix K represents the global CDFI image feature. US The key matrix representing the target global US image features, V US Represents the value matrix of the target global US image feature, Q US The query matrix K represents the target global US image features. CDFI The key matrix representing the global CDFI image features, V CDFI The value matrix representing the global CDFI image features, d k Indicates the dimension of the key matrix, which is used to prevent the gradient from exploding due to excessive data size;

[0109] A second target global US image feature and a target global CDFI image feature are generated according to the US cross-attention and CDFI cross-attention.

[0110] In this embodiment, due to the unique but complementary information in the US and CDFI modes, the present invention designs a CMA module to promote cross-modal interaction and achieves the effect of cross-learning by cross-using the query matrix for self-attention calculation.

[0111] This module uses a cross-attention mechanism to guide which additional regions in the CDFI image should be given extra attention by the US image. Specifically, the query matrix of the US image patch self-attention is replaced by the query matrix of the CDFI image patch self-attention, and the query matrix of the CDFI image patch self-attention is replaced by the query matrix of the US image patch self-attention. The cross-learning effect is achieved by cross-using the query matrix for self-attention calculation. This two-way information exchange is crucial to enhance the network's ability to identify key regional features.

[0112] It can be seen that the present invention has the following advantages:

[0113] (1) The present invention takes into account that the features of lymph nodes in different regions are diverse and interrelated, and uses CNN branches and ViT branches to extract deep and global features to obtain multi-scale features.

[0114] (2) In order to solve the problem of difficulty in identifying key areas, the present invention uses a convolutional block attention (CBA) module and a cross-modal attention (CMA) module to allow the additional blood flow information from the CDFI image to guide the US image to learn additional edge information at each stage. At the same time, the US image guides the selection of additional features in the CDFI image to narrow the recognition range of the key areas.

[0115] (3) In order to enable interactive learning of the multi-scale features obtained by the CNN branch and the ViT branch, the present invention uses a feature fusion module (FFM) to fuse the CNN feature map with the ViT patch at each stage to achieve mutual complementation of different types of features. A series of experiments were conducted on a self-collected dataset, and the results showed that the MMFN of the present invention is superior to the existing lymph node classification method.

[0116] Furthermore, if Figure 6 As shown, based on the above-mentioned superficial lymph node classification method based on multi-feature fusion, the present invention also provides a superficial lymph node classification system based on multi-feature fusion, wherein the superficial lymph node classification system based on multi-feature fusion includes:

[0117] A model training module 51 is used to construct a multimodal fusion network model, train the multimodal fusion network model, and obtain a superficial lymph node classification model;

[0118] A data acquisition module 52 is used to acquire a US image and a CDFI image of a target to be predicted, and pre-process the US image and the CDFI image to obtain a target US image and a target CDFI image;

[0119] The lymph node classification module 53 is used to input the target US image and the target CDFI image into the superficial lymph node classification model for feature extraction and fusion, and output the target superficial lymph node classification result.

[0120] Furthermore, if Figure 7 As shown, based on the above-mentioned superficial lymph node classification method and system based on multi-feature fusion, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 7 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0121] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed in the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a superficial lymph node classification program 40 based on multi-feature fusion is stored on the memory 20, and the superficial lymph node classification program 40 based on multi-feature fusion can be executed by the processor 10, thereby realizing the superficial lymph node classification method based on multi-feature fusion in the present application.

[0122] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the superficial lymph node classification method based on multi-feature fusion.

[0123] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0124] In one embodiment, when the processor 10 executes the superficial lymph node classification program 40 based on multi-feature fusion in the memory 20, the following steps are implemented:

[0125] constructing a multimodal fusion network model, and training the multimodal fusion network model to obtain a superficial lymph node classification model;

[0126] Acquire a US image and a CDFI image of a target to be predicted, and preprocess the US image and the CDFI image to obtain a target US image and a target CDFI image;

[0127] The target US image and the target CDFI image are input into the superficial lymph node classification model for feature extraction and fusion, and the target superficial lymph node classification result is output.

[0128] Wherein, the training of the multimodal fusion network model also includes:

[0129] Acquiring historical data of a lymph node region, the historical data including historical US images, historical CDFI images, and superficial lymph node types corresponding to the historical US images and historical CDFI images;

[0130] Using the historical US images and the historical CDFI images as features, using the superficial lymph node types corresponding to the historical US images and the historical CDFI images as labels, and constructing a data set according to the features and the labels;

[0131] The data set is divided into a training set, a test set and a validation set according to a preset ratio. The training set is used to train the multimodal fusion network model, the test set is used to evaluate the multimodal fusion network model in each round of training, and the validation set is used to evaluate the multimodal fusion network model after training.

[0132] The superficial lymph node classification model includes: a CNN branch, a ViT branch, a CBA feature fusion module, a FFM feature fusion module and a CMA feature fusion module;

[0133] The data dimension of the feature map of the CNN branch is (C, H, W), where c represents the number of channels, H represents the height, and W represents the width;

[0134] The data dimension of the feature map of the ViT branch is (K+1, E), where K+1 represents the number of patches, 1 represents an additional separate feature for determining the patch position, and E represents the length of the patch.

[0135] Wherein, the CNN branch includes a first CNN branch and a second CNN branch, and the ViT branch includes a first ViT branch and a second ViT branch;

[0136] The step of inputting the target US image and the target CDFI image into the superficial lymph node classification model for feature extraction and fusion, and outputting the target superficial lymph node classification result, specifically includes:

[0137] Input the target US image into the first CNN branch for feature extraction to obtain local US image features, and input the target CDFI image into the second CNN branch for feature extraction to obtain local CDFI image features;

[0138] Input the target US image into the first ViT branch for feature extraction to obtain global US image features, and input the target CDFI image into the second ViT branch for feature extraction to obtain global CDFI image features;

[0139] Aligning and fusing the global US image features and the local US image features through the FFM feature fusion module to obtain a first target global US image feature and a target local US image feature;

[0140] The target local US image feature and the local CDFI image feature are fused by the CBA feature fusion module to obtain the target local CDFI image feature that provides additional blood flow information, and the target local CDFI image feature and the target local US image feature are used as the input of the next round of CNN respectively, and the final local feature is output after repeated iterations;

[0141] The target global US image feature and the global CDFI image feature are cross-learned by the CMA feature fusion module to obtain a second target global US image feature and a target global CDFI image feature, and the second target global US image feature and the target global CDFI image feature are used as inputs of the next round of ViT, respectively, and the final global feature is output after repeated iterations;

[0142] The final local features and the final global features are fused to generate a target superficial lymph node classification result.

[0143] Wherein, the FFM feature fusion module includes an upsampling unit and a downsampling unit;

[0144] The aligning and fusing the global US image features and the local US image features by the FFM feature fusion module to obtain a first target global US image feature and a target local US image feature specifically includes:

[0145] Input the local US image features into the downsampling unit, use Conv1x1 to align the local US image features with the embedding length of the patch to obtain an aligned local US image, and use Flatten to planarize the aligned local US image to obtain a planarized local US image;

[0146] Normalizing the planarized local US image by LayerNorm and GeLU to obtain a two-dimensional local US image, and concatenating the two-dimensional local US image with the global US image feature to obtain a first target global US image feature;

[0147] Inputting the patch of the first target global US image feature into the upsampling unit for reshape operation to obtain a reshaped patch, and using Conv1x1 to align the number of the reshaped patches with the number of channels of the feature map to obtain an aligned patch;

[0148] The aligned patch is normalized, and the local US image feature is concatenated with the normalized patch to obtain the target local US image feature.

[0149] The step of fusing the target local US image features and the local CDFI image features through the CBA feature fusion module to obtain the target local CDFI image features that provide additional blood flow information specifically includes:

[0150] The target local US image features are input into the CBA feature fusion module to calculate the channel attention weight matrix:

[0151] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F)));

[0152] Among them, M c (F) represents the channel attention weight matrix, F represents the target local US image feature, σ represents the Sigmoid activation function, MLP represents the multi-layer perceptron, AvgPool() represents the global average pooling operation, and MaxPool() represents the maximum pooling operation;

[0153] The target local US image feature is multiplied by the channel attention weight matrix to obtain the channel local US image feature F′, and the spatial attention weight matrix is ​​calculated according to the channel local US image feature F′:

[0154] M s (F′)=σ(f 7×7 ([AvgPool(F′);MaxPool(F′)]));

[0155] Among them, M s (F′) represents the spatial attention weight matrix, f 7×7 Represents a 7x7 convolution operation, [AvgPool(F′); MaxPool(F′)] means concatenating the average pooling and maximum pooling results along the channel axis;

[0156] The spatial attention weight matrix M s(F′) is multiplied with the channel local US image feature F′ to obtain a spatial local US image feature, and the spatial local US image feature is fused with the local CDFI image feature to obtain a target local CDFI image feature that provides additional blood flow information.

[0157] The cross-learning of the target global US image feature and the global CDFI image feature by the CMA feature fusion module to obtain a second target global US image feature and a target global CDFI image feature specifically includes:

[0158] The target global US image features and the global CDFI image features are input into the CMA feature fusion module for calculation to obtain US cross attention and CDFI cross attention:

[0159]

[0160] Among them, F US represents US cross attention, F CDFI represents CDFI cross attention, softmax() represents the normalized exponential function, Q CDFI The query matrix K represents the global CDFI image feature. US The key matrix representing the target global US image features, V US Represents the value matrix of the target global US image feature, Q US The query matrix K represents the target global US image features. CDFI The key matrix representing the global CDFI image features, V CDFI The value matrix representing the global CDFI image features, d k Indicates the dimension of the key matrix;

[0161] A second target global US image feature and a target global CDFI image feature are generated according to the US cross-attention and CDFI cross-attention.

[0162] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a superficial lymph node classification program based on multi-feature fusion, and when the superficial lymph node classification program based on multi-feature fusion is executed by a processor, the steps of the superficial lymph node classification method based on multi-feature fusion as described above are implemented.

[0163] In summary, the present invention proposes a superficial lymph node classification method, system and terminal based on multi-feature fusion, the method comprising: constructing a multimodal fusion network model, training the multimodal fusion network model, and obtaining a superficial lymph node classification model; acquiring a US image and a CDFI image of a target to be predicted, preprocessing the US image and the CDFI image, and obtaining a target US image and a target CDFI image; inputting the target US image and the target CDFI image into the superficial lymph node classification model for feature extraction and fusion, and outputting the target superficial lymph node classification result; the present invention applies a multi-feature multi-mode fusion network to superficial lymph node classification, the network integrates a convolutional neural network branch and a visual branch, extracts multi-scale features from US and CDFI images, and promotes cross-scale feature interaction and fusion through a CBA feature fusion module and a CMA feature fusion module, and uses blood flow information to enhance edge area detection, thereby improving the accuracy of superficial lymph node classification.

[0164] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0165] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0166] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A superficial lymph node classification method based on multi-feature fusion, characterized in that: The superficial lymph node classification method based on multi-feature fusion includes: constructing a multimodal fusion network model, and training the multimodal fusion network model to obtain a superficial lymph node classification model; Acquire a US image and a CDFI image of a target to be predicted, and preprocess the US image and the CDFI image to obtain a target US image and a target CDFI image; The target US image and the target CDFI image are input into the superficial lymph node classification model for feature extraction and fusion, and the target superficial lymph node classification result is output.

2. The method for superficial lymph node classification based on multi-feature fusion according to claim 1, characterized in that: The step of training the multimodal fusion network model also includes: Acquiring historical data of a lymph node region, the historical data including historical US images, historical CDFI images, and superficial lymph node types corresponding to the historical US images and historical CDFI images; Using the historical US images and the historical CDFI images as features, using the superficial lymph node types corresponding to the historical US images and the historical CDFI images as labels, and constructing a data set according to the features and the labels; The data set is divided into a training set, a test set and a validation set according to a preset ratio. The training set is used to train the multimodal fusion network model, the test set is used to evaluate the multimodal fusion network model in each round of training, and the validation set is used to evaluate the multimodal fusion network model after training.

3. The superficial lymph node classification method based on multi-feature fusion according to claim 1, characterized in that: The superficial lymph node classification model includes: a CNN branch, a ViT branch, a CBA feature fusion module, a FFM feature fusion module and a CMA feature fusion module; The data dimension of the feature map of the CNN branch is (C, H, W), where C represents the number of channels, H represents the height, and W represents the width; The data dimension of the feature map of the ViT branch is (K+1, E), where K+1 represents the number of patches, 1 represents an additional separate feature for determining the patch position, and E represents the length of the patch.

4. The method for superficial lymph node classification based on multi-feature fusion according to claim 3, characterized in that: The CNN branch includes a first CNN branch and a second CNN branch, and the ViT branch includes a first ViT branch and a second ViT branch; The step of inputting the target US image and the target CDFI image into the superficial lymph node classification model for feature extraction and fusion, and outputting the target superficial lymph node classification result, specifically includes: Input the target US image into the first CNN branch for feature extraction to obtain local US image features, and input the target CDFI image into the second CNN branch for feature extraction to obtain local CDFI image features; Input the target US image into the first ViT branch for feature extraction to obtain global US image features, and input the target CDFI image into the second ViT branch for feature extraction to obtain global CDFI image features; Aligning and fusing the global US image features and the local US image features through the FFM feature fusion module to obtain a first target global US image feature and a target local US image feature; The target local US image feature and the local CDFI image feature are fused by the CBA feature fusion module to obtain the target local CDFI image feature that provides additional blood flow information, and the target local CDFI image feature and the target local US image feature are used as the input of the next round of CNN respectively, and the final local feature is output after repeated iterations; The target global US image feature and the global CDFI image feature are cross-learned by the CMA feature fusion module to obtain a second target global US image feature and a target global CDFI image feature, and the second target global US image feature and the target global CDFI image feature are used as inputs of the next round of ViT, respectively, and the final global feature is output after repeated iterations; The final local features and the final global features are fused to generate a target superficial lymph node classification result.

5. The method for superficial lymph node classification based on multi-feature fusion according to claim 4, characterized in that: The FFM feature fusion module includes an upsampling unit and a downsampling unit; The aligning and fusing the global US image features and the local US image features by the FFM feature fusion module to obtain a first target global US image feature and a target local US image feature specifically includes: Input the local US image features into the downsampling unit, use Conv1x1 to align the local US image features with the embedding length of the patch to obtain an aligned local US image, and use Flatten to planarize the aligned local US image to obtain a planarized local US image; Normalizing the planarized local US image by LayerNorm and GeLU to obtain a two-dimensional local US image, and concatenating the two-dimensional local US image with the global US image feature to obtain a first target global US image feature; Inputting the patch of the first target global US image feature into the upsampling unit for reshape operation to obtain a reshaped patch, and using Conv1x1 to align the number of the reshaped patches with the number of channels of the feature map to obtain an aligned patch; The aligned patch is normalized, and the local US image feature is concatenated with the normalized patch to obtain the target local US image feature.

6. The superficial lymph node classification method based on multi-feature fusion according to claim 4 is characterized in that: The step of fusing the target local US image feature and the local CDFI image feature through the CBA feature fusion module to obtain the target local CDFI image feature that provides additional blood flow information specifically includes: The target local US image features are input into the CBA feature fusion module to calculate the channel attention weight matrix: M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))); Among them, M c (F) represents the channel attention weight matrix, F represents the target local US image feature, σ represents the Sigmoid activation function, MLP represents the multi-layer perceptron, AvgPool() represents the global average pooling operation, and MaxPool() represents the maximum pooling operation; The target local US image feature is multiplied by the channel attention weight matrix to obtain the channel local US image feature F′, and the spatial attention weight matrix is ​​calculated according to the channel local US image feature F′: M s (F′)=σ(f 7×7 ([AvgPool(F′);MaxPool(F′)])); Among them, M s (F′) represents the spatial attention weight matrix, f 7×7 Represents a 7x7 convolution operation, [AvgPool(F′); MaxPool(F′)] means concatenating the average pooling and maximum pooling results along the channel axis; The spatial attention weight matrix M s (F′) is multiplied with the channel local US image feature F′ to obtain a spatial local US image feature, and the spatial local US image feature is fused with the local CDFI image feature to obtain a target local CDFI image feature that provides additional blood flow information.

7. The method for superficial lymph node classification based on multi-feature fusion according to claim 4, characterized in that: The cross-learning of the target global US image feature and the global CDFI image feature by the CMA feature fusion module to obtain a second target global US image feature and a target global CDFI image feature specifically includes: The target global US image features and the global CDFI image features are input into the CMA feature fusion module for calculation to obtain US cross attention and CDFI cross attention: Among them, F US represents US cross attention, F CDFI represents CDFI cross attention, softmax() represents the normalized exponential function, Q CDFI The query matrix K represents the global CDFI image feature. US The key matrix representing the target global US image features, V US Represents the value matrix of the target global US image feature, Q US The query matrix K represents the target global US image features. CDFI The key matrix representing the global CDFI image features, V CDFI The value matrix representing the global CDFI image features, d k Indicates the dimension of the key matrix; A second target global US image feature and a target global CDFI image feature are generated according to the US cross-attention and CDFI cross-attention.

8. A superficial lymph node classification system based on multi-feature fusion, characterized in that: The superficial lymph node classification system based on multi-feature fusion includes: A model training module, used for constructing a multimodal fusion network model, training the multimodal fusion network model, and obtaining a superficial lymph node classification model; A data acquisition module, used for acquiring a US image and a CDFI image of a target to be predicted, and preprocessing the US image and the CDFI image to obtain a target US image and a target CDFI image; The lymph node classification module is used to input the target US image and the target CDFI image into the superficial lymph node classification model for feature extraction and fusion, and output the target superficial lymph node classification result.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a superficial lymph node classification program based on multi-feature fusion stored in the memory and executable on the processor. When the superficial lymph node classification program based on multi-feature fusion is executed by the processor, the steps of the superficial lymph node classification method based on multi-feature fusion as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a superficial lymph node classification program based on multi-feature fusion, and when the superficial lymph node classification program based on multi-feature fusion is executed by a processor, the steps of the superficial lymph node classification method based on multi-feature fusion as described in any one of claims 1-7 are implemented.

Images