A system for identifying metastatic cervical lymph nodes in cancer based on multi-scale feature fusion

Abstract

This invention discloses a multi-scale feature fusion-based cervical lymph node identification system for cancer metastasis, relating to the fields of artificial intelligence and ultrasound image analysis. It includes three sets of encoded neural networks, two feature enhancement modules, a dual attention module, and a decoding module. The multi-scale feature fusion network can extract lymph node features from different viewpoints and fuse features from different viewpoints. The dual attention module controls the information flow between low-level and high-level features by generating descriptive factors, selecting useful feature channels and avoiding the adverse effects of useless channel information on the model output, resulting in finer localization contours and more accurate diagnosis. This system can locate lymph nodes on cervical ultrasound images and identify lymph nodes with cancer metastasis, thus better assisting doctors in quickly identifying and diagnosing cervical lymph nodes during ultrasound examinations, avoiding excessive subjective factors, and making the diagnostic results more reliable.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of artificial intelligence and ultrasonic image analysis, in particular to a cancer metastasis neck lymph node identification system based on multi-scale feature fusion. BACKGROUND

[0002] Lymph nodes are an important part of the human immune defense system, and their main functions are to accept antigen stimulation, produce lymphocytes and plasma cells, participate in immune reactions, and filter lymph. Lymph nodes are distributed throughout the human body and are mainly composed of superficial lymph node groups and deep lymph node groups, among which the head and neck contain a large number of lymph node groups.

[0003] The neck lymph nodes have rich drainage and various pathologies. Common benign pathologies include reactive hyperplasia, lymphadenitis, tuberculosis, etc., and malignant pathologies include lymphoma and metastasis of malignant tumors. Neck lymph nodes are not only common sites of metastasis of head and neck malignant tumors, but also common sites of distant metastasis of systemic malignant tumors. Studies have shown that in head and neck tumors, patients with unilateral neck lymph node metastasis have a nearly 50% reduction in 5-year survival rate. When bilateral neck lymph node metastasis occurs, the 5-year survival rate of patients is only 25% of that of patients without lymph node metastasis. Therefore, the nature of the neck lymph nodes is not only related to the differential diagnosis of benign and malignant diseases, but also an important factor in determining the prognosis of tumors, and has an important influence on the treatment and prognosis of patients.

[0004] At present, the imaging methods for checking neck lymph nodes in clinical practice include CT, MRI and ultrasound, etc. Ultrasound examination has become the preferred method for checking neck lymph nodes due to its advantages of non-invasiveness, non-radiation, convenient operation, low cost, real-time performance, etc. Neck lymph nodes have complex physiological characteristics, and their shapes are small and irregular on ultrasonic images. At the same time, ultrasonic images often have low contrast, blurred edges, uneven gray scale, and serious speckle noise. In the face of a large amount of image data, the workload of ultrasonic doctors is huge, and the diagnosis process is time-consuming and laborious. Moreover, doctors mainly interpret image features according to personal experience, and the subjectivity is relatively strong. The diagnosis results are usually affected by factors such as imaging parameters of imaging equipment, display equipment, acquisition conditions, etc., which can easily lead to misdiagnosis or missed diagnosis.

[0005] In recent years, with the development of computer hardware and deep learning algorithms, computer-aided diagnosis technology has been gradually applied to the field of medical imaging and has achieved remarkable results. The traditional medical image interpretation method requires doctors to analyze and judge the lymph node cortex-medulla ratio, lymph node portal structure and calcification and other characteristic dimensions. Doctors mainly interpret image features according to personal experience, which is highly subjective, and the diagnosis result is usually affected by factors such as imaging parameters of imaging equipment, display equipment, acquisition conditions and the like. Unlike the traditional medical image interpretation method, the deep learning technology can automatically extract quantitative features associated with clinical information from the image to provide support for disease diagnosis, thereby avoiding the interference caused by human factors in the extraction of features in the traditional method, and making the diagnosis result more reliable. SUMMARY

[0006] In order to solve the above technical problems, the application provides a cancer metastasis neck lymph node recognition system based on multi-scale feature fusion, which comprises an ultrasonic image acquisition module, a model construction module, a model training module, an encoding module, a semantic feature extraction module, a feature information fusion module, an information fusion module and a decoding module.

[0007] The ultrasonic image acquisition module is used for acquiring neck ultrasonic images and pre-processing the neck ultrasonic images.

[0008] The model construction module is used for constructing a neck lymph node recognition network model, and the neck lymph node recognition network model comprises an encoding neural network ENET_1, an encoding neural network ENET_2, an encoding neural network ENET_3, a feature enhancement module Module_1, a feature enhancement module Module_2, a double attention module M_ATT and a decoding module DNET.

[0009] The model training module is used for training the neck lymph node recognition network model to obtain an optimal model.

[0010] The model construction module comprises an encoding module, a semantic feature extraction module, a feature information fusion module, an information fusion module and a decoding module.

[0011] The encoding module is used for inputting the pre-processed neck ultrasonic images into the trained neck lymph node recognition network model, and the neck ultrasonic images are sequentially input into the encoding neural network ENET_1, the encoding neural network ENET_2 and the encoding neural network ENET_3. Each group of encoding neural networks is reduced by half through convolution integration operation, and outputs feature maps F1, F2 and F3 respectively.

[0012] The semantic feature extraction module is used for inputting the feature map F3 output by the encoding neural network ENET_3 into the feature enhancement module Module_1, extracting semantic features and outputting a feature map F M1-6 ;

[0013] The feature information fusion module is used to simultaneously input feature maps F1, F2, and F3 into the feature enhancement module Module_2 while feature map F3 is input into feature enhancement module Module_1, and to fuse and calculate feature information at three scales, outputting feature map F. M2-9 ;

[0014] The information fusion module is used to integrate feature maps F M1-6 and feature map F M2-9 Simultaneously, the input is fed into the dual attention module M_ATT, where the control vector generated by the attention mechanism regulates the information flow between different semantic features. Information is selected from different semantic features for fusion to obtain the feature map F. M3-2 ;

[0015] The decoding module is used to decode the feature map F. M3-2 The input is fed into the decoding module DNET for upsampling and decoding to obtain the final output of the model.

[0016] The technical solution further defined in this invention is:

[0017] Furthermore, in the ultrasound image acquisition module, the size of the neck ultrasound image is adjusted to 512×512, which is the unified input size of the model. After the size adjustment, the neck ultrasound image is standardized and normalized, and the value of each pixel is converted to the range [0, 1].

[0018] The aforementioned cervical lymph node identification system based on multi-scale feature fusion includes the following steps in its model training module for training the cervical lymph node identification network model:

[0019] A1. Construct the dataset and divide it into training and test sets in an 8:2 ratio;

[0020] A2. Train the cervical lymph node identification network model using the training set data. During training, use the Adam optimizer and weighted cross-entropy as the loss function. The specific formula is as follows:

[0021]

[0022]

[0023] Among them, y c For the label value, y′ c The value is the model's predicted value; N is the total number of pixels in the training set. c This represents the total number of pixels in class c in the training set.

[0024] The Adam optimizer formula is as follows:

[0025] v t =β2v t-1 +(1-β2)g t 2

[0026] m t =β1m t-1 +(1-β1)g t

[0027]

[0028]

[0029]

[0030] Among them, g t The gradient at time t is represented by m. t v represents the gradient exponential moving average at time t, with an initial value of m0 = 0; t Let vt represent the squared exponential moving average of the gradient at time t, with an initial value of v0 = 0; β1 and β2 are the exponential decay rates, set to 0.9 and 0.999 respectively; η represents the learning rate; ε = 10 -8 θ is used to avoid division by zero; t Let θ represent the model parameters at time t. t-1 Represents the model parameters at time t-1;

[0031] A3. Use test set data to evaluate the model generated during training. When the evaluation result reaches the preset target, the model training is complete. If the evaluation result does not reach the preset target, adjust the parameters and repeat the training until the preset target is reached.

[0032] The aforementioned cervical lymph node identification system based on multi-scale feature fusion includes a cervical lymph node identification network model. The encoding neural networks ENET_1, ENET_2, and ENET_3 each contain several convolutional layers. These convolutional layers include three operations: convolution integration, batch normalization, and activation functions. Each encoding neural network reduces the feature map size by half through a single convolution integration operation with a stride of 2. Specifically, ENET_1, ENET_2, and ENET_3 output feature maps F1 (256×256), F2 (128×128), and F3 (64×64), respectively.

[0033] The aforementioned multi-scale feature fusion-based cervical lymph node metastasis identification system includes the following steps in its semantic feature extraction module:

[0034] B1. In the feature enhancement module Module_1, feature map F3 first extracts features through a convolutional layer of length 2, resulting in a feature map F with a size of 32×32. M1-1 Then the feature map F M1-1 The input is fed into three parallel dilated convolutional layers to extract features, resulting in feature maps F. M1-2 Feature map F M1-3 and feature map F M1-4 ;

[0035] B2. The feature map F M1-2 Feature map F M1-3 and feature map F M1-4 The feature map is concatenated by channel and then fused using a convolutional layer with a kernel size of 1×1 to reduce the number of channels in the feature map, resulting in a feature map F. M1-1 Feature maps F with the same size and number of channels M1-5 ;

[0036] B3. Transfer feature map F M1-1 and feature map F M1-5 The feature values ​​at corresponding channels and pixel indices are summed. A convolutional layer is then used to extract the summed features, and bilinear interpolation is used to augment the feature map, resulting in a 64×64 feature map F. M1-6 .

[0037] In the aforementioned multi-scale feature fusion-based cervical lymph node identification system for cancer metastasis, step B1 sets the dilation rates of the three parallel dilated convolutional layers to 2, 4, and 6, respectively.

[0038] The aforementioned multi-scale feature fusion-based cervical lymph node identification system for cancer metastasis includes the following steps in its feature information fusion module for fusing and calculating feature information at three scales:

[0039] C1. In the feature enhancement module Module_2, feature map F1 first extracts features through a convolutional layer to obtain feature map F. M2-1 ;

[0040] C2, Transfer feature map F M2-1 The input is fed into the feature weighting module F_W, where the weight vector guides the model to selectively learn the feature map, and the output is the feature map F. M2-3 ;

[0041] C3. Transfer feature map F M2-3 With feature map F M2-1 The data is concatenated by channel, then fed into a convolutional layer to extract features, resulting in feature map F. M2-4 ;

[0042] C4. Use a pooling layer with a kernel size of 2 and a stride of 2 to apply the feature map F. M2-4 Downsampling is performed to obtain a feature map F with a size of 128×128. M2-5 ;

[0043] C5. After extracting features from feature map F2 through a convolutional layer, input it into the feature weighting module F_W; concatenate the output of feature map F2 after feature extraction through a convolutional layer with the output of feature weighting module F_W by channel, and then input it into a convolutional layer to extract features to obtain feature map F. M2-6 ;

[0044] C6. Transfer feature map F M2-6 With feature map F M2-5 The feature values ​​of corresponding channels are summed, then fed into a convolutional layer to extract features. These features are then downsampled using a pooling layer with a kernel size of 2 and a stride of 2, resulting in a feature map F with a size of 64×64. M2-7 ;

[0045] C7. After extracting features from feature map F3 through a convolutional layer, input it into the feature weighting module F_W; concatenate the output of feature map F3 after feature extraction through a convolutional layer with the output of feature weighting module F_W by channel, and then input it into a convolutional layer to extract features to obtain feature map F. M2-8 ;

[0046] C8. Transfer feature map F M2-8 With feature map F M2-7 The feature values ​​of corresponding channels are summed, and then fed into a convolutional layer to extract features, resulting in the final output of the feature enhancement module Module_2, which is a feature map F with a size of 64×64. M2-9 .

[0047] In the aforementioned multi-scale feature fusion-based cervical lymph node identification system for cancer metastasis, in step C2, in the feature weighting module F_W, the feature map F... M2-1 Feature extraction is performed consecutively through two convolutional layers. The extracted features are then input into a global average pooling layer and a fully connected layer for global feature mapping, generating a weight vector α(α1, α2, ..., α3) for each channel. n Next, the feature map F M2-1 The feature map of each channel is multiplied by the corresponding channel weight vector α to obtain the feature map F. M2-2 Simultaneously, feature map F M2-2 With feature map F M2-1 The information from each channel is added together, and then fused using a convolutional layer with a kernel size of 1×1 to obtain the feature map F. M2-3 .

[0048] The aforementioned multi-scale feature fusion-based cervical lymph node identification system for cancer metastasis includes the following steps performed by the information fusion module:

[0049] D1. In the dual attention module M_ATT, the feature map F is first... M1-6 and feature map F M2-9 The concatenation is performed channel-wise, and a convolutional layer is used to fuse the features of the concatenated result, while reducing the number of channels to obtain feature map F. M3-1 ;

[0050] D2. For feature map F M3-1 Perform global average pooling to obtain the channel feature vector S;

[0051] D3. Set up two fully connected layers to learn the channel feature vector S, and output control feature maps F from the two fully connected layers respectively. M1-6 and feature map F M2-9 The feature descriptor vectors β and γ, and the lengths of the feature descriptor vectors β and γ and their corresponding feature maps F M1-6 and feature map F M2-9 The number of channels is the same;

[0052] D4. Connect the feature description factor vectors β and γ to the feature map F respectively. M1-6 Feature map F M2-9 Multiplication involves multiplying the eigenvalues ​​of the factor vector with all values ​​in the corresponding channel of the feature map. The resulting feature maps are then concatenated by channel to obtain the final output feature map F of the dual attention module M_ATT. M3-2 The feature map has a size of 64×64.

[0053] The aforementioned multi-scale feature fusion-based cervical lymph node identification system for cancer metastases includes the following steps performed by the decoding module:

[0054] F1. In the decoding module DNET, the feature map F M3-2 First, a convolutional layer is used to extract features and reduce the number of channels; then, a transposed convolution is used to enlarge the feature map size, resulting in a 128×128 feature map F. M4-1 ;

[0055] E2. Use a convolutional layer on feature map F M4-1 Feature extraction was performed and the number of channels was reduced to obtain a feature map F with 3 channels and a size of 128×128. M4-2 ;

[0056] E3. Use bilinear interpolation to transform the feature map F M4-2 The size is increased by 4 times, resulting in a feature map F with 3 channels and a size of 512×512. M4-3 ;

[0057] E4. Compare Feature Maps by Channel F M4-3 The element values ​​of the first channel are used to determine the feature value. If the element value of the first channel is greater than that of the other two channels, the feature value is 0; if the element value of the second channel is greater than that of the other two channels, the feature value is 1; if the element value of the third channel is greater than that of the other two channels, the feature value is 2. Finally, the feature map F, which indicates the location and nature of lymph nodes, is obtained. M4-4 ;

[0058] A feature value of 0 indicates that the pixel in the neck ultrasound image is not a lymph node and is a background area; a feature value of 1 indicates that the pixel in the neck ultrasound image is a benign lymph node lesion; and a feature value of 2 indicates that the pixel in the neck ultrasound image is a metastatic lymph node.

[0059] The beneficial effects of this invention are:

[0060] This invention includes three sets of encoding neural networks, two feature enhancement modules, a dual attention module, and a decoding module. The multi-scale feature fusion network can extract lymph node features from different viewpoints and fuse features from these different viewpoints. The dual attention module controls the information flow between low-level and high-level features by generating descriptive factors, selecting useful feature channels and avoiding the adverse effects of useless channel information on the model output, resulting in finer localization contours and more accurate diagnosis. This system can locate lymph nodes on cervical ultrasound images and identify lymph nodes with cancer metastases, thus better assisting doctors in quickly identifying and diagnosing cervical lymph nodes during ultrasound examinations, avoiding excessive subjective factors and making the diagnostic results more reliable. Attached Figure Description

[0061] Figure 1 This is a schematic diagram of the overall framework of the present invention;

[0062] Figure 2 This is a schematic diagram of the cervical lymph node identification network model in an embodiment of the present invention;

[0063] Figure 3 This is a schematic diagram of the feature enhancement module Module_1 in an embodiment of the present invention;

[0064] Figure 4 This is a schematic diagram of the feature enhancement module Module_2 in an embodiment of the present invention;

[0065] Figure 5 This is a schematic diagram of the feature weighting module F_W in an embodiment of the present invention;

[0066] Figure 6This is a schematic diagram of the dual attention module M_ATT in an embodiment of the present invention. Detailed Implementation

[0067] This embodiment provides a cervical lymph node identification system based on multi-scale feature fusion, such as... Figure 1 As shown, it includes an ultrasound image acquisition module, a model building module, a model training module, an encoding module, a semantic feature extraction module, a feature information fusion module, an information fusion module, and a decoding module.

[0068] The ultrasound image acquisition module is used to acquire neck ultrasound images and preprocess the neck ultrasound images. Specifically, the size of the neck ultrasound image is adjusted to 512×512, which is the uniform input size of the model. After the size adjustment, the neck ultrasound image is standardized and normalized, and the value of each pixel is converted to the range [0, 1].

[0069] The model building module is used to build a cervical lymph node identification network model, such as... Figure 2 As shown, the cervical lymph node identification network model includes an encoding neural network ENET_1, an encoding neural network ENET_2, an encoding neural network ENET_3, a feature enhancement module Module_1, a feature enhancement module Module_2, a dual attention module M_ATT, and a decoding module DNET.

[0070] The model training module is used to train the cervical lymph node identification network model to obtain the optimal model. The specific method for training the cervical lymph node identification network model includes the following steps.

[0071] A1. Construct the dataset and divide it into training and test sets in an 8:2 ratio;

[0072] A2. Train the cervical lymph node identification network model using the training set data. During training, use the Adam optimizer and weighted cross-entropy as the loss function. The specific formula is as follows:

[0073]

[0074]

[0075] Among them, y c For the label value, y′ c The value is the model's predicted value; N is the total number of pixels in the training set. c This represents the total number of pixels in class c in the training set.

[0076] The Adam optimizer formula is as follows:

[0077] v t =β2vt-1 +(1-β2)g t 2

[0078] m t =β1m t-1 +(1-β1)g t

[0079]

[0080]

[0081]

[0082] Among them, g t The gradient at time t is represented by m. t v represents the gradient exponential moving average at time t, with an initial value of m0 = 0; t Let vt represent the squared exponential moving average of the gradient at time t, with an initial value of v0 = 0; β1 and β2 are the exponential decay rates, set to 0.9 and 0.999 respectively; η represents the learning rate; ε = 10 -8 θ is used to avoid division by zero; t Let θ represent the model parameters at time t. t-1 Represents the model parameters at time t-1;

[0083] A3. Use test set data to evaluate the model generated during training. When the evaluation result reaches the preset goal, that is, the model's diagnostic level is better than that of middle-aged doctors, the model training is complete. If the evaluation result does not reach the preset goal, adjust the parameters and repeat the training of the model until the preset goal is reached.

[0084] The model building module includes an encoding module, a semantic feature extraction module, a feature information fusion module, an information fusion module, and a decoding module.

[0085] The encoding module is used to input the preprocessed neck ultrasound image into the trained neck lymph node recognition network model. The neck ultrasound image is sequentially input into the encoding neural network ENET_1, encoding neural network ENET_2 and encoding neural network ENET_3. Each group of encoding neural networks reduces the feature map by half through convolution integration operation and outputs feature maps F1, F2 and F3 respectively.

[0086] The ENET_1, ENET_2, and ENET_3 coding neural networks each contain several convolutional layers. The convolutional layers include three operations: convolution integration, batch normalization, and activation functions. Each coding neural network reduces the feature map by half through a convolution integration operation with a stride of 2. That is, the ENET_1, ENET_2, and ENET_3 coding neural networks output feature maps F1 (256×256), F2 (128×128), and F3 (64×64) with the same size, respectively.

[0087] The semantic feature extraction module is used to input the feature map F3 output by the ENET_3 encoding neural network into the feature enhancement module Module_1 to extract high-level semantic features and output the feature map F. M1-6 .

[0088] like Figure 3 The diagram shown is a structural schematic of the feature enhancement module Module_1, where the ◎ symbol represents the addition of feature maps. Extracting semantic features includes the following steps:

[0089] B1. In the feature enhancement module Module_1, feature map F3 first extracts features through a convolutional layer of length 2, resulting in a feature map F with a size of 32×32. M1-1 Then the feature map F M1-1 The input is fed into three parallel dilated convolutional layers to extract features, resulting in feature maps F. M1-2 Feature map F M1-3 and feature map F M1-4 .

[0090] The dilated convolution operation inserts holes into the ordinary convolution kernel. Its hyperparameter dilation rate represents the interval distance of parameter values ​​on the convolution kernel. The dilated convolution operation can expand the receptive field of the convolution kernel without increasing the amount of computation, extract more global semantic information, and at the same time retain the relative spatial position information of the feature map. The dilation rates of the three parallel dilated convolution layers in the feature enhancement module Module_1 are set to 2, 4 and 6, respectively.

[0091] B2. The feature map F M1-2 Feature map F M1-3 and feature map F M1-4 The feature map is concatenated by channel and then fused using a convolutional layer with a kernel size of 1×1 to reduce the number of channels in the feature map, resulting in a feature map F. M1-1 Feature maps F with the same size and number of channels M1-5 .

[0092] B3. Transfer feature map F M1-1 and feature map FM1-5 The feature values ​​at corresponding channels and pixel indices are summed. A convolutional layer is then used to extract the summed features, and bilinear interpolation is used to augment the feature map, resulting in a 64×64 feature map F. M1-6 .

[0093] The feature information fusion module is used to simultaneously input feature maps F1, F2, and F3 into the feature enhancement module Module_2 while feature map F3 is input into feature enhancement module Module_1. It then fuses and calculates feature information at three scales to output feature map F. M2-9 .

[0094] like Figure 4 The diagram shown is a structural schematic of the feature enhancement module Module_2. The ◎ symbol indicates the addition of feature maps. The fusion calculation of feature information at three scales includes the following steps:

[0095] C1. In the feature enhancement module Module_2, feature map F1 first extracts features through a convolutional layer to obtain feature map F. M2-1 ;

[0096] C2, Transfer feature map F M2-1 The input is fed into the feature weighting module F_W, where the weight vector guides the model to selectively learn the feature map, and the output is the feature map F. M2-3 ;

[0097] C3. Transfer feature map F M2-3 With feature map F M2-1 The data is concatenated by channel, then fed into a convolutional layer to extract features, resulting in feature map F. M2-4 ;

[0098] C4. Use a pooling layer with a kernel size of 2 and a stride of 2 to apply the feature map F. M2-4 Downsampling is performed to obtain a feature map F with a size of 128×128. M2-5 ;

[0099] C5. After extracting features from feature map F2 through a convolutional layer, input it into the feature weighting module F_W; concatenate the output of feature map F2 after feature extraction through a convolutional layer with the output of feature weighting module F_W by channel, and then input it into a convolutional layer to extract features to obtain feature map F. M2-6 ;

[0100] C6. Transfer feature map F M2-6 With feature map F M2-5 The feature values ​​of corresponding channels are summed, then fed into a convolutional layer to extract features. These features are then downsampled using a pooling layer with a kernel size of 2 and a stride of 2, resulting in a feature map F with a size of 64×64.M2-7 ;

[0101] C7. After extracting features from feature map F3 through a convolutional layer, input it into the feature weighting module F_W; concatenate the output of feature map F3 after feature extraction through a convolutional layer with the output of feature weighting module F_W by channel, and then input it into a convolutional layer to extract features to obtain feature map F. M2-8 ;

[0102] C8. Transfer feature map F M2-8 With feature map F M2-7 The feature values ​​of corresponding channels are summed, and then fed into a convolutional layer to extract features, resulting in the final output of the feature enhancement module Module_2, which is a feature map F with a size of 64×64. M2-9 .

[0103] like Figure 5 The diagram shown is a structural schematic of the feature weighting module F_W, where the ◎ symbol indicates the addition of feature maps. The symbol represents feature map multiplication. In step C2, the feature weighting module F_W contains feature map F. M2-1 Feature extraction is performed consecutively through two convolutional layers. The extracted features are then input into a global average pooling layer and a fully connected layer for global feature mapping, generating a weight vector α(α1, α2, ..., α3) for each channel. n Next, the feature map F M2-1 The feature map of each channel is multiplied by the corresponding channel weight vector α to obtain the feature map F. M2-2 Simultaneously, feature map F M2-2 With feature map F M2-1 The information from each channel is added together, and then fused using a convolutional layer with a kernel size of 1×1 to obtain the feature map F. M2-3 .

[0104] The information fusion module is used to integrate feature maps F M1-6 and feature map F M2-9 Simultaneously, the input is fed into the dual attention module M_ATT, where the control vector generated by the attention mechanism regulates the information flow between low-level and high-level semantic features. Important information is selected from the two semantic features and fused to obtain the feature map F. M3-2 .

[0105] like Figure 6 The diagram shown is a structural schematic of the dual attention module M_ATT. The information fusion module performs the following steps.

[0106] D1. In the dual attention module M_ATT, the feature map F is first... M1-6 and feature map F M2-9The concatenation is performed channel-wise, and a convolutional layer is used to fuse the features of the concatenated result, while reducing the number of channels to obtain feature map F. M3-1 ;

[0107] D2. For feature map F M3-1 Perform global average pooling to obtain the channel feature vector S;

[0108] D3. Set up two fully connected layers to learn the channel feature vector S, and output control feature maps F from the two fully connected layers respectively. M1-6 and feature map F M2-9 The feature descriptor vectors β and γ, and the lengths of the feature descriptor vectors β and γ and their corresponding feature maps F M1-6 and feature map F M2-9 The number of channels is the same;

[0109] D4. Connect the feature description factor vectors β and γ to the feature map F respectively. M1-6 Feature map F M2-9 Multiplication involves multiplying the eigenvalues ​​of the factor vectors with all values ​​in the corresponding channels of the feature map, using the descriptive factor vectors to guide the learning of the feature maps. The multiplied feature maps are then concatenated channel by channel to obtain the final output feature map F of the dual attention module M_ATT. M3-2 The feature map has a size of 64×64.

[0110] The decoding module is used to decode the feature map F. M3-2 The input is fed into the decoding module DNET for upsampling and decoding to obtain the final output of the model.

[0111] The decoding module performs the following steps

[0112] F1. In the decoding module DNET, the feature map F M3-2 First, a convolutional layer is used to extract features and reduce the number of channels; then, a transposed convolution is used to enlarge the feature map size, resulting in a 128×128 feature map F. M4-1 ;

[0113] E2. Use a convolutional layer on feature map F M4-1 Feature extraction was performed and the number of channels was reduced to obtain a feature map F with 3 channels and a size of 128×128. M4-2 ;

[0114] E3. Use bilinear interpolation to transform the feature map F M4-2 The size is increased by 4 times, resulting in a feature map F with 3 channels and a size of 512×512. M4-3 ;

[0115] E4. Compare Feature Maps by Channel F M4-3The element values ​​of the first channel are used to determine the feature value. If the element value of the first channel is greater than that of the other two channels, the feature value is 0; if the element value of the second channel is greater than that of the other two channels, the feature value is 1; if the element value of the third channel is greater than that of the other two channels, the feature value is 2. Finally, the feature map F, which indicates the location and nature of lymph nodes, is obtained. M4-4 ;

[0116] A feature value of 0 indicates that the pixel in the neck ultrasound image is not a lymph node and is a background area; a feature value of 1 indicates that the pixel in the neck ultrasound image is a benign lymph node lesion; and a feature value of 2 indicates that the pixel in the neck ultrasound image is a metastatic lymph node.

[0117] In this embodiment, the multi-scale feature fusion network can extract lymph node features from different fields of view and fuse the features from different fields of view. The dual attention module controls the information flow between low-level and high-level features by generating descriptive factors, selects useful feature channels, and avoids the adverse effects of useless channel information on the model output results, making the localization contour line more refined and the property diagnosis more accurate. This method can locate lymph nodes on cervical ultrasound images and identify lymph nodes with cancer metastasis, thereby better assisting doctors in quickly identifying and diagnosing cervical lymph nodes during ultrasound examinations, avoiding too many subjective factors, and making the diagnostic results more reliable.

[0118] In addition to the embodiments described above, the present invention may have other implementations. All technical solutions formed by equivalent substitution or equivalent transformation fall within the protection scope claimed by the present invention.

Claims

1. A cancer metastasis cervical lymph node identification system based on multi-scale feature fusion, characterized by: The model construction module comprises an encoding module, a semantic feature extraction module, a feature information fusion module, an information fusion module, and a decoding module. The ultrasonic image acquisition module is configured to acquire a neck ultrasonic image and pre-process the neck ultrasonic image. The model construction module is configured to construct a neck lymph node recognition network model, wherein the neck lymph node recognition network model comprises an encoding neural network ENET_1, an encoding neural network ENET_2, an encoding neural network ENET_3, a feature enhancement module Module_1, a feature enhancement module Module_2, a double-attention module M_ATT, and a decoding module DNET. The model training module is configured to train the neck lymph node recognition network model to obtain an optimal model. The model construction module comprises an encoding module, a semantic feature extraction module, a feature information fusion module, an information fusion module, and a decoding module. The encoding module is configured to input the pre-processed neck ultrasonic image into the trained neck lymph node recognition network model, and the neck ultrasonic image is sequentially input into the encoding neural network ENET_1, the encoding neural network ENET_2, and the encoding neural network ENET_3. The semantic feature extraction module is configured to input the feature map F3 output by the encoding neural network ENET_3 into the feature enhancement module Module_1, extract semantic features, and output a feature map F M1-6 ; The feature information fusion module is configured to input the feature maps F1, F2 and F3 into the feature enhancement module Module_2 while the feature map F3 is input into the feature enhancement module Module_1, fuse and calculate the feature information of the three scales, and output a feature map F M2-9 ; The information fusion module is configured to fuse the feature maps F M1-6 and the feature maps F M2-9 into the double-attention module M ATT, and generate a control vector by using an attention mechanism to regulate information flow between different semantic features, select information between different semantic features for fusion, and obtain the feature maps F M3-2 ; The decoding module is configured to input the feature map F M3-2 to the decoding module DNET for upsampling and decoding to obtain a final output result of the model.

2. The cancer metastasis cervical lymph node recognition system based on multi-scale feature fusion according to claim 1, characterized in that: In the ultrasonic image acquisition module, the size of the neck ultrasonic image is adjusted to 512*512, which is the unified input size of the model.

3. The cancer metastasis cervical lymph node recognition system based on multi-scale feature fusion of claim 1, characterized in that: In the model training module, the training of the neck lymph node recognition network model comprises the following steps A1, constructing a data set and dividing the data set into a training set and a test set according to a ratio of 8:2; A2, training the neck lymph node recognition network model using the training set data, using an Adam optimizer during the training process, and using a cross-entropy function with a weight as the loss function, with the specific formula being: where y c is the label value, y′ c is the model prediction; N is the total number of pixels in the training set, N c is the total number of pixels of class c in the training set; The Adam optimizer formula is as follows: v t = β2v t-1 + (1 - β2)g t 2 m t = β1m t-1 + (1 - β1)g t where g t denotes the gradient at time t; m t denotes the exponential moving average of the gradient at time t, with initial value m0=0; v t denotes the exponential moving average of the squared gradient at time t, with initial value v0=0; β1and β2are exponential decay rates, set to 0.9 and 0.999, respectively; η denotes the learning rate; ε=10 -8 , to avoid division by zero; θ t denotes the model parameter at time t; θ t-1 denotes the model parameter at time t-1. A3, evaluating the model generated during the training process using the test set data, and when the evaluation result reaches a preset target, the model training is completed; if the evaluation result does not reach the preset target, adjusting the parameters and repeating the model training until the preset target is reached.

4. The cancer metastasis cervical lymph node recognition system based on multi-scale feature fusion of claim 1, wherein: In the neck lymph node recognition network model, the encoding neural network ENET_1, the encoding neural network ENET_2, and the encoding neural network ENET_3 each comprise a plurality of convolution layers, and the convolution layer comprises convolution integration, batch normalization, and an activation function; each group of encoding neural networks is reduced by half through a convolution integration operation with a step of 2, i.e., the encoding neural network ENET 1, the encoding neural network ENET_2, and the encoding neural network ENET 3 output feature maps F1 with a size of 256*256, feature maps F2 with a size of 128*128, and feature maps F3 with a size of 64*64, respectively.

5. The cancer metastasis cervical lymph node recognition system based on multi-scale feature fusion according to claim 1, characterized in that: In the semantic feature extraction module, the semantic feature extraction comprises the following steps B1. In the feature enhancement module Module_1, feature map F3 first extracts features through a convolutional layer of length 2, resulting in a feature map F with a size of 32×32. M1-1 Then the feature map F M1-1 The input is fed into three parallel dilated convolutional layers to extract features, resulting in feature maps F. M1-2 Feature map F M1-3 and feature map F M1-4 ; B2, the feature map F M1-2 , the feature map F M1-3 , and the feature map F M1-4 are spliced by channel, and a convolution layer with a convolution kernel size of 1x1 is used to fuse channel information, reduce the number of feature map channels, and obtain a feature map F M1-1 with the same size and the same number of channels as the feature map F M1-5 ; B3, the feature map F M1-1 and the feature map F M1-5 are added, i.e. the feature values of the corresponding channels, corresponding pixel index positions are added, then a convolutional layer is used to extract the added features, and a bilinear interpolation operation is used to expand the size of the feature map, obtaining a feature map F M1-6 of size 64x64.

6. The cancer metastasis cervical lymph node identification system based on multi-scale feature fusion according to claim 5, characterized in that: In step B1, the expansion rates of the three parallel dilated convolution layers are set to 2, 4 and 6 respectively.

7. The cancer metastasis cervical lymph node recognition system based on multi-scale feature fusion of claim 1, wherein: In the feature information fusion module, the step of fusing and calculating the feature information of three scales includes the following steps C1, in the feature enhancement module Module_2, the feature map F1 is first extracted via a convolutional layer to obtain a feature map F M2-1 ; C2, the feature map F M2-1 is input into a feature weighting module F W, and the model is guided to selectively learn the feature map using a weight vector, and a feature map F M2-3 is output. C3, the feature map F M2-3 with the feature map F M2-1 concatenated along the channels, followed by a convolutional layer to extract features, resulting in a feature map F M2-4 ; C4, the pooling layer with the pooling kernel size of 2 and the step size of 2 is used to pool the feature map F M2-4 down-sampling is performed to obtain the feature map F with the size of 128x128 M2-5 ; C5, input the feature map F2 after feature extraction via a convolutional layer into a feature weighting module F W; concatenate the output of the feature map F2 after feature extraction via a convolutional layer and the output of the feature weighting module F W by channel, and then input into a convolutional layer to obtain a feature map F M2-6 ; C6, the feature map F M2-6 with the feature map F M2-5 The feature values of the corresponding channels are added, and then a convolutional layer is used to extract features, and a pooling layer with a kernel size of 2 and a step size of 2 is used for down-sampling, to obtain a feature map F with a size of 64x64 M2-7 ; C7, input the feature map F3 after feature extraction via a convolutional layer into the feature weighting module F W; concatenate the output of the feature map F3 after feature extraction via a convolutional layer and the output of the feature weighting module F W by channel, and then input into a convolutional layer to obtain the feature map F M2-8 ; C8, the feature map F M2-8 with the feature map F M2-7 The feature values of the corresponding channels are added, and then a convolutional layer is extracted to obtain the final output of the feature enhancement module Module_2, that is, a feature map F with a size of 64x64 M2-9 .

8. The cancer metastasis cervical lymph node recognition system based on multi-scale feature fusion according to claim 7, characterized in that: In step C2, in the feature weighting module F_W, the feature map F M2-1 is continuously extracted through two convolution layers, and the extracted features are input into a global average pooling layer and a fully connected layer for global mapping of the features to generate a weight vector α (α1, α2, …, α n ) corresponding to each channel; then the feature map F M2-1 is multiplied by the weight vector α corresponding to each channel to obtain a feature map F M2-2 ; meanwhile, the feature map F M2-2 is added to the feature map F M2-1 by channel, and then a convolution layer with a 1×1 convolution kernel is used to fuse the information of each channel to obtain a feature map F M2-3 .

9. The cancer metastasis cervical lymph node recognition system based on multi-scale feature fusion of claim 1, wherein: The information fusion module performs the following steps D1、In the double attention module M ATT, first, the feature map F M1-6 and the feature map F M2-9 are spliced by channel, and the spliced result is fused by a convolution layer to reduce the number of channels and obtain the feature map F M3-1 ; D2, on the feature map F M3-1 performing a global average pooling operation to obtain a channel feature vector S; D3, set two fully connected layers to learn the channel feature vector S, and the two fully connected layers output the control feature map F M1-6 and the feature description factor vector β, γ of the feature map F M2-9 , the length of the feature description factor vector β, γ is the same as the channel number of the corresponding feature map F M1-6 and the feature map F M2-9 ; D4, multiply the feature description factor vectors β, γ respectively with the feature map F M1-6 , multiply the feature map F M2-9 , that is, multiply the factor vector feature value with all values on the corresponding channel of the feature map, splice the multiplied feature map by channel to obtain the final output feature map F of the double attention module M ATT M3-2 , and the size of the feature map is 64x64.

10. The cancer metastasis cervical lymph node recognition system based on multi-scale feature fusion of claim 1, wherein: The decoding module performs the following steps F1, in the decoding module DNET, the feature map F M3-2 First, via a convolutional layer processing, features are extracted and the channel number is reduced; subsequently, a transposed convolution is used to expand the size of the feature map, obtaining a feature map F M4-1 ; E2, using a convolutional layer on the feature map F M4-1 perform feature extraction and reduce the number of channels to obtain a feature map F with a channel number of 3 and a size of 128x128 M4-2 ; E3. The feature map F is resized to 3x512x512 using bilinear interpolation M4-2 The feature map F is resized 4 times, resulting in a feature map F with 3 channels and size 512x512 M4-3 ; E4. Comparing the feature map F by channel M4-3 If the element value of the first channel is greater than the other two channels, the feature value is 0; if the element value of the second channel is greater than the other two channels, the feature value is 1; if the element value of the third channel is greater than the other two channels, the feature value is 2; finally, the indication feature map F of the lymph node position and property is obtained M4-4 ; The feature value of 0 indicates that the pixel point is not a pixel point of a lymph node in the neck ultrasound image, that is, a background region. The feature value of 1 indicates that the pixel point is a pixel point of a benign lymph node lesion in the neck ultrasound image; and the feature value of 2 indicates that the pixel point is a pixel point of a metastatic lymph node in the neck ultrasound image.

Images