A prediction system for central cervical lymph node metastasis of thyroid cancer based on ultrasound images

Through the deep learning model based on ultrasound images, the cross-layer guide module and multi-layer feature aggregation module are used to solve the accuracy of lymph node metastasis diagnosis in the central area of ​​neck of thyroid cancer, which improves prediction performance and reduces the subjectivity of clinical applications. Work burden.

CN117766134BActive Publication Date: 2024-11-29TIANJIN UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202410059973.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-16
Publication Date
2024-11-29
Estimated Expiration
2044-01-16

AI Technical Summary

Technical Problem

The prior art is difficult to accurately diagnose lymph node metastasis in the central area of ​​the neck of patients with thyroid cancer, resulting in problems of overtreatment and low predictive performance.

Method used

A system for lymph node metastasis prediction in the central region of thyroid cancer neck based on ultrasound images is designed, including image encoder, nodule mask decoder, nodule boundary decoder and lymph node metastasis state predictor, through a cross-layer guide module and multiple Layer feature aggregation module, a deep learning model is constructed to locate the internal and boundary areas of thyroid nodules, extract features and predict lymph node metastasis status.

Benefits of technology

Improve the accuracy of predicting lymph node metastasis in the central area of ​​neck of thyroid cancer, especially the prediction performance of small nodules has been significantly improved, maintaining the accuracy of prediction of large nodules, reducing the work burden of doctors and subjective clinical application sex.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117766134B_ABST
    Figure CN117766134B_ABST
Patent Text Reader

Abstract

The present invention discloses a system for predicting lymph node metastasis in the central neck of thyroid cancer based on ultrasound images, including an image encoder, a nodule mask decoder, a nodule boundary decoder, and a lymph node metastasis state predictor; the image encoder is used to input a thyroid ultrasound image and encode the thyroid ultrasound image. The present invention constructs and trains a deep learning model, designs a cross-layer guidance module, so that the model can accurately locate the internal and boundary areas of the thyroid nodule, and designs a multi-layer feature fusion module, which improves the prediction performance of the model for small nodules while maintaining the accuracy of large nodule prediction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the field of software, and in particular to a prediction system for central neck lymph node metastasis of thyroid cancer based on ultrasonic images. Background Art

[0002] In the past 30 years, the incidence of thyroid cancer has risen sharply. In 2020 alone, the number of new cases of thyroid cancer has exceeded 586,000, and about 50% of these new cases are prone to central cervical lymph node metastasis. The status of central cervical lymph node metastasis is an important reference for determining the patient's surgical plan. For patients with lymph node metastasis, therapeutic central lymph node dissection should be performed. For patients with uninvolved central lymph nodes, there is great controversy as to whether preventive lymph node dissection should be performed. A more conservative surgical plan may lead to disease recurrence and the risk of secondary surgery, while a more radical surgical plan may reduce the patient's quality of life due to recurrent laryngeal nerve damage or hypoparathyroidism.

[0003] Ultrasound examination has the advantages of simple operation, no radiation, and high local resolution, and is the preferred imaging examination method for the thyroid gland. However, due to the deep location of central lymph nodes, ultrasound is easily shielded by air, bones, etc., making it difficult to directly observe the central lymph nodes of patients. Therefore, ultrasound examination is difficult to accurately diagnose the central lymph node metastasis status of patients. Meta-analysis shows that the sensitivity of ultrasound for the diagnosis of central lymph node metastasis is only 0.33. CT scanning is of great value in evaluating the extent of thyroid tumors, the relationship between nodules and surrounding important structures such as trachea, esophagus, and carotid artery, and the presence or absence of lymph node metastasis. However, CT has a low resolution and is difficult to accurately diagnose the central lymph node metastasis status of patients. Meta-analysis shows that the sensitivity of CT for lymph node metastasis is 0.43. Although clinical guidelines recommend the use of ultrasound-guided fine-needle aspiration biopsy to confirm the patient's central lymph node metastasis status, fine-needle aspiration biopsy requires experienced radiologists and cytologists, and fine-needle aspiration biopsy is an invasive procedure with low patient acceptance, so fine-needle aspiration biopsy is less popular in clinical practice. Given the high incidence of central cervical lymph node metastasis in thyroid cancer patients and the lack of clinical methods to accurately diagnose the patient's lymph node metastasis status, preventive central lymph node dissection is widely used in clinical practice, which has led to widespread overtreatment.

[0004] Although it is difficult to observe the central cervical lymph nodes using ultrasound, thyroid nodules are located superficially on the skin, and ultrasound can well reflect the nodule's echo, cysticity, calcification, size, shape, aspect ratio, border regularity and other information. Therefore, in recent years, some scholars have begun to use ultrasound images of thyroid nodules to predict the patient's central cervical lymph node metastasis status, which currently mainly includes two types of methods. The first type is non-radiomics methods, in which researchers incorporate factors such as age, gender, thyroid nodule size, location, diameter, multifocality, and extraglandular invasion, and construct a machine learning model to predict the central lymph node metastasis status, with an AUC between 0.70 and 0.772. The second category is radiomics methods. In this type of method, researchers extracted high-dimensional radiomics features from ultrasound images, covering nodule size, shape, edge, direction, location, echo pattern, calcification characteristics, etc., selected the extracted features, and then used classifiers such as support vector machines and random forests to complete the prediction of lymph node metastasis status, and the obtained AUC was between 0.78 and 0.90.

[0005] Although the above methods have achieved certain results, they still have some shortcomings. First, non-radiomics methods require manual selection of features to be included in the model, and radiomics methods also require manual definition of high-dimensional features. Both methods are highly subjective and are prone to losing features related to the metastasis status of cervical lymph nodes, which in turn leads to reduced prediction performance. Second, these methods require doctors to outline the boundaries of nodules in ultrasound images in advance, which increases the workload of doctors and reduces the convenience of clinical application. Finally, most of the above methods have low AUCs and unsatisfactory prediction effects. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a system for predicting central cervical lymph node metastasis of thyroid cancer based on ultrasound images.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] A system for predicting lymph node metastasis in the central neck of thyroid cancer based on ultrasound images, comprising an image encoder, a nodule mask decoder, a nodule boundary decoder and a lymph node metastasis state predictor; the image encoder is used to input a thyroid ultrasound image and encode the thyroid ultrasound image;

[0009] The nodule mask decoder is used to segment the thyroid nodules, guide the image encoder to locate the thyroid nodules in the ultrasound image, and extract the internal structural features of the thyroid nodules. The nodule boundary decoder is used to segment the boundaries of the thyroid nodules, guide the image encoder to locate the boundaries of the thyroid nodules in the ultrasound image, and extract the boundary features of the thyroid nodules. The lymph node metastasis status predictor is used to predict whether there is thyroid cancer cell metastasis to the lymph nodes based on the thyroid ultrasound image; the image encoder is connected to the nodule mask decoder and the nodule boundary decoder respectively through a cross-layer guidance module; the cross-layer guidance module is used to assist the nodule mask decoder to locate the internal area of ​​the thyroid nodule in the thyroid ultrasound image and assist the nodule boundary decoder to locate the thyroid nodule boundary area in the thyroid ultrasound image.

[0010] As a further improvement, the image encoder is a ResNet34 network.

[0011] As a further improvement, the image encoder comprises a convolution module and a pooling layer 1, a first residual module 2, a second residual module 3, a third residual module 5 and a fourth residual module 7 which are sequentially arranged along a data processing direction;

[0012] The lymph node metastasis status predictor sequentially sets a multi-layer feature aggregation module and a fully connected layer along the data processing direction; the output of the fourth residual module 7 is the input of the third input layer of the multi-layer feature aggregation module;

[0013] The input of the first input layer 4 of the multi-layer feature aggregation module is the sum of the output of the first cross-layer guidance module 16, the output of the second cross-layer guidance module 17 and the output of the second residual module 3;

[0014] The input of the second input layer 6 of the multi-layer feature aggregation module is the sum of the output of the third cross-layer guidance module 18, the output of the fourth cross-layer guidance module 19 and the output of the third residual module 5; the input of the first input layer 4 of the multi-layer feature aggregation module is also used as the input of the third residual module 5, and the input of the second input layer 6 of the multi-layer feature aggregation module is also used as the input of the fourth residual module 7;

[0015] The nodule mask decoder and the nodule boundary decoder have the same structure, and both include a first upsampling module 8, a first feature splicing module 9, a second upsampling module 10, a second feature splicing module 11, a third upsampling module 12, a third feature splicing module 13, a fourth upsampling module 14 and a 1×1 convolution module 15 along the data processing direction;

[0016] The cross-layer guidance module includes a first cross-layer guidance module 16, a second cross-layer guidance module 17, a third cross-layer guidance module 18 and a fourth cross-layer guidance module 19;

[0017] The output of the first residual module 2 is used as the input of the first convolution normalization activation module 20, the second convolution normalization activation module 21 and the second residual module 3 respectively;

[0018] The output of the first convolution normalized activation module 20 is used as the input of the first cross-layer guidance module 16 and the third feature splicing module of the nodule mask decoder respectively; the output of the first cross-layer guidance module 16 is used as the input of the second feature splicing module and the third cross-layer guidance module 18 of the nodule mask decoder;

[0019] The output of the second convolution normalization activation module 21 is used as the input of the second cross-layer guidance module 17 and the third feature splicing module of the nodule boundary decoder; the output of the second cross-layer guidance module 17 is used as the input of the second feature splicing module and the fourth cross-layer guidance module 19 of the nodule boundary decoder;

[0020] The output of the first cross-layer guidance module 16 is used as the input of the second feature concatenation module of the nodule mask decoder;

[0021] The output of the second cross-layer guidance module 17 is used as the input of the second feature splicing module of the nodule boundary decoder;

[0022] The output of the third cross-layer guidance module 18 is used as the input of the first feature concatenation module of the nodule mask decoder;

[0023] The output of the fourth cross-layer guidance module 19 is used as the input of the first feature concatenation module of the nodule boundary decoder.

[0024] As a further improvement, the convolution module and the pooling layer 1 sequentially include a convolution layer, a normalization layer, a Relu activation layer and a pooling layer along the data processing direction; the first convolution normalization activation module 20 and the second convolution normalization activation module 21 sequentially include a convolution layer, a normalization layer and a Relu activation layer along the data processing direction;

[0025] The first upsampling module 8, the second upsampling module 10, the second feature concatenation module 11, the third upsampling module 12 and the fourth upsampling module 14 respectively include an upsampling layer, a convolution layer and a dropout layer along the data processing direction.

[0026] As a further improvement, the data processing method of the cross-layer guidance module is as follows:

[0027] The output feature x'0 of the previous cross-layer guidance module is used to guide the nodule mask response in the output feature x1 of the residual module of the image encoder:

[0028] 5.1 First, calculate the query element vector Q of the nodule mask from x'0:

[0029]

[0030] Among them, Adapooling() represents adaptive pooling, Conv() represents convolution, A query meta-vector representing the nodule mask;

[0031] 5.2 Calculate the key vector K from the output feature x1 of the current residual module:

[0032] K=conv(x1) (2)

[0033] The attention vector A between the query meta-vector Q and the key vector K is:

[0034]

[0035] Among them, d k is the dimension of the key vector K; T is the matrix transpose, and softmax is the softmax activation function;

[0036] 5.3 The cross-layer guidance module includes two outputs, where the first output y1 is used to input the multi-layer feature aggregation module of the image encoder to enhance the response of the internal area of ​​the nodule in the image encoder, and the calculation method is:

[0037] y1=α·A·V (4)

[0038] Among them, α is a trainable parameter, V is the Value matrix, and the calculation method is:

[0039] V = Relu(BN(Conv(x1))) (5)

[0040] Relu is the activation function, BN is the batch normalization layer;

[0041] The second output x'1 is passed to the feature concatenation module of the nodule mask decoder or the nodule boundary decoder; x1' is calculated as:

[0042] x′1=Relu(BN(Conv(A·V))) (6).

[0043] As a further improvement, the inputs of the first input layer 4, the second input layer 6 and the third input layer 22 of the multi-layer feature aggregation module are f3, f4 and f5 respectively; f3, f4 and f5 are first subjected to a convolution operation to obtain their respective key tensors.

[0044]

[0045] And further concatenate the three key tensors to obtain the key tensor that aggregates multiple layers of features

[0046]

[0047] concat means tensor concatenation;

[0048] Get the Value tensors corresponding to f3, f4, and f5 respectively

[0049]

[0050]

[0051] A Value tensor representing the aggregation of multiple layers of features;

[0052] Among them, f5 is closest to the lymph node metastasis status label and can obtain the most direct supervision signal. The multi-layer feature aggregation module further performs global average pooling on f5 to obtain the query element vector about the lymph node metastasis status

[0053]

[0054] Adapooling represents the adaptive pooling layer

[0055] Query Meta Vector and key vector The attention vector A between H for:

[0056]

[0057] Finally, we get the aggregated feature f Agg for:

[0058]

[0059] Aggregate feature f Agg It is then further fed into the fully connected layer of the lymph node metastasis status predictor to complete the prediction of lymph node metastasis.

[0060] The beneficial effects of the present invention are:

[0061] The present invention constructs and trains a deep learning model, designs a cross-layer guidance module to enable the model to accurately locate the internal and boundary areas of thyroid nodules, and designs a multi-layer feature fusion module to improve the model's prediction performance for small nodules while maintaining the accuracy of large nodule prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The present invention is further described with reference to the accompanying drawings, but the contents in the accompanying drawings do not constitute any limitation to the present invention.

[0063] Figure 1 It is a schematic diagram of the overall structure of the present invention;

[0064] Figure 2 It is a schematic diagram of the structure of the cross-layer guidance module;

[0065] Figure 3 Schematic diagram of the structure of the multi-layer feature aggregation module. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of the invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0067] When diagnosing thyroid nodules, clinicians mainly focus on the echo, calcification, cysticity, size, shape, and boundary information of the nodules. Among them, the echo, calcification, cysticity, and boundary information of the nodules are mainly distributed inside the nodules, while the size, shape, and boundary information of the nodules are mainly distributed in the boundary area of ​​the nodules. Based on this prior knowledge, Figure 1 As shown in the figure, the present invention designs a multi-task model ThyLymNet, which uses the nodule mask segmentation task to guide the model to focus on the internal features of the nodule, and uses the nodule boundary segmentation task to guide the model to focus on the boundary features of the nodule, and integrates the two features to complete the final lymph node metastasis prediction. Specifically, as shown in the attached figure Figure 1 As shown, the model designed by the present invention includes four branches, namely, image encoder (Image Encoder), nodule mask decoder (MaskDecoder), nodule boundary decoder (BoundaryDecoder) and lymph node metastasis status predictor (Predictor). In view of the good performance of ResNet in computer vision tasks, the present invention adopts ResNet34 as the image encoder. The nodule mask decoder and the nodule boundary decoder have the same structure, both of which include four upsampling modules, each of which includes an upsampling layer, a convolutional layer and a dropout layer. The lymph node metastasis status predictor consists of a cross-layer feature fusion module, a fully connected layer, and an activation layer, wherein the activation function is a Sigmoid function.

[0068] Cross-layer guidance module

[0069] In order to better locate the internal and boundary areas of thyroid nodules in ultrasound images, the present invention designs a cross-layer guidance module. Since the four cross-layer guidance modules used in the present invention have the same structure, this section takes the cross-layer guidance module connecting the image encoder branch and the nodule mask decoder branch as an example to introduce the cross-layer guidance module.

[0070] The structure of the cross-layer guidance module is shown in the attached figure. Figure 2As shown in Figure 1, the input of this module includes the output feature x'0 of the previous cross-layer guidance module (for the first cross-layer guidance module, this input is the output of the previous Conv-BN-Relu component) and the output feature x1 of the current residual module. x'0 comes from a shallower network layer and contains more fine-grained nodule mask position information. Therefore, the present invention uses x'0 to guide the nodule mask response in x1. First, the query element vector of the nodule mask is calculated from x'0:

[0071]

[0072] Among them, Adapooling(·) represents adaptive pooling, and Conv(·) represents convolution. After that, the key vector is calculated from the current layer feature x1:

[0073] K = conv(x1) # (2)

[0074] The attention vector between the query meta-vector and the key vector is:

[0075]

[0076] Among them, d k is the dimension of the key vector.

[0077] The cross-layer guidance module includes two outputs, where the first output y1 is used to enhance the response of the internal area of ​​the nodule in the image encoder, and its calculation method is:

[0078] y1=α·A·V#(4)

[0079] Among them, α is a trainable parameter, V is the Value matrix, and the calculation method is:

[0080] V = Relu(BN(Conv(x1))) #(5)

[0081] The second output x'1 is further passed to the mask decoder for downstream mask segmentation tasks, and further passed to the cross-layer decoding module of the next layer to guide the positioning of the nodule mask in the next network layer. The calculation method of x1' is:

[0082] x′1=Relu(BN(Conv(A·V)))#(6)

[0083] The computational complexity of ordinary attention is O(N 2 ), where N is the number of pixels in the Query and Key (generally, the number of pixels in the Query and Key are equal). The computational complexity of the attention in the cross-layer guidance module is only O(N), which is only The cross-layer guidance module has low computational complexity and can be applied multiple times between multiple adjacent network layers and on multiple tasks such as nodule masking and boundary segmentation, allowing the model to focus more accurately on the internal and boundary areas of nodules in ultrasound images.

[0084] Multi-layer feature aggregation module

[0085] In order to improve the prediction accuracy of small nodules while maintaining the prediction performance of large nodules, the present invention proposes a multi-layer feature aggregation module, which can adaptively aggregate features of different network layers according to the size of nodules. The specific structure is shown in the attached figure. Figure 3 In the present invention, the multi-layer feature aggregation module adaptively aggregates the output features from the 3rd, 4th, and 5th residual modules, which are represented as f3, f4, and f5, respectively. Inside the multi-layer feature aggregation module, f3, f4, and f5 are first convolved to obtain their respective key tensors:

[0086]

[0087] And further concatenate the key tensors of the three layers to obtain the key tensor that aggregates multiple layers of features:

[0088]

[0089] Similarly, f3, f4, and f5 further obtain the Value tensor:

[0090]

[0091]

[0092] Because f5 is closest to the lymph node metastasis status label and can obtain the most direct supervisory signal, the present invention further performs global average pooling on f5 to obtain the query element vector about the lymph node metastasis status:

[0093]

[0094] Correspondingly, the attention vector between the query meta-vector and the key vector is:

[0095]

[0096] Finally, the aggregated features are:

[0097]

[0098] The aggregated features are then further fed into the downstream fully connected layer to complete the prediction of lymph node metastasis.

[0099] Implementation details

[0100] In the present invention, the basic learning rate of model training is 1×10 -4 , and adjusted by cosine decay. The batch size is set to 30, the optimizer is Adam, and the weight decay is set to 1×10 -5 , which is used to mitigate the risk of overfitting of the model. For classification tasks, the present invention uses a binary cross entropy loss function, and for segmentation tasks, the present invention uses a combination of a binary cross entropy loss and an intersection-over-union loss function. The model is trained for a total of 300 epochs, of which the first 100 epochs are only trained for the segmentation task, and the next 200 epochs are trained for both the segmentation and lymph node metastasis prediction tasks.

[0101] Prediction effect

[0102] The present invention uses accuracy, AUC (area under the curve), sensitivity, specificity, precision, F1-score and other indicators to evaluate the prediction performance of the model. In order to prove the effect of the model proposed in the present invention, the prediction results of the model are compared with the existing general classification model, and the results are shown in Table 1. Among them, the optimal indicators are represented by bold fonts. As shown in Table 1, compared with the existing general classification model, the ThyLymNet proposed in the present invention has achieved the best performance in multiple indicators, which fully proves the effectiveness of the present invention.

[0103] Table 1. Comparison of prediction effects between ThyLymNet and existing general classification models

[0104]

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the essence and scope of the technical solution of the present invention.

Claims

1. A prediction system for central cervical lymph node metastasis of thyroid cancer based on ultrasound images, characterized in that: It includes an image encoder, a nodule mask decoder, a nodule boundary decoder and a lymph node metastasis status predictor; the image encoder is used to input a thyroid ultrasound image and encode the thyroid ultrasound image; The nodule mask decoder is used to segment the thyroid nodules, and guide the image encoder to locate the thyroid nodules in the ultrasound image, and extract the internal structural features of the thyroid nodules. The nodule boundary decoder is used to segment the boundaries of the thyroid nodules, and guide the image encoder to locate the boundaries of the thyroid nodules in the ultrasound image, and extract the boundary features of the thyroid nodules. The lymph node metastasis state predictor is used to predict whether there is a situation in which thyroid cancer cells metastasize to the lymph nodes based on the thyroid ultrasound image; the image encoder is respectively connected to the nodule mask decoder and the nodule boundary decoder through a cross-layer guidance module; the cross-layer guidance module is used to assist the nodule mask decoder to locate the internal area of ​​the thyroid nodule in the thyroid ultrasound image and assist the nodule boundary decoder to locate the thyroid nodule boundary area in the thyroid ultrasound image; The image encoder comprises a convolution module and a pooling layer (1), a first residual module (2), a second residual module (3), a third residual module (5) and a fourth residual module (7) which are sequentially arranged along a data processing direction; The lymph node metastasis status predictor sequentially sets a multi-layer feature aggregation module and a fully connected layer along the data processing direction; the output of the fourth residual module (7) is the input of the third input layer of the multi-layer feature aggregation module; The input of the first input layer (4) of the multi-layer feature aggregation module is the sum of the output of the first cross-layer guidance module (16), the output of the second cross-layer guidance module (17) and the output of the second residual module (3); The input of the second input layer (6) of the multi-layer feature aggregation module is the sum of the output of the third cross-layer guidance module (18), the output of the fourth cross-layer guidance module (19) and the output of the third residual module (5); the input of the first input layer (4) of the multi-layer feature aggregation module is also used as the input of the third residual module (5), and the input of the second input layer (6) of the multi-layer feature aggregation module is also used as the input of the fourth residual module (7); The nodule mask decoder and the nodule boundary decoder have the same structure, and both include a first upsampling module (8), a first feature splicing module (9), a second upsampling module (10), a second feature splicing module (11), a third upsampling module (12), a third feature splicing module (13), a fourth upsampling module (14) and a 1×1 convolution module (15) along the data processing direction; The cross-layer guiding module comprises a first cross-layer guiding module (16), a second cross-layer guiding module (17), a third cross-layer guiding module (18) and a fourth cross-layer guiding module (19); The output of the first residual module (2) is respectively used as the input of the first convolution normalization activation module (20), the second convolution normalization activation module (21) and the second residual module (3); The output of the first convolution normalized activation module (20) is used as the input of the first cross-layer guidance module (16) and the third feature splicing module of the nodule mask decoder respectively; the output of the first cross-layer guidance module (16) is used as the input of the second feature splicing module and the third cross-layer guidance module (18) of the nodule mask decoder; The output of the second convolution normalization activation module (21) is used as the input of the second cross-layer guidance module (17) and the third feature splicing module of the nodule boundary decoder; the output of the second cross-layer guidance module (17) is used as the input of the second feature splicing module and the fourth cross-layer guidance module (19) of the nodule boundary decoder; The output of the first cross-layer guidance module (16) is used as the input of the second feature concatenation module of the nodule mask decoder; The output of the second cross-layer guidance module (17) is used as the input of the second feature splicing module of the nodule boundary decoder; The output of the third cross-layer guidance module (18) is used as the input of the first feature concatenation module of the nodule mask decoder; The output of the fourth cross-layer guiding module (19) is used as the input of the first feature splicing module of the nodule boundary decoder; the convolution module and the pooling layer (1) sequentially include a convolution layer, a normalization layer, a Relu activation layer and a pooling layer along the data processing direction; the first convolution normalization activation module (20) and the second convolution normalization activation module (21) sequentially include a convolution layer, a normalization layer and a Relu activation layer along the data processing direction; The first upsampling module (8), the second upsampling module (10), the second feature concatenation module (11), the third upsampling module (12) and the fourth upsampling module (14) each sequentially include an upsampling layer, a convolution layer and a dropout layer along the data processing direction; The data processing method of the cross-layer guidance module is as follows: The output feature x'0 of the previous cross-layer guidance module is used to guide the nodule mask response in the output feature x1 of the residual module of the image encoder: 5.1 First, calculate the query element vector Q of the nodule mask from x'0: Among them, Adapooling() represents adaptive pooling, Conv() represents convolution, A query meta-vector representing the nodule mask; 5.2 Calculate the key vector K from the output feature x1 of the current residual module: K=conv(x1) (2) The attention vector A between the query meta-vector Q and the key vector K is: Among them, d k is the dimension of the key vector K; T is the matrix transpose, and softmax() is the softmax activation function; 5.3 The cross-layer guidance module includes two outputs, where the first output y1 is used to input the multi-layer feature aggregation module of the image encoder to enhance the response of the internal area of ​​the nodule in the image encoder, and the calculation method is: y1=α·A·V (4) Among them, α is a trainable parameter, V is the Value matrix, and the calculation method is: V = Relu(BN(Conv(x1))) (5) Relu() is the activation function, BN() is the batch normalization layer; The second output x'1 is passed to the feature concatenation module of the nodule mask decoder or the nodule boundary decoder; x1' is calculated as: x′1=Relu(BN(Conv(A·V))) (6).

2. The ultrasound image-based prediction system for thyroid cancer central cervical lymph node metastasis according to claim 1, characterized in that: The image encoder is a ResNet34 network.

3. The ultrasound image-based prediction system for thyroid cancer central cervical lymph node metastasis according to claim 1, characterized in that: The inputs of the first input layer (4), the second input layer (6) and the third input layer (22) of the multi-layer feature aggregation module are f3, f4, and f5 respectively; f3, f4, and f5 are first subjected to a convolution operation to obtain their respective key tensors And further concatenate the three key tensors to obtain the key tensor that aggregates multiple layers of features concat() means tensor concatenation; Get the Value tensors corresponding to f3, f4, and f5 respectively A Value tensor representing the aggregation of multiple layers of features; Among them, f5 is closest to the lymph node metastasis status label and can obtain the most direct supervision signal. The multi-layer feature aggregation module further performs global average pooling on f5 to obtain the query element vector about the lymph node metastasis status Adapooling() represents the adaptive pooling layer Query Meta Vector and key vector The attention vector A between H for: Finally, we get the aggregated feature f Agg for: Aggregate feature f Agg It is then further sent to the fully connected layer of the lymph node metastasis status predictor to complete the prediction of lymph node metastasis.

Citation Information

Patent Citations

  • Ultrasonic image thyroid nodule intelligent detection system based on Fast-RCNN

    CN111612752A

  • Calculation device and method

    CN111626413A

  • Lymph node metastasis prediction method and device, equipment and storage medium

    CN113920137A