An interpretable ultrasound image nodule recognition method based on a hierarchical network
Through the self-supervised learning method based on hierarchical network, the problem of doctors' subjective dependence and difficulty in identifying etiology in thyroid nodules is solved, and the automated, standardized characteristic characterization and precise grading of thyroid nodules are realized, which enhances the interpretability of the diagnosis, especially the prediction of thyroid cancer and cervical lymph node metastasis.
Patent Information
- Application Number
- CN202210967012.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-11
- Publication Date
- 2025-08-01
- Estimated Expiration
- 2042-08-11
AI Technical Summary
In the diagnosis of thyroid nodules, there is a doctor's subjective experience dependence on doctors, lack of standardization of diagnostic feature descriptors, insufficient and inaccurate diagnostic basis, resulting in large errors in diagnostic results and insufficient interpretability. In the recognition of deep learning models, there are difficulties in identifying data labeling, low lesion recognition accuracy and co-occurrence of causes in medical image recognition.
A hierarchical network-based method is adopted, including a hierarchical model of self-supervised learning, through recursive self-supervised learning subnet and etiology multi-attribute quantization subnet, mining lesions and etiology characteristics from unlabeled ultrasound images, and using puzzle learning strategies and multi-layer perceptron to establish models to obtain etiology multi-attribute quantitative information and prediction results of thyroid nodules.
It improves the automation of thyroid nodules diagnosis, standardization of characteristic characterization and accuracy of grading evaluation, and enhances the interpretability of medical images, especially the disclosure of etiological factors of thyroid cancer and the interpretability of cervical lymph node metastasis auxiliary diagnosis.
Smart Images
Figure CN115187580B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical fields of medical ultrasound image processing, deep learning, artificial intelligence, etc., and particularly relates to an interpretable ultrasound image nodule recognition method based on a hierarchical network. Background Art
[0002] As the preferred imaging examination method for thyroid nodules, ultrasound examination has important value in clinical diagnosis and treatment decisions. In clinical diagnosis, doctors determine descriptors information such as whether the nodule is solitary or multiple, the size of the nodule, the morphological characteristics of the nodule, the composition of the nodule, and the echo quality through 2D ultrasound images, and combine its blood supply characteristics, structural relationships such as the spatial position of the lesion, etc., to achieve the malignant risk grading of thyroid nodules, which is crucial for formulating surgical resection or ablation treatment strategies.
[0003] There are still two key problems in obtaining the descriptor information of thyroid lesions and diagnosing the malignant risk grading of nodules: First, the diagnostic basis for the malignant risk grading of thyroid lesions is limited by the subjective experience of doctors, and the diagnostic descriptors lack standardization. Second, the diagnostic basis for the malignant risk grading is insufficient, inaccurate, and the diagnostic efficiency is low, resulting in a large error in the diagnostic result and insufficient interpretability of the diagnostic basis. Therefore, using deep learning technology to mine features and study the correlation of feature factors from thyroid 2D ultrasound image data, realizing the malignant risk grading of thyroid lesions, and further realizing the interpretable analysis of the diagnostic process can effectively promote the automation of lesion recognition, the standardization of feature representation, and the precision of grading evaluation.
[0004] Deep learning has shown great potential in the field of medical image analysis. However, its big data-driven characteristics and implicit representativeness have significant limitations in medical image recognition. First, due to the time-consuming and laborious nature of fine annotation of a large number of medical images, which requires professional personnel to complete, the annotation of a large amount of high-quality data has become a difficult problem. Second, the similarity of tissue anatomical structures in medical images leads to low accuracy of lesion recognition by the model. Third, the co-occurrence of lesion etiology descriptors in medical images means that doctors only annotate the main cause, which makes it difficult for network models based on multi-label recognition to achieve good results. On the one hand, self-supervised learning provides a new paradigm for mining the feature representations of unlabeled data. In particular, self-supervised learning based on contrastive learning strategies can learn valuable image feature representations from unlabeled data, thereby enhancing the performance of downstream tasks. However, the high similarity of medical images in anatomical structures results in a very small difference distance between images, which limits the learning space of self-supervised methods based on contrastive learning. On the other hand, although multi-label recognition models provide possible solutions for identifying the etiological factors of thyroid nodules, the co-occurrence of etiological factors and the lack of secondary factor labels make it difficult for existing models to adapt to the multi-factor recognition and quantitative characterization of nodule etiology. In addition, interpretable deep learning methods have been developed to explain the lesion recognition in ultrasound images. Unfortunately, almost all of these studies only focus on the visualization of heatmaps to explain the end-to-end prediction results, failing to explain the etiology of disease occurrence. In particular, there is a lack of interpretable auxiliary diagnostic methods for diagnosing thyroid cancer based on investigating etiological factors and even predicting cervical lymph node metastasis. Summary of the Invention
[0005] The purpose of the present invention is to provide a new hierarchical model method based on self-supervised learning to explain ultrasound images, including a perception layer for lesion localization, a cognitive layer for quantifying etiological factors, and a diagnostic layer for interpretability, to reveal the factors leading to thyroid cancer and predict cervical lymph node metastasis. Specifically, a recursive self-supervised learning sub-network and an etiological multi-attribute quantification sub-network are proposed in the present invention to mine the feature representations of lesions and their etiologies from unlabeled ultrasound images, so as to improve the performance of the perception layer and the cognitive layer, and further improve the reliability of interpretable lesion recognition and analysis in medical images.
[0006] To achieve the above purpose, the present invention adopts the following technical solutions:
[0007] An interpretable ultrasound image nodule recognition method based on a hierarchical network, comprising the following steps:
[0008] Step 1: Establish a desensitized unlabeled and labeled thyroid ultrasound image data set, specifically including an unlabeled thyroid ultrasound image data set Data_A, a small sample data set Data_N based on nodule contour segmentation, and a small sample data set Data_F based on multi-attribute recognition of nodule etiology;
[0009] Step 2: Establish a self-supervised learning sub-network layer based on 2D ultrasound feature encoder (RSSL-N): To address the problem of difficult extraction of diverse features due to the anatomical structure similarity of medical images, this method uses a residual network to establish a 2D ultrasound feature encoder, and constructs a self-supervised learning model and objective function using the jigsaw learning strategy to drive the model to learn the feature representations of 2D ultrasound images, and then obtain important feature representations such as anatomical structures, tissue textures, and brightness intensities of 2D ultrasound images from unlabeled medical image data;
[0010] Step 3: Establish a nodule localization and measurement sub-network layer based on 2D ultrasound feature encoder (NMN): Using the 2D ultrasound feature encoder established in Step 2, further establish a nodule segmentation sub-network based on the UNet++ structure through feature upsampling and feature fusion methods to obtain the contour of the thyroid nodule, and use the post-processing layer to obtain the azimuth measurement information of the nodule;
[0011] Step 4: Establish a self-supervised sub-network layer based on nodule ROI feature encoder (RSSL-F): Use the NMN network established in Step 3 to measure unlabeled 2D ultrasound images, obtain the ROI region of the suspicious thyroid nodule as input, use a residual network to establish a nodule ROI feature encoder, and construct a self-supervised learning model and objective function using the jigsaw learning strategy to obtain key feature representations such as the edges, components, and calcifications of the thyroid nodule;
[0012] Step 5: Establish an etiology multi-attribute quantification sub-network layer (FQN): Using the nodule ROI feature encoder established in Step 4, further establish an asymmetric multi-factor recognition loss function based on clinical prior knowledge to alleviate the problems of difficult network learning and low accuracy caused by factor label imbalance, label errors, and losses, and establish an etiology multi-attribute quantification sub-network FQN with a double prediction head through a multi-layer perceptron to obtain the etiology multi-attribute quantification information of the thyroid nodule and its corresponding prediction probability value;
[0013] Step 6: Establish an etiology analysis module based on a multi-layer perceptron: Using the azimuth measurement information and etiology multi-attribute quantification information of the nodule as input, establish an etiology analysis module using a multi-layer perceptron to simultaneously obtain interpretable prediction results for benign and malignant and lymph node metastasis.
[0014] In step 1, the unlabeled dataset Data_A consists of a large number of 2D ultrasound images normalized into png images; the small sample dataset Data_N based on nodule contour segmentation annotates the nodules with contours and benign labels; the small sample dataset Data_F based on multi-attribute identification of nodule etiology annotates the nodule contour, benign or malignant, edge, composition, echo quality and calcification; the three datasets are divided into training set, validation set and test set in a ratio of 7:2:1 respectively.
[0015] Step 2 further includes the following steps:
[0016] Step 2.1, jigsaw puzzle of input data: This method uses jigsaw puzzle preprocessing to preprocess the unlabeled input data to obtain image blocks and their jigsaw puzzle labels with jigsaw puzzle attribution ori and position pos;
[0017] Step 2.2: Establish the RSSL-N self-supervised learning network framework: This method uses a residual convolutional layer network as a 2D ultrasound image feature encoder. On this basis, a puzzle piece attribution prediction head and a position prediction head based on a multi-layer perceptron are established. The self-supervised learning network model can be expressed as:
[0018]
[0019] Where f(*) represents the 2D ultrasound image feature encoder, x ib Indicates that the input puzzle piece comes from the i-th puzzle position of the b-th image, g ori (*) and g pos (*) represent the puzzle piece belonging prediction head and position prediction head, respectively, z ib and A vector representation of each prediction;
[0020] Step 3.3: Establish the loss function L of the self-supervised learning network layer stot :This method uses a puzzle strategy to drive model learning and uses similarity and distance metrics to model the model's loss function, thereby prompting the model to carve out anatomical structures and intensity differences from medical images to improve the model's accuracy in predicting image block attribution and location.
[0021] The image block attribution prediction loss function is expressed as L ori :
[0022]
[0023]
[0024] In the formula, the predicted probability of belonging can be expressed as p ib , cos(z ib ,z0b ) represents the cosine similarity between the image patch and the image vector representation, τ represents the temperature parameter for smoothing or exacerbating the distance, N represents the number of images or batch size, k h and k w represents dividing the h×w image into k×k patches, represents that the attribution prediction is successful as 1 or incorrect as 0;
[0025] The image patch position prediction loss function is denoted as L ori :
[0026]
[0027]
[0028] L stot = αL oir +(1 - α)L pos (6)
[0029] In the formula, the position prediction probability is denoted as represents that the position prediction is successful as 1 or incorrect as 0; α represents the balance factor, N represents the number of images or batch size, k h and k w represents dividing the h×w image into k×k patches;
[0030] Step three further includes the following steps:
[0031] Step 3.1, Input data: The 2D ultrasound png image after normalization preprocessing;
[0032] Step 3.2, Establish the NMN network structure design: Use the 2D ultrasound feature encoder established in step 2.2 as the feature extraction backbone network layer of this model. Further, through feature upsampling and feature fusion methods, and combined with the channel attention mechanism to enhance the output features of the C1 - C5 layers, establish a nodule segmentation sub - network based on the UNet++ structure to obtain the contour of the thyroid nodule, and use the post - processing layer to obtain the azimuth measurement information of the nodule: the quantization values of the contour, size, and aspect ratio;
[0033] Step four further includes the following steps:
[0034] Step 4.1, Input data tiling: Use the NMN network established in step 3.2 to measure the unlabeled 2D ultrasound image, obtain the contour information of the suspected thyroid nodule, and crop its multi - scale rectangular ROI area based on the contour and use it as the input data. Then, perform tiling preprocessing on the input data of the unlabeled suspected nodule ROI area image to obtain the image patches and their tiling attribution ori and position pos tiling labels;
[0035] Step 4.2, Establish an RSSL-F network structure: Based on Step 2.2, use a residual network to establish a nodule ROI feature encoder, and use the loss function in Step 2.3 to learn this network model to obtain a feature representation based on the image of the suspicious nodule ROI area;
[0036] Step five further includes the following steps:
[0037] Step 5.1, Input data: Use the measurement result of the NMN network established in Step 3.2 for the 2D ultrasound image, that is, the suspicious nodule ROI area, as the input data;
[0038] Step 5.2, Establish an FQN network layer structure: Use the nodule ROI feature encoder in Step 4.2 as the feature extraction backbone network layer f(*), and establish a multi-factor attribute prediction head g mul and the benign and malignant prediction head g as a prior constraint bin , to obtain the multi-attribute quantification information of the etiology of thyroid nodules: margin, composition, echo quality, and calcification factor and their corresponding prediction probability values;
[0039] Step 5.3, Establish the loss function L of the FQN layer mtot : Use a positive and negative sample weighting strategy, probability shift, and clinical prior constraint to establish an asymmetric multi-factor recognition loss function based on clinical prior knowledge to alleviate the problems of difficult network learning and low accuracy caused by factor label imbalance, label errors, and losses. Its multi-factor loss function L mul The mathematical model is expressed as:
[0040]
[0041]
[0042]
[0043] In the formula, p i is the multi-factor prediction probability value in this figure, δ represents the sigmoid function, and p ij represents the weighting of the multi-factor prediction probability under the clinical prior constraint j∈C k , represents the sum of the main cause C and the secondary cause K r + and r - represent the adjustment factors for the contributions of positive and negative samples to prompt the network to focus on the learning of difficult samples. p s = max(p ij -Δp, 0), Δp is the shift probability, aiming to ignore simple negative samples and reduce the tolerance rate of mislabeled samples. u ijThe true value label representing the factor, N represents the number of images or batch size; with a similar design concept, the benign and malignant prediction loss function L bin The mathematical model is expressed as:
[0044]
[0045] L mtot = βL mul +(1 - β)L bin (11)
[0046] In the formula, u i is the true value label of benign and malignant, and β represents the constraint adjustment factor;
[0047] Step six further includes the following steps:
[0048] Step 5.1, Input data: The nodule orientation measurement information obtained by measuring the 2D ultrasound image with the NMN network layer established in step 3.2: the quantization values of the contour, size, and aspect ratio; and the multi-factor information of the nodule etiology obtained by measuring the nodule ROI area with the FQN network layer established in step 5.2: the edge, composition, echo quality, and calcification factor and their corresponding predicted probability values are used as the input data for this step;
[0049] Step 5.2, Establish an interpretable prediction layer for thyroid nodules: Use a multi-layer perceptron to establish a benign and malignant prediction layer and a lymph node metastasis prediction layer to achieve interpretable identification of thyroid nodules based on a hierarchical network. Description of the Drawings
[0050] Figure 1 is a schematic diagram of interpretable ultrasound image nodule identification based on a hierarchical network;
[0051] Figure 2 is a schematic diagram of the sub-network layer based on self-supervised learning; among them, Figure 2(a) is the image mosaic strategy, and Figure 2(b) is the schematic diagram of the self-supervised learning sub-network structure;
[0052] Figure 3 is a schematic diagram of the etiology multi-attribute quantization sub-network layer (FQN);
[0053] Figure 4 is a schematic diagram of the measurement results of the nodule by the nodule localization and measurement sub-network layer (NMN) based on the 2D ultrasound feature encoder;
[0054] Figure 5 is a schematic diagram of the identification and quantization results of the nodule combined factors by the etiology multi-attribute quantization sub-network layer (FQN). Detailed Implementation Manner
[0055] This design is further described in detail according to medical image analysis and deep learning methods.
[0056] The specific implementation of the present invention includes two key aspects: training and testing. The training is divided into 4 detailed steps: The first step is RSSL-N model training. The unlabeled dataset Data_A is preprocessed by tiling and then fed into the self-supervised learning network RSSL-N to mine the features of the unlabeled data, so as to obtain the 2D ultrasound image feature representation. The second step is NMN model training and testing. Using the obtained 2D ultrasound image feature representation to enhance the feature extraction ability of the thyroid nodule localization network NMN, and training and fine-tuning NMN on Data_N, thereby improving the localization and contour measurement accuracy of the NMN model for suspicious nodules. Further, NMN is used to test the unlabeled data Data_A to obtain the ROI region of the suspicious nodule. The third step is RSSL-F network training. Based on the suspicious nodule ROI image obtained in the previous step, the RSSL-F network is used to mine and train the nodule features to obtain the feature representation of the nodule ROI image. The fourth step is FQN model training. Using the nodule feature representation obtained in the previous step to further enhance the nodule factor feature extraction ability of the FQN network model, and training and fine-tuning FQN on Data_F to improve the recognition accuracy of FQN for thyroid nodule factors. The testing is divided into 3 detailed steps: The first step is suspicious nodule localization. The thyroid 2D ultrasound image is fed into the trained NMN model for testing to obtain the position information of the suspicious nodule: contour, size, and aspect ratio. The second step is nodule multi-factor recognition. Based on the contour information obtained in the previous step, the ROI region of the suspicious nodule is extracted and tested by the trained FQN model to obtain the recognition results and probabilities of nodule edge, composition, echo quality, and calcification factors. The third step is interpretable qualitative recognition. Using the multi-layer perceptron model to comprehensively predict the nodule localization information obtained in the first step of testing and the multi-factor recognition information obtained in the second step of testing to obtain the benign and malignant results of the nodule and the prediction results of whether there is lymph node metastasis. At the same time, the heat map of multi-factors is obtained by using the maximum class activation mapping to further enhance the interpretability of the method model of the present invention for qualitative prediction of thyroid ultrasound nodules.
[0057] See Figure 1 , the schematic diagram of the interpretable ultrasound image nodule recognition based on the hierarchical network of the present invention, where label (a) is the input layer of thyroid ultrasound image data; (b) is the sub-network layer (RSSL-N) of self-supervised learning based on the 2D ultrasound feature encoder, corresponding to the second step; (c) is the nodule localization and measurement sub-network layer (NMN) based on the 2D ultrasound feature encoder, corresponding to the third step; (d) is the self-supervised sub-network layer (RSSL-F) based on the nodule ROI feature encoder, corresponding to the fourth step; (e) is the etiological multi-attribute quantification sub-network layer (FQN), corresponding to the fifth step; (f) is the interpretable result prediction layer based on the multi-layer perceptron, corresponding to the sixth step.
[0058] Referring to Figure 2, the method for constructing the sub-network layer based on self-supervised learning of the present invention is divided into the following three steps, and each step is as follows:
[0059] Step 1), input data tiling: As shown in Figure 2(a), this method uses tiling preprocessing to preprocess the unlabeled input data to obtain image patches and their corresponding tiling attribution ori and position pos tiling labels;
[0060] Step 2), establishing the RSSL-N self-supervised learning network framework: As shown in Figure 2(b), this method uses the residual convolutional layer network as the 2D ultrasound image feature encoder, and on this basis, establishes a tiling patch attribution prediction head g ori (*) and g pos (*) position prediction head. This self-supervised learning network model can be expressed as:
[0061]
[0062] In the formula, f(*) represents the 2D ultrasound image feature encoder, x ib represents that the input tiling patch comes from the i-th tiling position of the b-th image, g ori (*) and g pos (*) respectively represent the tiling patch attribution prediction head and the position prediction head, z ib and represent the vector representations predicted respectively;
[0063] Step 3), establishing the loss function L of the self-supervised learning network layer stot : This method uses the tiling strategy to drive the model to learn, and uses similarity and distance metrics to model the loss function of the model, so as to prompt the model to carve out the differences in anatomical structure and intensity from medical images, so as to improve the accuracy of the model's prediction of tiling patch attribution and position. Among them, [[ID=З5]]
[0064] The tiling patch attribution prediction loss function is expressed as L ori :
[0065]
[0066]
[0067] In the formula, the attribution prediction probability can be expressed as p ib , cos(z ib ,z 0b ) represents the cosine similarity between the image patch and the image vector representation, τ represents the temperature parameter used to smooth or exacerbate the distance, N represents the number of images or batch size, k h and k wIndicates that the h×w image is divided into k×k blocks, Indicates that the attribution prediction is successful as 1 or wrong as 0;
[0068] The image block position prediction loss function is expressed as L ori :
[0069]
[0070]
[0071] L stot =αL oir +(1-α)L pos (6)
[0072] In the formula, the position prediction probability is expressed as Indicates that the position prediction is successful as 1 or wrong as 0; α is the balance factor, N is the number of images or batch size, k h and k w Indicates dividing the h×w image into k×k blocks;
[0073] See also Figure 3 The method for constructing the etiology multi-attribute quantization sub-network layer (FQN) of the present invention is divided into the following three steps, each of which is detailed as follows:
[0074] Step 1) Input data: Figure 3 As shown, based on the measurement results of the 2D ultrasound image by the NMN network layer, the suspicious nodule ROI area of the 2D ultrasound image is cropped and used as input data;
[0075] Step 2) Establish the network structure of FQN factor identification and quantization layer: Figure 3 As shown in the figure, the nodule ROI feature encoder in the ASSL-N network layer is used as the feature extraction backbone network layer f(*), and the multi-factor attribute prediction head g is established through the multi-layer perceptron mul and the benign and malignant prediction head g as a priori constraint bin , to obtain the quantitative information of multiple attributes of the etiology of thyroid nodules: edge, composition, echo quality and calcification factor and their corresponding predicted probability values;
[0076] Step 3) Establish the loss function L of the FQN factor identification and quantization layer mtot :like Figure 3 As shown in the figure, the positive and negative sample weighting strategy, probability offset and clinical prior constraints are used to establish an asymmetric multi-factor recognition loss function based on clinical prior knowledge to alleviate the problem of factor label imbalance, label error and loss, which leads to difficult network learning and low accuracy. The multi-factor loss function L mul The mathematical model is expressed as:
[0077]
[0078]
[0079]
[0080] In the formula, p i is the multi-factor prediction probability value in this figure, δ represents the sigmoid function, and p ij represents the weighting of the multi-factor prediction probability under the clinical prior constraint j∈C k and represents the sum of the main cause C and the secondary cause K r + and r - represent the adjustment factors for the contributions of positive and negative samples to prompt the network to focus on the learning of difficult samples. p s = max(p ij -Δp, 0), where Δp is the shift probability, aiming to ignore simple negative samples and reduce the tolerance rate of mislabels. u ij represents the true value label of the factor, and N represents the number of images or batch size; with a similar design concept, the loss function L[[ID=ORIGINAL=31]] bin of the benign and malignant prediction is mathematically expressed as:
[0081]
[0082] L mtot = βL mul + (1 - β)L bin (11)
[0083] In the formula, u i [[ID=ORIGINAL=49]]is the true value label of benign and malignant, and β represents the constraint adjustment factor.
[0084] See Figure 4 , for the azimuth test results of thyroid nodules on the thyroid ultrasound image dataset of the present invention, which mainly include the immersion depth, width, height, and transverse-longitudinal ratio of the nodules, and the prediction contour of the nodules is visualized.
[0085] See Figure 5 , for the test results of the etiological factors of thyroid nodules on the thyroid ultrasound image dataset of the present invention, which mainly include the edges, components, echo quality, and calcification factors of the nodules and their corresponding prediction probability values, and the malignancy and lymph node metastasis of the nodules are predicted. Finally, the heat maps corresponding to each etiological factor are visualized to further enhance the interpretability analysis of thyroid nodule recognition.
[0086] In the present invention, an interpretable ultrasonic image nodule recognition method based on a hierarchical network is provided, and Python and PyTorch are used to model and experimentally verify this interpretable recognition method. From Figures 4 - 5 the test results, it can be seen that this method measures the azimuth of thyroid nodules in ultrasonic images through a hierarchical mode, obtains the quantification values of the contour, size and aspect ratio; measures the nodule ROI region, and obtains the multi-factor information of the nodule etiology: edge, composition, echo quality and calcification factor and their corresponding prediction probability values; based on this quantification value, further predicts the benign and malignant nature and lymph node metastasis of thyroid nodules, so as to realize the interpretable recognition of thyroid nodules based on a hierarchical network.
[0087] The above embodiments are specific embodiments of the present invention. It should be clear that the implementation manners of the present invention are not limited to the above embodiments, and any modifications, equivalent replacements, improvements, etc. made subsequently to the embodiments should be included in the protection scope of the present invention.
Claims
1. An interpretable ultrasound image nodule recognition method based on a hierarchical network, characterized in that, It includes the following steps: Step 1: Establish a desensitized unlabeled and labeled thyroid ultrasound image dataset, specifically including an unlabeled thyroid ultrasound image dataset Data_A, a small-sample dataset Data_N based on nodule contour segmentation, and a small-sample dataset Data_F based on multi-attribute recognition of nodule etiology; Step 2: Establish a self-supervised learning sub-network layer RSSL-N based on a 2D ultrasound feature encoder: To solve the problem of difficult extraction of diverse features due to the anatomical structure similarity of medical images, this method uses a residual network to establish a 2D ultrasound feature encoder, and constructs a self-supervised learning model and objective function using the jigsaw learning strategy to drive the model to learn the feature representations of 2D ultrasound images, and then obtain important feature representations of the anatomical structure, tissue texture, and light and dark intensity of 2D ultrasound images from unlabeled medical image data; It further includes the following steps: Step 2.1: Input data jigsawing: This method uses jigsaw preprocessing to preprocess unlabeled input data to obtain image patches and their jigsaw labels with jigsaw attribution ori and position pos; Step 2.2: Establish the RSSL-N self-supervised learning network framework: This method uses a residual convolutional layer network as the 2D ultrasound image feature encoder, and on this basis, establishes a jigsaw patch attribution prediction head and a position prediction head based on a multi-layer perceptron; This self-supervised learning network framework can be expressed as: where f(*) represents the 2D ultrasound image feature encoder, and x ib indicates that the input patch is from the i-th patch position of the b-th image, and g ori (*) and g pos (*) represent the patch attribution prediction head and the position prediction head respectively, and z ib and represent the vector representations predicted respectively; Step 2.
3. Establish the loss function L of the self-supervised learning network layer stot : This method uses a jigsaw puzzle strategy to drive the model to learn, and models the loss function of the model using similarity and distance metrics, so as to prompt the model to carve out the differences in anatomical structures and intensities from medical images, in order to improve the accuracy of the model's prediction of the attribution and position of image patches. Among them, The prediction loss function for image block attribution is denoted as L ori : where the attribution prediction probability can be expressed as p ib , cos(z ib , z 0b ) represents the cosine similarity between the image patch and the image vector representation, τ represents the temperature parameter for smoothing or exacerbating the distance, N represents the number of images or batch size, k h and k w represent dividing the h×w image into k×k patches, represents that the attribution prediction is successful as 1 or incorrect as 0; The loss function for predicting the position of the image block is expressed as L ori : L stot = αL oir + (1 - α)L pos (6) In the formula, the position prediction probability is expressed as indicating that the position prediction is successful as 1 or incorrect as 0; α represents the balance factor, N represents the number of images or batch size, and k h and k w indicating that an h×w image is divided into k×k blocks; Step 3: Establish a nodule localization and measurement sub-network layer NMN based on a 2D ultrasound feature encoder: Utilize the 2D ultrasound feature encoder established in Step 2, and further establish a nodule segmentation sub-network based on the UNet++ structure through feature upsampling and feature fusion methods to obtain the contour of the thyroid nodule, and use the post-processing layer to obtain the azimuth measurement information of the nodule; Step 4: Establish a self-supervised learning sub-network layer RSSL-F based on a nodule ROI feature encoder: Use the NMN network established in Step 3 to measure unlabeled 2D ultrasound images, obtain the ROI region of the suspicious thyroid nodule as input, use a residual network to establish a nodule ROI feature encoder, and construct a self-supervised learning model and objective function with the jigsaw learning strategy to obtain key feature representations of the edge, composition, and calcification of the thyroid nodule; Step 5: Establish an etiology multi-attribute quantification sub-network layer FQN: Use the nodule ROI feature encoder established in Step 4 to further establish an asymmetric multi-factor recognition loss function based on clinical prior knowledge to alleviate the problems of difficult network learning and low accuracy caused by factor label imbalance, label errors, and losses, and establish an etiology multi-attribute quantification sub-network FQN with a dual prediction head through a multi-layer perceptron to obtain the etiology multi-attribute quantification information of the thyroid nodule and its corresponding prediction probability value; It further includes the following steps: Step 5.1: Input data: Use the measurement result of the NMN network established in Step 3.2 for 2D ultrasound images, that is, the ROI region of the suspicious nodule, as the input data; Step 5.
2. Establish the FQN network layer structure: Use the nodular ROI feature encoder in Step 4.2 as the feature extraction backbone network layer f(*), and establish a multi-factor attribute prediction head g through a multi-layer perceptron mul and the benign and malignant prediction head g as a prior constraint bin , so as to obtain the multi-attribute quantification information of the etiology of thyroid nodules: margin, composition, echo quality, and calcification factor and their corresponding predicted probability values; Step 5.3: Establish the loss function L of the FQN layer mtot : Using the positive and negative sample weighting strategy, probability offset, and clinical prior constraint, establish an asymmetric multi-factor recognition loss function based on clinical prior knowledge to alleviate the problems of difficult network learning and low accuracy caused by factor label imbalance, label errors, and losses. Its multi-factor loss function L mul The mathematical model is expressed as: Where p i is the multi-factor prediction probability value, δ represents the sigmoid function, and p ij represents the weighting of the multi-factor prediction probability under the clinical prior constraint j ∈ C k , which is expressed as the sum of the main cause C and the secondary cause K r + and r - represent the adjustment factors for the contributions of positive and negative samples to prompt the network to focus on the learning of difficult samples. p s = max(p ij - Δp, 0), where Δp is the shift probability, aiming to ignore simple negative samples and reduce the tolerance rate of mislabeled samples. u ij represents the true label of the factor, and N represents the number of images or the batch size. With a similar design concept, the benign and malignant prediction loss function L bin is mathematically modeled as: L mtot = βL mul +(1 - β)L bin (11) where \(u\) i is the true label of benign and malignant, and \(\beta\) represents the constraint adjustment factor; Step 6: Establish an interpretable result prediction layer based on a multi-layer perceptron: Using the azimuth measurement information of the nodules and the multi-attribute quantification information of the etiology as inputs, establish an etiology analysis module using a multi-layer perceptron to simultaneously obtain interpretable prediction results for benign / malignant and lymph node metastasis.
2. The interpretable ultrasound image nodule recognition method based on a hierarchical network according to claim 1, wherein: In Step 1, the unlabeled dataset Data_A consists of a large number of 2D ultrasound images normalized into png images; in the small-sample dataset Data_N based on nodule contour segmentation, the contours and benign / malignant pathology of the nodules are labeled; in the small-sample dataset Data_F based on multi-attribute identification of nodule etiology, the contours, benign / malignancy, margins, components, echo quality, and calcifications of the nodules are further labeled; the three datasets are divided into training sets, validation sets, and test sets according to the ratio of 7:2:1 respectively.
3. The interpretable ultrasound image nodule recognition method based on a hierarchical network according to claim 1, wherein: Step 3 further includes the following steps: Step 3.1, Input data: 2D ultrasound png images after normalization preprocessing; Step 3.2, Establish the NMN network structure design: Use the 2D ultrasound feature encoder established in Step 2.2 as the feature extraction backbone network layer of this model. Further, through feature upsampling and feature fusion methods, and combined with a channel attention mechanism to enhance the output features of layers C1 - C5, establish a nodule segmentation sub-network based on the UNet++ structure to obtain the contours of thyroid nodules, and use the post-processing layer to obtain the azimuth measurement information of the nodules: the quantification values of the contours, sizes, and aspect ratios.
4. The interpretable ultrasound image nodule recognition method based on a hierarchical network according to claim 1, characterized in that: Step 4 further includes the following steps: Step 4.1, Input data tiling: Use the NMN network established in Step 3.2 to measure the unlabeled 2D ultrasound images to obtain the contour information of the suspicious thyroid nodules. Based on the contours, perform multi-scale rectangular ROI region cropping and use it as the input data. Then, perform tiling preprocessing on the input data of the unlabeled suspicious nodule ROI region images to obtain image patches and their tiling attribution ori and position pos tiling labels; Step 4.2, Establish the RSSL-F network structure: Based on Step 2.2 and use a residual network to establish a nodule ROI feature encoder, and use the loss function in Step 2.3 to learn the self-supervised learning network framework to obtain the feature representation based on the suspicious nodule ROI region images.
5. The method for identifying nodules in an interpretable ultrasound image based on a hierarchical network according to claim 1, characterized in that: Step 6 further includes the following steps: Step 5.1, Input data: The quantification values of the azimuth measurement information of the nodules obtained by measuring the 2D ultrasound images using the NMN network layer established in Step 3.2: the contours, sizes, and aspect ratios; and the multi-factor information of nodule etiology obtained by measuring the nodule ROI region using the FQN network layer established in Step 5.2: the margin, component, echo quality, and calcification factors and their corresponding prediction probability values are used as the input data for this step; Step 5.2, Establish an interpretable prediction layer: Use a multi-layer perceptron to establish a benign / malignant prediction layer and a lymph node metastasis prediction layer to achieve interpretable identification of thyroid nodules based on a hierarchical network.