Dynamic Image Thyroid Cancer Risk Stratification Prediction Method Based on Graph Convolutional Network

Abstract

The present invention proposes a method for predicting the risk stratification of thyroid cancer in dynamic images based on a graph convolutional network, which includes the following steps: Step 1: Use image filtering enhancement and a super-resolution network to preprocess the ultrasound images, design an algorithm based on a graph convolutional neural network structure for thyroid ultrasound image segmentation, and establish a corresponding network model for training; Step 2: Use a feature extraction method of dynamic ultrasound radiomics based on artificial intelligence technology to extract features from the dynamic video sequence, and introduce a deep learning method to fuse the dynamic ultrasound radiomics to make the extracted radiomics features more abundant; Step 3: Use a multi-branch neural network model to comprehensively integrate various different modalities of data, and calculate multi-modal biomarkers by mining the complementary information of different modalities of data, so as to establish a risk stratification prediction model of thyroid cancer that can guide clinical decisions; The present invention can improve the accuracy and precision of thyroid cancer risk stratification.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a dynamic image thyroid cancer risk stratification prediction method based on a graph convolutional network. Background Art

[0002] Thyroid cancer is one of the most common malignant tumors of the head, neck, and endocrine system. In recent years, the diagnosis and treatment of thyroid cancer have been subject to ongoing controversy. Some scholars believe that it is overdiagnosed and treated, while others believe that the aggressiveness of thyroid cancer should not be underestimated.

[0003] Pathological type (subtype) and molecular biological characteristics are important indicators for assessing the aggressiveness and prognosis of thyroid cancer. Treatment and prognosis vary significantly for thyroid cancers of varying degrees of malignancy. Therefore, effective preoperative screening of thyroid cancer patients and accurate stratification of their aggressiveness and prognosis are important clinical and scientific issues that need to be addressed.

[0004] Clinically, preoperative risk stratification of thyroid nodules primarily involves imaging, cytology, and genetic testing. Ultrasound is currently the preferred imaging modality for thyroid nodule screening and plays a crucial role in clinical decision-making. Various countries and organizations have developed different versions of the TIRADS risk stratification system (e.g., ACR-TIRADS, C-TIRADS, ATA, and AACE / ACE / AME guidelines). These systems define methods for assessing different levels of malignant risk in thyroid nodules and their corresponding management strategies. However, these systems do not further assess the malignancy of thyroid cancer. Currently, when suspicious nodules are identified by thyroid ultrasound, ultrasound features such as lesion size, presence of capsule invasion, and / or cervical lymph node metastasis can only be used to roughly assess the risk of malignancy and invasiveness. However, the pathological type and subtype cannot be effectively predicted. Therefore, conventional ultrasound has certain limitations in assessing the malignancy of thyroid nodules.

[0005] Currently, neither ultrasound examination, cytopathological diagnosis nor genetic testing can provide a comprehensive and accurate assessment of the risk stratification of thyroid cancer, as they can only reflect some of the characteristics of thyroid cancer.

[0006] In recent years, the deep integration of artificial intelligence and medicine has played an important role in the intelligentization and precision of medical decision-making. The radiomics technology based on artificial intelligence can capture all the characteristics of tumors through non-invasive means, which is more representative than traditional puncture biopsy methods. However, in the ultrasonic diagnosis of thyroid diseases, the application of artificial intelligence mainly focuses on the differential diagnosis of benign and malignant lesions. Although relevant studies have shown high diagnostic efficacy, most of them adopt the method of "radiomics (static images) + machine learning or deep learning", and the method of "radiomics (dynamic images) + deep learning" is less used. In addition, most of the studies are retrospective, and more prospective studies are needed to verify the practicability and effectiveness of these methods. Therefore, the present invention proposes a new method for predicting the risk stratification of thyroid cancer based on dynamic images of graph convolutional networks.

[0007] The present invention aims to conduct research using multimodal data, fuse the characteristics of multimodal data such as ultrasound, pathology (including gene detection), and other clinical data, and apply some innovative artificial intelligence technologies to achieve intelligent and precise stratification prediction of thyroid cancer. Through the technical solution of the present invention, the accuracy and precision of thyroid cancer risk stratification can be improved, providing more powerful support and guidance for clinical treatment. Summary of the Invention

[0008] The present invention proposes a method for predicting the risk stratification of thyroid cancer based on dynamic images of graph convolutional networks. Through the automatic recognition and analysis of clinical imaging data, the accuracy and precision of thyroid cancer risk stratification can be improved, providing more powerful support and guidance for clinical treatment.

[0009] The present invention adopts the following technical solutions.

[0010] A method for predicting the risk stratification of thyroid cancer based on dynamic images of graph convolutional networks, which is used for the analysis of ultrasonic examination image results for thyroid nodule screening, is characterized in that the prediction method comprises the following steps;

[0011] Step 1: Use image filtering enhancement and super-resolution networks to preprocess ultrasonic images, design an algorithm based on the graph convolutional neural network structure for thyroid ultrasonic image segmentation, and establish a corresponding network model for training;

[0012] Step 2: Use a feature extraction method of dynamic ultrasonic radiomics based on artificial intelligence technology to extract features from dynamic video sequences, and introduce deep learning methods to fuse dynamic ultrasonic radiomics to make the extracted radiomics features more abundant;

[0013] Step 3: Use a multi-branch neural network model to integrate various different modalities of data, calculate multi-modal biomarkers by mining complementary information from different modalities of data, and thus establish a thyroid cancer risk stratification prediction model that can guide clinical decisions.

[0014] The algorithm based on the graph convolutional neural network structure for thyroid ultrasound image segmentation in Step 1 adopts a graph convolutional neural network segmentation framework constrained by anatomical prior information, that is, it uses the spatial correlation of the thyroid structure in the ultrasound image to extract semantic feature representations, and obtains the structural dependence between the thyroid and adjacent tissues by introducing prior knowledge of the thyroid, thereby improving the segmentation performance of the network and the accuracy of thyroid lesion segmentation.

[0015] When training the segmentation stage of the network model, it includes the following steps;

[0016] Step A1: Train the backbone network without the graph convolutional module;

[0017] Step A2: Use the training weights of the backbone network as pre-training weights and integrate them with the graph convolutional module to train the network together;

[0018] In the training of the graph convolutional neural network, first obtain the anatomical prior information of the thyroid to learn the structure and its characteristics of the thyroid; then extract and model this anatomical prior information, and design a loss function according to parameters such as shape and position. Then integrate this information into the graph convolutional neural network model to design a graph convolutional neural network segmentation model with anatomical feature constraints.

[0019] By using graph convolution in thyroid image segmentation to improve the segmentation performance of the thyroid nodule part in the image, the specific method is as follows:

[0020] Step B1: Use the output of the last layer of the convolutional neural network as the input of the graph convolutional node, construct graph nodes, and use the similarity of the nodes to construct the edges of the graph, and input them into the graph convolutional neural network together;

[0021] Step B2: By calculating the loss and backpropagating the gradient, iterate the network parameters to complete the training of the graph convolution-based segmentation network model, so that the form of graph convolution can expand the receptive field of the segmentation network, effectively combine the global and local information of the image to avoid the loss of local position information;

[0022] Step B3: Combine the graph convolutional neural network with multiple other convolutional neural network structures respectively, integrate the graph convolutional neural network designed after combining other convolutional neural network structures into the network for comparison, and further adjust and improve the core parameters of the graph convolutional neural network according to the comparison results to improve the performance of thyroid nodule segmentation; the multiple other convolutional neural network structures include U-net, FCN, and SegNet.

[0023] In step 2, a dynamic video sequence is used for ultrasonic radiomics feature extraction to obtain the full ultrasonic signs of the lesion and the adjacent relationship between the lesion and the surrounding tissue structures.

[0024] The feature extraction of the dynamic video sequence in step 2 is the feature extraction of the dynamic image. Specifically, when extracting the features of the dynamic image, based on the encoding-decoding structure of the Swin Transformer network, a dual-path Transformer structure is used to ensure temporal consistency so as to extract the dynamic image features; the pixel-level features are decoded into the object-level representation of each frame. Before the backbone features enter the Transformer decoder, a linear embedding layer is applied to map it from the backbone dimension to the decoder hidden dimension; then its spatial and temporal dimensions are flattened so as to feed it into the Transformer decoder; the video sequence feature extraction includes four stages, and in each stage, the spatial dimension of the video sequence is downsampled to achieve feature extraction.

[0025] In step 3, multiple different modalities of data include dynamic images, case type data, and genotype data; the input data of the dynamic image, case type data, and genotype data, three different modalities of data, are input into the classification network; a global average pooling GAP operation is performed, and after passing through the fully connected FC layer, a hierarchical prediction result of thyroid cancer is given, reducing the doctor's film reading time and diagnostic difficulty.

[0026] In step 3, a multi-modal deep neural network framework is used to calculate multi-modal biomarker. By adding branches to the neural network structure, it can accept different modalities of data input; and a loss function is designed to fully mine the non-linear complementary information between multi-modal data. In the loss function, the optimization direction of the non-linear function is driven by the four-classification accuracy rate to better mine the complementary information of multi-modal data, so as to improve the diagnostic accuracy.

[0027] The prediction method uses a deep convolutional network to extract the ultrasonic radiomics image features related to the dynamic image, pathological type, and molecular features of thyroid cancer respectively, integrates the information of the three, performs effective fusion, and inputs them into the multi-branch neural network model in a parallel manner, integrates the feature information for end-to-end learning, and makes a more comprehensive and objective comprehensive judgment.

[0028] The present invention can analyze biological data more carefully, can also better understand biological processes and regulatory effects, and promotes the further development of the biomedical field.

[0029] Through the automatic recognition and analysis of clinical imaging data, the present invention can improve the accuracy and precision of thyroid cancer risk stratification, and provide more powerful support and guidance for clinical treatment.

[0030] The present invention combines multi-modal data such as ultrasonic dynamic images, pathology, and other clinically relevant data, as well as some innovative intelligent technologies for research, enabling the complementary advantages of different types of information to achieve intelligent and accurate prediction of the risk stratification of thyroid cancer.

[0031] The present invention introduces a graph convolution module into the convolutional neural network architecture in the segmentation stage, and utilizes the spatial correlation of the thyroid structure in the ultrasonic image to extract semantic feature representations. By introducing prior knowledge of the thyroid to obtain the structural dependence between the thyroid and adjacent tissues, the segmentation performance of the network is improved, and the accuracy of thyroid lesion segmentation is enhanced.

[0032] The present invention uses dynamic video sequences for feature extraction of ultrasonic radiomics, which can obtain the overall ultrasonic signs of the lesion and the adjacent relationship between the lesion and the surrounding tissue structure. While comprehensively scanning the nodule, a multi-task comprehensive judgment is made based on more diverse data forms, and the risk stratification level of thyroid cancer is predicted more accurately, serving as the basis for clinical decision-making.

[0033] The present invention uses a deep convolutional network to separately extract ultrasonic radiomics image features related to the dynamic image, pathological type, and molecular features of thyroid cancer, integrates the information of the three, performs effective fusion, and inputs them into the network in a parallel manner. By integrating feature information for end-to-end learning, a more comprehensive and objective comprehensive judgment can be made.

[0034] The discrimination of the risk level of thyroid cancer plays a crucial role in the diagnosis and treatment of thyroid diseases. The present invention fills the gap in the discrimination of the cancer risk level in the field of thyroid image processing by current deep learning algorithms, laying a foundation for the application of artificial intelligence technology in the field of thyroid cancer assistance. For some local hospitals with a relatively shortage of doctors, it can assist doctors in diagnosis, speed up the doctor's diagnosis speed, quickly discriminate the risk level of thyroid cancer, greatly shortening the time and energy spent by doctors in discriminating the risk level of thyroid cancer, and effectively assisting doctors in the diagnosis of the condition and the planning and analysis of the prognosis. It has wide application value in the processes of intraoperative, diagnostic, and postoperative rehabilitation of the thyroid. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The following further details the present invention in conjunction with the drawings and specific embodiments:

[0036] Attached Figure 1 is a schematic flow chart of the present invention;

[0037] Attached Figure 2 is a schematic diagram of a graph convolution neural network segmentation framework constructed with anatomical prior information constraints;

[0038] Attached Figure 3 is a schematic diagram of dynamic image feature extraction;

[0039] Attached Figure 4 is a schematic diagram of biomarker analysis based on multimodal data fusion. Specific implementation manners

[0040] As shown in the figure, a dynamic image thyroid cancer risk stratification prediction method based on a graph convolutional network is used for analyzing the ultrasound examination image results for thyroid nodule screening, and is characterized in that: the prediction method comprises the following steps;

[0041] Step 1, preprocess the ultrasound image by using image filtering enhancement and a super-resolution network, design an algorithm based on a graph convolutional neural network structure for thyroid ultrasound image segmentation, and establish a corresponding network model for training;

[0042] Step 2, extract features from a dynamic video sequence by using a feature extraction method of dynamic ultrasound radiomics based on artificial intelligence technology, and introduce a deep learning method to fuse the dynamic ultrasound radiomics to make the extracted radiomics features richer;

[0043] Step 3, utilize a multi-branch neural network model to comprehensively integrate various data of different modalities, calculate multimodal biomarkers by mining complementary information of different modality data, and thereby establish a thyroid cancer risk stratification prediction model that can guide clinical decisions.

[0044] The algorithm based on the graph convolutional neural network structure for thyroid ultrasound image segmentation in the above step 1 adopts a graph convolutional neural network segmentation framework constrained by anatomical prior information, that is, utilizes the spatial correlation of the thyroid structure in the ultrasound image to extract semantic feature representations, and obtains the structural dependence relationship between the thyroid and adjacent tissues by introducing prior knowledge of the thyroid, thereby improving the segmentation performance of the network and the accuracy of thyroid lesion segmentation;

[0045] When training the segmentation stage of the network model, it includes the following steps;

[0046] Step A1, train the backbone network without a graph convolutional module;

[0047] Step A2, use the training weights of the backbone network as pre-training weights, and incorporate a graph convolutional module to train the network together;

[0048] In the training of the graph convolutional neural network, first obtain the anatomical prior information of the thyroid to learn the structure and its features of the thyroid; then extract and model these anatomical prior information, and design a loss function according to parameters such as shape and position. Then integrate this information into the graph convolutional neural network model to design a graph convolutional neural network segmentation model constrained by anatomical features.

[0049] Improve the segmentation performance of the thyroid nodule part in the image by using graph convolution in thyroid image segmentation. The specific method is as follows:

[0050] Step B1: Use the output of the last layer of the convolutional neural network as the input of the graph convolution node, construct graph nodes, and construct the edges of the graph using the similarity of the nodes, and input them into the graph convolutional neural network together;

[0051] Step B2: By calculating the loss and backpropagating the gradient, iterate the network parameters to complete the training of the graph convolution segmentation network model, so that the form of graph convolution can expand the receptive field of the segmentation network, effectively combine the global and local information of the image to avoid the loss of local position information;

[0052] Step B3: Combine the graph convolutional neural network with multiple other convolutional neural network structures respectively, integrate the designed graph convolutional neural network after combining other convolutional neural network structures into the network for comparison, and further adjust and improve the core parameters of the graph convolutional neural network according to the comparison results to improve the performance of thyroid nodule segmentation; the multiple other convolutional neural network structures include U-net, FCN, and SegNet.

[0053] In the second step, a dynamic video sequence is used for feature extraction of ultrasound radiomics to obtain the overall ultrasound signs of the lesion and the adjacent relationship between the lesion and the surrounding tissue structure;

[0054] The feature extraction of the dynamic video sequence in the second step is the feature extraction of the dynamic image. Specifically, when extracting the features of the dynamic image, based on the encoding-decoding structure of the Swin Transformer network, a dual-path Transformer structure is used to ensure temporal consistency to extract the dynamic image features; decode the pixel-level features into the object-level representation of each frame, and apply a linear embedding layer before the backbone features enter the Transformer decoder to map it from the backbone dimension to the decoder hidden dimension; then flatten its spatial and temporal dimensions so that it can be fed into the Transformer decoder; the video sequence feature extraction includes four stages, and the spatial dimension of the video sequence is downsampled in each stage to achieve feature extraction.

[0055] In the third step, multiple different modalities of data include dynamic images, case type data, and genotype data; input the input data of the dynamic image, case type data, and genotype data of three different modalities into the classification network; perform the global average pooling GAP operation, and give the hierarchical prediction results of thyroid cancer after passing through the fully connected FC layer, reducing the doctor's reading time and diagnostic difficulty.

[0056] In step 3, a multi-modal deep neural network framework is used to calculate multi-modal biomarkers. By adding branches to the neural network structure, it can accept data inputs of different modalities. A loss function is designed to fully exploit the non-linear complementary information between multi-modal data. In the loss function, the optimization direction of the non-linear function is driven by the four-class accuracy rate to better mine the complementary information of multi-modal data, so as to improve the diagnostic accuracy rate.

[0057] The prediction method uses a deep convolutional network to extract ultrasound omics image features related to dynamic images, pathological types, and molecular features of thyroid cancer respectively, integrates the information of the three, performs effective fusion, and inputs it into a multi-branch neural network model in a parallel manner. Integrate the feature information for end-to-end learning to make a more comprehensive and objective comprehensive judgment.

[0058] Example:

[0059] In this example, first, image filtering enhancement and super-resolution network are used to preprocess the input image. Improve the quality and resolution of the input image to assist subsequent convolutional neural network for segmentation and classification processing.

[0060] In the segmentation stage, two steps are required for training. The first step is to train the backbone network without the graph convolution module. The second step is to use the training weights of the backbone network as pre-trained weights and integrate the graph convolution module to train the network together. Due to the characteristics of the graph convolution module, if the entire network is trained together, the network model cannot converge well. Therefore, training the network in two steps can ensure that the graph convolution module obtains more significant feature maps.

[0061] In the graph convolution network, we first need to obtain the anatomical prior information of the thyroid gland to learn the structure and its characteristics of the thyroid gland. Then extract this prior information and model it, and design a loss function according to parameters such as shape and position. Then integrate this information into the graph convolution neural network model to design a graph convolution neural network segmentation model with anatomical feature constraints.

[0062] The segmented images of the attention regions with high information content are input into the classification network for hierarchical prediction. In the final prediction network, dynamic video sequences are used for feature extraction of ultrasound imaging omics. In the classification network, three different modalities of data, namely dynamic input data, case type data, and genotype data, can be input, which can more accurately assist in predicting the risk level of thyroid cancer.

[0063] In the preprocessing stage, we resize the dynamic image to 224×224. In the training and validation stages, the Adam optimizer is used, with the initial learning rate set to 0.1 and the weight decay set to 0.001. To avoid overfitting during training, we terminate the training process after 200 epochs. It is recommended to perform classification prediction on an Ubuntu system with a GPU above Nvidia RTX 2080TI to speed up the prediction. The overall process is divided into three steps. First, preprocess the image to resize it to a size suitable for the network. Second, perform segmentation using a segmentation network incorporating a graph convolution module. Finally, input three different modalities of data, namely dynamic input data, case type data, and genotype data, into the classification network. Finally, perform global average pooling (GAP) operation, and after passing through the fully connected (FC) layer, give the hierarchical prediction results of thyroid cancer, automatically assisting doctors and greatly reducing the doctors' reading time and diagnostic difficulty.

[0064] The above are the preferred embodiments of the present invention. All changes made according to the technical solution of the present invention, when the functions and effects produced do not exceed the scope of the technical solution of the present invention, shall fall within the protection scope of the present invention.

Claims

1. A dynamic image thyroid cancer risk stratification prediction method based on a graph convolutional network, which is used for analyzing the ultrasound examination image results of thyroid nodule screening, and is characterized in that: The prediction method comprises the following steps; Step 1: Use image filtering enhancement and super-resolution network to preprocess the ultrasound image, design an algorithm based on the graph convolutional neural network structure for thyroid ultrasound image segmentation, and establish a corresponding network model for training; Step 2: Use the feature extraction method of dynamic ultrasound radiomics based on artificial intelligence technology to extract features from the dynamic video sequence, and introduce deep learning methods to fuse the dynamic ultrasound radiomics to make the extracted radiomics features more abundant; Step 3: Use a multi-branch neural network model to comprehensively integrate various data of different modalities, calculate multi-modal biomarkers by mining the complementary information of different modality data, and thus establish a thyroid cancer risk stratification prediction model that can guide clinical decisions; The algorithm based on the graph convolutional neural network structure for thyroid ultrasound image segmentation in Step 1 adopts a graph convolutional neural network segmentation framework constrained by anatomical prior information, that is, uses the spatial correlation of the thyroid structure in the ultrasound image to extract semantic feature representations, and obtains the structural dependence between the thyroid and adjacent tissues by introducing prior knowledge of the thyroid, thereby improving the segmentation performance of the network and the accuracy of thyroid lesion segmentation; When training the segmentation stage of the network model, it includes the following steps; Step A1: Train the backbone network without the graph convolutional module; Step A2: Use the training weights of the backbone network as pre-training weights and integrate them into the graph convolutional module to train the network together; In the training of the graph convolutional neural network, first obtain the anatomical prior information of the thyroid to learn the structure and its features of the thyroid; then extract and model these anatomical prior information, and design a loss function according to the shape and position parameters; Then combine this information and integrate it into the graph convolutional neural network model to design a graph convolutional neural network segmentation model with anatomical feature constraints; Improve the segmentation performance of the thyroid nodule part in the image by using graph convolution in thyroid image segmentation. The specific method is: Step B1: Use the output of the last layer of the convolutional neural network as the input of the graph convolutional node, construct graph nodes, and use the similarity of the nodes to construct the edges of the graph, and input them into the graph convolutional neural network together; Step B2: Iterate the network parameters by calculating the loss and backpropagating the gradient, complete the training of the graph convolution-based segmentation network model, so that the form of graph convolution can expand the receptive field of the segmentation network, effectively combine the global and local information of the image to avoid the loss of local position information; Step B3: Combine the graph convolutional neural network with multiple other convolutional neural network structures respectively, integrate the graph convolutional neural network designed after combining other convolutional neural network structures into the network for comparison, and further adjust and improve the core parameters of the graph convolutional neural network according to the comparison results to improve the performance of thyroid nodule segmentation; the multiple other convolutional neural network structures include U-net, FCN, and SegNet; In Step 2, use the dynamic video sequence to extract the features of ultrasound radiomics to obtain the overall ultrasound signs of the lesion and the adjacent relationship between the lesion and the surrounding tissue structures; Step 2: Feature extraction of the dynamic video sequence is the feature extraction of the dynamic image, specifically: When extracting features of the dynamic image, based on the encoding-decoding structure of the Swin Transformer network, a dual-path Transformer structure is used to ensure temporal consistency so as to extract dynamic image features; the pixel-level features are decoded into the object-level representation of each frame. Before the backbone features enter the Transformer decoder, a linear embedding layer is applied to map it from the backbone dimension to the decoder hidden dimension; then its spatial and temporal dimensions are flattened so as to feed it into the Transformer decoder; the video sequence feature extraction includes four stages, and in each stage, the spatial dimension of the video sequence is downsampled to achieve feature extraction. In Step 3, multiple different modalities of data include dynamic images, case type data, and genotype data; the input data of the dynamic image, case type data, and genotype data, these three different modalities of data, are input into the classification network. The global average pooling (GAP) operation is performed, and after passing through the fully connected (FC) layer, the hierarchical prediction results of thyroid cancer are given, reducing the doctors' reading time and diagnostic difficulty.

2. The method for predicting the risk stratification of dynamic image thyroid cancer based on the graph convolutional network according to claim 1, wherein: In Step 3, a multi-modal deep neural network framework is used to calculate multi-modal biomarkers. By adding branches to the neural network structure, it can accept data inputs of different modalities; and a loss function is designed to fully exploit the non-linear complementary information between multi-modal data. In the loss function, the optimization direction of the non-linear function is driven by the four-class classification accuracy to better mine the complementary information of multi-modal data, so as to improve the diagnostic accuracy.

3. The dynamic image thyroid cancer risk stratification prediction method based on a graph convolutional network according to claim 1, characterized in that: The prediction method uses a deep convolutional network to extract ultrasound omics image features related to thyroid cancer dynamic images, pathological types, and molecular features respectively, integrates the information of the three, performs effective fusion, and inputs them into the multi-branch neural network model in a parallel manner, integrates the feature information for end-to-end learning, and makes a more comprehensive and objective comprehensive judgment.

Images