Thyroid nodule braf gene mutation prediction system and method based on multi-modal fusion
By integrating ultrasound images, real-time shear wave elastography, and cytological images into a deep learning model, the technical challenge of non-invasively predicting BRAF gene mutations in thyroid nodules has been solved, achieving high-precision prediction and reducing detection costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ANHUI PROVINCIAL HOSPITAL
- Filing Date
- 2026-05-08
- Publication Date
- 2026-06-05
AI Technical Summary
Current technologies cannot accurately predict the BRAF gene mutation status of thyroid nodules without relying on invasive molecular testing, and there are risks and high costs associated with invasive procedures.
By fusing conventional ultrasound images, real-time shear wave elastography quantitative data, and puncture cytology images, a feature-level adaptive fusion was performed using an attention-based deep learning model to predict the BRAF gene mutation status.
It enables non-invasive and accurate prediction of BRAF gene mutations, reduces testing costs and time, improves diagnostic accuracy and consistency, and provides interpretable prediction results.
Smart Images
Figure CN122157768A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical auxiliary diagnostic technology, and in particular to a BRAF gene mutation prediction system and method for thyroid nodules based on multimodal fusion. Background Technology
[0002] Currently, the Thyroid Imaging Reporting and Data System (TI-RADS) is widely used in clinical practice to standardize the scoring of ultrasound images for preliminary assessment of the malignancy risk of thyroid nodules. However, TI-RADS grading relies on the subjective judgment of physicians, and there is limited consistency among physicians with different levels of experience. Furthermore, the risk assessment for gray zone nodules that fall between benign and malignant is still not precise enough.
[0003] To improve diagnostic accuracy, ultrasound-guided fine-needle aspiration (FNA) is commonly used in clinical practice to obtain cytological specimens for pathological diagnosis, and it is considered an important reference for preoperative diagnosis. However, FNA still has an indeterminate result of about 20%-30% (Bethesda III and IV), which cannot determine whether the condition is benign or malignant, leading to patients needing diagnostic surgical resection to obtain a final diagnosis, thus creating a risk of overtreatment.
[0004] In recent years, molecular biology research has revealed that the BRAF V600E mutation is the most common driver gene mutation in papillary thyroid carcinoma. It is not only directly related to malignancy but also closely associated with tumor invasiveness, lymph node metastasis, recurrence risk, and resistance to radioactive iodine therapy. Therefore, detecting BRAF mutation status has significant clinical value for the precise diagnosis and treatment of thyroid nodules. However, BRAF mutation detection relies on molecular pathological testing of FNA samples (such as PCR sequencing), which is costly, time-consuming, and increases the number of invasive procedures, limiting its application in routine clinical screening.
[0005] With the development of artificial intelligence technology, existing research has attempted to use deep learning models to automate the analysis of thyroid ultrasound images, aiming to improve the accuracy and consistency of diagnosis. Some studies have further explored multimodal information fusion, combining ultrasound images with clinical indicators or known gene testing results to improve the accuracy of benign / malignant grading. For example, Chinese patent application CN120452757A discloses a benign / malignant nodule grading assessment system based on a large-scale model fusion of ultrasound imaging and thyroid gene markers. This system uses ultrasound images and detected gene markers (such as BRAF mutations, TERT mutations, circRNA, etc.) as input, and fuses them through a multimodal Transformer model to output the benign / malignant risk grading results of the nodules. However, this type of technical solution still relies on invasively obtained gene testing results as input. Its essence is to re-grade the risk based on known molecular information, and it has not escaped the dependence on invasive molecular testing, nor can it provide predictive information before gene testing.
[0006] Furthermore, existing AI models based on ultrasound imaging generally employ a single modality input, neglecting the intrinsic correlation between the biomechanical properties of tumor tissue (such as stiffness) and BRAF mutations. Studies have shown that BRAF mutations can induce fibroblast activation and collagen deposition, leading to a significant increase in the stiffness of the tumor microenvironment. This biomechanical fingerprint can be quantitatively captured using real-time shear wave elastography. Simultaneously, BRAF mutations also cause changes in cytopathological features such as nuclear morphology and chromatin distribution; this microscopic information is contained within biopsy images. However, current technologies lack a feature-level deep fusion of conventional ultrasound images, real-time shear wave elastography quantitative data, and biopsy images to predict BRAF mutation status in a non-invasive or minimally invasive manner.
[0007] It is evident that the key technical challenge in this field is how to construct an end-to-end predictive model by comprehensively utilizing multi-dimensional medical data without relying on invasive molecular testing, to achieve non-invasive and accurate prediction of the BRAF gene mutation status of thyroid nodules, and to endow the model with interpretability to improve clinical credibility. Summary of the Invention
[0008] The technical problem to be solved by this invention is how to achieve non-invasive and accurate prediction of the BRAF gene mutation status of thyroid nodules by integrating conventional ultrasound images, real-time shear wave elastography quantitative data and puncture cytology images without relying on invasive molecular detection.
[0009] To address the aforementioned technical problems, this invention provides a thyroid nodule BRAF gene mutation prediction system based on multimodal fusion, comprising: The data acquisition module is configured to acquire conventional ultrasound images, real-time shear wave elastography quantitative data, and fine-needle aspiration cytology images of the target thyroid nodule. The multimodal feature extraction module includes: An ultrasound feature extraction unit is used to extract ultrasound image depth features from the conventional ultrasound image; The elastic feature extraction unit is used to extract elastic features from the real-time shear wave elastography quantitative data, wherein the elastic features include at least one of the following: the average Young's modulus value within the nodule, the maximum Young's modulus value within the nodule, the hardness ratio of the nodule to the surrounding tissue, and the hardness heterogeneity coefficient within the nodule. A cytological feature extraction unit is used to extract cytological depth features from the puncture cytology image; The feature fusion module is configured to perform feature-level adaptive fusion of the ultrasound image depth features, the elasticity features, and the cytological depth features using an attention-based deep learning model to obtain fused features. The prediction module is configured to input the fused features into a classifier and output a prediction result indicating whether the target thyroid nodule has a BRAF gene mutation.
[0010] Furthermore, the attention-based deep learning model includes a cross-modal Transformer encoder, which is configured to perform the following operations: The ultrasound image depth features, elastic features, and cytological depth features are each mapped to the same feature space through independent linear mapping layers to obtain ultrasound modality tokens, elastic modality tokens, and cytological modality tokens. The ultrasound modal token, elastic modal token, and cytology modal token are concatenated into an input sequence, and learnable modal position codes are added. The input sequence is fed into the cross-modal Transformer encoder, and the attention weights between each modal token are calculated through a multi-head self-attention mechanism, so that each modal token can absorb information from other modal tokens when updating, so as to perform feature-level adaptive fusion; and the ultrasonic modal token processed by the cross-modal Transformer encoder is output as the fused feature.
[0011] Furthermore, the multi-head self-attention mechanism dynamically adjusts the contribution weights of each modality feature to BRAF gene mutation prediction by performing the following operations: The ultrasound modality token, elasticity modality token, and cytology modality token are each transformed linearly to generate corresponding query vectors. Key vector Sum value vector ,in Representing different modes; Calculate the attention weight between any two modal tokens It satisfies the calculation formula:
[0012] In the formula, Indicates the first When the modal token updates its own representation, it adjusts the representation of the first modal token. The level of attention given to each modal token; Indicates the first The query vector of the modality and the first modality The dot product of the key vectors of each modality. For each dimension of attention head, It is an exponential function.
[0013] Furthermore, the multi-head self-attention mechanism employs multiple parallel attention heads, each independently calculating attention weights, and concatenating and linearly transforming the outputs of each attention head to capture cross-modal association patterns in different dimensions.
[0014] Furthermore, the cross-modal Transformer encoder comprises multiple cascaded Transformer encoder layers, with each encoder layer achieving layer-by-layer fusion through iterative optimization of the attention weights, wherein: In the first Transformer encoder layer, the initial attention weights between each modality token are calculated to complete shallow cross-modal feature interaction; In the second Transformer encoder layer, the attention weights are recalculated based on the updated modal representation from the first layer to complete the cross-modal feature interaction in the middle layer. In the third Transformer encoder layer, attention weights are calculated again to complete deep cross-modal feature interaction and output the final fused features; In this process, the attention weights of each layer are dynamically recalculated based on the modal representation of the current layer, so that the contribution weights of each modal feature are optimized layer by layer with the fusion depth.
[0015] Furthermore, each of the Transformer encoder layers includes: A multi-head self-attention sub-layer is used to calculate the attention weights between modal tokens in the input sequence, thereby completing cross-modal information interaction; A feedforward sublayer is used to perform a nonlinear transformation on the output of the multi-head self-attention sublayer; Residual connections and layer normalization are respectively set after the multi-head self-attention sub-layer and the feedforward network sub-layer to stabilize the training process.
[0016] Furthermore, the multimodal feature extraction module also includes a TI-RADS scoring unit, which extracts artificial features based on the conventional ultrasound images and scores them according to the thyroid imaging report and data system standards to obtain TI-RADS grading and quantification results; The feature fusion module is further used to perform feature-level fusion of the TI-RADS graded quantization results with the ultrasound image depth features, the elasticity features, and the cytological depth features.
[0017] Furthermore: The ultrasound image depth features are obtained by inputting the conventional ultrasound image into a pre-trained deep convolutional neural network. The cytological depth features are obtained by segmenting the puncture cytology image into multiple image blocks, inputting each image block into a pre-trained deep convolutional neural network to extract block-level features, and then aggregating them through a block-level attention pooling layer. The real-time shear wave elastography quantitative data includes the average Young's modulus value within the nodule. Maximum Young's modulus value within the nodule The hardness ratio of the nodule to the surrounding tissue and the coefficient of hardness heterogeneity within nodules ,in: The nodule's hardness ratio compared to the surrounding tissue The calculation formula is: In the formula, The mean Young's modulus value of the normal thyroid tissue surrounding the nodule; The intranodal hardness heterogeneity coefficient The calculation formula is: In the formula, This represents the standard deviation of Young's modulus values within the nodule.
[0018] Furthermore, the data acquisition module also includes a preprocessing unit, which is configured to perform the following operations: The conventional ultrasound images are subjected to image denoising, grayscale normalization, and size normalization. The nodule region was delineated and Young's modulus value was extracted from the real-time shear wave elastic imaging quantitative data. The puncture cytology images were subjected to full-field digital slice scanning, image segmentation, and cell nucleus percentage screening.
[0019] Furthermore, the classifier of the prediction module is a two-layer fully connected network, and its output layer uses the Softmax activation function to output the probability distribution of BRAF gene mutation positive and negative.
[0020] Furthermore, the system also includes an interpretation module configured to generate a key region heatmap on the conventional ultrasound image based on the attention weights or gradient information of the deep learning model, the heatmap indicating the image region that contributes most to the prediction of BRAF gene mutations.
[0021] Furthermore, the key region heatmap is generated using a gradient-weighted class activation mapping method, including: Obtain the feature map of the last convolutional layer in the ultrasound image depth feature extraction unit; Calculate the gradient of the feature map relative to the BRAF gene mutation positive category, and perform global average pooling on the gradient to obtain the weight of each channel; The feature maps of each channel are summed according to their weights, processed by an activation function, and then upsampled to the size of the conventional ultrasound image to obtain the heat map.
[0022] Furthermore, the system also includes a training module configured to train the deep learning model using a multimodal sample dataset labeled with BRAF gene mutation states; wherein, during the training process, a weighted cross-entropy loss function is used, and class weights are set according to the ratio of positive to negative samples in the training set.
[0023] Furthermore, the system is configured as follows: When only the conventional ultrasound image and the real-time shear wave elastography quantitative data are obtained, but the puncture cytology image is not obtained, the input of the cytology feature extraction unit is empty or a placeholder, and the feature fusion module only fuses the depth features of the ultrasound image and the elastic features; When only the puncture cytology image is acquired and the conventional ultrasound image and the real-time shear wave elastography quantitative data are not acquired, the input of the ultrasound feature extraction unit and the elastography feature extraction unit is empty or a placeholder, and the feature fusion module only fuses the cytology depth features.
[0024] The present invention also provides a multimodal fusion-based method for predicting BRAF gene mutations in thyroid nodules applied to the system, comprising the following steps: Acquire routine ultrasound images, real-time shear wave elastography quantitative data, and fine-needle aspiration cytology images of the target thyroid nodule; Ultrasound image depth features, elastic features, and cytological depth features are extracted from the conventional ultrasound images, the real-time shear wave elastography quantitative data, and the puncture cytology images, respectively. By using a deep learning model based on an attention mechanism, the ultrasound image depth features, elasticity features, and cytological depth features are adaptively fused at the feature level to obtain fused features. The fused features are input into the classifier, which outputs a prediction result of whether the target thyroid nodule has a BRAF gene mutation. Based on the attention weights or gradient information of the deep learning model, a heat map of key areas on the conventional ultrasound image is generated.
[0025] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: This invention constructs an auxiliary diagnostic system that includes a data acquisition module, a multimodal feature extraction module, an attention-based feature fusion module, and a prediction module. For the first time, it performs feature-level adaptive fusion of conventional ultrasound images, real-time shear wave elastography quantitative data, and puncture cytology images. Without relying on invasive molecular detection, it achieves high-precision non-invasive prediction of macroscopic morphology, biomechanical properties, and microscopic pathological features, as well as specific driver gene mutation states.
[0026] Compared to existing technologies that rely on known gene testing results or single-modal input, this invention not only avoids invasive procedures and reduces testing costs and cycles, but also uncovers the deep correlation between phenotype and genotype through multimodal feature-level fusion, providing innovative technical support for the precise diagnosis and treatment of thyroid nodules, personalized treatment decisions, and optimization of clinical procedures. Attached Figure Description
[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0028] Figure 1 This is a system architecture diagram disclosed in this invention; Figure 2 This is a flowchart disclosed in the present invention. Detailed Implementation
[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0030] Please see Figure 1 This invention aims to provide a BRAF gene mutation prediction system for thyroid nodules based on multimodal fusion, mainly comprising a data acquisition module, a multimodal feature extraction module, a feature fusion module, a prediction module, and an interpretation module. The data acquisition module acquires conventional ultrasound images, real-time shear wave elastography quantitative data, and fine-needle aspiration cytology images of the target thyroid nodule. The multimodal feature extraction module includes an ultrasound feature extraction unit, an elastography feature extraction unit, and a cytology feature extraction unit, each used to extract deep features from the aforementioned three modalities. The feature fusion module uses a deep learning model based on an attention mechanism to adaptively fuse the features of the three modalities at the feature level, obtaining fused features. The prediction module inputs the fused features into a classifier and outputs a prediction result indicating whether the target thyroid nodule has a BRAF gene mutation. The interpretation module generates a heatmap of key regions on the conventional ultrasound image based on the attention weights or gradient information of the deep learning model. This heatmap indicates the image region that contributes the most to the BRAF gene mutation prediction. The following provides a detailed description of each module.
[0031] First, the data acquisition module will be explained in detail.
[0032] The data acquisition module interfaces with the hospital information system (HIS / PACS / LIS) to automatically extract patients' thyroid ultrasound images (DICOM format) and corresponding fine-needle aspiration cytology digital images. Simultaneously, it acquires quantitative data from real-time shear wave elastography via the laboratory information system. All data is de-identified and matched using the patient ID as an index to ensure data privacy and security.
[0033] The data acquisition module also includes a preprocessing unit for cleaning and format conversion of data from different modalities. This includes image denoising, grayscale normalization, and size normalization for conventional ultrasound images; nodule delineation and Young's modulus extraction for real-time shear wave elastography quantitative data; and full-field digital slice scanning, image segmentation, and cell nucleus proportion screening for puncture cytology images.
[0034] Specifically: Routine ultrasound image acquisition was performed using a Philips EPIQ 7 ultrasound system with an L12-5 linear array probe and a frequency of 5-12 MHz. Acquisition included grayscale B-mode images, one cross-section of the nodule's maximum diameter, one vertical cross-section, and one color Doppler flow image. The preprocessing unit performed the following operations on the ultrasound images: first, median filtering (kernel size 3×3 pixels) was used to remove speckle noise; then, grayscale normalization was performed, linearly mapping pixel values to the [0,1] interval; finally, the images were uniformly scaled to a size of 224×224 pixels. If the original image was a single-channel grayscale image, the copied channel was converted to a three-channel format to meet the input requirements of the subsequent deep convolutional neural network.
[0035] Real-time shear wave elastography data were obtained from the SuperSonic Imagine Aixplorer ultrasound diagnostic system equipped with an SL15-4 linear array probe. After a routine ultrasound examination, the system was switched to elastography mode. The patient was instructed to breathe calmly, and the probe was lightly touched to the skin without applying pressure. The image was frozen after it stabilized. Three representative elastography images were saved for each nodule—the maximum diameter section, the vertical section, and a color elastography overlay. The specific operation included: first, two sonographers independently delineated the region of interest (ROI) of the nodule, and the intersection of their delineations was taken as the final ROI. Artifacts caused by cystic areas and calcifications were avoided during delineation. Pixel-level Young's modulus values were extracted from the DICOM format elastography exported from the device, and the following four quantitative indicators were calculated: (1) Average Young's modulus value within the nodule :
[0036] In the formula, The total number of pixels in the region of interest. For the first The Young's modulus value of each pixel.
[0037] (2) Maximum Young's modulus value within the nodule : Take the maximum value of Young's modulus of all pixels in the region of interest, which reflects the hardness of the hardest region of the nodule.
[0038] (3) The hardness ratio of the nodule to the surrounding tissue :
[0039] In the formula, The average Young's modulus value of the normal thyroid tissue surrounding the nodule is calculated by delineating a ring-shaped region with a width of 2-3 mm around the nodule.
[0040] (4) Hardness heterogeneity coefficient within nodules :
[0041] In the formula, The standard deviation of Young's modulus values within the nodule is denoted as . This coefficient reflects the uniformity of hardness within the nodule; a larger value indicates a more uneven distribution of hardness.
[0042] The cytology data were obtained from cell smears obtained by ultrasound-guided fine-needle aspiration biopsy. A 23G needle was used for the aspiration procedure, with 2-3 punctures per nodule. After smear preparation, the smears were processed using a liquid-based thin-layer cytology preparation system (ThinPrep 2000) and stained with Papanicolaou. Digital pathology images were acquired using a Leica Aperio AT2 full-view digital slide scanner, scanned under a 40× objective lens at a resolution of 0.25 μm / pixel, with output in SVS or TIFF format. The preprocessing unit first used the open-source pathology image analysis software QuPath for semi-automatic annotation: a pre-trained nucleus segmentation model was used to identify all cell nuclei, and then pathologists selected diagnostically valuable cell clusters, removing mucus, blood, and areas without diagnostic value. Then, the full-view digital slide image corresponding to each nodule was segmented into 512×512 pixel patches with a stride of 256 pixels, retaining only patches with a nucleus content greater than 30% as valid input. Ultimately, the cytological data of each nodule is represented as a set of N patches, where N varies from approximately 50 to 200 depending on the number of cells, with each patch serving as an independent input unit.
[0043] Next, the multimodal feature extraction module will be described in detail.
[0044] The multimodal feature extraction module includes three parallel branches: an ultrasound feature extraction unit, an elasticity feature extraction unit, and a cytology feature extraction unit, which respectively process conventional ultrasound images, real-time shear wave elastography quantitative data, and puncture cytology images.
[0045] The ultrasound feature extraction unit employs a ResNet-50 deep convolutional neural network as its backbone. This network was pre-trained on the ImageNet dataset to fully leverage the general feature representation capabilities learned from large-scale natural images. The input image size is 224×224×3. Features are extracted through five convolutional stages. Each residual block consists of two or three convolutional layers, interspersed with batch normalization and ReLU activation functions. A shortcut connection using an identity mapping effectively mitigates the gradient vanishing problem in deep networks. After the final convolutional stage, the network transforms the feature map (7×7×2048) into a 2048-dimensional feature vector using a global average pooling layer. This vector represents the depth features of the ultrasound image. The feature vector has a dimension of 2048. This feature vector encapsulates key radiographic information such as the nodule's texture pattern, shape structure, boundary features, and echo distribution.
[0046] The elasticity feature extraction unit employs a dual-path feature extraction strategy, simultaneously extracting deep features and handcrafted features. For deep feature extraction, the elasticity image (224×224×1) is input into a ResNet-18 deep convolutional neural network. ResNet-18 has an 18-layer convolutional structure, is lighter than ResNet-50, and is suitable for images with relatively simple textures like elasticity images. The network is initialized with ImageNet pre-trained weights, and the first convolutional kernel of the input layer is adjusted to accept single-channel input. After global average pooling, a 512-dimensional deep feature vector is obtained. For handcrafted feature extraction, four elasticity quantitative indicators (…) calculated in the preprocessing stage are used… , , , This forms a 4-dimensional handcrafted feature vector. This handcrafted feature vector is then concatenated with a 512-dimensional depth feature vector to obtain a 516-dimensional elastic feature. .
[0047] The cytological feature extraction unit employs a tile-level feature extraction and aggregation strategy. For each nodule, N 512×512×3 pathological tiles are obtained after preprocessing. Each tile is independently input into a ResNet-18 deep convolutional neural network (pre-trained on ImageNet) to extract a 2048-dimensional tile-level feature vector. Subsequently, the N tile features are aggregated into a global feature vector through a tile-level attention pooling layer. Specifically, for the ... Feature vectors of each patch First, its attention score is calculated through a fully connected layer:
[0048] In the formula, and These are learnable weight parameters; For bias terms; The activation function is hyperbolic tangent. This attention score... Indicates the first The importance weight of each patch in the overall nodule cytological features. Final cytological depth features. We obtain the following by weighted summation:
[0049] This aggregation strategy enables the model to automatically focus on the most diagnostically valuable cytological patches, such as those containing atypical nuclei or cell clusters, while ignoring background or non-diagnostic areas.
[0050] Secondly, the feature fusion module will be explained in detail.
[0051] The feature fusion module is used to adaptively fuse ultrasound image depth features, elasticity features, and cytological depth features at the feature level using an attention-based deep learning model to obtain fused features. The attention-based deep learning model includes a cross-modal Transformer encoder, which is configured to perform the following operations: a. Map the depth features, elastic features, and cytological depth features of ultrasound images to the same feature space through independent linear mapping layers to obtain ultrasound modality tokens, elastic modality tokens, and cytological modality tokens.
[0052] Because the feature vectors of the three modalities have different dimensions, among which, 2048 dimensions It is 516-dimensional. Since the feature space is 2048 dimensions, direct feature interaction is not possible. Therefore, we first use three independent linear mapping layers to map the features of each modality to a unified 512-dimensional feature space:
[0053]
[0054]
[0055] in, , , These are referred to as the ultrasound modal token, the elastic modal token, and the cytology modal token, respectively, and each token is a 512-dimensional vector; , , The weight matrix is a learnable matrix; , , This is a bias term.
[0056] b. Concatenate the ultrasound modal token, elastic modal token, and cytology modal token into an input sequence, and add learnable modal position codes.
[0057] Specifically, the three modal tokens are concatenated into an input sequence:
[0058] To enable the model to distinguish tokens of different modalities, learnable modal position encoding is added. Add to the corresponding Token:
[0059]
[0060]
[0061] This positional encoding is optimized along with the model parameters during training, enabling the model to learn the semantic role that each modality should have.
[0062] c. Input the input sequence into the cross-modal Transformer encoder, calculate the attention weights between each modal token through a multi-head self-attention mechanism, so that each modal token can absorb information from other modal tokens when updating, so as to perform feature-level adaptive fusion; and output the ultrasonic modal token processed by the cross-modal Transformer encoder as the fusion feature.
[0063] Specifically, the input sequence with position encoding is input into the cross-modal Transformer encoder. This embodiment uses a three-layer cascaded Transformer encoder layer, each of which consists of a multi-head self-attention sub-layer and a feedforward network sub-layer, with residual connections and layer normalization added after each sub-layer.
[0064] Regarding the multi-head self-attention quantum layer: Multi-head self-attention is key to achieving adaptive fusion. For an input sequence X (where each layer's input is the output of the previous layer), the query matrix Q, key matrix K, and value matrix V are first generated through three different linear transformations:
[0065]
[0066]
[0067] in, These are learnable parameters. Then, Q, K, and V are divided into h attention heads, where h=8 in this embodiment, and the dimension of each attention head is... .
[0068] For the Each attention head is used to calculate attention weights.
[0069] in, For the first The projection matrix of each attention head.
[0070] Attention weight matrix Its elements Indicates the first When updating itself, the modal token... The level of attention given to each modal token. This weight is dynamically calculated by the model based on the input samples, rather than being a fixed parameter. Specifically, The calculation formula is:
[0071] in, For the first The query vector corresponding to each modality For the first The key vectors corresponding to each mode For each dimension of attention head, It is an exponential function. This formula ensures that for each query token, the sum of its attention weights over all key tokens is 1, i.e. This normalization mechanism gives the fusion process a clear semantic allocation of information. Subsequently, the outputs of the eight attention heads are concatenated along the feature dimension and then subjected to a linear transformation.
[0072] Regarding residual connectivity and layer normalization: Residual connections and layer normalization are added after each sublayer to ensure stable training of deep networks. Layer normalization normalizes the feature dimensions of each token, and the calculation formula is as follows:
[0073] in, and They are respectively The mean and variance, To prevent division by zero for extremely small constants, and These are learnable scaling and translation parameters.
[0074] Regarding feedforward network sublayers: Following the multi-head self-attention sublayer is a feedforward sublayer containing two fully connected layers, with the GELU activation function used in between.
[0075] in, , , and This is the bias term. The GELU activation function is defined as:
[0076] in, This is the cumulative distribution function of the standard normal distribution.
[0077] Similarly, residual connections and layer normalization are added after the feedforward network sublayers.
[0078] After iterative processing through three Transformer encoder layers, each modal token incorporates information from the other two modalities. The ultrasound modal token was selected as the fused nodule comprehensive representation. Experimental results show that the ultrasonic modal token retains the highest predictive performance after fusion, and as an image modality, it is convenient for subsequent heatmap generation.
[0079] The aforementioned attention mechanism enables dynamic adjustment of the contribution weights of each modality, and its clinical semantics are reflected in the following aspects: When the real-time shear wave elastography quantitative data in the input sample reflects a significant increase in nodule stiffness, attention weighting is applied. and The increase in stiffness leads to a higher weighting of the elastic modal token in relation to the ultrasound and cytological modal tokens. This indicates that the model has learned that stiffness information has greater predictive value for BRAF mutations when nodule stiffness is abnormally high.
[0080] When the cytology images in the input sample reflect significant nuclear atypia (e.g., enlarged nuclei, increased nuclear-cytoplasmic ratio, irregular nuclear membrane), the attention weight is applied. and The increase in the size of the cytological modality token leads to a higher weighting of the contribution of the ultrasound modality token and the elasticity modality token. This reflects the importance of cytopathological features in BRAF mutation prediction.
[0081] When the input sample contains conventional ultrasound images that reflect typical malignant features (such as microcalcifications, aspect ratio > 1, irregular borders), the attention weight is adjusted. and The increase in the size of the ultrasound modality token leads to a higher weighting of the contribution of the elasticity modality token and the cytology modality token. This reflects the central role of ultrasound imaging as the basic modality for thyroid nodule assessment.
[0082] In this scheme, the dynamic weight adjustment mechanism enables the model to adaptively allocate the fusion weights of each modality according to the specific features of each sample, thereby achieving personalized fusion that varies from sample to sample and improving the accuracy and robustness of prediction.
[0083] The prediction module will then be explained in detail.
[0084] The prediction module will fuse features Input classifier. This classifier is a two-layer fully connected network—the first layer maps 512-dimensional features to 256-dimensional features using the ReLU activation function; the second layer maps 256-dimensional features to 2-dimensional features and outputs a logits vector. The Softmax function is used to convert logits into a probability distribution.
[0085]
[0086] in, This indicates a negative BRAF gene mutation. This indicates a positive BRAF gene mutation. The final prediction result corresponds to the category with the higher probability.
[0087] Finally, the explanation module is explained in detail.
[0088] The explanation module uses the gradient-weighted class activation mapping method to generate heatmaps of key regions. The specific steps are as follows: First, obtain the feature map of the last convolutional layer of ResNet-50 in the ultrasound feature extraction unit. Then, the gradient of the feature map relative to the BRAF gene mutation positive category is calculated, and global average pooling is performed on the gradient to obtain the weights of each channel:
[0089] in, This represents the model's output probability for the BRAF positive category. For the first Each channel is located at ( The eigenvalues of the feature map are given by Z = 7 × 7, which is the spatial size of the feature map.
[0090] Next, the feature maps of each channel are summed according to their weights, and then processed by the ReLU activation function to obtain the heatmap:
[0091] Finally, the heat map The image was upsampled to the original ultrasound image size and pseudo-color mapped using jet colormap, then overlaid on the original ultrasound image. This heatmap highlights the image regions that contribute most to BRAF gene mutation prediction, such as nodule edges, microcalcification clusters, and hypoechoic areas, providing clinicians with intuitive decision-making support.
[0092] Please see Figure 2 The present invention also provides a multimodal fusion-based method for predicting BRAF gene mutations in thyroid nodules applied to the above-mentioned system, comprising the following steps: S1. Obtain routine ultrasound images, real-time shear wave elastography quantitative data, and fine-needle aspiration cytology images of the target thyroid nodule.
[0093] S2. Extract ultrasound image depth features, elastic features, and cytological depth features from conventional ultrasound images, real-time shear wave elastography quantitative data, and puncture cytology images, respectively.
[0094] S3. Using a deep learning model based on an attention mechanism, feature-level adaptive fusion of ultrasound image depth features, elasticity features, and cytological depth features is performed to obtain fused features.
[0095] S4. Input the fused features into the classifier and output the prediction result of whether the target thyroid nodule has a BRAF gene mutation.
[0096] S5. Based on the attention weights or gradient information of the deep learning model, generate heat maps of key areas on conventional ultrasound images.
[0097] To verify the effectiveness of this technical solution, the present invention has also conducted relevant verifications: In one specific embodiment, a multimodal dataset containing 1150 thyroid nodules was constructed. All nodules had complete conventional ultrasound images, real-time shear wave elastography quantitative data, and fine-needle aspiration cytology digital images, and their BRAF mutation status was confirmed by postoperative pathological or molecular testing. BRAF mutation detection was performed using real-time quantitative PCR with a human BRAF V600E gene mutation detection kit (fluorescent PCR method), achieving a sensitivity of 1% and a specificity greater than 99%. The dataset was divided as follows: Training set: 800 cases, of which 320 were BRAF positive (40%) and 480 were BRAF negative (60%).
[0098] Validation set: 200 cases, of which 80 were BRAF positive (40%) and 120 were BRAF negative (60%).
[0099] Internal test set: 200 cases, of which 80 were BRAF positive (40%) and 120 were BRAF negative (60%).
[0100] External test set: 150 cases, of which 60 were BRAF positive (40%) and 90 were BRAF negative (60%), from two collaborating hospitals, used to verify the model's generalization ability.
[0101] First, the model was trained using an end-to-end approach, simultaneously optimizing the parameters of the feature extraction network and the fusion network. The performance metrics of the model on the test set are shown in Table 1 below: Table 1
[0102] As shown in Table 1, the AUC of the trimodal cross-modal attention fusion scheme of this invention is 0.942. However, the AUC using only ultrasound images is 0.812, using only elastography images is 0.754, using only cytology images is 0.869, and the AUC of trimodal stitching without attention fusion is 0.901. Comparative experimental results demonstrate that the multimodal adaptive fusion scheme of this invention has significant advantages over single-modal or simple stitching schemes.
[0103] It should be noted that in actual clinical scenarios, there may be situations where certain modal data are missing. This system is flexibly adaptable; specifically: When only conventional ultrasound images and real-time shear wave elastography quantitative data are acquired, but puncture cytology images are not obtained, the input to the cytology feature extraction unit is empty or a placeholder, and the feature fusion module only fuses the depth and elastic features of the ultrasound images. In this case, the input sequence of the cross-modal Transformer encoder contains only two tokens, and the attention mechanism interacts between the two.
[0104] When only puncture cytology images are obtained without conventional ultrasound images and real-time shear wave elastography quantitative data, the inputs to the ultrasound feature extraction unit and the elastography feature extraction unit are empty or placeholders, and the feature fusion module only fuses cytology depth features. In this case, the model degenerates into a pure cytology prediction mode, which is suitable for auxiliary judgment when large-scale population screening or imaging examination results are unclear.
[0105] During implementation, this system can be deployed within the internal network of a tertiary hospital. Through integration with the hospital's information system and ultrasound equipment, it enables multimodal intelligent diagnosis of thyroid nodules using imaging and cytology. The specific workflow is as follows: First, when an ultrasound physician performs a thyroid ultrasound examination on a patient, the real-time images acquired by the ultrasound equipment are automatically stored on the PACS server. Once the system's data interface module detects that a new examination has been completed, it automatically retrieves the patient's ultrasound images.
[0106] If an ultrasound examination reveals a suspicious nodule and the patient consents to a fine-needle aspiration biopsy, the pathology department will send the obtained nodule tissue sample to the cytology laboratory for liquid-based thin-layer cytology preparation and Papanicolaou staining. The digital pathology image will then be scanned and uploaded to the system.
[0107] Subsequently, the system automatically performs the following steps: data preprocessing, multimodal feature extraction, cross-modal adaptive fusion, BRAF mutation prediction, and heatmap generation.
[0108] Finally, the system automatically generates a structured diagnostic report. This report includes basic patient information, ultrasound imaging findings, quantitative elastography indicators, cytological image analysis, BRAF mutation prediction results, and heat maps of key areas.
[0109] This invention is the first to adaptively fuse conventional ultrasound images, real-time shear wave elastography quantitative data, and puncture cytology images at the feature level. Utilizing the attention mechanism in a deep learning model, it automatically learns and dynamically adjusts the contribution weights of each modality feature to BRAF gene mutation prediction. This achieves high-precision, non-invasive prediction from macroscopic morphology, biomechanical properties, and microscopic pathological features to specific driver gene mutation states, while avoiding invasive molecular testing operations and reducing testing costs and time. Furthermore, the system has a good modular structure and flexibility, adapting to modality loss situations in different clinical scenarios. It can be used as a comprehensive risk assessment tool or as an independent molecular risk assessment product.
[0110] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A BRAF gene mutation prediction system for thyroid nodules based on multimodal fusion, characterized in that, include: The data acquisition module is configured to acquire conventional ultrasound images, real-time shear wave elastography quantitative data, and fine-needle aspiration cytology images of the target thyroid nodule. The multimodal feature extraction module includes: An ultrasound feature extraction unit is used to extract ultrasound image depth features from the conventional ultrasound image; The elastic feature extraction unit is used to extract elastic features from the real-time shear wave elastography quantitative data, wherein the elastic features include at least one of the following: the average Young's modulus value within the nodule, the maximum Young's modulus value within the nodule, the hardness ratio of the nodule to the surrounding tissue, and the hardness heterogeneity coefficient within the nodule. A cytological feature extraction unit is used to extract cytological depth features from the puncture cytology image; The feature fusion module is configured to perform feature-level adaptive fusion of the ultrasound image depth features, the elasticity features, and the cytological depth features using an attention-based deep learning model to obtain fused features. The prediction module is configured to input the fused features into a classifier and output a prediction result indicating whether the target thyroid nodule has a BRAF gene mutation.
2. The BRAF gene mutation prediction system for thyroid nodules based on multimodal fusion according to claim 1, characterized in that, The attention-based deep learning model includes a cross-modal Transformer encoder, which is configured to perform the following operations: The ultrasound image depth features, elastic features, and cytological depth features are each mapped to the same feature space through independent linear mapping layers to obtain ultrasound modality tokens, elastic modality tokens, and cytological modality tokens. The ultrasound modal token, elastic modal token, and cytology modal token are concatenated into an input sequence, and learnable modal position codes are added. The input sequence is input into the cross-modal Transformer encoder, and the attention weights between each modal token are calculated through a multi-head self-attention mechanism, so that each modal token can absorb information from other modal tokens when updating, in order to perform feature-level adaptive fusion. The ultrasonic modal token processed by the cross-modal Transformer encoder is then output as a fusion feature.
3. The BRAF gene mutation prediction system for thyroid nodules based on multimodal fusion according to claim 2, characterized in that, The multi-head self-attention mechanism dynamically adjusts the contribution weights of each modality feature to BRAF gene mutation prediction by performing the following operations: The ultrasound modality token, elasticity modality token, and cytology modality token are each transformed linearly to generate corresponding query vectors. Key vector Sum value vector ,in Representing different modes; Calculate the attention weight between any two modal tokens It satisfies the calculation formula: In the formula, Indicates the first When the modal token updates its own representation, it adjusts the representation of the first modal token. The level of attention given to each modal token; Indicates the first The query vector of the modality and the first modality The dot product of the key vectors of each modality. For each dimension of attention head, It is an exponential function.
4. The BRAF gene mutation prediction system for thyroid nodules based on multimodal fusion according to claim 2, characterized in that, The multi-head self-attention mechanism employs multiple parallel attention heads, each independently calculating attention weights, and concatenating and linearly transforming the outputs of each attention head to capture cross-modal association patterns across different dimensions.
5. The BRAF gene mutation prediction system for thyroid nodules based on multimodal fusion according to claim 2, characterized in that, The cross-modal Transformer encoder comprises multiple cascaded Transformer encoder layers. Each encoder layer achieves layer-by-layer fusion by iteratively optimizing the attention weights, wherein: In the first Transformer encoder layer, the initial attention weights between each modality token are calculated to complete shallow cross-modal feature interaction; In the second Transformer encoder layer, the attention weights are recalculated based on the updated modal representation from the first layer to complete the cross-modal feature interaction in the middle layer. In the third Transformer encoder layer, attention weights are calculated again to complete deep cross-modal feature interaction and output the final fused features; In this process, the attention weights of each layer are dynamically recalculated based on the modal representation of the current layer, so that the contribution weights of each modal feature are optimized layer by layer with the fusion depth.
6. The BRAF gene mutation prediction system for thyroid nodules based on multimodal fusion according to claim 5, characterized in that, Each Transformer encoder layer includes: A multi-head self-attention sub-layer is used to calculate the attention weights between modal tokens in the input sequence, thereby completing cross-modal information interaction; A feedforward sublayer is used to perform a nonlinear transformation on the output of the multi-head self-attention sublayer; Residual connections and layer normalization are respectively set after the multi-head self-attention sub-layer and the feedforward network sub-layer to stabilize the training process.
7. The BRAF gene mutation prediction system for thyroid nodules based on multimodal fusion according to claim 1, characterized in that, The multimodal feature extraction module also includes a TI-RADS scoring unit, which extracts artificial features based on the conventional ultrasound images and scores them according to the thyroid imaging report and data system standards to obtain TI-RADS grading and quantification results. The feature fusion module is further used to perform feature-level fusion of the TI-RADS graded quantization results with the ultrasound image depth features, the elasticity features, and the cytological depth features.
8. The BRAF gene mutation prediction system for thyroid nodules based on multimodal fusion according to claim 1, characterized in that: The ultrasound image depth features are obtained by inputting the conventional ultrasound image into a pre-trained deep convolutional neural network. The cytological depth features are obtained by segmenting the puncture cytology image into multiple image blocks, inputting each image block into a pre-trained deep convolutional neural network to extract block-level features, and then aggregating them through a block-level attention pooling layer. The real-time shear wave elastography quantitative data includes the average Young's modulus value within the nodule. Maximum Young's modulus value within the nodule The hardness ratio of the nodule to the surrounding tissue and the coefficient of hardness heterogeneity within nodules ,in: The nodule's hardness ratio compared to the surrounding tissue The calculation formula is: In the formula, The mean Young's modulus value of the normal thyroid tissue surrounding the nodule; The intranodal hardness heterogeneity coefficient The calculation formula is: In the formula, This represents the standard deviation of Young's modulus values within the nodule.
9. The BRAF gene mutation prediction system for thyroid nodules based on multimodal fusion according to claim 1, characterized in that, The data acquisition module further includes a preprocessing unit, which is configured to perform the following operations: The conventional ultrasound images are subjected to image denoising, grayscale normalization, and size normalization. The nodule region was delineated and Young's modulus value was extracted from the real-time shear wave elastic imaging quantitative data. The puncture cytology images were subjected to full-field digital slice scanning, image segmentation, and cell nucleus percentage screening.
10. The BRAF gene mutation prediction system for thyroid nodules based on multimodal fusion according to claim 1, characterized in that, The predictive module's classifier is a two-layer fully connected network, and its output layer uses the Softmax activation function to output the probability distribution of BRAF gene mutation positive and negative.
11. The BRAF gene mutation prediction system for thyroid nodules based on multimodal fusion according to claim 1, characterized in that, The system also includes an interpretation module configured to generate a key region heatmap on the conventional ultrasound image based on the attention weights or gradient information of the deep learning model. The heatmap is used to indicate the image region that contributes the most to the prediction of BRAF gene mutations.
12. The BRAF gene mutation prediction system for thyroid nodules based on multimodal fusion according to claim 11, characterized in that, The key region heatmap is generated using a gradient-weighted class activation mapping method, including: Obtain the feature map of the last convolutional layer in the ultrasound image depth feature extraction unit; Calculate the gradient of the feature map relative to the BRAF gene mutation positive category, and perform global average pooling on the gradient to obtain the weight of each channel; The feature maps of each channel are summed according to their weights, processed by an activation function, and then upsampled to the size of the conventional ultrasound image to obtain the heat map.
13. The BRAF gene mutation prediction system for thyroid nodules based on multimodal fusion according to claim 1, characterized in that, The system also includes a training module configured to train the deep learning model using a multimodal sample dataset labeled with BRAF gene mutation states; wherein, during training, a weighted cross-entropy loss function is used, and class weights are set according to the ratio of positive to negative samples in the training set.
14. The BRAF gene mutation prediction system for thyroid nodules based on multimodal fusion according to claim 1, characterized in that, The system is configured as follows: When only the conventional ultrasound image and the real-time shear wave elastography quantitative data are obtained, but the puncture cytology image is not obtained, the input of the cytology feature extraction unit is empty or a placeholder, and the feature fusion module only fuses the depth features of the ultrasound image and the elastic features; When only the puncture cytology image is acquired and the conventional ultrasound image and the real-time shear wave elastography quantitative data are not acquired, the input of the ultrasound feature extraction unit and the elastography feature extraction unit is empty or a placeholder, and the feature fusion module only fuses the cytology depth features.
15. A method for predicting BRAF gene mutations in thyroid nodules based on multimodal fusion, applicable to the system described in any one of claims 1-14, characterized in that, Includes the following steps: Acquire routine ultrasound images, real-time shear wave elastography quantitative data, and fine-needle aspiration cytology images of the target thyroid nodule; Ultrasound image depth features, elastic features, and cytological depth features are extracted from the conventional ultrasound images, the real-time shear wave elastography quantitative data, and the puncture cytology images, respectively. By using a deep learning model based on an attention mechanism, the ultrasound image depth features, elasticity features, and cytological depth features are adaptively fused at the feature level to obtain fused features. The fused features are input into the classifier, which outputs a prediction result of whether the target thyroid nodule has a BRAF gene mutation. Based on the attention weights or gradient information of the deep learning model, a heat map of key areas on the conventional ultrasound image is generated.