Interval perception multi-instance learning based longitudinal prediction method for hashimoto's thyroiditis
Patent Information
- Application Number
- CN202611403311.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-10
- Publication Date
- 2026-10-30
AI Technical Summary
[0010]为解决现有技术中早期超声变化细微、多视角图像贡献不一致以及不规则随访间隔难以表达的问题，本发明提出一种基于区间感知多实例学习的桥本甲状腺炎纵向预测方法，能够形成更稳定的随访级表示并提高不规则随访序列的建模能力，能够增强细微超声特征，降低多视角信息冗余和成像伪影对聚合结果的影响，并表达不规则随访间隔所对应的时间依赖关系
[0024]1、本发明以目标随访之前的多次历史随访记录为输入，依次形成实例级超声特征、随访级超声表示和区间感知时间表示，并输出目标随访时点的桥本甲状腺炎状态预测结果。
Smart Images
Figure CN123015523A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to data processing technology, specifically to a longitudinal prediction method for Hashimoto's thyroiditis based on interval-aware multi-instance learning. Background Technology
[0002] Clinical examination methods for Hashimoto's thyroiditis include pathological examination, biochemical examination, and thyroid ultrasound. Pathological examination is invasive and is usually not used as a routine follow-up method; biochemical examination can assess thyroid function and immune abnormalities, but there is no absolute correlation between biochemical indicators and the state of Hashimoto's thyroiditis. Thyroid ultrasound is non-invasive and easy to repeat, making it suitable for disease identification and continuous follow-up assessment.
[0003] Longitudinal prediction of Hashimoto's thyroiditis utilizes multiple historical examinations prior to the target follow-up to estimate the disease status at the future target follow-up time. A single historical follow-up typically includes thyroid ultrasound images from multiple perspectives, each image constituting an instance; different historical follow-ups for the same patient have actual occurrence times, and the time intervals between adjacent and non-adjacent follow-ups are not fixed. Therefore, longitudinal prediction needs to simultaneously process instance-level ultrasound information, multiple instance relationships within the same follow-up, and relationships across follow-up times.
[0004] Early Hashimoto's thyroiditis shows subtle ultrasound changes during continuous follow-up, with related responses exhibiting weak texture changes and diffuse structural alterations, easily confused with background noise and imaging artifacts. Conventional feature extraction networks tend to focus on regions with significant responses, making it difficult to simultaneously preserve low-level texture details, high-level semantic information, and local changes at different scales.
[0005] When channel attention is used alone, the model can highlight feature channels with high information content, but it cannot directly determine the spatial location of the response. When spatial attention is used alone, the model can highlight spatial regions, but it cannot fully determine whether the channels corresponding to those regions are stable. Simply superimposing either attention result directly onto the input features will amplify unreliable responses. Therefore, it is necessary to simultaneously utilize channel reliability and spatial reliability to constrain local enhancement locations and preserve the original features through residual paths.
[0006] The same historical follow-up includes ultrasound images from different sections and viewing angles. Different instances contribute differently to the disease state; some images contain clear lesion clues, while others are affected by artifacts or viewing angle differences. Direct stitching or simple averaging cannot establish correlations between instances, nor can it reduce the impact of instances with low information content on the follow-up level representation, easily leading to unstable aggregation results.
[0007] Existing multi-instance learning methods typically obtain set-level representations through instance weights, but when relying solely on the weights of a single instance to calculate weights, they lack the context between different perspectives within the same follow-up.
[0008] Longitudinal disease prediction methods need to represent the time dependence between different historical follow-ups. Fixed-position coding typically only represents the sequential order of follow-ups in the sequence, with a fixed coding distance between two adjacent positions, failing to reflect the fact that actual follow-ups may be separated by weeks, months, or even longer. Ignoring the true time interval weakens the model's ability to express the rate of disease progression and changes across follow-ups.
[0009] Furthermore, existing longitudinal analysis methods have improved time-dependent modeling through time difference weighting, cross-temporal attention, or continuous-time models. However, under irregular follow-up conditions, it is still necessary to simultaneously represent the specific time of a single follow-up and the relative intervals between different follow-ups. Therefore, a longitudinal prediction method for Hashimoto's thyroiditis is needed that connects reliable subtle feature enhancement, multi-view instance aggregation, and irregular time modeling into a complete processing chain. Summary of the Invention
[0010] To address the issues of subtle early ultrasound changes, inconsistent contributions from multi-view images, and difficulty in representing irregular follow-up intervals in existing technologies, this invention proposes a longitudinal prediction method for Hashimoto's thyroiditis based on interval-aware multi-instance learning. This method can form a more stable follow-up-level representation and improve the modeling ability of irregular follow-up sequences. It can enhance subtle ultrasound features, reduce the redundancy of multi-view information and the impact of imaging artifacts on the aggregation results, and express the time dependence corresponding to irregular follow-up intervals.
[0011] On one hand, embodiments of the present invention provide a longitudinal prediction method for Hashimoto's thyroiditis based on interval-aware multi-instance learning, comprising the following steps:
[0012] S1. Obtain multiple historical follow-up records of the patient before the target follow-up. Each historical follow-up record includes multiple multi-view thyroid ultrasound images and the corresponding follow-up time.
[0013] S2. Reliability-guided cross-scale feature enhancement is performed on each multi-view thyroid ultrasound image to obtain instance-level ultrasound features;
[0014] S3. Perform self-attention-gated multi-instance aggregation on the instance-level ultrasound features corresponding to the same historical follow-up to obtain the follow-up-level ultrasound representation;
[0015] S4. Based on the specific follow-up time of each historical follow-up and the relative time interval between different historical follow-ups, the follow-up ultrasound representation is encoded by interval sensing location to obtain the interval sensing time representation.
[0016] S5. The follow-up-level ultrasound representation and interval-aware time representation of each historical follow-up are fused to obtain the fused historical follow-up sequence. The fused historical follow-up sequence is subjected to longitudinal dependency modeling, and the prediction result of Hashimoto's thyroiditis status at the target follow-up time point is output based on the sequence representation of the last historical follow-up obtained from the longitudinal dependency modeling.
[0017] On the other hand, embodiments of the present invention also provide a longitudinal prediction system for Hashimoto's thyroiditis based on interval-aware multi-instance learning, implemented based on the above-mentioned longitudinal prediction method, the system comprising:
[0018] The data acquisition module is used to acquire multiple historical follow-up records of patients before the target follow-up. Each historical follow-up record includes multiple multi-view thyroid ultrasound images and the corresponding follow-up time.
[0019] The cross-scale feature enhancement module performs reliability-guided cross-scale feature enhancement on each multi-view thyroid ultrasound image to obtain instance-level ultrasound features.
[0020] The multi-instance aggregation module performs self-attention-gated multi-instance aggregation on instance-level ultrasound features corresponding to the same historical follow-up to obtain a follow-up-level ultrasound representation.
[0021] The interval sensing coding module performs interval sensing location coding on the follow-up ultrasound representation based on the specific follow-up time of each historical follow-up and the relative time interval between different historical follow-ups, to obtain the interval sensing time representation.
[0022] The longitudinal prediction module integrates the follow-up-level ultrasound representation and interval-aware time representation of each historical follow-up to obtain the fused historical follow-up sequence; it performs longitudinal dependency modeling on the fused historical follow-up sequence and outputs the prediction result of Hashimoto's thyroiditis status at the target follow-up time point based on the sequence representation corresponding to the last historical follow-up obtained from the longitudinal dependency modeling.
[0023] Compared with the prior art, the technical effects achieved by the present invention include:
[0024] 1. This invention takes multiple historical follow-up records prior to the target follow-up as input, sequentially forms instance-level ultrasound features, follow-up-level ultrasound representation, and interval-aware time representation, and outputs the prediction result of Hashimoto's thyroiditis status at the target follow-up time point.
[0025] 2. This invention establishes instance associations before aggregation and then uses gating weights to highlight instances with higher information content, which can reduce redundancy and artifact effects while retaining multi-perspective information.
[0026] 3. This invention, through joint constraints of channel reliability and spatial reliability, focuses local enhancement on feature responses supported by dual reliability, and avoids the loss of original ultrasound information through residual paths; it utilizes both texture details and semantic information through cross-scale fusion; it aggregates multi-view images with high information content through gated instance aggregation; and it expresses the actual follow-up interval through relative temporal bias. Therefore, this invention can form a more stable follow-up-level ultrasound representation and improve the modeling ability of irregular follow-up sequences.
[0027] 4. This invention employs a self-attention-gated multi-instance learning module to first perform association modeling on instance-level ultrasound features from the same historical follow-up, combining each instance feature with related information from other perspectives. Then, it determines the contribution of each instance to the follow-up-level ultrasound representation through gated multi-instance weights, and performs residual fusion with the weighted result and the original instance feature mean. The resulting follow-up-level ultrasound representation simultaneously retains both multi-perspective context and original set information. Attached Figure Description
[0028] Figure 1 This is a flowchart of the longitudinal prediction method for Hashimoto's thyroiditis in an embodiment of the present invention;
[0029] Figure 2 This is a schematic diagram of the reliability-guided cross-scale feature enhancement framework in an embodiment of the present invention;
[0030] Figure 3 This is a schematic diagram of the framework for self-attention gating multi-instance learning in an embodiment of the present invention;
[0031] Figure 4 This is a schematic diagram of the framework for longitudinal modeling of interval sensing in an embodiment of the present invention;
[0032] Figure 5 This is a schematic diagram of the overall structure of the Hashimoto's thyroiditis longitudinal prediction system in an embodiment of the present invention. Detailed Implementation
[0033] The present invention will now be described in further detail with reference to the accompanying drawings. The embodiments described illustrate the input, operation, output, and connection relationships of each module.
[0034] Example
[0035] This embodiment provides a longitudinal prediction method for Hashimoto's thyroiditis based on interval-aware multi-instance learning. The prediction process includes, in sequence, acquiring historical follow-up records, generating instance-level ultrasound features, generating follow-up-level ultrasound representations, generating interval-aware time representations, and performing longitudinal dependency modeling and target state prediction.
[0036] like Figure 1 As shown, the prediction method in this embodiment specifically includes the following steps:
[0037] S1. Obtain the patient's multiple historical follow-up records prior to the target follow-up. Each historical follow-up record includes multiple multi-view thyroid ultrasound images and the corresponding follow-up time.
[0038] In this embodiment, for the patient's first This historical follow-up represents the set of multi-view thyroid ultrasound images as ,in Indicates the first The first historical follow-up 10 ultrasound images Indicates the first The number of instances included in each historical follow-up. The first The specific follow-up time for each historical follow-up is expressed as follows: The number of historical follow-ups prior to the target follow-up is expressed as The model will then learn to map the disease status from all historical follow-up records to the target follow-up period.
[0039] For target follow-up Before the target follow-up Using historical follow-up records as input, the historical follow-up records are represented as... . For the first A collection of multi-view thyroid ultrasound images from historical follow-ups. For the first The time information associated with each historical follow-up.
[0040] In subsequent prediction steps, historical follow-up records are used as a basis. Output the predicted value of the target state, and mark the positive or negative status of Hashimoto's thyroiditis at the target follow-up time point using the target label.
[0041] S2. Reliability-guided cross-scale feature enhancement is performed on each multi-view thyroid ultrasound image to obtain instance-level ultrasound features.
[0042] In this embodiment, a residual network is used as the backbone network to extract first-stage, second-stage, third-stage, and fourth-stage feature maps. Convolutional block attention processing is performed on the first-stage and third-stage feature maps to enhance the channel response and spatial region response related to thyroid structure and texture changes, resulting in refined feature maps for the corresponding stages. Reliability-guided cross-scale feature enhancement processing is performed on the second-stage and fourth-stage feature maps, resulting in refined feature maps for the second and fourth stages. The refined feature map of the fourth stage is used as the initial fusion feature map, and cross-scale fusion is performed in order from high to low layers. Global average pooling is then performed on the cross-scale fusion feature map corresponding to the first stage to obtain instance-level ultrasound features.
[0043] The above-described cross-scale fusion, performed from high to low levels, specifically includes: upsampling the fused feature map obtained in the previous stage to the spatial size of the refined feature map in the current stage, adding it to the refined feature map of the current stage, and then performing channel adjustment and feature integration using lightweight projection blocks to obtain the cross-scale fused feature map of the current stage. After completing the fusion of the third, second, and first stages in sequence, the cross-scale fused feature map corresponding to the first stage is used as the final cross-scale fused feature map; global average pooling is then performed on the final cross-scale fused feature map to obtain instance-level ultrasound features.
[0044] In this embodiment, for the first The first in the historical follow-up 100 ultrasound images The residual network outputs feature maps for four stages, represented as follows:
[0045] ;
[0046] in, In the residual network, the first... Feature maps output by the stage ,Right now This is the first stage feature map. This is the second-stage feature map. This is the third stage feature map. This is the fourth stage feature map. The feature maps output from the four stages have different spatial scales and semantic levels. Lower-level features preserve texture and structural details, while higher-level features form stronger semantic representations.
[0047] Alternating refinement is performed on the feature maps of the four stages output by the residual network. Convolutional block attention is applied to the first and third stage feature maps, while reliability-guided feature enhancement is applied to the second and fourth stage feature maps, as shown below:
[0048] ;
[0049] in, This is the refined feature map for stage s. This indicates attention processing for convolutional blocks. This indicates reliability-guided feature enhancement processing. Using the refined feature map from the fourth stage as the initial fused feature map, upsampling, summation, and projection are performed progressively from the third stage to the first stage:
[0050] ;
[0051] in, Indicates upsampling, This represents the lightweight projection in stage s. This represents the cross-scale fusion feature map corresponding to the first stage; This represents the cross-scale fusion feature maps corresponding to the second to fourth stages.
[0052] The cross-scale fusion feature map corresponding to the first stage is used as the final cross-scale fusion feature map, and instance-level ultrasound features are obtained through global average pooling:
[0053] ;
[0054] In the formula, This represents the final cross-scale fused feature map. Indicates global average pooling. For the first The first historical follow-up Instance-level ultrasound features of a single ultrasound image.
[0055] Thus, a four-stage feature refinement is first completed, followed by top-down cross-scale fusion, and a single ultrasound image is converted into a fixed-length instance-level ultrasound feature.
[0056] In this embodiment, the reliability-guided cross-scale feature enhancement performed on the second-stage feature map and the fourth-stage feature map includes the following process: The second-stage feature map (or the fourth-stage feature map) is sequentially subjected to a first pointwise convolution, a depthwise separable convolution, activation processing, and a second pointwise convolution to obtain local enhanced features; the second-stage feature map (or the fourth-stage feature map) is subjected to global average pooling and two learnable mappings, followed by activation processing and logistic function processing to obtain channel reliability weights; the second-stage feature map (or the fourth-stage feature map) is subjected to convolution processing, activation processing, pointwise convolution, and logistic function processing to obtain spatial reliability weights; the channel reliability weights are extended along the spatial dimension, and the spatial reliability weights are extended along the channel dimension; the extended channel reliability weights and spatial reliability weights are multiplied element-wise to obtain joint reliability modulation weights; the joint reliability modulation weights are used to element-wise weight the local enhanced features to suppress local responses with low reliability at channel or spatial locations; the element-wise weighted local enhanced features are then added to the second-stage feature map (or the fourth-stage feature map) through residual connections to obtain reliability enhanced features, i.e., refined feature maps.
[0057] like Figure 2 As shown, in the process of implementing reliability-guided feature enhancement, the second-stage feature map (or fourth-stage feature map) is received first. ; Execute first through local enhancement branches Convolution, then execute Depthwise separable convolution and activation processing, followed by execution Convolution, forming local enhancement features , represented as:
[0058] ;
[0059] in, express convolution, express Depthwise separable convolution, This represents the activation function. Weak texture variations and local structural responses are extracted through local enhancement branches.
[0060] Simultaneously, global average pooling is performed on the input second-stage feature map (or fourth-stage feature map) through the channel reliability branch, summarizing the spatial dimensions into a global description, and then forming the channel reliability weights through two fully connected layers and an activation function. , represented as:
[0061] ;
[0062] in, Indicates global average pooling. and This represents two learnable fully connected layers. This represents the activation function (e.g., the Sigmoid function). Channel reliability weights. The dimension is It is used to enhance channels with high information content and reduce the weight of unstable channels.
[0063] In the spatial reliability branch, the second-stage feature map (or the fourth-stage feature map) is sequentially processed... Convolution, activation function Convolution and activation functions (such as the sigmoid function) are used to form spatial reliability weights. , represented as:
[0064] ;
[0065] Among them, spatial reliability weight The dimension is This is used to determine the reliable spatial region in the second-stage feature map (or fourth-stage feature map). The joint reliability modulation weight is obtained by multiplying the channel reliability weight and spatial reliability weight element-wise after dimensional expansion. :
[0066] ;
[0067] The local enhancement features are modulated using joint reliability modulation weights and then added to the residuals of the input second-stage (or fourth-stage) feature map to obtain the reliability enhancement features. :
[0068] ;
[0069] in, and Both represent element-wise multiplication. The joint reliability modulation weights require that the local response simultaneously obtain channel reliability and spatial reliability support, and the residuals are summed to preserve the original information in the second or fourth stage feature maps of the input.
[0070] The reliability enhancement features of the second stage and the reliability enhancement features of the fourth stage are used as the refined feature maps of the second stage and the fourth stage, respectively.
[0071] S3. Perform self-attention-gated multi-instance aggregation on the instance-level ultrasound features corresponding to the same historical follow-up to obtain the follow-up-level ultrasound representation.
[0072] In this embodiment, the self-attention gating multi-instance aggregation includes: determining instance attention scores for each instance-level ultrasound feature corresponding to the same historical follow-up; normalizing the instance attention scores for the same historical follow-up to obtain instance aggregation weights; weighting and summing each instance-level ultrasound feature according to the instance aggregation weights to obtain weighted multi-view features; and adding the linear projection of the weighted multi-view features to the mean of the instance-level ultrasound features corresponding to the same historical follow-up to obtain the follow-up-level ultrasound representation.
[0073] like Figure 3 As shown, for the first This historical follow-up will The instance-level ultrasound features of the ultrasound images are stacked row-wise into a multi-view feature matrix:
[0074] ;
[0075] In the formula, For the first The first historical follow-up Instance-level ultrasound features of a single ultrasound image. .
[0076] To establish the correlation between different ultrasound cases within the same historical follow-up, a multi-view feature matrix was used. The query features are obtained by mapping the data to the query space, key space, and value space using three linear transformations, respectively. Key features Sum value characteristics The scaled dot product of query features and key features is used to measure the relevance between different perspectives, while value features are used to provide cross-perspective context. The attention output is represented as follows:
[0077] ;
[0078] in, Represents the attention dimension. Represents the normalized exponential function, The symbol "" represents the transpose of the key feature. " represents division operation, symbol " "" indicates the square root operation. This process allows each ultrasound instance to retain its own information while incorporating the context provided by the relevant perspective.
[0079] Output attention Compared with the original multi-view feature matrix Residual summation and layer normalization are performed, followed by a multilayer perceptron, a second residual summation, and layer normalization to obtain the refined multi-view feature matrix. :
[0080] ;
[0081] In the formula, Representation layer normalization, Indicates multi-layered perception. This represents the multi-view matrix features after residual summation and layer normalization. The multi-view feature matrix is then refined. The intermediate representation obtained after attention and residual processing and multi-layer perception mainly optimizes the features of each viewpoint to make it more suitable for subsequent association modeling.
[0082] The refined multi-view feature matrix is formed by performing same-follow-up association modeling through self-attention. :
[0083] .
[0084] In the formula, For the first Each refined instance feature; It also includes its own instance information and contextual information provided by other relevant perspectives in the same follow-up.
[0085] In this embodiment, same-follow-up association modeling refers to using instance-level ultrasound features corresponding to different ultrasound perspectives as self-attention inputs within the same historical follow-up scope. By calculating the similarity between query features and key features, the correlation weight between different instances is calculated, and the value features are weighted and aggregated using the correlation weight, so that each instance-level ultrasound feature retains its own information while introducing contextual information provided by relevant perspectives within the same follow-up.
[0086] Regarding the first Refined instance features The gating response is formed through hyperbolic tangent function branching and activation function (Sigmoid function) branching, and the instance attention score is obtained through linear mapping. :
[0087] ;
[0088] In the formula, Represents the hyperbolic tangent function. This represents the activation function. This indicates element-wise multiplication. This represents the learnable transpose weight vector in instance attention computation. This represents the learnable mapping matrix in the hyperbolic tangent function branch. This represents the learnable mapping matrix in the activation function branch.
[0089] The attention scores of all instances in the same follow-up are normalized to obtain the instance aggregation weight. and refined instance features Weighted summation yields weighted multi-view features. :
[0090] ;
[0091] The weighted multi-view features, after linear projection, are compared with the original multi-view feature matrix. The mean values are added together to form the follow-up ultrasound representation. :
[0092] ;
[0093] In the formula, Represents a linear projection function. This indicates that the average value has been applied.
[0094] After averaging the original multi-view feature matrix, the original set mean is obtained. The weighted features after linear projection are added to the original set mean to achieve residual connection, thereby preserving the original set information of the same follow-up. The instance aggregation weight is used to reduce the effect of instances with low information content or affected by imaging artifacts.
[0095] S4. Based on the specific follow-up time of each historical follow-up and the relative time interval between different historical follow-ups, the follow-up-level ultrasound representation is encoded with interval sensing location to obtain the interval sensing time representation.
[0096] This embodiment achieves interval-aware location encoding through interval-aware longitudinal modeling, including: determining the difference between the specific follow-up times of any two historical follow-ups as the relative time interval between the two historical follow-ups; mapping the relative time interval to a time bias through a learnable interval encoding function; adding the time bias to the multi-head attention operation of the time encoding block; and sequentially passing through three time encoding blocks to obtain an interval-aware time representation containing ultrasound information and relative time interval information.
[0097] like Figure 4 As shown, the follow-up ultrasound data from each historical follow-up visit indicates... and specific follow-up time Implement interval-aware position encoding. For the first... The second historical follow-up and the first This historical follow-up will involve relative time intervals. Defined as:
[0098] ;
[0099] First, the follow-up ultrasound representation is projected onto the time coding space to obtain the initial time representation. :
[0100] ;
[0101] in, This represents a learnable bias vector in the temporal encoding space, used to adjust the baseline position of the initial temporal representation.
[0102] In the l-th time coding block, the specific follow-up time is... Input a Time-Conditional Feature Linear Modulation (Time-FiLM) network to generate scaling parameters corresponding to the feature dimensions. and offset parameters :
[0103] ;
[0104] in, This refers to the Time-FiLM network in the l-th time-coded block, used to transmit the follow-up time. Mapped to scaling parameters and offset parameters .
[0105] Scaling and offset parameters are used to perform differential modulation on follow-up features at different real times. Meanwhile, the relative time interval... Input the learnable interval coding function of the l-th time coding block to form the l-th time coding block. The second historical follow-up and the first Time bias between historical follow-ups :
[0106] ;
[0107] in, This represents the learnable relative time interval coding function in the l-th time coding block, used to convert the time interval... Mapped to attention time bias .
[0108] Temporal bias is incorporated into the multi-head attention computation, ensuring that the attention weights are influenced by both the follow-up-level ultrasound representation and the actual follow-up time interval. Each temporal coding block also includes layer normalization, a multilayer perceptron, and residual connections. After three temporal coding blocks, the... Interval-based time representation of historical follow-up for:
[0109] .
[0110] in, Indicates the first The hidden state obtained after the third time-coded block of the historical follow-up is the interval-aware time representation. .
[0111] S5. The follow-up-level ultrasound representation and interval-aware time representation of each historical follow-up are fused to obtain the fused historical follow-up sequence. The fused historical follow-up sequence is subjected to longitudinal dependency modeling, and the prediction result of Hashimoto's thyroiditis status at the target follow-up time point is output based on the sequence representation of the last historical follow-up obtained from the longitudinal dependency modeling.
[0112] In this embodiment, the longitudinal dependency modeling includes: concatenating the follow-up-level ultrasound representation of the same historical follow-up with the interval sensing time representation and performing linear projection to obtain a unified historical follow-up representation; assembling the unified historical follow-up representations into a historical follow-up sequence according to the follow-up order; encoding the historical follow-up sequence through a Transformer encoder, and inputting the encoding result corresponding to the last historical follow-up into a multilayer perceptron classifier and a normalized exponential function to obtain the predicted probability of the target follow-up time point being positive or negative for Hashimoto's thyroiditis.
[0113] The follow-up ultrasound at the same historical follow-up period is represented as follows. With interval sensing time representation By concatenating the data and obtaining a unified representation of historical follow-ups through linear mapping, a unified representation of the data can be obtained. :
[0114] ;
[0115] in, The weight matrix represents the linear mapping. This indicates a splicing operation. This represents the bias vector of a linear mapping.
[0116] Each historical follow-up visit is represented in chronological order to form a historical follow-up sequence. :
[0117] ;
[0118] in, This indicates a unified approach to all historical follow-up visits.
[0119] The Transformer encoder is used to establish the vertical dependencies between different historical follow-ups, and the historical follow-up sequence representation is output. :
[0120] ;
[0121] in, This indicates the Transformer encoding operation.
[0122] The sequence representation corresponding to the last historical follow-up is input into a multilayer perceptron classifier, and the state prediction probability at the target follow-up time point is obtained through normalization processing:
[0123] ;
[0124] in, This is a normalization operation; This represents the Transformer encoded output corresponding to the last historical follow-up, i.e., the historical follow-up sequence representation; This represents the predicted state probability at the target follow-up time point.
[0125] The above-mentioned instance-level ultrasound feature extraction, self-attention gating multi-instance aggregation, interval-aware longitudinal modeling, and follow-up sequence fusion, i.e., longitudinal dependency modeling, together constitute a closed-loop processing and prediction process from multi-view ultrasound images and time information to the target disease state.
[0126] Preferably, this embodiment trains and validates the model. The private longitudinal thyroid dataset contains 2726 patients, including 1452 Hashimoto's thyroiditis-negative patients and 1274 positive patients, with each patient having 3 to 7 follow-ups. The external validation dataset contains 1088 patients, including 754 negative patients and 334 positive patients. Both datasets have corresponding medical ethics review approvals.
[0127] The model uses a pre-trained 50-layer residual network (ResNet) for feature extraction and employs an adaptive moment estimation (AdamW) optimizer with weight decay of 1×10⁻⁶. -6 The initial learning rate is 1×10. -5 And the learning rate was reduced to 1×10 using cosine annealing. -6 The model was trained for 80 epochs with a batch size of 2.
[0128] In five-fold cross-validation on a private longitudinal thyroid dataset, the complete model achieved 87.69% accuracy, 93.21% area under the curve, 90.69% precision, 85.28% recall, 86.92% F1 score, and 88.14% specificity. These results demonstrate that the complete processing chain can achieve good longitudinal prediction performance under real-world data and training conditions.
[0129] When grouped by the number of follow-ups, the area under the curve for each group with 3 to 7 follow-ups exceeded 91%, and the accuracy exceeded 84%. This result indicates that the model can handle historical sequences with differences in the number of follow-ups and follow-up intervals.
[0130] Specifically, the accuracy and area under the curve (AUC) were 86.59% and 93.32% for the 3-follow-up group, 90.89% and 94.34% for the 4-follow-up group, 87.32% and 92.82% for the 5-follow-up group, 84.76% and 91.32% for the 6-follow-up group, and 85.29% and 95.75% for the 7-follow-up group. These results for different follow-up numbers demonstrate that the model can utilize historical sequences of varying lengths.
[0131] In external validation, the model achieved 85.73% accuracy, 87.29% area under the curve, 76.34% precision, 77.81% recall, and 86.98% specificity. External validation is only intended to illustrate the model's generalization performance under cross-center data conditions and does not constitute a guarantee of clinical performance.
[0132] Subject characteristics included age range, sex composition, and follow-up years. Private longitudinal thyroid data covered 2018 to 2024, with patients aged 22 to 91 years; external validation data covered 2018 to 2025, with patients aged 21 to 93 years. This information is used to illustrate the experimental sample range and is not intended to limit the input of the method or the applicable population.
[0133] In the ablation experiments, the accuracy and area under the curve of the baseline model were 80.74% and 84.86%, respectively; after adding reliability-guided feature enhancement, the corresponding metrics were 81.26% and 85.64%; after adding self-attention gating multi-instance learning, the corresponding metrics were 82.69% and 87.18%; after combining relevant modules and adding interval-aware position encoding, the complete model achieved 87.69% and 93.21%.
[0134] Further experiments with module combinations showed that the baseline model, when incorporating both reliability-guided feature enhancement and self-attention-gated multi-instance learning, achieved accuracy and area under the curve (AUC) of 83.93% and 89.27%, respectively; the combination of reliability-guided feature enhancement and interval-aware location encoding achieved 84.28% and 89.54%, respectively; and the combination of self-attention-gated multi-instance learning and interval-aware location encoding achieved 86.05% and 91.58%, respectively. The complete combination achieved the highest overall results among the listed metrics.
[0135] The branch ablation results of reliability-guided feature enhancement indicate that local enhancement, channel reliability, and spatial reliability each contribute to feature refinement, and the combined effect of these three factors yields a complete model result. The ablation results of multi-instance aggregation demonstrate that gated aggregation, same-follow-up association modeling, and mean residual projection together form a stable follow-up-level ultrasound representation.
[0136] In the reliability-guided feature enhancement branch experiments, the areas under the curve (AUC) were 91.86%, 91.27%, and 92.14% when using only local enhancement, only channel reliability, and only spatial reliability, respectively, reaching 93.21% when all three were combined. In the multi-instance aggregation component experiments, the AUC for different component combinations ranged from 91.23% to 91.89%, reaching 93.21% when self-attention, gated aggregation, and residual projection were all enabled. These results support the complementary relationship between the components.
[0137] Based on the same inventive concept, this embodiment also provides a longitudinal prediction system for Hashimoto's thyroiditis, such as... Figure 5 As shown, it includes:
[0138] The data acquisition module is used to acquire multiple historical follow-up records of patients before the target follow-up. Each historical follow-up record includes multiple multi-view thyroid ultrasound images and the corresponding follow-up time.
[0139] The cross-scale feature enhancement module performs reliability-guided cross-scale feature enhancement on each multi-view thyroid ultrasound image to obtain instance-level ultrasound features.
[0140] The multi-instance aggregation module performs self-attention-gated multi-instance aggregation on instance-level ultrasound features corresponding to the same historical follow-up to obtain a follow-up-level ultrasound representation.
[0141] The interval sensing coding module performs interval sensing location coding on the follow-up ultrasound representation based on the specific follow-up time of each historical follow-up and the relative time interval between different historical follow-ups, to obtain the interval sensing time representation.
[0142] The longitudinal prediction module integrates the follow-up-level ultrasound representation and interval-aware time representation of each historical follow-up to obtain the fused historical follow-up sequence; it performs longitudinal dependency modeling on the fused historical follow-up sequence and outputs the prediction result of Hashimoto's thyroiditis status at the target follow-up time point based on the sequence representation corresponding to the last historical follow-up obtained from the longitudinal dependency modeling.
[0143] The data acquisition module, cross-scale feature enhancement module, multi-instance aggregation module, interval-aware coding module, and longitudinal prediction module in the system correspond to the steps of the longitudinal prediction method in this embodiment. For detailed implementation process, please refer to the aforementioned steps.
[0144] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.
Claims
1. A longitudinal prediction method for Hashimoto's thyroiditis based on interval-aware multi-instance learning, characterized in that, Includes the following steps: S1. Obtain multiple historical follow-up records of the patient before the target follow-up. Each historical follow-up record includes multiple multi-view thyroid ultrasound images and the corresponding follow-up time. S2. Reliability-guided cross-scale feature enhancement is performed on each multi-view thyroid ultrasound image to obtain instance-level ultrasound features; S3. Perform self-attention-gated multi-instance aggregation on the instance-level ultrasound features corresponding to the same historical follow-up to obtain the follow-up-level ultrasound representation; S4. Based on the specific follow-up time of each historical follow-up and the relative time interval between different historical follow-ups, the follow-up ultrasound representation is encoded by interval sensing location to obtain the interval sensing time representation. S5. The follow-up-level ultrasound representation and interval-aware time representation of each historical follow-up are fused to obtain the fused historical follow-up sequence. The fused historical follow-up sequence is subjected to longitudinal dependency modeling, and the prediction result of Hashimoto's thyroiditis status at the target follow-up time point is output based on the sequence representation of the last historical follow-up obtained from the longitudinal dependency modeling.
2. The longitudinal prediction method for Hashimoto's thyroiditis according to claim 1, characterized in that, In step S2, feature maps of the first, second, third, and fourth stages are extracted using a residual network. Convolutional block attention processing is performed on the first and third stage feature maps to enhance the channel response and spatial region response related to changes in thyroid structure and texture, resulting in refined feature maps for the corresponding stages. Reliability-guided cross-scale feature enhancement processing is performed on the second and fourth stage feature maps, resulting in refined feature maps for the second and fourth stages. The refined feature map of the fourth stage is used as the initial fusion feature map, and cross-scale fusion is performed in order from high to low layers. Global average pooling is then performed on the cross-scale fusion feature map corresponding to the first stage to obtain instance-level ultrasound features.
3. The longitudinal prediction method for Hashimoto's thyroiditis according to claim 2, characterized in that, The cross-scale fusion, performed in descending order of layer height, specifically includes: After upsampling the fusion feature map obtained in the higher stage to the spatial size of the current stage refined feature map, it is added to the current stage refined feature map, and channel adjustment and feature integration are performed through lightweight projection blocks to obtain the current stage cross-scale fusion feature map. After completing the fusion of the third, second and first stages in sequence, the cross-scale fusion feature map corresponding to the first stage is taken as the final cross-scale fusion feature map; global average pooling is performed on the final cross-scale fusion feature map to obtain instance-level ultrasound features.
4. The longitudinal prediction method for Hashimoto's thyroiditis according to claim 3, characterized in that, Using the refined feature map from the fourth stage as the initial fused feature map, upsampling, summation, and projection are performed progressively from the third stage to the first stage: ； in, Indicates upsampling, This represents the lightweight projection in stage s. This represents the cross-scale fusion feature map corresponding to the first stage; This represents the cross-scale fusion feature maps corresponding to the second to fourth stages; This is the refined feature map for stage s; The cross-scale fusion feature map corresponding to the first stage is used as the final cross-scale fusion feature map, and instance-level ultrasound features are obtained through global average pooling: ； In the formula, This represents the final cross-scale fused feature map. Indicates global average pooling. For the first The first historical follow-up Instance-level ultrasound features of a single ultrasound image.
5. The longitudinal prediction method for Hashimoto's thyroiditis according to claim 2, characterized in that, Reliability-guided cross-scale feature enhancement performed on the second-stage and fourth-stage feature maps. Includes the following processes: The feature map is sequentially subjected to a first pointwise convolution, a depthwise separable convolution, activation processing, and a second pointwise convolution to obtain local enhanced features. The feature map is subjected to global average pooling and two learnable mappings, followed by activation and logistic function processing to obtain channel reliability weights. The feature map is then subjected to convolution, activation, pointwise convolution, and logistic function processing to obtain spatial reliability weights. The channel reliability weights are extended along the spatial dimension, and the spatial reliability weights are extended along the channel dimension. The extended channel reliability weights and spatial reliability weights are multiplied element-wise to obtain joint reliability modulation weights. These joint reliability modulation weights are then used to element-wise weight local enhancement features to suppress local responses with low reliability at channel or spatial locations. Finally, the element-wise weighted local enhancement features are added to the feature map via residual connections to obtain a refined feature map.
6. The longitudinal prediction method for Hashimoto's thyroiditis according to claim 1, characterized in that, Step S3, the self-attention gated multi-instance aggregation, includes: For each instance-level ultrasound feature corresponding to the same historical follow-up, an instance attention score is determined; the instance attention scores of the same historical follow-up are normalized to obtain instance aggregation weights; the instance-level ultrasound features are weighted and summed according to the instance aggregation weights to obtain weighted multi-view features; the linear projection of the weighted multi-view features is added to the mean of the instance-level ultrasound features corresponding to the same historical follow-up to obtain the follow-up-level ultrasound representation.
7. The longitudinal prediction method for Hashimoto's thyroiditis according to claim 6, characterized in that, For the This historical follow-up will The instance-level ultrasound features of the ultrasound images are stacked row-wise into a multi-view feature matrix: ； In the formula, For the first The first historical follow-up Instance-level ultrasound features of a single ultrasound image. ; Multi-view feature matrix The query features are obtained by mapping the data to the query space, key space, and value space using three linear transformations, respectively. Key features Sum value characteristics ; The scaled dot product of query features and key features is used to measure the relevance between different perspectives, the value features are used to provide cross-perspective context, and the attention output is represented as: ； in, Represents the attention dimension. Represents the normalized exponential function, The symbol "" represents the transpose of the key feature. "Indicates division operation, symbol " " indicates the square root operation; Output attention With multi-view feature matrix Residual summation and layer normalization are performed, followed by a multilayer perceptron, a second residual summation, and layer normalization to obtain the refined multi-view feature matrix. : ； In the formula, Representation layer normalization, Indicates multi-layered perception. The features of the multi-view matrix after residual summation and layer normalization; After performing same-perspective follow-up association modeling through self-attention, a refined multi-perspective feature matrix is formed. : ； In the formula, For the first Each refined instance feature; It also includes its own instance information and contextual information provided by other relevant perspectives in the same follow-up; Regarding the first Refined instance features The gating response is formed through hyperbolic tangent function branches and activation function branches, and the instance attention score is obtained through linear mapping. : ； In the formula, Represents the hyperbolic tangent function. This represents the activation function. This indicates element-wise multiplication. This represents the learnable transpose weight vector in instance attention computation. This represents the learnable mapping matrix in the hyperbolic tangent function branch. This represents the learnable mapping matrix in the activation function branch; The attention scores of all instances in the same follow-up are normalized to obtain the instance aggregation weight. and refined instance features Weighted summation yields weighted multi-view features. : ； The weighted multi-view features, after linear projection, are compared with the original multi-view feature matrix. The mean values are added together to form the follow-up ultrasound representation. : ； In the formula, Represents a linear projection function. This indicates that the average value has been applied.
8. The longitudinal prediction method for Hashimoto's thyroiditis according to claim 1, characterized in that, Step S4, the interval-aware location coding, includes: The difference between the specific follow-up times of any two historical follow-ups is determined as the relative time interval between the two historical follow-ups; the relative time interval is mapped to a time bias through a learnable interval coding function; the time bias is added to the multi-head attention operation of the time coding block; and after passing through three time coding blocks in sequence, an interval-sensing time representation containing ultrasound information and relative time interval information is obtained. Step S5, the vertical dependency modeling, includes: The follow-up ultrasound representation of the same historical follow-up visit is concatenated with the interval sensing time representation and linearly projected to obtain a unified historical follow-up representation. The unified historical follow-up representations are then combined into a historical follow-up sequence according to the follow-up order. The historical follow-up sequence is encoded using a Transformer encoder, and the encoding result corresponding to the last historical follow-up visit is input into a multilayer perceptron classifier and a normalized exponential function to obtain the predicted probability of the target follow-up time point being positive or negative for Hashimoto's thyroiditis.
9. The longitudinal prediction method for Hashimoto's thyroiditis according to claim 8, characterized in that, For the The second historical follow-up and the first This historical follow-up will involve relative time intervals. Defined as: ； First, the follow-up ultrasound was presented. Projecting onto the time encoding space yields the initial time representation. : ； in, This represents a learnable bias vector in the temporal encoding space, used to adjust the baseline position of the initial temporal representation; In the l-th time coding block, the specific follow-up time is... Input a temporal conditional feature linear modulation network to generate scaling parameters corresponding to the feature dimensions. and offset parameters : ； in, This represents the temporal conditional feature linear modulation network in the l-th temporal coding block, used to control the follow-up time. Mapped to scaling parameters and offset parameters ; relative time interval Input the learnable interval coding function of the l-th time coding block to form the l-th time coding block. The second historical follow-up and the first Time bias between historical follow-ups : ； in, This represents the learnable relative time interval coding function in the l-th time coding block; Incorporating temporal bias into multi-head attention computation ensures that attention weights are simultaneously influenced by both follow-up ultrasound representation and actual follow-up time intervals; after three time-coding blocks, the... Interval-based time representation of historical follow-up for: ； in, Indicates the first The hidden state obtained after the third time-coded block of the next historical follow-up.
10. A longitudinal prediction system for Hashimoto's thyroiditis based on interval-aware multi-instance learning, implemented according to the method of any one of claims 1-9, characterized in that, The system includes: The data acquisition module is used to acquire multiple historical follow-up records of the patient before the target follow-up. Each historical follow-up record includes multiple multi-view thyroid ultrasound images and the corresponding follow-up time. The cross-scale feature enhancement module performs reliability-guided cross-scale feature enhancement on each multi-view thyroid ultrasound image to obtain instance-level ultrasound features. The multi-instance aggregation module performs self-attention-gated multi-instance aggregation on instance-level ultrasound features corresponding to the same historical follow-up to obtain a follow-up-level ultrasound representation. The interval sensing coding module performs interval sensing location coding on the follow-up ultrasound representation based on the specific follow-up time of each historical follow-up and the relative time interval between different historical follow-ups, to obtain the interval sensing time representation. The longitudinal prediction module integrates the follow-up-level ultrasound representation and interval-aware time representation of each historical follow-up to obtain the fused historical follow-up sequence; it performs longitudinal dependency modeling on the fused historical follow-up sequence and outputs the prediction result of Hashimoto's thyroiditis status at the target follow-up time point based on the sequence representation corresponding to the last historical follow-up obtained from the longitudinal dependency modeling.