Bridge-based longitudinal prediction method for hashimoto's thyroiditis based on cross-modal dual-graph interaction learning

Abstract

The application belongs to the technical field of image data processing, and is a Hashimoto's thyroiditis longitudinal prediction method based on cross-modal double graph interaction learning, comprising the following steps: obtaining historical follow-up data corresponding to a target object at multiple historical follow-up time points; sequentially performing multi-receptive field feature extraction, channel and spatial attention enhancement and double pooling fusion on multi-view thyroid ultrasound images of each historical follow-up time point to obtain enhanced ultrasound features of each ultrasound view, and performing self-attention processing and attention-based weighted aggregation to obtain follow-up level ultrasound representation; constructing a longitudinal prediction model comprising a cross-modal double graph encoding unit, a double graph interaction fusion unit and a longitudinal prediction unit, and training the model; predicting the real-time input multi-view thyroid ultrasound images by using the trained model to output Hashimoto's thyroiditis state prediction results of the target object at a target follow-up time point. The application improves the accuracy and robustness of Hashimoto's thyroiditis prediction.

Description

Technical Field

[0001] This invention belongs to the field of image data processing technology, specifically relating to a longitudinal prediction method for Hashimoto's thyroiditis based on cross-modal dual-image interactive learning. Background Technology

[0002] Hashimoto's thyroiditis (HT) is a common autoimmune thyroid disease with a typically slow progression and often subtle early clinical symptoms and imaging findings. As the disease progresses, chronic inflammation, follicular cell damage, and thyroid dysfunction may occur in the thyroid tissue. Therefore, identifying potential disease progression trends from multiple patient follow-up data is crucial for early risk assessment and management of Hashimoto's thyroiditis.

[0003] Currently, the clinical assessment of Hashimoto's thyroiditis mainly relies on thyroid ultrasound and biochemical marker tests. Thyroid ultrasound images can reflect changes in the texture, echogenicity, and morphology of thyroid tissue, and are characterized by being non-invasive, convenient, and repeatable. Biochemical markers can provide information on thyroid function and autoimmune status. However, ultrasound abnormalities and biochemical abnormalities do not always occur simultaneously, and relying on only one type of examination data is insufficient to comprehensively characterize the disease status of Hashimoto's thyroiditis. Therefore, it is necessary to comprehensively utilize ultrasound images, biochemical markers, and time information obtained during multiple follow-up visits for longitudinal prediction.

[0004] In scenarios with irregular follow-up, longitudinal prediction of Hashimoto's thyroiditis still faces the following challenges:

[0005] 1) The problem of extracting subtle multi-scale ultrasound features. Early Hashimoto's thyroiditis often presents as slight texture changes and echogenic abnormalities in ultrasound images, lacking obvious focal lesion features, and these changes may appear at different spatial scales. Using a single receptive field for feature extraction makes it difficult to simultaneously capture both local subtle textures and large-scale echogenic changes. Furthermore, a single follow-up visit typically includes ultrasound images from multiple scanning sections, and the effective information and diagnostic relevance contained in different images vary. Simply averaging or max pooling multiple images fails to fully utilize the correlation information between different views.

[0006] 2) The problem of cross-modal heterogeneous information fusion. Thyroid ultrasound images are high-dimensional visual data, while biochemical indicators such as anti-thyroid peroxidase antibody, anti-thyroglobulin antibody, free triiodothyronine, free thyroxine, and thyroid-stimulating hormone are low-dimensional numerical data. There are significant differences between the two in terms of data structure, feature dimension, and semantic expression. Existing feature splicing or attention fusion methods typically integrate multimodal information implicitly, making it difficult to simultaneously characterize the direct relationship between ultrasound features and various biochemical indicators, as well as the global and higher-order correlations among all multimodal features.

[0007] 3) Modeling problems with irregular follow-up times. The number of follow-up visits and the time intervals between adjacent follow-up time points are usually inconsistent among different patients. Existing longitudinal prediction methods mostly perform sequence modeling of each examination data according to the follow-up order, which easily treats adjacent follow-up records with different time spans as equally spaced observations. This makes it difficult to accurately characterize the rate of change and evolution trend of the disease at different time intervals, thus affecting the accuracy and stability of the prediction of the state at the target follow-up time point.

[0008] Existing technologies struggle to simultaneously address challenges such as subtle multi-scale ultrasound feature extraction, heterogeneous multimodal information interaction, and modeling irregular follow-up times, resulting in underutilization of complementary and temporal evolution information within historical follow-up data. While some studies have attempted to employ deep learning methods to analyze thyroid ultrasound images or fuse clinical indicators, these methods are mostly geared towards single-examination or equally spaced follow-up scenarios, primarily focusing on single-image recognition, simple multimodal feature fusion, or general sequence modeling. These approaches fall short of meeting the needs for predicting the longitudinal status of Hashimoto's thyroiditis in irregular follow-up scenarios. Summary of the Invention

[0009] To address the problems existing in the prior art, this invention proposes a longitudinal prediction method and system for Hashimoto's thyroiditis based on cross-modal dual-graph interactive learning. This method comprehensively utilizes multi-view thyroid ultrasound images, clinical variables, and time intervals between adjacent follow-up points obtained from multiple historical follow-up time points for the target subject. Through multi-receptive field ultrasound feature enhancement, self-attention multi-instance aggregation, dual-graph interactive fusion, and time interval-aware longitudinal modeling, it extracts ultrasound image features and clinical variable features from different follow-up time points, establishes direct and higher-order associations between different modal data, and explicitly encodes irregular follow-up time information, thereby obtaining the predicted Hashimoto's thyroiditis status of the target subject at the future target follow-up time point.

[0010] On one hand, embodiments of the present invention provide a longitudinal prediction method for Hashimoto's thyroiditis based on cross-modal dual-graph interactive learning, comprising the following steps:

[0011] S1. Obtain historical follow-up data for the target subject at multiple historical follow-up time points prior to the target follow-up time point; wherein, the historical follow-up data for each historical follow-up time point includes multi-view thyroid ultrasound images, clinical variables, and time intervals between adjacent historical follow-up time points;

[0012] S2. For each historical follow-up time point, multi-receptive field feature extraction, channel and spatial attention enhancement, and dual pooling fusion are performed sequentially on the multi-view thyroid ultrasound images to obtain the enhanced ultrasound features of each ultrasound view; self-attention processing and attention-based multi-instance weighted aggregation are performed on the enhanced ultrasound features of each ultrasound view at the same historical follow-up time point to obtain the corresponding follow-up-level ultrasound representation.

[0013] S3. Construct a longitudinal prediction model, which includes:

[0014] A cross-modal dual-graph encoding unit is used to normalize or numerically encode clinical variables, mapping follow-up-level ultrasound representations to ultrasound nodes and each clinical variable to clinical variable nodes. Based on the ultrasound nodes and clinical variable nodes, local and global graphs are constructed, and graph attention encoding is performed on both to obtain enhanced node representations for the local and global graphs. The enhanced node representations for the local graphs include both ultrasound node representations and clinical variable node representations, while the enhanced node representations for the global graphs include both ultrasound node representations and clinical variable node representations.

[0015] The dual-graph interactive fusion unit includes an ultrasound branch and a clinical variable branch. The ultrasound branch performs a weighted fusion based on the ultrasound node representations of the local graph and the global graph to obtain a fused ultrasound representation. The clinical variable branch performs a weighted fusion based on the clinical variable node representations of the local graph and the global graph to obtain a fused clinical variable representation. The fused ultrasound representation and the fused clinical variable representation are concatenated along the feature dimension to obtain a follow-up-level multimodal representation of the current historical follow-up time point.

[0016] The longitudinal prediction unit is used to embed and encode the time interval between each historical follow-up time point, embed and encode the time interval obtained by embedding and encode it and fuse it with the corresponding follow-up-level multimodal representation, and perform gated loop modeling according to the time order of each historical follow-up time point; based on the hidden state corresponding to the last historical follow-up time point obtained by the gated loop processing, the prediction result of the Hashimoto's thyroiditis status of the target object at the target follow-up time point is output.

[0017] S4. Train the constructed longitudinal prediction model;

[0018] S5. For real-time input multi-view thyroid ultrasound images, use the trained longitudinal prediction model to make predictions.

[0019] On the other hand, based on the same inventive concept, embodiments of the present invention also provide a longitudinal prediction system for Hashimoto's thyroiditis based on cross-modal dual-graph interactive learning, comprising:

[0020] The data acquisition unit is used to acquire historical follow-up data of the target object at multiple historical follow-up time points before the target follow-up time point; wherein, the historical follow-up data of each historical follow-up time point includes multi-view thyroid ultrasound images, clinical variables, and time intervals between adjacent historical follow-up time points;

[0021] The ultrasound feature enhancement and aggregation unit is used to sequentially perform multi-receptive field feature extraction, channel and spatial attention enhancement, and dual pooling fusion on multi-view thyroid ultrasound images for each historical follow-up time point to obtain enhanced ultrasound features for each ultrasound view; and to perform self-attention processing and attention-based multi-instance weighted aggregation on the enhanced ultrasound features of each ultrasound view at the same historical follow-up time point to obtain the corresponding follow-up-level ultrasound representation.

[0022] The longitudinal prediction model includes a cross-modal dual-graph coding unit, a dual-graph interactive fusion unit, and a longitudinal prediction unit;

[0023] The cross-modal dual-graph encoding unit is used to normalize or numerically encode clinical variables, mapping the follow-up ultrasound representation to ultrasound nodes and mapping each clinical variable to clinical variable nodes. Based on the ultrasound nodes and clinical variable nodes, local and global graphs are constructed, and graph attention encoding is performed on both to obtain enhanced node representations for the local and global graphs. The enhanced node representations for the local graphs include both ultrasound node representations and clinical variable node representations, while the enhanced node representations for the global graphs include both ultrasound node representations and clinical variable node representations.

[0024] The dual-graph interactive fusion unit includes an ultrasound branch and a clinical variable branch. The ultrasound branch performs a weighted fusion based on the ultrasound node representations of the local graph and the global graph to obtain a fused ultrasound representation. The clinical variable branch performs a weighted fusion based on the clinical variable node representations of the local graph and the global graph to obtain a fused clinical variable representation. The fused ultrasound representation and the fused clinical variable representation are concatenated along the feature dimension to obtain a follow-up-level multimodal representation of the current historical follow-up time point.

[0025] The longitudinal prediction unit is used to embed and encode the time interval between each historical follow-up time point, embed and encode the time interval obtained by embedding and encode it and fuse it with the corresponding follow-up-level multimodal representation, and perform gated loop modeling according to the time order of each historical follow-up time point; based on the hidden state corresponding to the last historical follow-up time point obtained by the gated loop processing, the prediction result of the Hashimoto's thyroiditis status of the target object at the target follow-up time point is output.

[0026] The technical solution of this invention employs a multi-field ultrasound feature enhancement, multi-view adaptive aggregation, cross-modal dual-image interactive fusion, and explicit modeling strategy for irregular time intervals, enabling full utilization of imaging information, clinical variable information, and temporal evolution information from multiple historical follow-up points. Compared to existing technologies, the technical effects achieved by this invention specifically include:

[0027] 1. Multi-receptive field dual-pooling ultrasound feature enhancement

[0028] Early ultrasound findings in Hashimoto's thyroiditis are often subtle, mainly manifesting as weak texture changes and mild echogenicity abnormalities, and these changes may be distributed across different spatial scales. If a single-size convolutional kernel is used for feature extraction, it is difficult to simultaneously characterize local texture details and a large range of echogenic context.

[0029] To address the aforementioned issues, this invention employs multiple parallel convolutional branches with different receptive fields to perform multi-scale processing on the skeletal feature map of thyroid ultrasound images. These parallel convolutional branches include 1×1, 3×3, and 5×5 convolutional branches, as well as dilated convolutional branches with a preset dilation rate. Each branch extracts ultrasound features at different scales and concatenates them along the channel dimension. After obtaining the multi-scale features, this invention sequentially modulates the feature responses using channel attention and spatial attention, and preserves the original feature information through residual connections. Furthermore, global average pooling and global max pooling are performed in parallel: global average pooling preserves overall echo and contextual information, while global max pooling highlights local salient responses. The results of the two pooling operations are then fused to generate enhanced ultrasound features, thereby improving the characterization ability of subtle, diffuse, and multi-scale ultrasound manifestations of Hashimoto's thyroiditis.

[0030] 2. Multi-view ultrasound aggregation based on self-attention multi-instance learning

[0031] A single follow-up examination typically includes thyroid ultrasound images from multiple scanning sections. These different views are complementary, but the degree of correlation between each view and the state of Hashimoto's thyroiditis varies. Average pooling treats all views as equally important, while max pooling may only retain locally strong responses; neither approach fully utilizes the correlation information between different views.

[0032] To address the aforementioned issues, this invention employs a self-attention multi-instance learning strategy to aggregate multi-view ultrasound features at the same historical follow-up time point. First, feature associations between ultrasound views are established through self-attention operations, enabling each view's features to be updated in conjunction with contextual information from other views. Then, normalized importance weights are calculated based on the updated view features, and a weighted sum is performed according to these importance weights to generate a follow-up-level ultrasound representation. By establishing dependencies between different views before weighting, this invention can adaptively utilize complementary information from different ultrasound images within the same follow-up, reducing the interference of ultrasound views with low information content or weak correlation on the follow-up-level representation.

[0033] 3. Cross-modal local-global dual-graph interactive fusion

[0034] Significant differences exist between thyroid ultrasound images and clinical variables in terms of data structure, feature dimensions, and medical semantics. Directly concatenating ultrasound features and clinical variables makes it difficult to explicitly describe the direct relationship between ultrasound manifestations and individual clinical variables, and also makes it difficult to establish global and higher-order relationships between all clinical variables and ultrasound features.

[0035] To address the aforementioned issues, this invention maps follow-up-level ultrasound representations to ultrasound nodes and preprocessed clinical variables to clinical variable nodes. Based on these ultrasound nodes and clinical variable nodes, a local graph and a global graph are constructed. The local graph employs a star topology centered on the ultrasound nodes, connecting each ultrasound node to a specific clinical variable node, representing the direct correlation between ultrasound manifestations and individual clinical variables. The global graph uses a fully connected topology, connecting each ultrasound node and each clinical variable node pairwise, representing the global and higher-order correlations among all multimodal nodes.

[0036] This invention employs a graph attention encoder to encode features in both the local and global graphs, and establishes independent gating fusion mechanisms in the ultrasound and clinical variable branches. The ultrasound branch determines the fusion weights of ultrasound nodes in the local and global graphs based on the follow-up-level ultrasound representation, and further performs interactive fusion through multi-head attention. The clinical variable branch determines the fusion weights of the local and global graphs based on the aggregation results of each clinical variable node, and adaptively fuses the clinical variable node representations in both graphs. Through this approach, this invention can dynamically balance local direct relationships and global higher-order relationships based on the multimodal state at different historical follow-up points, obtaining a follow-up-level multimodal representation that includes both ultrasound and clinical variable information.

[0037] 4. Longitudinal prediction based on the perception of irregular time intervals

[0038] The number of follow-up visits and the time intervals between adjacent follow-up points often vary among different target subjects. Sequence modeling based solely on the follow-up order will treat adjacent follow-up data across different time spans as equally spaced observations, making it difficult to accurately reflect the speed and trend of disease status changes over time.

[0039] To address the aforementioned issues, this invention calculates the time interval between each historical follow-up point and the previous historical follow-up point, and converts this time interval into a time interval embedding through a learnable mapping layer. The time interval embedding is then concatenated and feature-mapped with the follow-up-level multimodal representation of the corresponding historical follow-up point, and input into a gated recurrent unit for longitudinal modeling in real-time order. The gated recurrent unit sequentially updates the hidden state corresponding to each historical follow-up point, using the hidden state of the last historical follow-up point as the longitudinal feature representation of the target object. This longitudinal feature representation is input into a classifier to obtain the positive and negative probabilities of Hashimoto's thyroiditis corresponding to the target follow-up point, thereby outputting a specific prediction result for the Hashimoto's thyroiditis status. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the process of the longitudinal prediction method for Hashimoto's thyroiditis based on cross-modal dual-graph interactive learning in an embodiment of the present invention. Detailed Implementation

[0041] This invention can make full use of multi-view thyroid ultrasound images, clinical variables, and irregular longitudinal follow-up information to improve the accuracy and robustness of Hashimoto's thyroiditis prediction, and provide technical support for early screening and clinical decision support for Hashimoto's thyroiditis.

[0042] The present invention will be further described in detail below with reference to embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto. Equivalent substitutions or adjustments made to the network structure, feature dimensions, training parameters, and data processing methods without departing from the technical concept of the present invention should be included within the scope of protection of the present invention.

[0043] Example

[0044] This embodiment provides a longitudinal prediction method for Hashimoto's thyroiditis based on cross-modal dual-graph interactive learning, which is applicable to irregular follow-up scenarios. The method comprehensively processes multi-view thyroid ultrasound images, clinical variables, and time intervals between adjacent follow-up time points obtained by the target object at multiple historical follow-up time points, and constructs a longitudinal model for sequentially performing multi-receptive field enhanced feature representation, cross-modal dual-graph interactive fusion, and irregular time interval perception, and outputs the prediction result of the target object's Hashimoto's thyroiditis status at the future target follow-up time point.

[0045] like Figure 1 As shown, the specific steps include the following:

[0046] S1. Obtain historical follow-up data for the target subject at multiple historical follow-up time points prior to the target follow-up time point; wherein, the historical follow-up data for each historical follow-up time point includes multi-view thyroid ultrasound images, clinical variables, and time intervals between adjacent historical follow-up time points.

[0047] This step obtains the target object at the target follow-up time point. Previous historical follow-up data, corresponding to multiple historical follow-up time points, should include at least multi-view thyroid ultrasound images, clinical variables (i.e., clinical laboratory indicators), and follow-up time information. This historical follow-up data... Represented as:

[0048] ;

[0049] in, This indicates the number of historical follow-up points prior to the target follow-up point; Indicates a historical follow-up index; Indicates the first A collection of multi-view thyroid ultrasound images at historical follow-up time points; Indicates the first A vector of clinical variables at each historical follow-up time point; Indicates the first The time interval corresponding to each historical follow-up point.

[0050] In this embodiment, the first A collection of multi-view thyroid ultrasound images at historical follow-up time points includes , , , ,in Indicates the first The number of ultrasound images at each historical follow-up time point. Clinical variables included anti-thyroid peroxidase antibody (TPOAb), anti-thyroglobulin antibody (TgAb), free triiodothyronine (FT3), free thyroxine (FT4), thyroid-stimulating hormone (TSH), sex, and age. Continuous clinical variables were normalized, and categorical clinical variables were numerically coded.

[0051] The time interval corresponding to the first historical follow-up point Set to 0; for the time intervals corresponding to the remaining historical follow-up points. , Set as the current historical follow-up point Compared with the previous historical follow-up point The time difference between them.

[0052] S2. For each historical follow-up time point, multi-receptive field feature extraction, channel and spatial attention enhancement, and dual pooling fusion are performed sequentially on the multi-view thyroid ultrasound images to obtain the enhanced ultrasound features of each ultrasound view; self-attention processing and attention-based multi-instance weighted aggregation are performed on the enhanced ultrasound features of each ultrasound view at the same historical follow-up time point to obtain the corresponding follow-up-level ultrasound representation.

[0053] S21. For each historical follow-up time point, a pre-trained backbone network is used to extract ultrasound image feature maps. The ultrasound image feature maps are processed through different convolutional branches to extract local fine texture features and global echo context features. The outputs of each convolutional branch are concatenated by channel dimension, group normalized and linearly activated to obtain multi-scale ultrasound features.

[0054] In this embodiment, the size of the ultrasound image feature map is Where C represents the number of channels, and H and W represent the height and width of the feature map, respectively. In other embodiments, other feature extraction networks capable of outputting two-dimensional medical image feature maps may also be used.

[0055] For the The first historical follow-up time point 4 thyroid ultrasound images Pre-trained backbone network is used to extract feature maps from ultrasound images. The ultrasound image feature maps are input in parallel into 1×1 convolution branches, 3×3 convolution branches, 5×5 convolution branches, and 3×3 dilated convolution branches with an inflation rate of 2. Local fine texture features and global echo context features are extracted through different convolution branches. The outputs of each convolution branch are concatenated along the channel dimension and then subjected to group normalization and ReLU (Rectified Linear Unit) linear activation processing in sequence to obtain multi-scale ultrasound features.

[0056] ;

[0057] in, Indicates the first The output of each convolutional branch ; Indicates multi-scale ultrasound characteristics; Indicates splicing, Indicates group normalization processing, This indicates linear activation processing.

[0058] S22. Channel attention mapping is used to modulate the channel dimension of multi-scale ultrasound features to enhance the semantic channel response related to Hashimoto's thyroiditis; spatial attention mapping is used to modulate the spatial dimension of the channel dimension modulated features to highlight spatial regions with discriminative value, and the spatial dimension modulated results are residually connected with the ultrasound image feature map output by the backbone network to obtain the attention-enhanced feature map.

[0059] Channel dimension modulation is represented as: ;

[0060] in, This represents a channel attention mapping; This indicates element-wise multiplication; This indicates the characteristics after channel modulation.

[0061] The residual connection between the spatial dimension modulation result and the ultrasound image feature map is represented as follows: ;

[0062] in, Represents spatial attention mapping; This represents the ultrasound image feature map extracted by the backbone network. This represents the feature map after connecting the spatial dimension modulation result with the residual of the ultrasound image feature map, i.e., the attention-enhanced feature map.

[0063] S23. Perform global average pooling and global max pooling in parallel on the attention-enhanced feature map. Global average pooling is used to preserve the overall echo and global contextual information, while global max pooling is used to highlight local salient responses. After concatenating the two pooling results (i.e., the results of global average pooling and global max pooling) along the channel dimension, the input includes a convolutional fusion head consisting of a 1×1 convolutional layer, a 3×3 depth convolutional layer, a linear activation layer, and a 1×1 convolutional layer connected in sequence. The output of the convolutional fusion head is flattened to obtain the enhanced ultrasound features of the ultrasound view.

[0064] In this embodiment, the dual-pooling fusion based on global context information and local salient responses is represented as:

[0065] ;

[0066] in, Indicates global average pooling; Indicates global max pooling; This indicates the characteristics after dual-pooling fusion.

[0067] Flattening is represented as: ;

[0068] in, Indicates the convolution fusion head; Indicates flattening; This represents the enhanced ultrasound features of the generated single ultrasound image.

[0069] S24. Arrange the enhanced ultrasound features of each ultrasound view at the same historical follow-up time point into an enhanced ultrasound feature matrix by row. Establish the dependency relationship between different ultrasound views by scaling dot product self-attention operation to obtain a multi-view feature matrix containing contextual information between ultrasound views. Then, use an attention-based multi-instance learning network to calculate the normalized importance weight of each ultrasound view, and perform weighted aggregation of the enhanced ultrasound features of each ultrasound view according to the normalized importance weight to obtain the follow-up-level ultrasound representation.

[0070] In this embodiment, the dependency relationship between multiple ultrasound views is modeled as follows:

[0071] ;

[0072] in, An enhanced ultrasound feature matrix representing the same historical follow-up time point; These represent the query projection matrix, the key projection matrix, and the value projection matrix, respectively. Indicates the dimension of attention projection; superscript Represents the transpose of a matrix; This represents the normalized exponential function; This represents the feature matrix of the multi-view diagram.

[0073] The normalized importance weights of each ultrasound view are determined by the following formula:

[0074] ;

[0075] in, Representing the feature matrix of multiple views The first in Multiple view features; Representing the feature matrix of multiple views The first in Multiple view features; Indicates the first Number of ultrasound images at each historical follow-up time point; This represents the transpose of the learnable attention weight vector; Represents the hyperbolic tangent function; This represents an exponential function with the natural constant e as its base. This represents the learnable feature transformation matrix used to map multi-view image features to the attention latent space; This represents the normalized importance weight.

[0076] The follow-up ultrasound results obtained by weighted aggregation are represented as follows: ;

[0077] in, This indicates a follow-up ultrasound.

[0078] S3. Construct a longitudinal prediction model, which includes:

[0079] A cross-modal dual-graph encoding unit is used to normalize or numerically encode clinical variables, mapping follow-up-level ultrasound representations to ultrasound nodes and each clinical variable to clinical variable nodes. Based on the ultrasound nodes and clinical variable nodes, local and global graphs are constructed, and graph attention encoding is performed on both to obtain enhanced node representations for the local and global graphs. The enhanced node representations for the local graphs include both ultrasound node representations and clinical variable node representations, while the enhanced node representations for the global graphs include both ultrasound node representations and clinical variable node representations.

[0080] The dual-graph interactive fusion unit includes an ultrasound branch and a clinical variable branch. The ultrasound branch performs a weighted fusion based on the ultrasound node representations of the local graph and the global graph to obtain a fused ultrasound representation. The clinical variable branch performs a weighted fusion based on the clinical variable node representations of the local graph and the global graph to obtain a fused clinical variable representation. The fused ultrasound representation and the fused clinical variable representation are concatenated along the feature dimension to obtain a follow-up-level multimodal representation of the current historical follow-up time point.

[0081] The longitudinal prediction unit is used to embed and encode the time interval between each historical follow-up time point, embed and encode the time interval obtained by embedding and encode it and fuse it with the corresponding follow-up-level multimodal representation, and perform gated loop modeling according to the time order of each historical follow-up time point; based on the hidden state corresponding to the last historical follow-up time point obtained by the gated loop processing, the prediction result of the Hashimoto's thyroiditis status of the target object at the target follow-up time point is output.

[0082] (i) Cross-modal bi-graph coding unit, constructing and coding cross-modal local graphs and global graphs.

[0083] In this embodiment, the data processing procedure of the cross-modal dual-graph coding unit includes:

[0084] (1) The follow-up ultrasound representation is mapped to the graph feature space through linear projection to obtain ultrasound nodes; each preprocessed clinical variable is input into the clinical variable mapping network to obtain the corresponding clinical variable nodes.

[0085] In this embodiment, the clinical variable mapping network includes a linear layer, an activation layer, and a regularization layer. Clinical variable nodes with the same dimension as the ultrasound nodes are generated through linear projection. The linear projection process can be represented as follows:

[0086] ;

[0087] in, Indicates an ultrasonic node; Indicates the first One clinical variable node; Indicates the first preprocessed... One clinical variable; Represents a linear projection matrix; Indicates the bias term; This indicates multi-layer sensing processing.

[0088] (2) Construct local and global graphs based on ultrasound nodes and clinical variable nodes.

[0089] The local graph employs a star topology centered on ultrasound nodes, connecting each ultrasound node to a specific clinical variable node while preserving self-connections between ultrasound nodes; clinical variable nodes are not directly connected to each other. The local graph is used to characterize the direct association between ultrasound findings and individual clinical variables. Several ultrasound nodes and several clinical variable nodes form a multimodal node feature matrix.

[0090] A local graph using a star topology is defined as follows:

[0091] ;

[0092] in, Represents the elements of the adjacency matrix of the local graph; This represents the node index; 0 indicates an ultrasound node. It means "otherwise".

[0093] The global graph employs a fully connected topology, linking ultrasound nodes and clinical variable nodes pairwise to represent global and higher-order associations among all multimodal nodes. The global graph using a fully connected topology is defined as follows:

[0094] ;

[0095] in, This represents the elements of the global graph adjacency matrix.

[0096] (3) Input the multimodal node feature matrix and the adjacency matrix of the local graph into the local graph attention encoder to obtain the local graph enhanced node representation; input the multimodal node feature matrix and the adjacency matrix of the global graph into the global graph attention encoder to obtain the global graph enhanced node representation.

[0097] Obtain enhanced node representations of the local and global graphs, specifically:

[0098] ;

[0099] in, Represents the feature matrix of multimodal nodes; Represents the adjacency matrix of a local graph; This represents a local graph augmented node representation; Represents the adjacency matrix of the global graph; This represents the enhanced node representation of the global graph.

[0100] (ii) Dual-graph interaction fusion unit, used to perform cross-modal dual-graph interaction fusion of local graph and global graph.

[0101] In the dual-graph interactive fusion unit, ultrasound branch and clinical variable branch are set up. Based on the multimodal state (i.e. ultrasound representation and clinical variable representation) at the current historical follow-up time point, the local graph enhancement node representation and the global graph enhancement node representation are adaptively fused.

[0102] The data processing procedure of the dual-image interactive fusion unit includes:

[0103] (1) The ultrasound branch extracts follow-up ultrasound node representations from the local graph enhancement node representations and the global graph enhancement node representations respectively; the follow-up ultrasound representations are mapped to ultrasound query vectors, and the ultrasound normalization gating weights corresponding to the local graph ultrasound nodes and the global graph ultrasound nodes are calculated based on the ultrasound query vectors; the ultrasound normalization gating weights are used to gating and weight the follow-up ultrasound node representations in the local graph and the global graph, the ultrasound query vectors are used as queries, and the gating and weighted follow-up ultrasound node representations are used as keys and values ​​to perform multi-head attention operations, and after residual connection, layer normalization and linear activation, the fused ultrasound representation is obtained.

[0104] In ultrasound branching, the fusion ratio between local and global images is determined as follows:

[0105] ;

[0106] in, Represents the ultrasound query vector; This represents the normalized gating weights for ultrasound. This represents the linear projection matrix used to generate the ultrasound query vector; Indicates the bias term; This represents the output of the multilayer perceptron used to generate the gating weights in the ultrasound branch.

[0107] Using ultrasound query vectors as queries to guide the fusion of ultrasound nodes in two graphs can be represented as follows:

[0108] ;

[0109] in, The key represents the sequence of ultrasound nodes at the follow-up level in the gated weighted local and global graphs. This represents the values ​​of the follow-up ultrasound nodes in the local and global graphs after gating and weighting. Indicates fusion ultrasound representation; This represents the multi-head attention mechanism; Presentation layer normalization processing; This represents the exponential linear unit activation function.

[0110] (2) The clinical variable branch performs mean pooling on each clinical variable node to obtain the clinical variable query vector, and calculates the clinical normalization gating weights corresponding to the clinical variable nodes in the local graph and the global graph based on the clinical variable query vector; the clinical variable node representations in the local graph and the global graph are weighted and fused using the clinical normalization gating weights, and then the mean is aggregated along the dimension of the clinical variable nodes to obtain the fused clinical variable representation.

[0111] In the clinical variable branch, the fusion ratio of the local and global plots is determined as follows:

[0112] ;

[0113] in, This indicates a query obtained by pooling the mean values ​​of clinical variable nodes; Indicates the clinically normalized gating weights; This represents the multilayer perceptron output used to generate the gating weights in the clinical variable branch.

[0114] Adaptive fusion and aggregation of clinical variable nodes, represented as follows:

[0115] ;

[0116] in, This represents the clinical variable nodes in the local plot; This represents the clinical variable nodes in the global graph; The clinical variable nodes in the local graph of the clinical variable branch represent the corresponding gating weights. The clinical variable nodes in the global graph represent the corresponding gating weights in the clinical variable branch. This indicates the mean pooling operation; This indicates the fusion of clinical variables.

[0117] (3) The fused ultrasound representation and the fused clinical variable representation are spliced ​​along the feature dimension to obtain the follow-up-level multimodal representation of the current historical follow-up time point.

[0118] ;

[0119] in, This indicates a follow-up-level multimodal representation. Indicates splicing.

[0120] (iii) Longitudinal prediction unit, used for longitudinal modeling of irregular time interval perception.

[0121] The longitudinal prediction unit consists of a learnable linear mapping layer, a gated recurrent unit, a multilayer perceptron classifier, and a normalized exponential function (such as the Softmax function) connected in sequence.

[0122] First, the time interval corresponding to each historical follow-up point is input into a learnable linear mapping layer, and encoded through the learnable linear mapping layer to obtain the time interval embedding. In this embodiment, the dimension of the time interval embedding is set to 128.

[0123] Next, the time interval embedding is concatenated with the follow-up-level multimodal representation of the corresponding historical follow-up time point, and mapped to the input space of the gated recurrent unit to obtain the gated recurrent input representation of each historical follow-up time point. Specifically, the scalar time interval is mapped to the learnable feature space as follows:

[0124] ;

[0125] In the formula, Indicates the first The time interval between historical follow-up points; Indicates time interval embedding; Represents a linear mapping matrix. This indicates the bias term.

[0126] The fusion of time interval embedding and follow-up-level multimodal representation is expressed as:

[0127] ;

[0128] in, Indicates the first A gated cyclic input representation of each historical follow-up time point. Represents a linear mapping matrix. This indicates the bias term.

[0129] Then, the gated cyclic input representations of each historical follow-up time point are sequentially input into the gated cyclic unit according to the real time order to model the vertical dependency relationship between different historical follow-up time points.

[0130] Modeling vertical dependencies in chronological order, specifically represented as follows:

[0131] ;

[0132] in, Indicates the first Hidden status of each historical follow-up point; Indicates the first Hidden status of each historical follow-up point; Indicates the number of historical follow-up points; This indicates gated loop processing.

[0133] Finally, the hidden state corresponding to the last historical follow-up point is used as the longitudinal feature representation of the target object and input into a multilayer perceptron classifier. After normalization, the positive and negative probabilities of Hashimoto's thyroiditis at the target follow-up point are obtained. Based on the predicted probabilities, the predicted Hashimoto's thyroiditis status at the target follow-up point is output.

[0134] The state prediction results in this embodiment are used to provide supplementary longitudinal risk reference and do not replace clinical diagnosis made by medical personnel. The specific Hashimoto's thyroiditis state prediction results can be represented as follows:

[0135] ;

[0136] in, This indicates the hidden state at the last historical follow-up point; Indicates the predicted probability of Hashimoto's thyroiditis status at the target follow-up time point; The parameter is Predictive models; This represents the follow-up sequence consisting of the first to the Tth historical follow-up points; This indicates multi-layer sensing processing.

[0137] S4. Train the constructed longitudinal prediction model.

[0138] Using the actual Hashimoto's thyroiditis status at the target follow-up time point as the supervision label, the model was trained end-to-end using the cross-entropy loss function. The parameters of each module were updated through backpropagation, including the parameters of the backbone network, multi-scale dual-pooling attention module, self-attention multi-instance aggregation module, graph attention encoder, dual-graph interaction fusion module, time interval embedding layer, gated recurrent unit, and classifier.

[0139] S5. For real-time input multi-view thyroid ultrasound images, use the trained longitudinal prediction model to make predictions.

[0140] Based on the same inventive concept, this embodiment also provides a longitudinal prediction system for Hashimoto's thyroiditis based on cross-modal dual-graph interactive learning, including:

[0141] The data acquisition unit is used to acquire historical follow-up data of the target object at multiple historical follow-up time points before the target follow-up time point; wherein, the historical follow-up data of each historical follow-up time point includes multi-view thyroid ultrasound images, clinical variables, and time intervals between adjacent historical follow-up time points;

[0142] The ultrasound feature enhancement and aggregation unit is used to sequentially perform multi-receptive field feature extraction, channel and spatial attention enhancement, and dual pooling fusion on multi-view thyroid ultrasound images for each historical follow-up time point to obtain enhanced ultrasound features for each ultrasound view; and to perform self-attention processing and attention-based multi-instance weighted aggregation on the enhanced ultrasound features of each ultrasound view at the same historical follow-up time point to obtain the corresponding follow-up-level ultrasound representation.

[0143] The longitudinal prediction model includes a cross-modal dual-graph coding unit, a dual-graph interactive fusion unit, and a longitudinal prediction unit;

[0144] The cross-modal dual-graph encoding unit is used to normalize or numerically encode clinical variables, mapping the follow-up ultrasound representation to ultrasound nodes and mapping each clinical variable to clinical variable nodes. Based on the ultrasound nodes and clinical variable nodes, local and global graphs are constructed, and graph attention encoding is performed on both to obtain enhanced node representations for the local and global graphs. The enhanced node representations for the local graphs include both ultrasound node representations and clinical variable node representations, while the enhanced node representations for the global graphs include both ultrasound node representations and clinical variable node representations.

[0145] The dual-graph interactive fusion unit includes an ultrasound branch and a clinical variable branch. The ultrasound branch performs a weighted fusion based on the ultrasound node representations of the local graph and the global graph to obtain a fused ultrasound representation. The clinical variable branch performs a weighted fusion based on the clinical variable node representations of the local graph and the global graph to obtain a fused clinical variable representation. The fused ultrasound representation and the fused clinical variable representation are concatenated along the feature dimension to obtain a follow-up-level multimodal representation of the current historical follow-up time point.

[0146] The longitudinal prediction unit is used to embed and encode the time interval between each historical follow-up time point, embed and encode the time interval obtained by embedding and encode it and fuse it with the corresponding follow-up-level multimodal representation, and perform gated loop modeling according to the time order of each historical follow-up time point; based on the hidden state corresponding to the last historical follow-up time point obtained by the gated loop processing, the prediction result of the Hashimoto's thyroiditis status of the target object at the target follow-up time point is output.

[0147] The aforementioned units can be implemented by software programs within the same electronic device, or they can be deployed separately on a medical imaging server, a clinical data server, and a model inference server. The specific processing procedures for each unit are the same as the corresponding method steps in this embodiment.

[0148] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, substitutions, or combinations made to the module structure, number of network layers, feature dimensions, clinical variable combinations, or parameter configurations without departing from the technical principles of the present invention are all within the protection scope of the present invention.

Claims

1. A longitudinal prediction method for Hashimoto's thyroiditis based on cross-modal dual-graph interactive learning, characterized in that, Includes the following steps: S1. Obtain historical follow-up data for the target subject at multiple historical follow-up time points prior to the target follow-up time point; wherein, the historical follow-up data for each historical follow-up time point includes multi-view thyroid ultrasound images, clinical variables, and time intervals between adjacent historical follow-up time points; S2. For each historical follow-up time point, multi-receptive field feature extraction, channel and spatial attention enhancement, and dual pooling fusion are performed sequentially on the multi-view thyroid ultrasound images to obtain the enhanced ultrasound features of each ultrasound view; self-attention processing and attention-based multi-instance weighted aggregation are performed on the enhanced ultrasound features of each ultrasound view at the same historical follow-up time point to obtain the corresponding follow-up-level ultrasound representation. S3. Construct a longitudinal prediction model, which includes: A cross-modal dual-graph encoding unit is used to normalize or numerically encode clinical variables, mapping follow-up-level ultrasound representations to ultrasound nodes and each clinical variable to clinical variable nodes. Based on the ultrasound nodes and clinical variable nodes, local and global graphs are constructed, and graph attention encoding is performed on both to obtain enhanced node representations for the local and global graphs. The enhanced node representations for the local graphs include both ultrasound node representations and clinical variable node representations, while the enhanced node representations for the global graphs include both ultrasound node representations and clinical variable node representations. The dual-graph interactive fusion unit includes an ultrasound branch and a clinical variable branch. The ultrasound branch performs a weighted fusion based on the ultrasound node representations of the local graph and the global graph to obtain a fused ultrasound representation. The clinical variable branch performs a weighted fusion based on the clinical variable node representations of the local graph and the global graph to obtain a fused clinical variable representation. The fused ultrasound representation and the fused clinical variable representation are concatenated along the feature dimension to obtain a follow-up-level multimodal representation of the current historical follow-up time point. The longitudinal prediction unit is used to embed and encode the time interval between each historical follow-up time point, embed and encode the time interval obtained by embedding and encode it and fuse it with the corresponding follow-up-level multimodal representation, and perform gated loop modeling according to the time order of each historical follow-up time point; based on the hidden state corresponding to the last historical follow-up time point obtained by the gated loop processing, the prediction result of the Hashimoto's thyroiditis status of the target object at the target follow-up time point is output. S4. Train the constructed longitudinal prediction model; S5. For real-time input multi-view thyroid ultrasound images, use the trained longitudinal prediction model to make predictions.

2. The longitudinal prediction method for Hashimoto's thyroiditis according to claim 1, characterized in that, Step S2 includes: S21. For each historical follow-up time point, a pre-trained backbone network is used to extract ultrasound image feature maps. The ultrasound image feature maps are processed through different convolutional branches to extract local fine texture features and global echo context features. The outputs of each convolutional branch are concatenated by channel dimension, group normalization and linear activation to obtain multi-scale ultrasound features. S22. Channel attention mapping is used to modulate the channel dimension of multi-scale ultrasound features, and spatial attention mapping is used to modulate the spatial dimension of the channel dimension modulated features. The spatial dimension modulation result is then residually connected with the ultrasound image feature map output by the backbone network to obtain the attention-enhanced feature map. S23. Perform global average pooling and global max pooling in parallel on the attention-enhanced feature map. After concatenating the pooling results of global average pooling and global max pooling along the channel dimension, input them into the convolutional fusion head and flatten the output of the convolutional fusion head to obtain the enhanced ultrasound features of the ultrasound view. S24. Arrange the enhanced ultrasound features of each ultrasound view at the same historical follow-up time point into an enhanced ultrasound feature matrix by row. Establish the dependency relationship between different ultrasound views by scaling dot product self-attention operation to obtain a multi-view feature matrix containing contextual information between ultrasound views. Then, use an attention-based multi-instance learning network to calculate the normalized importance weight of each ultrasound view, and perform weighted aggregation of the enhanced ultrasound features of each ultrasound view according to the normalized importance weight to obtain the follow-up-level ultrasound representation.

3. The longitudinal prediction method for Hashimoto's thyroiditis according to claim 2, characterized in that, In step S21, the ultrasound image feature map is input in parallel into a 1×1 convolution branch, a 3×3 convolution branch, a 5×5 convolution branch, and a 3×3 dilated convolution branch with an inflation rate of 2. Local fine texture features and global echo context features are extracted through different convolution branches.

4. The longitudinal prediction method for Hashimoto's thyroiditis according to claim 2, characterized in that, In step S22, channel dimension modulation is represented as: ; in, This represents a channel attention mapping; Indicates multi-scale ultrasound characteristics; This indicates element-wise multiplication; This indicates the characteristics after channel modulation; The residual connection between the spatial dimension modulation result and the ultrasound image feature map is represented as follows: ; in, Represents spatial attention mapping; This represents the ultrasound image feature map extracted by the backbone network. This represents the attention-enhanced feature map obtained after residual connection.

5. The longitudinal prediction method for Hashimoto's thyroiditis according to claim 2, characterized in that, The convolutional fusion head in step S23 includes a 1×1 convolutional layer, a 3×3 depth convolutional layer, a linear activation layer, and a 1×1 convolutional layer connected in sequence.

6. The longitudinal prediction method for Hashimoto's thyroiditis according to claim 2, characterized in that, Step S24 models the dependency relationship as follows: ； in, An enhanced ultrasound feature matrix representing the same historical follow-up time point; These represent the query projection matrix, the key projection matrix, and the value projection matrix, respectively. Indicates the dimension of attention projection; superscript Represents the transpose of a matrix; This represents the normalized exponential function; Represents the feature matrix of multiple views; The normalized importance weights of each ultrasound view are determined by the following formula: ； in, Representing the feature matrix of multiple views The first in Multiple view features; Representing the feature matrix of multiple views The first in Multiple view features; Indicates the first Number of ultrasound images at each historical follow-up time point; This represents the transpose of the learnable attention weight vector; Represents the hyperbolic tangent function; This represents an exponential function with the natural constant e as its base. This represents the learnable feature transformation matrix used to map multi-view image features to the attention latent space; This represents the normalized importance weight.

7. The longitudinal prediction method for Hashimoto's thyroiditis according to claim 1, characterized in that, The data processing procedure for the cross-modal bi-graph coding unit in step S3 includes: (1) The follow-up ultrasound representation is mapped to the graph feature space through linear projection to obtain ultrasound nodes; each preprocessed clinical variable is input into the clinical variable mapping network to obtain the corresponding clinical variable nodes; (2) Construct local and global graphs based on ultrasound nodes and clinical variable nodes; the local graph adopts a star topology centered on ultrasound nodes, so that ultrasound nodes are connected to each clinical variable node respectively, and the self-connection of each ultrasound node is retained, and the clinical variable nodes are not directly connected to each other; the global graph adopts a fully connected topology, so that ultrasound nodes and each clinical variable node are connected in pairs; a number of ultrasound nodes and a number of clinical variable nodes form a multimodal node feature matrix; (3) Input the multimodal node feature matrix and the adjacency matrix of the local graph into the local graph attention encoder to obtain the local graph enhanced node representation; input the multimodal node feature matrix and the adjacency matrix of the global graph into the global graph attention encoder to obtain the global graph enhanced node representation.

8. The longitudinal prediction method for Hashimoto's thyroiditis according to claim 1, characterized in that, The data processing procedure of the dual-image interactive fusion unit in step S3 includes: (1) The ultrasound branch extracts follow-up ultrasound node representations from the local graph enhancement node representations and the global graph enhancement node representations respectively; the follow-up ultrasound representations are mapped to ultrasound query vectors, and the ultrasound normalization gating weights corresponding to the local graph ultrasound nodes and the global graph ultrasound nodes are calculated based on the ultrasound query vectors; the ultrasound normalization gating weights are used to gating and weight the follow-up ultrasound node representations in the local graph and the global graph, the ultrasound query vectors are used as queries, and the gating and weighted follow-up ultrasound node representations are used as keys and values ​​to perform multi-head attention operations, and after residual connection, layer normalization and linear activation, the fused ultrasound representation is obtained. (2) The clinical variable branch performs mean pooling on each clinical variable node to obtain the clinical variable query vector, and calculates the clinical normalization gating weights corresponding to the clinical variable nodes in the local graph and the global graph based on the clinical variable query vector; the clinical variable node representations in the local graph and the global graph are weighted and fused using the clinical normalization gating weights, and then the mean is aggregated along the dimension of the clinical variable nodes to obtain the fused clinical variable representation. (3) The fused ultrasound representation and the fused clinical variable representation are spliced ​​along the feature dimension to obtain the follow-up-level multimodal representation of the current historical follow-up time point.

9. The longitudinal prediction method for Hashimoto's thyroiditis according to claim 1, characterized in that, In step S3, the longitudinal prediction unit includes a learnable linear mapping layer, a gated recurrent unit, a multilayer perceptron classifier, and a normalized exponential function connected in sequence. The time interval corresponding to each historical follow-up point is input into a learnable linear mapping layer, and then encoded through the learnable linear mapping layer to obtain the time interval embedding. The time interval is embedded and concatenated with the follow-up-level multimodal representation of the corresponding historical follow-up time point, and mapped to the input space of the gated loop unit to obtain the gated loop input representation of each historical follow-up time point; The gated cyclic input representations of each historical follow-up time point are sequentially input into the gated cyclic unit according to the real time order in order to model the vertical dependency relationship between different historical follow-up time points; The hidden state corresponding to the last historical follow-up time point is used as the vertical feature representation of the target object and input into the multilayer perceptron classifier. After normalization, the positive and negative probabilities of Hashimoto's thyroiditis at the target follow-up time point are obtained.

10. A longitudinal prediction system for Hashimoto's thyroiditis based on cross-modal dual-graph interactive learning, characterized in that, include: The data acquisition unit is used to acquire historical follow-up data of the target object at multiple historical follow-up time points before the target follow-up time point; wherein, the historical follow-up data of each historical follow-up time point includes multi-view thyroid ultrasound images, clinical variables, and time intervals between adjacent historical follow-up time points; The ultrasound feature enhancement and aggregation unit is used to sequentially perform multi-receptive field feature extraction, channel and spatial attention enhancement, and dual pooling fusion on multi-view thyroid ultrasound images for each historical follow-up time point to obtain enhanced ultrasound features for each ultrasound view; and to perform self-attention processing and attention-based multi-instance weighted aggregation on the enhanced ultrasound features of each ultrasound view at the same historical follow-up time point to obtain the corresponding follow-up-level ultrasound representation. The longitudinal prediction model includes a cross-modal dual-graph coding unit, a dual-graph interactive fusion unit, and a longitudinal prediction unit; The cross-modal dual-graph encoding unit is used to normalize or numerically encode clinical variables, mapping the follow-up ultrasound representation to ultrasound nodes and mapping each clinical variable to clinical variable nodes. Based on the ultrasound nodes and clinical variable nodes, local and global graphs are constructed, and graph attention encoding is performed on both to obtain enhanced node representations for the local and global graphs. The enhanced node representations for the local graphs include both ultrasound node representations and clinical variable node representations, while the enhanced node representations for the global graphs include both ultrasound node representations and clinical variable node representations. The dual-graph interactive fusion unit includes an ultrasound branch and a clinical variable branch. The ultrasound branch performs a weighted fusion based on the ultrasound node representations of the local graph and the global graph to obtain a fused ultrasound representation. The clinical variable branch performs a weighted fusion based on the clinical variable node representations of the local graph and the global graph to obtain a fused clinical variable representation. The fused ultrasound representation and the fused clinical variable representation are concatenated along the feature dimension to obtain a follow-up-level multimodal representation of the current historical follow-up time point. The longitudinal prediction unit is used to embed and encode the time interval between each historical follow-up time point, embed and encode the time interval obtained by embedding and encode it and fuse it with the corresponding follow-up-level multimodal representation, and perform gated loop modeling according to the time order of each historical follow-up time point; based on the hidden state corresponding to the last historical follow-up time point obtained by the gated loop processing, the prediction result of the Hashimoto's thyroiditis status of the target object at the target follow-up time point is output.

Images