Target change evaluation system and method based on multi-time point images

Abstract

The application discloses a target change evaluation system and method based on multi-time point images, which comprises a change-before and change-after image segmentation branch and a corresponding segmentation attention guide module, a feature extractor of the change-before and change-after images, an interactive spatial feature enhancement module and a time sequence feature classification module. The application is aimed at the complexity of the task of target evaluation based on multi-time point images, and solves the model design by introducing a segmentation auxiliary task to guide classification attention, cross-time point feature interaction and using time sequence feature classification to complete feature fusion, and divides the model design into target positioning, obtaining the correlation between local regions at different time points and evaluating changes.

Description

Technical Field

[0001] This invention relates to technologies in the field of image processing, specifically a target change evaluation system and method based on multi-time point images. Background Technology

[0002] With the development of deep learning, it has been increasingly widely used in the field of image analysis. However, existing technologies have failed to fully tap into the deep information of MRI and have failed to achieve ideal classification performance for assessing changes in targets within images. Summary of the Invention

[0003] To address the aforementioned shortcomings of existing technologies, this invention proposes a target change assessment system and method based on multi-time point images. To tackle the complexity of target assessment based on multi-time point images, this invention introduces segmentation-assisted tasks to guide classification attention, cross-time point feature interaction, and uses time feature sequence classification to complete feature fusion. The model design is broken down into target localization, obtaining the correlation between local regions at different time points, and assessing changes.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a target change assessment system based on multiple time points, comprising: image segmentation branches before and after the change and their corresponding segmentation attention guidance modules, feature extractors for the images before and after the change, an interactive spatial feature enhancement module, and a temporal feature classification module. Specifically: the image segmentation branch before the change performs encoding and decoding operations based on image information containing the target at the previous time point to obtain the target segmentation result before the change; the image segmentation branch after the change performs encoding and decoding operations based on image information containing the target at the next time point to obtain the target segmentation result after the change; the segmentation attention guidance module calculates the linear separability of each feature element in the multi-scale intermediate layer feature map output by the decoder in the two image segmentation branches to obtain the attention score map result; then the change... In the main network of change evaluation, two feature extractors, one for the image before the change and the other for the image after the change, focus on the target region based on the attention score map results to perform feature extraction operations, obtaining the hidden layer features corresponding to the input images at the two time points. Then, two interactive spatial feature enhancement modules in the main network of change evaluation use the image before the change and the image after the change as references, respectively, to calculate the cross-attention score between the two hidden layer features obtained by the feature extractors and enhance the previously obtained hidden layer features accordingly, realizing the correspondence between the local related regions of the target before and after the change. Finally, the temporal feature classification network composed of gated recursive units and multilayer perceptrons in the main network of change evaluation generates the evaluation result based on the feature-enhanced hidden layer features and auxiliary indicators.

[0006] The segmentation branch structures for the images before and after the change are the same, both including an encoder and a decoder. The encoder performs convolution-based encoding processing on the images before or after the change, respectively. The decoder decodes the encoded features and obtains the segmentation result of the target to be evaluated for change. The spatial location information of the target is extracted through the attention guidance module to limit the classification network's attention to key regions.

[0007] The linear separability of each feature element is specifically defined as follows: for each element s in the multi-scale intermediate layer feature map, through... The energy map is calculated, where: μ c and λ represents the channel mean and variance containing element s, respectively, and λ is the regularization term.

[0008] The cross-attention score is obtained in the following way:

[0009] Step 1: Divide the images before and after the change into S×S×D non-overlapping regions based on the hidden layer features obtained by the feature extractor.

[0010] Step 2: Calculate the correlation matrix between sub-regions at different time points to obtain the correspondence between local areas before and after the target change. Taking the image features before the change as a reference, specifically: Q is obtained by linearly mapping the pre-change features of the divided regions using learnable parameters. r Then, K is obtained from the changed features of the divided regions using the same method. r and V r Establish a region-level correlation matrix A between features before and after the change. r =Q r (K r ) T Each element in the matrix represents the correlation score between the regions before and after the change. Based on the correlation score, for each non-overlapping region of the hidden layer feature before the change, select the k non-overlapping regions of the hidden layer feature after the change that have the highest correlation with it.

[0011] Step 3: After further dividing these regions into image blocks, perform linear mapping again and calculate the similarity (dot product) between the query and the key. After normalization, obtain the attention weights. Multiply the calculated attention weights by the values ​​to obtain the enhanced features, thereby achieving the correspondence between the target local related regions before and after the change.

[0012] The evaluation results are obtained by converting the enhanced hidden features output by the interactive spatial feature enhancement module into a one-dimensional vector through the temporal feature classification module. At this point, the vector can be concatenated with auxiliary indicators according to the task requirements. The temporal feature classification network is then used to evaluate the changes that occur between two time points of the target. Specifically, the input is fed into the temporal feature classification network, which consists of a gated recurrent unit and a multilayer perceptron. Through the memory capability of the gated recurrent unit, the target-related information from the previous time point is used to capture the changes in the features at the later time point and then make predictions.

[0013] The auxiliary indicator refers to one-dimensional information related to the task. In possible implementations, it can be the target size information calculated based on the segmentation results output by the segmentation branch.

[0014] This invention relates to a target change assessment method for the aforementioned system. In the offline stage, a sample set is constructed and a target change assessment system is trained. In the online stage, the trained target change assessment system extracts features from input images at two different time points in real time and focuses on key regions. An interactive spatial feature enhancement module calculates the correlation between key regions at the two time points at the feature level to achieve feature enhancement. After problem transformation, further classification is performed, thereby realizing change assessment based on multi-time point images.

[0015] The training mentioned refers to: by designing a task-related loss function, calculating the error loss between the model output and the true label, and then using gradient backpropagation to automatically update the network model parameters. The training is considered complete when the loss converges with each training round.

[0016] Technical effect

[0017] This invention effectively guides the classification network's attention to the target region closely related to the evaluation by introducing segmentation-guided classification attention, cross-time point feature interaction, and using temporal feature sequence classification instead of traditional feature fusion methods. It also captures the correspondence between local regions of images from different time points and evaluates the changes that occur. Compared with existing technologies, this invention can effectively evaluate changes in targets in images from different time points, especially when image registration between different time points is not possible, and only local regions of the target to be evaluated may change. Attached Figure Description

[0018] Figure 1 This is a flowchart of the present invention;

[0019] Figure 2 This is a schematic diagram of the invention;

[0020] Figure 3 A schematic diagram of the cross-time feature interaction module structure at different points in time;

[0021] Figure 4 A structural diagram for classifying time-series features. Detailed Implementation

[0022] like Figure 1 and Figure 2 As shown, this embodiment illustrates a target change assessment method based on multi-time-point images, which specifically includes:

[0023] Step 1: Extract the input image X at different time points using two independent feature extractors. pre and X post While performing feature extraction, the input image X is respectively... pre and X post The input consists of independent parallel segmentation branches containing encoders and decoders, which are used for segmentation to obtain the segmentation result of the target. and In this process, the decoder obtains a multi-scale intermediate layer feature map S; for each element s in the multi-scale intermediate layer feature map S, the decoder then... The energy map is calculated, where: μ c and These represent the channel mean and variance of element s, respectively, with λ being a regularization term set to 1e-6. Taking the reciprocal of each element in the energy map and normalizing it yields the final attention map, containing the spatial location information of the target. Then, the multi-scale attention map is multiplied by the corresponding scale features in the feature extractor of the main classification network using a segmentation auxiliary task. This guides the classification network to focus on the target region more critical to change assessment. Thus, the hidden features obtained from the segmentation-guided feature extractor at two time points before and after the change can be obtained. and

[0024] The segmentation is achieved by outputting features from the intermediate layer of the decoder to the segmentation attention guidance module to obtain a multi-scale attention map.

[0025] The multi-scale attention map is implemented by, but not limited to, measuring the linear separability of each element in the input feature map.

[0026] Step 2, through Figure 3 The interactive spatial feature enhancement shown establishes the correlation between local regions of images at different time points, that is, it obtains features F from the two time points obtained in step 1. pre and F post After dividing the region into multiple non-overlapping regions, the correspondence between the local regions of the two regions is found, specifically including:

[0027] 2.1 Taking the image features before the change as the Query as an example, the features F obtained at two time points are... pre and Fpost The image is divided into S×S×D non-overlapping regions. S and D can be adjusted according to the size of the input image. Q is obtained from the features of the regions before the changes through learnable parameters. r K is obtained from the changed features of the divided regions. r and V r .

[0028] 2.2 Establish the region-level correlation matrix A between features before and after the change. r =Q r (K r ) T Each element in the matrix represents the correlation score between the region before and after the change.

[0029] 2.3 For each element in the query region before the change, cross-attention is calculated with all elements in the k elements of the k most relevant regions after the change to further explore the correlation between features. In this implementation, k is set to 8, but this can be adjusted based on the complexity of the input image information and the size of the target.

[0030] Step 3, as follows Figure 4 As shown, in order to obtain the final change assessment result, feature fusion is performed by constructing a time series classification problem, specifically including:

[0031] 3.1 The features obtained from the segmentation-guided feature extractor and the cross-temporal interaction module are transformed into two one-dimensional time series f through global average pooling. pre and f post .

[0032] 3.2 The auxiliary indicators c related to the change assessment at two different time points. pre and c post It is then spliced ​​with the corresponding time series.

[0033] 3.3 After concatenation, the two feature sequences are output to a gated recursive unit, with the specific structure as follows: Figure 4 As shown.

[0034] 3.4 Obtaining the final result of target change assessment using a multilayer perceptron containing three fully connected layers.

[0035] The change evaluation system, consisting of the two segmented branches and the main change evaluation network, is trained together, and the loss function is as follows: Wherein: Dice coefficient loss includes Focus loss L focal =-α t (1-p t ) γ log(pt ), Y pre and Y post These represent reference standards for the target region at different time points, with p referring to all pixels in the image. t It represents the probability that the model correctly classifies the current sample. When the label is 1, then p... t =p, when the label is 0, then p t = 1 - p, where p is the positive class probability output by the model. α t It is a balancing factor used to adjust the weights of positive and negative samples.

[0036] Through practical experiments, the aforementioned network model was implemented using the PyTorch framework and trained and tested on an Intel(R) Core(TM) i7-12700H@2.30GHz CPU and an NVIDIA GeForce RTX 3060. To reduce the complexity of joint optimization for segmentation and classification tasks, we pre-trained the two segmentation branches separately. During the main training process, the segmentation network was updated using a smaller learning rate of 1e-4, while the classification network used a learning rate of 1e-3. Gradient descent employed the Adam optimizer. Furthermore, an additional dropout rate of 0.5 was introduced into the network, and L2 regularization was added to the loss function to prevent overfitting. In the example, the input image is an annotated (e.g., class and segmentation labels) T2-weighted modal magnetic resonance image with a spatial resolution of 0.396 × 0.396 × 3 mm. 3 .

[0037] One possible implementation is to standardize the input image before it is fed into the network to reduce the impact of differences in image grayscale values. Simultaneously, to optimize GPU memory usage and computational efficiency, the input image can be downsampled while maintaining its quality. For example, the size of the input 3D image can be downsampled from 512×512×24 pixels to 320×320×24 pixels.

[0038] To evaluate the model's performance, four metrics are calculated: balanced precision, precision, recall, and area under the curve (AUC). The definitions of balanced precision, precision, and recall are as follows: In this context, TP, FP, TN, and FN represent true positive, false positive, true negative, and false negative, respectively. Using balanced precision instead of traditional accuracy as the evaluation metric aims to avoid the impact of data imbalance on model evaluation. The AUC metric quantifies the overall model performance by calculating the area under the Receiver Operating Characteristic (ROC) curve, which is plotted using the false positive rate and true positive rate at different thresholds. An AUC value close to 1 indicates good model performance, while an AUC value close to 0.5 indicates that the model performance is similar to random guessing.

[0039] To verify the effectiveness of each module in the proposed network, multiple ablation experiments were conducted. The baseline model refers to the simplest model, which extracts features from inputs at two different time points and directly inputs these features into the time series classification layer to obtain the classification results. As shown in Table 1, based on the baseline model, the classification performance is significantly improved by combining the cross-time feature interaction module with segmentation attention guidance: balanced precision is improved by 10.56%, precision by 12.72%, recall by 18.18%, and AUC by 10.00%. Furthermore, to verify the effectiveness of using time series classification to replace feature fusion of input images at different time points, it was compared with two of the most commonly used feature fusion methods: feature concatenation and element-wise addition. Table 2 shows that the time series classification method achieved the best classification performance, with balanced precision, precision, recall, and AUC of 81.95%, 72.72%, 72.72%, and 83.21%, respectively. Compared to the second-best performing feature concatenation method, the time series classification strategy improves precision, accuracy, recall, and AUC by 6.02%, 9.09%, 9.09%, and 3.12%, respectively.

[0040] To demonstrate the superior performance of this invention in target change assessment, it was compared with traditional machine learning methods and existing deep learning methods for change assessment. The quantitative results of the comparative experiments are shown in Table 3. It can be observed that the model used in this invention achieved the most accurate results, with balanced precision, accuracy, recall, and AUC of 81.95%, 72.72%, 72.72%, and 83.21%, respectively. Compared with traditional machine learning methods, it showed significant improvements of 16.44%, 29.87%, 18.18%, and 15.13% in b-Acc, precision, recall, and AUC, respectively. Compared with MPNet, it also improved by 12.04%, 18.18%, 18.18%, and 0.11 in b-Acc, precision, recall, and AUC, respectively. Compared with MMDL, the best-performing model among all compared methods, it also improved by 6.02%, 14.39%, 9.08%, and 5.43% in these metrics, respectively.

[0041] Table 1

[0042]

[0043]

[0044] Table 2

[0045]

[0046] Table 3

[0047]

[0048] In summary, to address the problems in existing research on change assessment based on multi-time-point input images, this invention focuses on three aspects: First, it utilizes parallel segmentation branches to extract spatial information of the target, thereby guiding the classification network to assess change based on images from different time points. To obtain the correspondence between local regions from images at two different time points, a cross-temporal feature interaction module is designed, employing different levels of attention mechanisms. Furthermore, temporal feature classification replaces traditional feature fusion to capture change. Experimental results show that the model used in this invention outperforms other state-of-the-art methods for change assessment.

[0049] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A multi-timepoint based target change evaluation system, characterized in that, The method comprises: a pre-change and post-change image segmentation branch and a corresponding segmentation attention guide module, a pre-change and post-change image feature extractor, an interactive spatial feature enhancement module, and a time sequence feature classification module, wherein: the pre-change image segmentation branch performs encoding and decoding operations on image information containing a target at a previous time point to obtain a segmentation result of the target before the change; the post-change image segmentation branch performs encoding and decoding operations on image information containing the target at a subsequent time point to obtain a segmentation result of the target after the change; the segmentation attention guide module calculates the linear separability of each feature element in the multi-scale intermediate layer feature map output by the decoder in the two image segmentation branches to obtain an attention score map result; then, two feature extractors in the change evaluation main network respectively for the pre-change image and the post-change image focus on the target region to perform feature extraction operations according to the attention score map result, to obtain hidden layer features corresponding to the input images at the two time points; next, two interactive spatial feature enhancement modules in the change evaluation main network take the pre-change image and the post-change image as references, respectively, calculate cross-attention scores between local regions of the two hidden layer features obtained by the feature extractors, and perform feature enhancement on the previously obtained hidden layer features according to the cross-attention scores, to realize the correspondence between the local associated regions of the target before and after the change; finally, a time sequence feature classification network composed of a gated recurrent unit and a multilayer perception mechanism in the change evaluation main network generates an evaluation result according to the feature-enhanced hidden layer features and auxiliary indicators. The cross-attention score is obtained by the following method: Step 1, the hidden layer features obtained by the feature extractor according to the before-change and after-change images are divided into non-overlapping regions; Step 2, calculate the correlation matrix between sub-regions at different time points to obtain the correspondence between local areas before and after the target change, taking the image features before the change as an example, specifically: the features of the divided areas before the change are linearly mapped through learnable parameters to obtain Then, the features of the divided areas after the change are obtained in the same way and , and the correlation matrix between the features before and after the change at the area level is established Each element in the matrix represents the correlation score between the area before the change and the area after the change. According to the correlation score, for each non-overlapping area of the hidden layer features before the change, the k non-overlapping areas of the hidden layer features after the change with the largest correlation are selected. Step 3: After further dividing the regions into image blocks, performing linear mapping again, calculating the similarity between the query and the key, i.e. the dot product normalization to obtain the attention weight, multiplying the calculated attention weight with the value to obtain the feature-enhanced post-change feature, thereby realizing the correspondence between the local associated regions of the target before and after the change. The evaluation result is converted into a one-dimensional vector by the time sequence feature classification module, and then the one-dimensional vector is spliced with the auxiliary indicators according to the task situation, and the change between the target at the two time points is evaluated by the time sequence feature classification network, specifically: the time sequence feature classification network composed of the gated recurrent unit and the multilayer perception mechanism is input, the memory capability of the gated recurrent unit is used to capture the change in the feature at the subsequent time point by using the target-related information at the previous time point, and then the change is predicted.

2. The multi-timepoint based target change evaluation system according to claim 1, characterized in that The pre-change and post-change image segmentation branches have the same structure and each comprises an encoder and a decoder, wherein: the encoder performs convolution-based encoding processing on the pre-change or post-change image; the decoder decodes the encoded features to obtain a segmentation result of the target to be evaluated for change, and the spatial position information of the target is extracted by the attention guide module to limit the attention of the classification network to the key region.

3. The multi-timepoint based target change evaluation system according to claim 1, characterized in that The linear separability of each feature element is defined as: for each element in the multi-scale intermediate layer feature map , the energy map is calculated by , where: and are the channel mean and variance containing the element , respectively, is a regularization term.

4. The multi-timepoint based target change evaluation method of any of claims 1-3, wherein, The sample set is constructed and the target change evaluation system is trained in an offline stage, and in an online stage, the features in the input images of two different time points are extracted in real time by the trained target change evaluation system, and the key areas are focused on, the correlation between the key areas of the two time points is calculated at the feature level by the interactive spatial feature enhancement module to realize feature enhancement, and after problem conversion, further classification is carried out, so as to realize change evaluation based on multi-time point images; The training is to calculate the error loss between the model output and the real label by designing a task-related loss function, and then realize the automatic update of the network model parameters by means of gradient back propagation, and after the loss converges with the training rounds, it is considered that the training is completed.

5. The target variation evaluation method according to claim 4, characterized in that, specifically Comprise: Step 1, feature extraction on input images at different time points by two independent feature extractors and Simultaneously, input images and are input into independent parallel segmentation branches containing encoders and decoders for segmentation and obtaining the segmentation results of the target and , where the multi-scale intermediate layer feature maps obtained by the decoder ; for the elements in the multi-scale intermediate layer feature map , the energy map is calculated by , where: and are the mean and variance of the channel containing the element , respectively, is a regularization term; after taking the inverse and normalizing each element in the energy map, the final attention map is obtained, which contains the spatial position information of the target. After that, the multi-scale attention map is multiplied by the corresponding scale features in the feature extractor of the main classification network to obtain the hidden layer features and obtained by the feature extractor guided by segmentation from the two time points before and after the change Step 2, the correlation between the local regions of the images at different time points is established by interactive spatial feature enhancement, that is, the features obtained at two time points in step 1 are combined and After the division into a plurality of non-overlapping regions, the corresponding relationship between the local regions of the two images is found, specifically including: 2.

1. Take the image features before changes as Query, obtain features at two time points and Divide into non-overlapping regions, S and D can be adjusted according to the size of the input image, and then obtain from the change before feature of the divided region and ; 2.2 Establishing the regional level correlation matrix between pre-change and post-change features Each element in the matrix represents a correlation score between a pre-change region and a post-change region; 2.3 Cross attention calculation is performed on all elements of each pre-change query area and all elements of the corresponding top k post-change areas with the largest correlation, to further mine the correlation between features; Step 3, constructing a time series classification problem for feature fusion, specifically comprising: 3.1 The features obtained from the split-guided feature extractor and the cross- temporal interaction module are converted into two one-dimensional time series by global average pooling and ; 3.2 Auxiliary indicators at two different time points and related to change assessment and splicing with corresponding time series; 3.3 The two feature sequences after splicing are output to a gated recurrent unit; 3.4 Obtaining the final result of the target change assessment by a multilayer perceptron comprising three fully connected layers .

6. The target change evaluation method according to claim 5, characterized in that the two segmentation branches and the change evaluation main network constitute a change evaluation system which is trained together, and the loss function is specifically: wherein: The Dice coefficient loss includes , , the focal loss , and respectively represent the reference standard of the target region at different time points, refers to all pixels in the image, is the classification probability of the correct class of the current sample by the model, when the label is 1, then when the label is 0, then , is the positive class probability output by the model, is a balance factor used to adjust the weights of positive and negative samples.

Images