Target change evaluation system and method based on multi-time-point image

Abstract

The invention discloses a target change evaluation system and method based on a multi-time-point image. The system comprises pre-change and post-change image segmentation branches and corresponding segmentation attention guiding modules thereof, pre-change and post-change image feature extractors, an interactive spatial feature enhancement module and a time sequence feature classification module. According to the method, for complexity of a task based on multi-time-point image target evaluation, classification attention is guided by introducing a segmentation auxiliary task, cross-time-point feature interaction is performed, and feature fusion is completed by using time feature sequence classification; model design is divided into target positioning, relevance between local areas at different time points is obtained, and evaluation changes are solved.

Description

Technical Field

[0001] The present invention relates to the technology in the field of image processing, and in particular to a target change evaluation system and method based on multi-time point images. Background Art

[0002] With the development of deep learning, it has been widely used in the field of image analysis. However, the existing technology fails to fully explore the deep information of MRI and fails to obtain ideal classification performance for evaluating target changes in images. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a target change assessment system and method based on multi-time point images. In view of the complexity of the task of target assessment based on multi-time point images, segmentation auxiliary tasks are introduced to guide classification attention, cross-time point feature interaction, and feature fusion is completed by using time feature sequence classification. The model design is split into target positioning, obtaining the correlation between local areas at different time points, and evaluating changes to solve this problem.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a target change evaluation system based on multiple time points, comprising: image segmentation branches before and after the change and corresponding segmentation attention guidance modules, feature extractors of the images before and after the change, an interactive spatial feature enhancement module and a temporal feature classification module, wherein: the image segmentation branch before the change performs encoding and decoding operations according to image information containing the target at a previous time point to obtain a segmentation result of the target before the change, and the image segmentation branch after the change performs encoding and decoding operations according to image information containing the target at a later time point to obtain a segmentation result of the target after the change; the segmentation attention guidance module calculates the linear separability of each feature element in a multi-scale intermediate layer feature map output by decoders in two image segmentation branches to obtain an attention score map result; and then the image segmentation branch after the change performs encoding and decoding operations according to image information containing the target at a later time point to obtain a segmentation result of the target after the change. The two feature extractors in the main network of change assessment, which are respectively for the image before and after the change, focus on the target area to perform feature extraction operations based on the attention score map results, and obtain the hidden features corresponding to the input images at two time points; then the two interactive spatial feature enhancement modules in the main network of change assessment take the image before and after the change as references, calculate the cross-attention scores between their local areas based on the two hidden features obtained by the feature extractor, and enhance the previously obtained hidden features accordingly to achieve the correspondence between the local associated areas of the target before and after the change; finally, the temporal feature classification network composed of a gated recurrent unit and a multi-layer perceptron in the main network of change assessment generates evaluation results based on the hidden features after feature enhancement and auxiliary indicators.

[0006] The image segmentation branches before and after the change have the same structure, both including an encoder and a decoder, wherein: the encoder performs encoding processing based on convolution operations according to the image before or after the change; the decoder decodes the encoded features and obtains the segmentation result of the target to be evaluated for change, and extracts the spatial position information of the target through the attention guidance module to limit the classification network's attention to the key area.

[0007] The linear separability of each feature element is specifically: for each element s in the multi-scale intermediate layer feature map, by The energy diagram is calculated, where: μ c and are the channel mean and variance of the element s, respectively, and λ is the regularization term.

[0008] The cross attention score is obtained by:

[0009] Step 1: Divide the image before and after the change into S×S×D non-overlapping regions according to the hidden layer features obtained by the feature extractor.

[0010] Step 2: Calculate the correlation matrix between sub-regions at different time points to obtain the correspondence between the local areas before and after the target changes. Take the image features before the change as an example, specifically: linearly map the features before the change of the divided regions through learnable parameters to obtain Q r Then, the K is obtained from the changed features of the divided regions in the same way. r and V r . Establish the regional level correlation matrix A between the features before and after the change r =Q r (K r ) T , each element in the matrix represents the correlation score between the region before the change and the region after the change. According to the correlation score, for each non-overlapping region of the hidden layer feature before the change, the k non-overlapping regions of the hidden layer features after the change with the largest correlation are selected.

[0011] Step 3: After further dividing these areas into image blocks, linear mapping is performed again to calculate the similarity (dot product) between the query and the key, and the attention weight is obtained after normalization. The calculated attention weight is weighted multiplied with the value to obtain the enhanced changed features, thereby achieving the purpose of corresponding between the local associated areas of the target before and after the change.

[0012] The evaluation results are obtained by converting the enhanced hidden features output by the interactive spatial feature enhancement module into a one-dimensional vector through a temporal feature classification module. At this time, the vector can be spliced ​​with auxiliary indicators according to the task situation, and the changes between the two target time points can be evaluated with the help of a temporal feature classification network. Specifically, the vector is input into a temporal feature classification network composed of a gated recurrent unit and a multi-layer perceptron. Through the memory capacity of the gated recurrent unit and the target-related information at the previous time point, the changes in the features at the later time point are captured and predicted.

[0013] The auxiliary index refers to: one-dimensional information related to the task, and in a possible implementation, it can be target size information calculated based on the segmentation result output by the segmentation branch.

[0014] The present invention relates to a target change assessment method for the above-mentioned system. In an offline stage, a sample set is constructed and a target change assessment system is trained. In an online stage, the trained target change assessment system is used to extract features in input images at two different time points in real time and focus on key areas. An interactive spatial feature enhancement module is used to calculate the correlation between key areas at two time points at the feature level to achieve feature enhancement. The method is further classified after problem conversion, thereby achieving change assessment based on images at multiple time points.

[0015] The training refers to: by designing a task-related loss function, calculating the error loss between the model output and the true label, and then automatically updating the network model parameters with the help of gradient back propagation. The training is considered completed after the loss converges with the training rounds. Technical Effects

[0016] The present invention effectively guides the classification network attention to focus on the target area closely related to the evaluation and captures the correspondence between local areas of images from different time points, and evaluates the changes that occur by introducing segmentation to guide classification attention, cross-time point feature interaction, and using time feature sequence classification instead of traditional feature fusion methods. Compared with the prior art, the present invention can effectively evaluate the changes of targets in images from different time points, especially when images at different time points cannot be registered and only local areas of the target to be evaluated may change. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of the present invention;

[0018] Figure 2 It is the principle diagram of the present invention;

[0019] Figure 3 It is a schematic diagram of the structure of the cross-time feature interaction module at different time points;

[0020] Figure 4 Structural diagram for classifying time series features. DETAILED DESCRIPTION

[0021] like Figure 1 and Figure 2 As shown in FIG. 1 , the target change assessment method based on multi-time point images involved in this embodiment specifically includes:

[0022] Step 1: Use two independent feature extractors to extract the input image X at different time points pre and X post While extracting features, the input image X pre and X post Input an independent parallel segmentation branch containing an encoder and a decoder to perform segmentation and obtain the target segmentation result and In this process, the decoder obtains a multi-scale intermediate layer feature map S; for the element s in the multi-scale intermediate layer feature map S, The energy diagram is calculated, where: μ c and are the channel mean and variance of element s, respectively. λ is the regularization term, which is set to 1e-6. After taking the inverse of each element in the energy map and normalizing it, we get the final attention map, which contains the spatial position information of the target. Then, we perform Hadamard product on the multi-scale attention map with the corresponding scale features in the feature extractor in the main classification network, so as to guide the classification network to focus on the target area that is more critical to the change assessment with the help of the auxiliary task of segmentation. So far, we can get the hidden features obtained by the feature extractor guided by segmentation at two time points before and after the release change. and

[0023] The segmentation, from the intermediate layer feature output of the decoder, is sent to the segmentation attention guidance module to obtain a multi-scale attention map.

[0024] The multi-scale attention map is implemented by, but not limited to, measuring the linear separability of each element in the input feature map.

[0025] Step 2: Pass Figure 3 The interactive spatial feature enhancement shown in the figure establishes the correlation between the local regions of the image at different time points, that is, the two time points obtained in step 1 are used to obtain the feature F pre and F post After being divided into multiple non-overlapping regions, the corresponding relationship between the two local regions is found, including:

[0026] 2.1, taking the image features before the change as the query, the features F obtained at two time points pre and F postIt is divided into S×S×D non-overlapping regions. S and D can be adjusted according to the size of the input image. The Q is obtained from the features before the division of the region through the learnable parameters. r , obtain K from the characteristics of the changes in the divided area r and V r .

[0027] 2.2 Establishing the regional level correlation matrix A between features before and after the change r =Q r (K r ) T , each element in the matrix represents the correlation score between the region before and after the change.

[0028] 2.3 Cross-attention calculations are performed on all elements of each query region before the change and all elements of the top k regions after the change with the greatest correlation, so as to further explore the correlation between features. In the implementation process, k is 8, which can be adjusted according to the complexity of the input image information and the size of the target.

[0029] Step 3: Figure 4 As shown in the figure, in order to obtain the final change assessment result, feature fusion is performed by constructing a time series classification problem, which includes:

[0030] 3.1 The features obtained from the segmentation-guided feature extractor and the cross-temporal interaction module are transformed into two one-dimensional time series f through global average pooling. pre and f post .

[0031] 3.2 Two auxiliary indicators c related to change assessment at different time points pre and c post Concatenate with the corresponding time series.

[0032] 3.3 After concatenation, the two feature sequences are output to the gated recursive unit. The specific structure is as follows: Figure 4 shown.

[0033] 3.4 The final result of target change evaluation is obtained by a multilayer perceptron containing three fully connected layers

[0034] The change assessment system composed of the two segmentation branches and the change assessment main network is trained together, and the loss function is specifically: Among them: Dice coefficient loss includes Focus loss L focal =-α t (1-p t ) γ log(p t), Y pre and Y post They represent the reference standards of the target area at different time points, and p refers to all pixels in the image. t is the probability of the model classifying the correct category of the current sample. When the label is 1, p t =p, when the label is 0, then p t =1-p, where p is the probability of the positive class output by the model. t It is a balancing factor used to adjust the weights of positive and negative samples.

[0035] After specific practical experiments, the above network model was implemented through the PyTorch framework, and the model was trained and tested on the Intel(R) Core(TM) i7-12700H@2.30GHz CPU and NVIDIA GeForce RTX 3060. In order to reduce the complexity of joint optimization of segmentation and classification tasks, we pre-trained the two segmentation branches separately. During the main training process, the segmentation network was updated with a smaller learning rate of 1e-4, while the learning rate of the classification network was 1e-3. The Adam optimizer was used for gradient descent. In addition, a dropout rate of 0.5 was additionally introduced to the network and L2 regularization was added to the loss function to prevent the influence of overfitting. In the example, the input image is an annotated (e.g., category annotation and segmentation annotation) T2-weighted modality magnetic resonance image with a spatial resolution of 0.396×0.396×3 mm 3 .

[0036] In a possible implementation, the input image can be standardized before being input to the network to reduce the impact of image grayscale value differences. At the same time, in order to optimize GPU memory usage and computing efficiency, the input image can be downsampled while ensuring the quality of the input image. For example, the size of the input 3D image is downsampled from the original 512×512×24 pixels to 320×320×24 pixels.

[0037] To evaluate the performance of the model, four metrics are calculated, including balanced accuracy, precision, recall, and area under the curve (AUC). The definitions of balanced accuracy, precision, and recall are as follows: Among them, TP, FP, TN and FN represent true positive, false positive, true negative and false negative, respectively. The purpose of using balanced precision instead of traditional accuracy as the evaluation metric is to avoid the impact of data imbalance on model evaluation. The AUC metric quantifies the overall performance of the model by calculating the area under the receiver operating characteristic curve (ROC curve), which is plotted by the false positive rate and true positive rate at different thresholds. AUC values ​​close to 1 indicate good model performance, while AUC values ​​close to 0.5 indicate that the model performance is similar to random guessing.

[0038] In order to verify the effectiveness of each module in the proposed network, multiple groups of ablation experiments were conducted. The baseline model refers to the simplest model, which extracts features from the input at two different time points and directly inputs these features into the time series classification layer to obtain the classification results. As shown in Table 1, on the basis of the baseline model, the classification performance is significantly improved by the synergy of the cross-time feature interaction module and the segmentation attention guidance, with a balanced accuracy of 10.56%, a precision of 12.72%, a recall of 18.18%, and an AUC of 10.00%. In addition, in order to verify the effectiveness of the method of using time series classification instead of the fusion of input image features at different time points, it is compared with the two most commonly used feature fusion methods, namely feature concatenation and corresponding element addition. It can be seen from Table 2 that the time series classification method achieves the best classification performance, with balanced accuracy, precision, recall, and AUC of 81.95%, 72.72%, 72.72%, and 83.21%, respectively. Compared with the second best performing feature splicing method, the time series classification strategy improves the balanced accuracy, precision, recall and AUC by 6.02%, 9.09%, 9.09% and 3.12%, respectively.

[0039] In order to demonstrate the superiority of the present invention in target change assessment performance, it is compared with traditional machine learning methods and existing deep learning methods for change assessment. The quantitative results of the comparative experiment are statistically shown in Table 3. It can be observed that the model adopted by the present invention achieves the most accurate results, with balanced accuracy, precision, recall and AUC of 81.95%, 72.72%, 72.72% and 83.21% respectively. Compared with traditional machine learning methods, there are significant improvements of 16.44%, 29.87%, 18.18% and 15.13% in b-Acc, precision, recall and AUC respectively. Compared with MPNet, it also improves by 12.04%, 18.18%, 18.18% and 0.11 in b-Acc, precision, recall and AUC respectively. Compared with MMDL, the best performing model among all the comparison methods, these indicators are also improved by 6.02%, 14.39%, 9.08% and 5.43% respectively.

[0040] Table 1

[0041] Table 2

[0042] Table 3

[0043] In summary, in order to solve the problems of existing research on change assessment based on multi-time point input images, the present invention focuses on three aspects: First, the spatial information of the target is extracted using parallel segmentation branches to guide the classification network to perform change assessment based on images at different time points. In order to obtain the correspondence between local regions from images at two different time points, a cross-temporal feature interaction module is designed, using attention mechanisms at different levels. In addition, temporal feature classification is used instead of traditional feature fusion to capture changes. Experimental results show that the model used in the present invention performs better than other state-of-the-art methods for change assessment.

[0044] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A target change assessment system based on multiple time points, characterized in that: The invention comprises: image segmentation branches before and after the change and their corresponding segmentation attention guidance modules, feature extractors of the images before and after the change, an interactive spatial feature enhancement module and a temporal feature classification module, wherein: the image segmentation branch before the change performs encoding and decoding operations according to the image information containing the target at the previous time point to obtain the segmentation result of the target before the change, and the image segmentation branch after the change performs encoding and decoding operations according to the image information containing the target at the next time point to obtain the segmentation result of the target after the change; the segmentation attention guidance module calculates the linear separability of each feature element in the multi-scale intermediate layer feature map output by the decoders in the two image segmentation branches to obtain the attention score map result; then the two changes in the evaluation main network are evaluated. The feature extractors for the image before and after the change focus on the target area according to the attention score map results to perform feature extraction operations, and obtain the hidden features corresponding to the input images at two time points; then the two interactive spatial feature enhancement modules in the change assessment main network use the image before and after the change as references, calculate the cross-attention scores between their local areas according to the two hidden features obtained by the feature extractor, and enhance the previously obtained hidden features accordingly to achieve the correspondence between the local associated areas of the target before and after the change; finally, the temporal feature classification network composed of a gated recurrent unit and a multi-layer perceptron in the change assessment main network generates an evaluation result based on the enhanced hidden features and auxiliary indicators.

2. The target change assessment system based on multiple time points according to claim 1 is characterized in that: The image segmentation branches before and after the change have the same structure, both including an encoder and a decoder, wherein: the encoder performs encoding processing based on convolution operations according to the image before or after the change; the decoder decodes the encoded features and obtains the segmentation result of the target to be evaluated for change, and extracts the spatial position information of the target through the attention guidance module to limit the classification network's attention to the key area.

3. The target change assessment system based on multiple time points according to claim 1 is characterized in that: The linear separability of each feature element means that for each element s in the multi-scale intermediate layer feature map, The energy diagram is calculated, where: μ c and are the channel mean and variance of the element s, respectively, and λ is the regularization term.

4. The target change assessment system based on multiple time points according to claim 1 is characterized in that: The cross attention score is obtained by: Step 1: Divide the image before and after the change into S×S×D non-overlapping regions according to the hidden layer features obtained by the feature extractor; Step 2: Calculate the correlation matrix between sub-regions at different time points to obtain the correspondence between the local areas before and after the target changes. Take the image features before the change as an example. Specifically, linearly map the features before the change of the divided regions through learnable parameters to obtain Q r Then, the K is obtained from the changed features of the divided regions in the same way. r and V r , establish the regional level correlation matrix A between the features before and after the change r =Q r (K r ) T , each element in the matrix represents the correlation score between the area before the change and the area after the change. According to the correlation score, for each non-overlapping area of ​​the hidden layer feature before the change, the k non-overlapping areas of the hidden layer features after the change with the largest correlation are selected; Step 3: After further dividing these areas into image blocks, linear mapping is performed again to calculate the similarity between the query and the key, that is, the dot product is normalized to obtain the attention weight, and the calculated attention weight is weighted multiplied with the value to obtain the enhanced changed feature, thereby achieving the purpose of corresponding between the local associated areas of the target before and after the change.

5. The target change assessment system based on multiple time points according to claim 1 is characterized in that: The evaluation results are obtained by converting the enhanced hidden features output by the interactive spatial feature enhancement module into a one-dimensional vector through a temporal feature classification module. At this time, after splicing it with auxiliary indicators according to the task situation, the changes between two target time points are evaluated with the help of a temporal feature classification network. Specifically, the vector is input into a temporal feature classification network composed of a gated recurrent unit and a multi-layer perceptron. Through the memory capacity of the gated recurrent unit and the target-related information at the previous time point, the changes in the features at the later time point are captured and predicted.

6. The target change assessment method based on multiple time points according to any one of claims 1 to 5, characterized in that: In the offline stage, a sample set is constructed and the target change assessment system is trained. In the online stage, the trained target change assessment system is used to extract features from input images at two different time points in real time and focus on key areas. The interactive spatial feature enhancement module calculates the correlation between key areas at two time points at the feature level to achieve feature enhancement, and then further classifies the images after problem conversion, thereby achieving change assessment based on images at multiple time points. The training mentioned above means: by designing a task-related loss function, calculating the error loss between the model output and the true label, and then automatically updating the network model parameters with the help of gradient back propagation. The training is considered completed when the loss converges with the training rounds.

7. The target change assessment method according to claim 6, characterized in that: include: Step 1: Use two independent feature extractors to extract the input image X at different time points pre and X post While extracting features, the input image X pre and X post Input an independent parallel segmentation branch containing an encoder and a decoder to perform segmentation and obtain the target segmentation result and The decoder obtains the multi-scale intermediate layer feature map S; for the element s in the multi-scale intermediate layer feature map S, The energy diagram is calculated, where: μ c and are the channel mean and variance of element s, respectively, and λ is the regularization term; the inverse of each element in the energy map is taken and normalized to obtain the final attention map, which contains the spatial position information of the target. The multi-scale attention map is Hadamard-multiplied with the corresponding scale features in the feature extractor in the main classification network to obtain the hidden layer features obtained by the segmentation-guided feature extractor at two time points before and after the change. and Step 2: Establish the correlation between local regions of images at different time points through interactive spatial feature enhancement, that is, obtain the feature F at the two time points obtained in step 1. pre and F post After being divided into multiple non-overlapping regions, the corresponding relationship between the two local regions is found, including: 2.1, taking the image features before the change as the query, the features F obtained at two time points pre and F post It is divided into S×S×D non-overlapping regions. S and D can be adjusted according to the size of the input image. The Q is obtained from the features before the division of the region through the learnable parameters. r , obtain K from the characteristics of the changes in the divided area r and V r ; 2.2 Establishing the regional level correlation matrix A between features before and after the change r =Q r (K r ) T , each element in the matrix represents the correlation score between the region before and after the change; 2.3 Perform cross-attention calculation on all elements of each query region before the change and all elements of the top k regions after the change with the largest correlation, so as to further explore the correlation between features; Step 3: Construct a time series classification problem for feature fusion, including: 3.1 The features obtained from the segmentation-guided feature extractor and the cross-temporal interaction module are transformed into two one-dimensional time series f through global average pooling. pre and f post ; 3.2 Two auxiliary indicators c related to change assessment at different time points pre and c post Splice with the corresponding time series; 3.3 The two concatenated feature sequences are output to the gated recurrent unit; 3.4 The final result of target change evaluation is obtained by a multilayer perceptron containing three fully connected layers 8. The target change assessment method according to claim 6, characterized in that the change assessment system composed of the two segmentation branches and the change assessment main network is trained together, and the loss function is specifically: in: The Dice coefficient loss includes Focus loss L focal =-α t (1-p t ) γ log(p t ), Y pre and Y post They represent the reference standards of the target area at different time points, p refers to all pixels in the image, and p t is the probability of the model classifying the correct category of the current sample. When the label is 1, p t =p, when the label is 0, then p t =1-p, p is the probability of the positive class output by the model, α t is a balancing factor used to adjust the weights of positive and negative samples.

Images