A post-ischemic conditioning method and system for multi-modal image evaluation

Abstract

This invention provides a method and system for post-ischemic processing regulation of multimodal image assessment. The regulation method includes: acquiring at least two modalities of spatially registered cerebral ischemia images; calculating and generating pixel-level modality difference maps; using a shared encoder convolutional neural network to extract features from the multimodal images and difference maps to obtain multi-source initial feature maps; generating an initial ischemic region probability mask and constructing a spatial weight map with the maximum value at the boundary between the ischemic core region and the penumbra; simultaneously extracting dynamic features from synchronous non-image clinical time-series data and mapping them as channel attention weights; constructing a dual-regulation attention module to achieve spatial bias and channel recalibration at multiple depth levels; combining hierarchical progressive gating regulation to fuse multi-source features; splicing multi-level fused features and decoding them; and outputting an ischemic region segmentation map and a clinical prognostic quantitative assessment value.

Description

Technical Field

[0001] This application belongs to the field of regulation, and in particular relates to a method and system for regulating post-ischemic treatment based on multimodal imaging assessment. Background Technology

[0002] In clinical practice, rapid diagnosis of ischemic lesions, especially distinguishing between the irreversibly damaged ischemic core and the salvageable ischemic penumbra, is crucial for developing reperfusion therapy plans such as thrombolysis and thrombectomy, and for predicting patient prognosis. Multimodal medical imaging technologies, such as diffusion-weighted imaging (DWI), perfusion-weighted imaging (PFI), and computed tomography (CT), have become the gold standard for diagnosis. Different modalities of imaging can reveal the physiological and pathological state of lesions from different dimensions. However, reliably integrating these diverse and complementary imaging data to delineate lesion boundaries, especially in areas with heterogeneous signal intensity and blurred boundaries, remains a significant challenge. Furthermore, non-imaging clinical data of patients, such as stroke scale scores, blood pressure, and heart rate, contain rich prognostic information about disease severity and trends. Traditional analysis methods have failed to establish an intrinsic link between spatial imaging features and clinical temporal dynamics, limiting the comprehensive assessment of the condition and hindering integrated and accurate diagnosis and treatment.

[0003] Early fusion strategies failed to adequately consider the complex dependencies and weight differences between different modalities, resulting in limited fusion effectiveness. Most attention modules lack guidance from prior clinical knowledge, leading to insufficient focus on key diagnostic regions and inability to accurately define boundaries. For the fusion of clinical time-series data, existing methods fail to effectively utilize data change information to regulate the extraction and screening of image features in real-time and in depth. Existing network structures are mostly single-shot, static feature fusion models, lacking a hierarchical, progressive, coarse-to-fine correction and gating mechanism. Therefore, an analytical method capable of deeply integrating clinical prior knowledge, fusing spatiotemporal information, and possessing hierarchical correction capabilities is urgently needed. Summary of the Invention

[0004] To address the problem that existing technologies fail to deeply integrate clinical prior knowledge and lack a hierarchical, coarse-to-fine correction and gating mechanism.

[0005] In the first aspect, the present invention proposes a method for modulating post-ischemic treatment based on multimodal imaging assessment, comprising: At least two modalities of spatially registered brain ischemia images are acquired, and pixel-level modality difference maps are generated. A convolutional neural network with a shared encoder is used to extract features from the multimodal images and the modality difference maps to obtain a multi-source initial feature map. The multi-source initial feature map is input into the auxiliary decoder to generate an initial ischemic region probability mask, and a spatial weight map is constructed based on the mask; at the same time, non-image clinical time-series data synchronized with the image is acquired, and time-series dynamic features are extracted through a one-dimensional convolutional and gated recurrent unit network, and the time-series dynamic features are mapped to channel attention weights; A dual-control attention module is constructed, which uses the spatial weight map to addatively spatially bias the attention score map and uses the channel attention weight to recalibrate the value feature map. The dual-control attention module is applied to multiple deep layers of the convolutional neural network to fuse multi-source features and obtain intermediate fused features at each layer. The intermediate fused features generated at the deep layers are input into the probability prediction head to generate the corrected ischemic region posterior probability map of the layer. The posterior probability map is used as a gating signal, upsampled, and multiplied element-wise with the intermediate fused features output by the shallow layer attention module to achieve hierarchical progressive gating control. The multi-level fusion features generated at different depths of the convolutional neural network, which have undergone hierarchical progressive gating, are spliced ​​and decoded to output an ischemic region segmentation map and a quantitative assessment value representing the clinical prognosis of the region.

[0006] Optionally, acquiring at least two modalities of spatially registered brain ischemia images and calculating and generating pixel-level modality difference maps includes: Intensity normalization was performed on the first modality diffusion-weighted imaging image and the second modality perfusion-weighted imaging image respectively; Obtain the absolute intensity difference between the two normalized images at each corresponding pixel to obtain the initial difference map; The initial difference map is smoothed using a Gaussian filter to reduce noise and generate a modal difference map.

[0007] Optionally, constructing a spatial weight map that maximizes the value at the boundary between the ischemic core region and the penumbra based on the mask includes: Pixels with a probability value greater than the first preset threshold in the initial ischemic region probability mask are identified as ischemic core region masks; Pixels with a probability value greater than the second preset threshold in the initial ischemic region probability mask are identified as the total ischemic region mask; Remove the corresponding part of the ischemic core region mask from the total ischemic region mask to obtain the ischemic half-dark band mask; Using the boundary between the ischemic core region and the penumbra as the zero contour line, calculate the signed Euclidean distance from each pixel in the image to the boundary. The calculated distance transformation map is nonlinearly transformed using a Gaussian function to generate a spatial weight map.

[0008] Optionally, the extraction of temporal dynamic features through a one-dimensional convolutional and gated recurrent unit network includes: Each patient's stroke scale score sequence, containing multiple time points, is used as input; The scoring sequence is input into a network containing two one-dimensional convolutional layers, with each convolutional layer followed by a modified linear unit activation function; The output sequence of the convolutional layer is input into a bidirectional gated recurrent unit layer to extract and output the temporal dynamic feature vector.

[0009] Optionally, the construction of the dual-regulation attention module, which uses the spatial weight map to apply an additive spatial bias to the attention score map and uses the channel attention weights to perform channel recalibration on the value feature map, includes: After calculating the dot product of the query feature map and the key feature map to generate the original attention score map, the spatial weight map is added element by element to the original attention score map to achieve spatial bias. The temporal dynamic features are mapped through a fully connected layer to a channel attention weight vector equal to the number of channels in the value feature map; The channel recalibration is achieved by multiplying the value feature map element-wise with the channel attention weight vector. The spatially biased attention score map is processed by an activation function and then multiplied by a matrix with the channel-recalibrated value feature map to obtain the attention fusion feature.

[0010] Optionally, the step of inputting the intermediate fusion features generated at the deep level into the probability prediction head to generate the corrected ischemic region posterior probability map of the level, and using the posterior probability map as a gating signal, upsampling it and multiplying it element-wise with the intermediate fusion features output by the shallow level attention module to achieve hierarchical progressive gating control includes: The intermediate fusion features generated at the current deep level are passed through a probability prediction head consisting of a convolutional layer and an activation function to generate a corrected posterior probability map. The posterior probability map is upsampled using bilinear interpolation to match the spatial resolution with the shallow-level intermediate fusion features. The upsampled posterior probability map is multiplied element-wise with the intermediate fusion features output by the shallow dual-modulation attention module.

[0011] Optionally, the step of splicing and decoding the multi-level fusion features generated at different depths of the convolutional neural network through hierarchical progressive gating to output an ischemic region segmentation map and a quantitative assessment value representing the clinical prognosis of the region includes: The fused feature maps from three different depth levels in the network decoder path are upsampled by transposed convolutional layers of corresponding numbers to restore the feature maps to the same spatial resolution as the input image. The three upsampled feature maps are concatenated along the channel dimension to form an aggregated feature map; The aggregated feature map is input in parallel into the segmentation branch and the prognostic evaluation branch; In the segmentation branch, pixel-level ischemic region segmentation maps are generated through convolutional layers and activation functions; In the prognostic assessment branch, a single scalar value representing the clinical prognosis is output through a global average pooling layer and a fully connected layer.

[0012] In another aspect, the present invention also proposes a multimodal imaging assessment system for post-ischemic treatment regulation, comprising the following modules: The acquisition module is used to acquire at least two modalities of spatially registered brain ischemia images and calculate and generate pixel-level modality difference maps; a convolutional neural network with a shared encoder is used to extract features from the multimodal images and the modality difference maps to obtain multi-source initial feature maps; The extraction module is used to input the multi-source initial feature map into the auxiliary decoder to generate an initial ischemic region probability mask, and construct a spatial weight map based on the mask; at the same time, it acquires non-image clinical time-series data synchronized with the image, extracts time-series dynamic features through a one-dimensional convolutional and gated recurrent unit network, and maps the time-series dynamic features into channel attention weights; A generation module is used to construct a dual-regulation attention module. It uses the spatial weight map to additively spatially bias the attention score map and uses the channel attention weights to recalibrate the value feature map. The dual-regulation attention module is applied to multiple deep layers of the convolutional neural network to fuse multi-source features, obtaining intermediate fusion features for each layer. The intermediate fusion features generated at the deep layers are input to a probability prediction head to generate a corrected posterior probability map of the ischemic region at that layer. This posterior probability map is used as a gating signal, upsampled, and then multiplied element-wise with the intermediate fusion features output by the shallow-level attention module to achieve layered, progressive gating control. The output module is used to splice and decode the multi-level fusion features generated at different depths of the convolutional neural network after hierarchical progressive gating, and output the ischemic region segmentation map and the quantitative assessment value representing the clinical prognosis of the region.

[0013] Preferably, the step of acquiring at least two modalities of spatially registered brain ischemia images and calculating and generating pixel-level modality difference maps includes: Intensity normalization was performed on the first modality diffusion-weighted imaging image and the second modality perfusion-weighted imaging image respectively; Obtain the absolute intensity difference between the two normalized images at each corresponding pixel to obtain the initial difference map; The initial difference map is smoothed using a Gaussian filter to reduce noise and generate a modal difference map.

[0014] Preferably, the step of constructing a spatial weight map that achieves the maximum value at the boundary between the ischemic core region and the penumbra based on the mask includes: Pixels with a probability value greater than the first preset threshold in the initial ischemic region probability mask are identified as ischemic core region masks; Pixels with a probability value greater than the second preset threshold in the initial ischemic region probability mask are identified as the total ischemic region mask; Remove the corresponding part of the ischemic core region mask from the total ischemic region mask to obtain the ischemic half-dark band mask; Using the boundary between the ischemic core region and the penumbra as the zero contour line, calculate the signed Euclidean distance from each pixel in the image to the boundary. The calculated distance transformation map is nonlinearly transformed using a Gaussian function to generate a spatial weight map.

[0015] Preferably, the extraction of temporal dynamic features through a one-dimensional convolutional and gated recurrent unit network includes: Each patient's stroke scale score sequence, containing multiple time points, is used as input; The scoring sequence is input into a network containing two one-dimensional convolutional layers, with each convolutional layer followed by a modified linear unit activation function; The output sequence of the convolutional layer is input into a bidirectional gated recurrent unit layer to extract and output the temporal dynamic feature vector.

[0016] Preferably, the construction of the dual-regulation attention module, which uses the spatial weight map to apply an additive spatial bias to the attention score map and uses the channel attention weights to perform channel recalibration on the value feature map, includes: After calculating the dot product of the query feature map and the key feature map to generate the original attention score map, the spatial weight map is added element by element to the original attention score map to achieve spatial bias. The temporal dynamic features are mapped through a fully connected layer to a channel attention weight vector equal to the number of channels in the value feature map; The channel recalibration is achieved by multiplying the value feature map element-wise with the channel attention weight vector. The spatially biased attention score map is processed by an activation function and then multiplied by a matrix with the channel-recalibrated value feature map to obtain the attention fusion feature.

[0017] Preferably, the step of inputting the intermediate fusion features generated at the deep level into the probability prediction head to generate the corrected ischemic region posterior probability map of the level, and using the posterior probability map as a gating signal, upsampling it, and multiplying it element-wise with the intermediate fusion features output by the shallow level attention module to achieve hierarchical progressive gating control includes: The intermediate fusion features generated at the current deep level are passed through a probability prediction head consisting of a convolutional layer and an activation function to generate a corrected posterior probability map. The posterior probability map is upsampled using bilinear interpolation to match the spatial resolution with the shallow-level intermediate fusion features. The upsampled posterior probability map is multiplied element-wise with the intermediate fusion features output by the shallow dual-modulation attention module.

[0018] Preferably, the step of splicing and decoding the multi-level fusion features generated at different depths of the convolutional neural network through hierarchical progressive gating to output an ischemic region segmentation map and a quantitative assessment value representing the clinical prognosis of the region includes: The fused feature maps from three different depth levels in the network decoder path are upsampled by transposed convolutional layers of corresponding numbers to restore the feature maps to the same spatial resolution as the input image. The three upsampled feature maps are concatenated along the channel dimension to form an aggregated feature map; The aggregated feature map is input in parallel into the segmentation branch and the prognostic evaluation branch; In the segmentation branch, pixel-level ischemic region segmentation maps are generated through convolutional layers and activation functions; In the prognostic assessment branch, a single scalar value representing the clinical prognosis is output through a global average pooling layer and a fully connected layer.

[0019] This invention integrates three types of heterogeneous information—multimodal images, pixel-level modal difference maps, and non-image-based clinical time-series data—by constructing a dual attention regulation and hierarchical gating mechanism. Specifically, by utilizing a spatial weight map generated by an auxiliary decoder, the model's attention is biased towards the boundary region between the ischemic core and the penumbra, enhancing the ability to identify key pathological boundaries. Simultaneously, inputting clinical time-series data recalibrates the image feature channels, allowing the feature fusion process to incorporate the patient's individualized pathological state. Through a hierarchical, progressive gating structure, the prediction results of shallow networks are used to filter and optimize deep features, suppressing background noise and artifacts layer by layer, achieving coarse-to-fine lesion localization. This improves the accuracy of ischemic lesion segmentation, especially in the penumbra region, and enables the output of quantitative assessment values ​​related to clinical prognosis, making the analysis results more clinically valuable. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the hardware device architecture; Figure 2 This is a schematic diagram of the architecture of a multimodal stroke analysis system; Figure 3 This is a schematic diagram of the internal calculations of the dual-regulation attention module. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] Hardware device architecture such as Figure 1 As shown, it mainly consists of a magnetic resonance scanner, a computing server, a display terminal, and a data storage module. In this application, a method for modulating ischemic post-treatment in multimodal image assessment is described, such as... Figure 2 As shown, it includes the following steps: S1, acquire at least two modalities of spatially registered brain ischemia images and calculate and generate pixel-level modality difference maps; use a convolutional neural network with a shared encoder to extract features from the multimodal images and the modality difference maps to obtain multi-source initial feature maps; Load diffusion-weighted imaging (DWI) and perfusion-weighted imaging (PWI) sequences registered using the ANTs registration tool into NIfTI format using medical image processing libraries such as SimpleITK or NiBabel. Convert the DWI and PWI image data into NumPy arrays, and calculate the difference map by subtracting pixels one by one and taking the absolute value. The formula is as follows: DWI, PWI, and differential... Figure 3 Image data from each channel are stacked along the channel dimension to form a multi-channel input tensor. An encoder based on the U-Net architecture is constructed, with the backbone network employing a ResNet-34 structure, including a series of torch.nn.Conv2d 2D convolutional layers, torch.nn.BatchNorm2d batch normalization layers, torch.nn.ReLU activation function, and torch.nn.MaxPool2d max pooling layers. The concatenated multi-channel tensor is fed as a single input into this shared encoder. At different downsampling stages of the encoder, such as after each residual block, multi-source initial feature maps of different sizes and semantic levels are output.

[0023] In some embodiments, acquiring at least two modalities of spatially registered brain ischemia images and calculating and generating pixel-level modal difference maps includes: Intensity normalization was performed on the first modality diffusion-weighted imaging image and the second modality perfusion-weighted imaging image respectively; Obtain the absolute intensity difference between the two normalized images at each corresponding pixel to obtain the initial difference map; The initial difference map is smoothed using a Gaussian filter to reduce noise and generate a modal difference map.

[0024] Intensity normalization was performed independently on DWI and PWI image slices for each patient, preferably using the Z-score normalization method, which calculates the mean of all pixels for each image slice. and standard deviation And through the formula The intensity value of each pixel is converted into a distribution centered at 0 with a standard deviation of 1, eliminating the overall brightness differences caused by different scanning devices and parameters. After normalization, at pixel positions i,j, the intensity is calculated... This generates an initial difference map that highlights areas where DWI high signal and PWI low perfusion do not match, i.e., potential ischemic penumbra.

[0025] To enhance feature stability and suppress random noise from physiological pulsations or image acquisition inputs, a two-dimensional Gaussian filter is applied to smooth the initial difference map. The preferred kernel size of the Gaussian filter is 5x5, with a standard deviation of [missing value]. Set to 1.5. This parameter combination can smooth noise while preserving clinically significant regional differences to the maximum extent, avoiding excessive blurring that could lead to the loss of key boundary information. The Gaussian-smoothed image is the modality map, which will be fed as an independent third input channel along with the original DWI and PWI images into the subsequent shared encoder network.

[0026] In an optional embodiment, the convolutional neural network employing a shared encoder extracts features from the multimodal image and the modality difference map to obtain a multi-source initial feature map, including: For example, the three types of image data are uniformly converted into NumPy arrays and stacked along the channel dimension in a channel-first format to form a 3-channel multimodal input tensor with dimensions of batch size × 3 × 256 × 256. A shared encoder convolutional neural network with ResNet-34 as the backbone is constructed. This encoder has no classification head and consists of an input convolutional module and four groups of residual blocks. The input convolutional module is a 7×7 convolutional layer + BatchNorm2d batch normalization layer + ReLU activation function + 3×3 max pooling layer. The four groups of residual blocks are as follows: a 64-channel residual block group with a stride of 1 and no downsampling; a 128-channel residual block group with a stride of 2 and downsampling; a 256-channel residual block group with a stride of 2 and downsampling; and a 512-channel residual block group with a stride of 2 and downsampling. Each group is composed of concatenated basic residual blocks of the corresponding number of channels, and each residual block is followed by a BatchNorm2d batch normalization layer and a ReLU activation function. The sample is fed into the shared encoder as a single input. Feature extraction is performed after the input convolution module of the encoder and at the output stage of the four residual block groups. The output features are B×64×64×64, B×64×64×64, B×128×32×32, B×256×16×16, and B×512×8×8 respectively. The feature maps obtained from different downsampling stages of the encoder, with progressively decreasing spatial resolution and progressively increasing semantic level and number of channels, are the multi-source initial feature maps.

[0027] S2, the multi-source initial feature map is input to the auxiliary decoder to generate an initial ischemic region probability mask, and a spatial weight map is constructed based on the mask; at the same time, non-image clinical time-series data synchronized with the image is acquired, and time-series dynamic features are extracted through a one-dimensional convolutional and gated recurrent unit network, and the time-series dynamic features are mapped to channel attention weights; The feature map output from the deepest layer of the encoder is input into a lightweight auxiliary decoder consisting of torch.nn.ConvTranspose2d deconvolutional layers and torch.nn.Conv2d convolutional layers. A single-channel initial probability mask is generated using the torch.nn.Sigmoid activation function. Core region and penumbra masks are extracted based on thresholding, the signed Euclidean distance is calculated, and a Gaussian transform is applied to obtain a spatial weight map, maximizing the weights at the core-penumbra boundary. Simultaneously, a CSV file recording the NIHSS stroke score over time is loaded, normalized, and then input into a temporal feature extraction network consisting of a torch.nn.Conv1d one-dimensional convolutional layer and a torch.nn.GRU gated recurrent unit layer. The hidden state vector of the last time step of the GRU is taken and mapped to a vector with a length equal to the number of channels in the subsequent image feature map through a multilayer perceptron containing two torch.nn.Linear fully connected layers and a torch.nn.Sigmoid activation function. This vector is the channel attention weight.

[0028] In some embodiments, constructing a spatial weight map that maximizes the value at the boundary between the ischemic core region and the penumbra based on the mask includes: Pixels with a probability value greater than the first preset threshold in the initial ischemic region probability mask are identified as ischemic core region masks; Pixels with a probability value greater than the second preset threshold in the initial ischemic region probability mask are identified as the total ischemic region mask; Remove the corresponding part of the ischemic core region mask from the total ischemic region mask to obtain the ischemic half-dark band mask; Using the boundary between the ischemic core region and the penumbra as the zero contour line, calculate the signed Euclidean distance from each pixel in the image to the boundary. The calculated distance transformation map is nonlinearly transformed using a Gaussian function to generate a spatial weight map.

[0029] Initial probability mask generated using the auxiliary decoder The pixel values ​​in this image range from [0,1]. Two thresholds are set, the first being a preset threshold. Preferably 0.85, the second preset threshold Preferably 0.45, to ensure .pass and Generate binary masks for the ischemic core region and the total ischemic region, respectively. This is achieved through Boolean operations. ,in For logical AND operation, The logical NOT operation is used to extract the ischemic penumbra mask. The boundary between the ischemic core region and the penumbra is determined; the boundary is... Zhongyu The set of directly adjacent pixels.

[0030] Based on this boundary, the signed Euclidean distance transform of the entire image is calculated, such that the pixel distance on the boundary is 0, the pixel distance within the half-dark band is positive, and the pixel distance within the core region is negative. After obtaining the distance transform map D(x,y), a Gaussian function is applied... The distance transformation graph is converted into a spatial weight graph. Standard deviation This is a key hyperparameter, with an optimal value of 5.0, which controls the rate of weight decay. It causes the weight map to peak at 1 at the boundary between the core region and the penumbra, and then smoothly decrease towards both sides, thus forcing the model to focus more on the clinically critical areas in the subsequent attention module.

[0031] In some embodiments, the extraction of temporal dynamic features through a one-dimensional convolutional and gated recurrent unit network includes: Each patient's stroke scale score sequence, containing multiple time points, is used as input; The scoring sequence is input into a network containing two one-dimensional convolutional layers, with each convolutional layer followed by a modified linear unit activation function; The output sequence of the convolutional layer is input into a bidirectional gated recurrent unit layer to extract and output the temporal dynamic feature vector.

[0032] A temporal dynamic feature extraction network is employed. The network structure consists of a sequentially connected one-dimensional convolutional layer, a modified linear unit activation function (MLU), a second one-dimensional convolutional layer, a second MLU, and a bidirectional gated recurrent unit layer. The network input is an NIHSS score sequence for each patient, containing multiple time points, with dimensions of batch size B, sequence length T, and feature dimension 1, represented as follows: The network output is a time-series dynamic feature vector with dimensions equal to the batch size B and 256, represented as follows: .

[0033] The input NIHSS score sequence is a vector sequence, such as a patient's scores at three time points: admission, 24 hours, and 72 hours. The input dimension is the batch size, 3, 1. The sequence is passed through a first one-dimensional convolutional layer with 32 output channels, a kernel size of 3, a stride of 1, and the same padding. This one-dimensional convolutional layer is followed by a ReLU activation function to detect local variation patterns between scores. The output feature sequence is then passed through a second one-dimensional convolutional layer with 64 output channels. This layer has the same kernel size, stride, and padding as the previous layer and also uses ReLU activation to extract higher-level abstract temporal relationships.

[0034] The feature sequence after two convolutional layers is fed into a bidirectional gated recurrent unit (Bi-GRU) layer with 128 hidden units. Due to its bidirectional structure, it processes the entire sequence from both the forward and backward directions, enabling simultaneous detection of the influence of early scores on later scores and the summary of the overall trend by later scores. The hidden states of the forward GRU at the last time step and the hidden states of the backward GRU at the first time step are concatenated to obtain a 256-dimensional vector. This vector comprehensively encodes the overall severity, trend, and long-term dependencies of the NIHSS scores, serving as a temporal dynamic feature.

[0035] S3. Construct a dual-control attention module, using the spatial weight map to additively spatially bias the attention score map, and using the channel attention weights to recalibrate the value feature map. Apply the dual-control attention module to multiple deep layers of the convolutional neural network to fuse multi-source features and obtain intermediate fusion features at each layer. Input the intermediate fusion features generated at the deep layers into the probability prediction head to generate the corrected ischemic region posterior probability map of the layer. Use the posterior probability map as a gating signal, upsample it, and multiply it element-wise with the intermediate fusion features output by the shallow layer attention module to achieve hierarchical progressive gating control. A dual-modulation attention module is constructed. For a specific level of feature map output by the encoder, three independent 1x1 convolutions (using the `torch.nn.Conv2d` function) are used to generate query Q, key K, and value V feature maps. The dot product of the query and key is then calculated to obtain the original attention score map. The spatial weight map generated in the preceding steps is adjusted to the same size as the attention score map using bilinear interpolation. It is then element-wise added to the original attention score map to achieve additive spatial bias. The biased score map is normalized using the Softmax function. For the value V feature map, the dimension of the channel attention weight vector generated in the preceding steps is expanded from B×C to B×C×1×1, and element-wise multiplied with the value V feature map to achieve channel recalibration. The normalized attention score map is then matrix-multiplied with the recalibrated value V feature map to obtain the intermediate fusion feature for this layer. Deep-level intermediate fusion features, such as the third layer (low resolution, high semantics), are selected and used to generate the posterior probability map for this layer using a probability prediction head containing 3x3 convolutions and sigmoid activation. This posterior probability map is then upsampled to the size of shallow-level intermediate fusion features, such as the second layer (high resolution, fine details), and element-wise multiplied with the intermediate fusion features output from the second layer to achieve progressive gating from coarse to fine.

[0036] In an alternative embodiment, the construction of the dual-modulation attention module, which uses the spatial weight map to additively spatially bias the attention score map and uses the channel attention weights to channel recalibrate the value feature map, includes: After calculating the dot product of the query feature map and the key feature map to generate the original attention score map, the spatial weight map is added element by element to the original attention score map to achieve spatial bias. The temporal dynamic features are mapped through a fully connected layer to a channel attention weight vector equal to the number of channels in the value feature map; The channel recalibration is achieved by multiplying the value feature map element-wise with the channel attention weight vector. The spatially biased attention score map is processed by an activation function and then multiplied by a matrix with the channel-recalibrated value feature map to obtain the attention fusion feature.

[0037] This is achieved through a dual-regulation attention module. The module's structure includes a linear projection layer for generating query, key, and value feature maps; an addition operation for fusing spatial weight maps; a fully connected layer with a sigmoid activation function for generating channel weights based on temporal dynamic features; and a matrix multiplication operation for feature fusion. The module's input is a feature map from the encoder. Spatial weighted graph and temporal dynamic feature vectors The module's output is an attention fusion feature map, with the same dimensions as the input feature map, i.e. .

[0038] In a standard multi-head self-attention computation process, the input feature map is linearly mapped to three tensors: Q, K, and V. The original attention score map is then processed... The calculation yielded, where This refers to the dimension of the key vector. Input spatial bias: The previously generated spatial weight map W is downsampled so that the spatial resolution of W is... Matching, via broadcast mechanism By adding element by element, we get This additive bias operation increases the weight of the ischemic boundary region in the attention calculation, guiding the model to focus on feature interactions at key locations.

[0039] The channel dimension of the value feature map V is recalibrated by inputting the previously extracted 256-dimensional temporal dynamic feature vector into a fully connected layer. The output dimension of this layer is exactly the same as the number of channels C of V. Then, a sigmoid activation function is applied to generate a channel attention weight vector of dimension C with a range of 0 and 1. Connect V with the channel attention weight vector. Perform element-wise multiplication ,in This represents broadcast multiplication along the channel dimension. This operation allows the model to enhance or suppress the importance of different feature channels based on the severity and trends of the patient's clinical condition. The spatially biased attention score map is normalized using the Softmax function and then multiplied by a matrix with the channel-recalibrated value feature map. This yields a dual-modulation attention feature that integrates spatial prior and clinical temporal information. The computation of the dual-modulation attention module is as follows: Figure 3 As shown.

[0040] In some embodiments, the step of inputting the intermediate fusion features generated at the deep level into the probability prediction head to generate the corrected ischemic region posterior probability map of the level, and using the posterior probability map as a gating signal, upsampling it, and multiplying it element-wise with the intermediate fusion features output by the shallow level attention module to achieve hierarchical progressive gating control includes: The intermediate fusion features generated at the current deep level are passed through a probability prediction head consisting of a convolutional layer and an activation function to generate a corrected posterior probability map. The posterior probability map is upsampled using bilinear interpolation to match the spatial resolution with the shallow-level intermediate fusion features. The upsampled posterior probability map is multiplied element-wise with the intermediate fusion features output by the shallow dual-modulation attention module.

[0041] The probability prediction head is a lightweight neural network, with a structure of a 3x3 convolutional layer followed by a sigmoid activation function; the network input is the intermediate fused feature map of the current layer, for example... The network output is a single-channel posterior probability map with the same spatial resolution as the input, for example... .

[0042] Gating mechanisms are implemented across multiple layers in the decoder path; for example, the gating signal is passed from the third layer (L3) to the second layer (L2). The intermediate fused features output from the dual-modulated attention module at layer L3 are then used. For example, a 32x32 pixel array with 256 channels is input into a lightweight probability prediction head. This prediction head consists of a 3x3 convolutional layer with 1 output channel, 1 padding, and a sigmoid activation function. The output is a single-channel posterior probability map. The size is 32x32, and the pixel value is between [0,1]. It represents the prediction of the ischemic area at the current feature level.

[0043] To use this probability map as a shallow-level gating signal, it needs to be size-matched. The probability map generated from layer L3... The 32x32 size is obtained by upsampling by 2x using bilinear interpolation. This achieves a spatial resolution of 64x64 for the sampled probability map, matching the output features of the L2 attention module. The dimensions are exactly the same. Perform element-wise multiplication. ,in This represents the Hadamard product. This operation utilizes predictions from deeper, more globally semantically informed levels to filter and weight shallower, spatially detailed features, suppressing the representation of noisy and irrelevant regions, thus achieving progressive feature optimization from coarse to fine.

[0044] S4 splices and decodes the multi-level fusion features generated at different depths of the convolutional neural network after hierarchical progressive gating, and outputs an ischemic region segmentation map and a quantitative assessment value representing the clinical prognosis of the region.

[0045] The upsampled features from each layer of the U-Net decoder are concatenated with the fused features from the corresponding encoder layer after dual attention modulation and hierarchical gating, using the `torch.cat` function along the channel dimension. The decoder consists of a series of `torch.nn.ConvTranspose2d` deconvolutional layers and cascaded residual blocks. The highest resolution output feature map of the decoder is passed through a 1x1 `torch.nn.Conv2d` convolutional layer and a `torch.nn.Sigmoid` activation function to generate a pixel-level ischemic region segmentation map. Simultaneously, to predict clinical prognosis, the feature map output from the deepest layer of the encoder is subjected to global average pooling (`torch.nn.AdaptiveAvgPool2d`) to obtain a feature vector. This vector is then input into a multilayer perceptron regression head consisting of multiple `torch.nn.Linear` fully connected layers and a `torch.nn.ReLU` activation function. The regression head outputs a single neuron; the linear output value of this neuron is the quantized evaluation value, such as the predicted modified Rankin Scale (mRS) score after 3 months. The entire network uses a weighted sum of Dice loss and binary cross-entropy loss as the loss function for the segmentation task, and uses mean squared error loss, i.e., MSELoss, as the loss function for the prognostic regression task, for end-to-end joint training.

[0046] In an alternative embodiment, the step of splicing and decoding the multi-level fusion features generated at different depths of the convolutional neural network through hierarchical progressive gating to output an ischemic region segmentation map and a quantitative assessment value representing the clinical prognosis of the region includes: The fused feature maps from three different depth levels in the network decoder path are upsampled by transposed convolutional layers of corresponding numbers to restore the feature maps to the same spatial resolution as the input image. The three upsampled feature maps are concatenated along the channel dimension to form an aggregated feature map; The aggregated feature map is input in parallel into the segmentation branch and the prognostic evaluation branch; In the segmentation branch, pixel-level ischemic region segmentation maps are generated through convolutional layers and activation functions; In the prognostic assessment branch, a single scalar value representing the clinical prognosis is output through a global average pooling layer and a fully connected layer.

[0047] The segmentation branch and the prognostic evaluation branch in this step are two independent neural networks; the structure of the segmentation branch is: a 3x3 convolutional layer, a 1x1 convolutional layer, and a sigmoid activation function, with the aggregated feature map as the input. The output is a pixel-level segmentation probability map. The structure of the prognostic evaluation branch is as follows: a global average pooling layer and two fully connected layers, wherein the first fully connected layer is followed by a modified linear unit activation function, and the input is also an aggregated feature map. The output is a single scalar value representing the clinical prognosis. .

[0048] Assuming the input image resolution is 256x256, the decoder selects output features from three key levels, denoted as follows: Size 64x64, number of channels 256 Size 128x128, number of channels 128 and Dimensions: 256x256, Number of channels: 64. Applying two 2x2 transposed convolutions, or deconvolutions, with a stride of 2, to achieve a 4x upsampling; Apply a 2x2 transposed convolution with a stride of 2 to achieve a 2x upsampling; Since the target resolution is already achieved, no further upsampling is needed. After all upsampling operations, the spatial resolution of the three feature maps is unified to 256x256. The three upsampled feature maps are then concatenated along the channel dimension to form a single aggregated feature map. The number of channels is 448.

[0049] The aggregated feature map Two parallel task branches are simultaneously fed in. In the split branch, First, a 3x3 convolutional layer reduces the number of channels to 64, then a 1x1 convolutional layer reduces the number of channels to 1. Finally, a sigmoid activation function is applied to generate a 256x256 pixel-level ischemic region segmentation probability map. In the prognostic evaluation branch... After passing through a global average pooling layer, the spatial dimension of the feature map is compressed to obtain a 448-dimensional feature vector. This vector is then fed into a multilayer perceptron (MLP) containing two fully connected layers. For example, the first fully connected layer reduces the dimension from 448 to 64 and uses ReLU activation, while the second fully connected layer maps the 64-dimensional vector to 1-dimensional vector, outputting a single scalar value without an activation function. This value is the quantitative prediction of the patient's clinical prognosis, such as the modified Rankin Scale score after 3 months.

[0050] The experiments were based on a dataset containing 350 patients with acute ischemic stroke, using a unified training, validation, and test set partitioning. All models were trained for 100 epochs on an NVIDIA A100 GPU. The baseline model was a standard two-channel U-Net network, using only DWI and PWI images as input, achieving a Dice similarity coefficient of 0.825 on the ischemic lesion segmentation task and a mean absolute error of 1.25 on the 3-month modified Rankin Scale score prediction task. In the control experiment, using pixel-level modality difference maps as the third input channel, the model's Dice coefficient on the segmentation task was improved to 0.841, and the mean absolute error of prognostic prediction was reduced to 1.22.

[0051] Experiment 2, based on the baseline model and modality difference map, spatially biased the attention score using only the spatial weight map. The Dice coefficient for the segmentation task increased to 0.863, and the mean absolute error of the prognosis prediction decreased to 1.18. This demonstrates that the strategy of forcing the model to focus on the ischemic core and the penumbra boundary region can improve the extraction efficiency of key features.

[0052] Experiment 3 further integrated temporal dynamic features based on NIHSS scoring sequences to achieve channel recalibration of the value feature map. This complete dual-regulation module reduced the mean absolute error of prognostic prediction to 1.05 and slightly increased the segmentation Dice coefficient to 0.868. The results analysis indicates that incorporating clinical temporal information allows the model to adjust its dependence on different image feature channels according to the patient's condition, which is particularly crucial for improving prognostic assessment tasks.

[0053] Experiment 4 incorporated a hierarchical gating mechanism into the model, guiding the selection of shallow features through deep features. This improved the precision of the segmentation results, increasing the Dice coefficient to 0.882 and optimizing the mean absolute error of prognosis to 1.02. This validated the complete scheme involving multi-level feature fusion and dual-task decoding. The model upsamples and concatenates feature maps from three different levels of the decoder before feeding them into parallel segmentation and prognosis branches. It achieved a Dice coefficient of 0.895 in the segmentation task and a mean absolute error of 0.96 in the prognosis prediction task.

[0054] Based on any of the above embodiments, this application also provides a multimodal image assessment post-ischemic treatment regulation system, comprising the following modules: The acquisition module is used to acquire at least two modalities of spatially registered brain ischemia images and calculate and generate pixel-level modality difference maps; a convolutional neural network with a shared encoder is used to extract features from the multimodal images and the modality difference maps to obtain multi-source initial feature maps; The extraction module is used to input the multi-source initial feature map into the auxiliary decoder to generate an initial ischemic region probability mask, and construct a spatial weight map based on the mask; at the same time, it acquires non-image clinical time-series data synchronized with the image, extracts time-series dynamic features through a one-dimensional convolutional and gated recurrent unit network, and maps the time-series dynamic features into channel attention weights; A generation module is used to construct a dual-regulation attention module. It uses the spatial weight map to additively spatially bias the attention score map and uses the channel attention weights to recalibrate the value feature map. The dual-regulation attention module is applied to multiple deep layers of the convolutional neural network to fuse multi-source features, obtaining intermediate fusion features for each layer. The intermediate fusion features generated at the deep layers are input to a probability prediction head to generate a corrected posterior probability map of the ischemic region at that layer. This posterior probability map is used as a gating signal, upsampled, and then multiplied element-wise with the intermediate fusion features output by the shallow-level attention module to achieve layered, progressive gating control. The output module is used to splice and decode the multi-level fusion features generated at different depths of the convolutional neural network after hierarchical progressive gating, and output the ischemic region segmentation map and the quantitative assessment value representing the clinical prognosis of the region.

[0055] In this specification, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise limited, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In this document, "a," "an," "the," "the," and "its" may also include plural forms unless the context clearly indicates otherwise. "Multiple" refers to at least two, such as 2, 3, 5, or 8, etc. "And / or" includes any and all combinations of the associated listed items.

[0056] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0057] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for modulating ischemic post-treatment through multimodal imaging assessment, characterized in that, Includes the following steps: At least two modalities of spatially registered brain ischemia images are acquired, and pixel-level modality difference maps are generated. A convolutional neural network with a shared encoder is used to extract features from the multimodal images and the modality difference maps to obtain a multi-source initial feature map. The multi-source initial feature map is input into the auxiliary decoder to generate an initial ischemic region probability mask, and a spatial weight map is constructed based on the mask; at the same time, non-image clinical time-series data synchronized with the image is acquired, and time-series dynamic features are extracted through a one-dimensional convolutional and gated recurrent unit network, and the time-series dynamic features are mapped to channel attention weights; A dual-regulation attention module is constructed, which uses the spatial weight map to perform additive spatial bias on the attention score map and uses the channel attention weight to perform channel recalibration on the value feature map. The dual-regulation attention module is applied to multiple deep layers of the convolutional neural network to fuse multi-source features and obtain intermediate fused features at each layer. The intermediate fusion features generated at the deep level are input into the probability prediction head to generate the corrected ischemic region posterior probability map of the level. The posterior probability map is then used as a gating signal, upsampled, and multiplied element-wise with the intermediate fusion features output by the shallow level attention module to achieve hierarchical and progressive gating control. The multi-level fusion features generated at different depths of the convolutional neural network, which have undergone hierarchical progressive gating, are spliced ​​and decoded to output an ischemic region segmentation map and a quantitative assessment value representing the clinical prognosis of the region.

2. The method according to claim 1, characterized in that, The acquisition of at least two modalities of spatially registered brain ischemia images and the calculation of pixel-level modality difference maps include: Intensity normalization was performed on the first modality diffusion-weighted imaging image and the second modality perfusion-weighted imaging image respectively; Obtain the absolute intensity difference between the two normalized images at each corresponding pixel to obtain the initial difference map; The initial difference map is smoothed using a Gaussian filter to reduce noise and generate a modal difference map.

3. The method according to claim 1, characterized in that, The construction of a spatial weight map based on the mask, which maximizes the value at the boundary between the ischemic core region and the penumbra, includes: Pixels with a probability value greater than the first preset threshold in the initial ischemic region probability mask are identified as ischemic core region masks; Pixels with a probability value greater than the second preset threshold in the initial ischemic region probability mask are identified as the total ischemic region mask; Remove the corresponding part of the ischemic core region mask from the total ischemic region mask to obtain the ischemic half-dark band mask; Using the boundary between the ischemic core region and the penumbra as the zero contour line, calculate the signed Euclidean distance from each pixel in the image to the boundary. The calculated distance transformation map is nonlinearly transformed using a Gaussian function to generate a spatial weight map.

4. The method according to claim 1, characterized in that, The extraction of temporal dynamic features through a one-dimensional convolutional and gated recurrent unit network includes: Each patient's stroke scale score sequence, containing multiple time points, is used as input; The scoring sequence is input into a network containing two one-dimensional convolutional layers, with each convolutional layer followed by a modified linear unit activation function; The output sequence of the convolutional layer is input into a bidirectional gated recurrent unit layer to extract and output the temporal dynamic feature vector.

5. The method according to claim 1, characterized in that, The construction of the dual-regulation attention module, which uses the spatial weight map to apply an additive spatial bias to the attention score map and uses the channel attention weights to perform channel recalibration on the value feature map, includes: After calculating the dot product of the query feature map and the key feature map to generate the original attention score map, the spatial weight map is added element by element to the original attention score map to achieve spatial bias. The temporal dynamic features are mapped through a fully connected layer to a channel attention weight vector equal to the number of channels in the value feature map; The channel recalibration is achieved by multiplying the value feature map element-wise with the channel attention weight vector. The spatially biased attention score map is processed by an activation function and then multiplied by a matrix with the channel-recalibrated value feature map to obtain the attention fusion feature.

6. The method according to claim 3, characterized in that, The process involves inputting the intermediate fusion features generated at the deep level into the probability prediction head to generate a corrected posterior probability map of the ischemic region at that level. This posterior probability map is then used as a gating signal, upsampled, and multiplied element-wise with the intermediate fusion features output from the shallow-level attention module to achieve layered, progressive gating control. This includes: The intermediate fusion features generated at the current deep level are passed through a probability prediction head consisting of a convolutional layer and an activation function to generate a corrected posterior probability map. The posterior probability map is upsampled using bilinear interpolation to match the spatial resolution with the shallow-level intermediate fusion features. The upsampled posterior probability map is multiplied element-wise with the intermediate fusion features output by the shallow dual-modulation attention module.

7. The method according to claim 1, characterized in that, The process of splicing and decoding the multi-level fusion features generated at different depths of the convolutional neural network through hierarchical progressive gating to output an ischemic region segmentation map and a quantitative assessment value representing the clinical prognosis of the region includes: The fused feature maps from three different depth levels in the network decoder path are upsampled by transposed convolutional layers of corresponding numbers to restore the feature maps to the same spatial resolution as the input image. The three upsampled feature maps are concatenated along the channel dimension to form an aggregated feature map; The aggregated feature map is input in parallel into the segmentation branch and the prognostic evaluation branch; In the segmentation branch, pixel-level ischemic region segmentation maps are generated through convolutional layers and activation functions; In the prognostic assessment branch, a single scalar value representing the clinical prognosis is output through a global average pooling layer and a fully connected layer.

8. A multimodal imaging assessment system for post-ischemic treatment regulation, characterized in that, Includes the following modules: The acquisition module is used to acquire at least two modalities of spatially registered brain ischemia images and calculate and generate pixel-level modality difference maps; a convolutional neural network with a shared encoder is used to extract features from the multimodal images and the modality difference maps to obtain multi-source initial feature maps; The extraction module is used to input the multi-source initial feature map into the auxiliary decoder to generate an initial ischemic region probability mask, and construct a spatial weight map based on the mask; at the same time, it acquires non-image clinical time-series data synchronized with the image, extracts time-series dynamic features through a one-dimensional convolutional and gated recurrent unit network, and maps the time-series dynamic features into channel attention weights; A generation module is used to construct a dual-regulation attention module, which uses the spatial weight map to perform additive spatial bias on the attention score map and uses the channel attention weight to perform channel recalibration on the value feature map. The dual-regulation attention module is applied to multiple deep layers of the convolutional neural network to fuse multi-source features and obtain intermediate fused features at each layer. The intermediate fusion features generated at the deep level are input into the probability prediction head to generate the corrected ischemic region posterior probability map of the level. The posterior probability map is then used as a gating signal, upsampled, and multiplied element-wise with the intermediate fusion features output by the shallow level attention module to achieve hierarchical and progressive gating control. The output module is used to splice and decode the multi-level fusion features generated at different depths of the convolutional neural network after hierarchical progressive gating, and output the ischemic region segmentation map and the quantitative assessment value representing the clinical prognosis of the region.

9. The system according to claim 8, characterized in that, The acquisition of at least two modalities of spatially registered brain ischemia images and the calculation of pixel-level modality difference maps include: Intensity normalization was performed on the first modality diffusion-weighted imaging image and the second modality perfusion-weighted imaging image respectively; Obtain the absolute intensity difference between the two normalized images at each corresponding pixel to obtain the initial difference map; The initial difference map is smoothed using a Gaussian filter to reduce noise and generate a modal difference map.

10. The system according to claim 8, characterized in that, The construction of a spatial weight map based on the mask, which maximizes the value at the boundary between the ischemic core region and the penumbra, includes: Pixels with a probability value greater than the first preset threshold in the initial ischemic region probability mask are identified as ischemic core region masks; Pixels with a probability value greater than the second preset threshold in the initial ischemic region probability mask are identified as the total ischemic region mask; Remove the corresponding part of the ischemic core region mask from the total ischemic region mask to obtain the ischemic half-dark band mask; Using the boundary between the ischemic core region and the penumbra as the zero contour line, calculate the signed Euclidean distance from each pixel in the image to the boundary. The calculated distance transformation map is nonlinearly transformed using a Gaussian function to generate a spatial weight map.

Images