An intent-driven multi-modal medical image fusion method and system based on a MIND model

Abstract

The application discloses an intention-driven multi-modal medical image fusion method based on a MIND model, which comprises the following steps: acquiring double-mode medical images and generating fusion text according to a diagnosis intention; a multi-scale latent adapter extracts spatial features from the images; the spatial features are decoupled into high-frequency and low-frequency sequences and frequency-domain-time-step joint affine modulation is performed to generate intention-modulated spatial topology Tokens; the fusion text, time-step embedding, noise, visual Tokens and spatial topology Tokens are input into a diffusion Transformer to predict a vector field, and meanwhile, in the diffusion Transformer, time-varying gating coefficients are used to inject spatial features into a hidden layer; an adaptive dynamic truncation threshold is calculated, a loss function is established, and network parameters are updated to generate a fusion image. The application solves the problem that physical fidelity and clinical semantic alignment are difficult to consider simultaneously, and significantly improves the accuracy of the fusion image in a lesion segmentation task.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing and computer vision technology, specifically relating to an intent-driven multimodal medical image fusion method and system based on the MIND model. Background Technology

[0002] Existing medical image fusion techniques have evolved from traditional methods to deep learning approaches. Convolutional neural network-based methods leverage their feature extraction capabilities to capture texture at multiple scales, but are limited by local receptive fields and cannot effectively establish global long-range dependencies. Generative adversarial network-based methods improve fusion quality through a game between the generator and discriminator, but often face problems such as training instability and cross-modal semantic inconsistencies. Recently, diffusion models, with their powerful generative capabilities, have been introduced into this field, generating fused images through progressive denoising in an unconstrained latent space.

[0003] To improve the targeting of image fusion, text-driven image fusion methods have emerged, which guide the network to focus on specific structures or regions by introducing textual cues. For example, the invention patent with publication number CN121639594A discloses a method and system for thyroid ultrasound-assisted diagnosis based on text-driven visual pre-training and cross-modal feature fusion. The model construction and training stages include: preprocessing to obtain unlabeled and labeled image-text pair datasets; constructing an image encoder and a text encoder, performing pre-training to achieve text-driven visual pre-training; using a cross-modal splicing attention fusion module (CMCAF) to fuse text features and image features and inputting them into a fully connected layer classifier, which is then optimized through supervised training; the application stage is to predict the benign or malignant nature of thyroid nodules based on the image-text pairs to be predicted.

[0004] For example, the invention patent with publication number CN122392886A discloses a multimodal glaucoma grading diagnosis method, which includes: First, generating the representation of the missing modality at the feature semantic level through a cross-modal adversarial generative network to construct a complete multimodal feature set; Second, mapping the features to anatomical regions based on a predefined structured anatomical atlas, and generating an integrated feature description with confidence through local sign recognition and graph neural network reasoning; Finally, adopting a hierarchical fusion framework based on evidence theory, first synthesizing evidence in clinical sub-dimensions such as optic disc morphology, retinal nerve fiber layer structure, and visual field function, and then performing global fusion based on the dynamic weighting of the uncertainty of each sub-dimension to output a graded diagnosis conclusion and a structured diagnosis report.

[0005] However, existing technologies still suffer from the following significant problems: First, most existing fusion methods rely on unified global fusion rules, lacking a deep understanding of specific diagnostic intentions and pathological structures. Second, while the emerging diffusion Transformer architecture is highly scalable in global context modeling, its core mechanism of flattening two-dimensional latent variables into one-dimensional sequences easily disrupts the necessary local physical topological continuity in medical images, leading to tissue boundary distortion and the generation of false textures. Third, existing semantic guidance typically involves implicit soft weight redistribution at the feature level, lacking explicit physical constraints on the pixel domain of the generated image. This easily leads to severe semantic shifts and loss of clinically critical features in complex multimodal feature interactions. These shortcomings make it difficult for existing methods to achieve a reliable balance between accurate physical fidelity and high-level clinical semantic alignment when dealing with complex pathological structures, ultimately affecting the accuracy of downstream tasks such as lesion segmentation. Summary of the Invention

[0006] In view of the above, the purpose of this invention is to provide an intent-driven multimodal medical image fusion method and system based on the MIND model. First, it utilizes a visual language model to generate intent-driven fusion text. Second, addressing the structural defect of one-dimensional serialization in the diffusion Transformer that disrupts two-dimensional anatomical topology, it employs a multi-scale latent adapter to extract and preserve continuous two-dimensional anatomical features. Based on a time-step-aware adaptive gating mechanism, it injects high-frequency details into the shallow layer and global semantics into the deep layer, achieving hierarchical topology compensation that dynamically adjusts with the denoising progress. Furthermore, it introduces an adaptive dynamic truncation mechanism, forcibly shutting down semantic loss in the early, high-noise stage of denoising to prioritize the complete reconstruction of the physical skeleton, allowing semantic constraints to intervene only after the contours stabilize in the later stage. This mathematically decouples the coupling conflict between the underlying physical manifold and the high-level diagnostic semantics. Through collaborative design at the model architecture and loss function levels, it solves the problems of spatial continuity loss and semantic control offset caused by serialization, achieving a perfect combination of high physical fidelity and high semantic consistency. Ultimately, it significantly improves the accuracy of the fused image in downstream tasks such as lesion segmentation.

[0007] To achieve the above-mentioned objectives, an embodiment provides an intent-driven multimodal medical image fusion method based on the MIND model, comprising the following steps: S1. Acquire the first modality medical image and the second modality medical image, and generate intent-driven fusion text based on the specified diagnostic intent; S2. Encode and stitch together the first modality medical image and the second modality medical image, and then extract multi-scale spatial features through a multi-scale latent adapter; S3. Decouple the multi-scale spatial features into high-frequency and low-frequency sequences, and perform frequency-time step joint affine modulation on the decoupled sequences to generate spatial topology tokens intended for modulation. S4. The text embedding, time step embedding, target image noise, visual tokens and spatial topology tokens are concatenated and input into the diffusion Transformer prediction vector field. At the same time, in the forward propagation of the diffusion Transformer, the time-varying gating coefficients generated at the current time step are used to dynamically inject multi-scale spatial features into the hidden layer of the diffusion Transformer according to the corresponding scale. S5. Calculate the adaptive dynamic truncation threshold based on the vector field. When the generation time step is greater than the dynamic truncation threshold, activate the medical semantic consistency loss and calculate the physical reconstruction loss. Update the parameters of the diffusion Transformer and the multi-scale potential adapter based on the continuous flow matching loss, physical reconstruction loss and medical semantic consistency loss. S6. In the inference stage, starting from pure noise, iterative sampling is performed in the vector field predicted by the updated diffusion Transformer to generate the final fused medical image.

[0008] Preferably, step 1 specifically includes: The visual language model is input with intent-driven prompts and first and second modal medical images. The prompts include the following constraints: constraining the visual language pre-trained large model to be a multimodal medical image expert, used to define the ideal visual features for fusing diagnostic intent and describing the target image; enabling the visual language pre-trained large model to explicitly state the high-resolution structural features to be retained from anatomical images and the metabolic or physiological signals to be highlighted from functional images in an academic medical tone, and limiting the output to a preset number of short sentences; using a first general placeholder and a second general placeholder to refer to the input first and second modal medical images, respectively. The intent description output by the visual language model is obtained, and a preset standard prefix containing the basic fusion task identifier and image order declaration is concatenated with the intent description to form fused text.

[0009] Preferably, the extraction of multi-scale spatial features through a multi-scale latent adapter as described in S2 includes: Two-dimensional residual blocks are used to extract continuous anatomical edge features of the latent variables after splicing; By downsampling continuous anatomical edge features through pooling layers of different lengths, a hierarchical feature pyramid is constructed to obtain a multi-scale feature map. Embedding convolutions are performed on feature maps at each scale and mapped to the target sequence dimension to obtain multi-scale spatial features.

[0010] Preferably, the decoupling of multi-scale spatial features into high-frequency and low-frequency sequences as described in S3 includes: Local average pooling is performed on the multi-scale spatial features to extract low-frequency sequences; The high-frequency sequence is obtained by subtracting the low-frequency sequence from the multi-scale spatial features.

[0011] Preferably, the process described in S3, which involves performing frequency-time step joint affine modulation on the decoupled sequence to generate spatial topology tokens intended for modulation, includes: The time step embedding is nonlinearly mapped using a multilayer perceptron to generate time step scaling vectors and global displacement vectors that independently control the low-frequency sequence and the low-frequency sequence, respectively. The time step scaling vector and the global displacement vector are used to perform affine modulation on the low-frequency sequence and the high-frequency sequence respectively, and project them into a dynamic query vector. The embedded text is modulated with a bias towards low-frequency semantics and projected into a dynamic key vector, and the value vector is obtained through conventional linear projection. Cross-modal relevance weights are generated based on dynamic query vectors and dynamic key vectors. These cross-modal relevance weights are then used to weight and aggregate value vectors to generate spatial topology tokens for intent modulation.

[0012] Preferably, the method described in S4, which uses time-varying gating coefficients generated at the current time step to dynamically inject multi-scale spatial features into the hidden layer of the diffusion Transformer according to the corresponding scale, includes: The gating network based on the current hidden layer dynamically maps the embedding of continuous time steps and generates time-varying gating coefficients through Sigmoid activation; Based on the depth of the hidden layer, a corresponding anatomical prior sequence is assigned, and the product of the time-varying gating coefficient and the preset basic scaling parameter is used as the injection intensity. The product of the injection intensity and the anatomical prior sequence is superimposed at the self-attention output of the hidden layer and the output of the multilayer perceptron, respectively. In this process, shallow hidden layers are assigned high-resolution feature sequences to supplement high-frequency details, while deep hidden layers are assigned low-resolution feature sequences to supplement global semantics.

[0013] Preferably, the adaptive dynamic truncation threshold described in S5 is calculated as follows: Based on the basic truncation threshold, the product of the sensitivity coefficient and the L2 norm of the partial derivative of the predicted vector field with respect to time is added, and the result is truncated and restricted to the interval between 0 and 1.

[0014] Preferably, the activation and calculation method for medical semantic consistency loss in S5 is as follows: Cosine similarity is calculated based on the features of the predicted fused image and the fused text; When the generation time step is greater than the adaptive dynamic truncation threshold and the cosine similarity has not reached the set upper limit, the negative value of the cosine similarity is calculated using the time step as the weight coefficient as the loss value; otherwise, the loss is set to zero.

[0015] On the other hand, the embodiment also provides an intent-driven multimodal medical image fusion system based on the MIND model, used to implement the aforementioned intent-driven multimodal medical image fusion method based on the MIND model, including: The fusion text generation module is used to acquire a first modality medical image and a second modality medical image, and generate intent-driven fusion text based on the specified diagnostic intent; The multi-scale feature adaptation module is used to encode and stitch together the first modality medical image and the second modality medical image, and then extract multi-scale spatial features through the multi-scale latent adapter; The spatial feature modulation module is used to decouple multi-scale spatial features into high-frequency and low-frequency sequences, and perform frequency-time step joint affine modulation on the decoupled sequences to generate spatial topology tokens intended for modulation. The joint sequence prediction module is used to input the fused text embedding, time step embedding, target image noise, visual tokens and spatial topology tokens into the diffusion Transformer prediction vector field; at the same time, in the forward propagation of the diffusion Transformer, the time-varying gating coefficients generated at the current time step are used to dynamically inject multi-scale spatial features into the hidden layer of the diffusion Transformer according to the corresponding scale. The training module is used to calculate an adaptive dynamic truncation threshold based on the vector field. When the generation time step is greater than the dynamic truncation threshold, the medical semantic consistency loss is activated and the physical reconstruction loss is calculated. The parameters of the diffusion Transformer and the multi-scale latent adapter are updated based on the continuous flow matching loss, the physical reconstruction loss and the medical semantic consistency loss. The generation module, used in the inference phase, starts from pure noise and iteratively samples in the vector field predicted by the updated diffusion Transformer to generate the final fused medical image.

[0016] On the other hand, the embodiment also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, performs the intent-driven multimodal medical image fusion method based on the MIND model.

[0017] Compared with the prior art, the beneficial effects of the present invention include at least the following: (1) By using a multi-scale latent adapter and an adaptive gated injection mechanism, the gap between one-dimensional sequence modeling and the continuous topology of two-dimensional medical images is bridged, and the local structure of medical images is preserved while retaining the global generation capability of the diffusion Transformer.

[0018] (2) The time step truncation mechanism separates the underlying physical manifold reconstruction from the high-level diagnostic semantic injection, ensuring the Lipschitz continuity of the probability flow from a mathematical perspective, eliminating the risk of discretization truncation error explosion, and enabling the model to still have significant physical defense robustness when facing extreme or erroneous text interference.

[0019] (3) The intent-driven text generation paradigm is directly geared towards clinical goals, enabling fused images to achieve the best balance between preserving anatomical texture and highlighting functional metabolic hotspots, which significantly improves the accuracy of downstream clinical medical segmentation tasks such as brain tumors. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of the intent-driven multimodal medical image fusion method based on the MIND model provided in the embodiment.

[0022] Figure 2 This is a flowchart of the data processing for the multi-scale potential adapter and multi-objective loss function provided in the embodiment.

[0023] Figure 3 This is a schematic diagram of the structure of the intent-driven multimodal medical image fusion system based on the MIND model provided in the embodiment. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0025] The technical concept of this invention lies in the following: In the data input and intent generation stage, for registered bimodal medical images, a pre-trained visual language model is used to prospectively generate intent text describing ideal fusion features based on specific clinical prompts, breaking the limitations of traditional unified fusion rules. Secondly, in the feature extraction and injection stage, to address the problem of one-dimensional serialization of the diffusion Transformer destroying the topological structure, a multi-scale latent adapter is designed. This adapter performs two-dimensional residual convolution and multi-scale pooling on the latent variables of the stitched image in the latent space, explicitly extracting continuous anatomical edge features, and mapping them to an adapter feature sequence with the same dimension as the text image sequence through non-overlapping patch embedding. Subsequently, a time-step-aware adaptive gating injection mechanism is adopted, dynamically adjusting the injection intensity of multi-scale anatomical prior features in each hidden layer through time-varying gating coefficients. High-resolution adapter feature sequences are injected into the shallow hidden layers of the diffusion Transformer to supplement high-frequency details, while low-resolution adapter feature sequences are injected into the deep hidden layers to supplement global semantics. Finally, in the continuous flow matching and network optimization stage, an ordinary differential equation trajectory from Gaussian noise to the target data distribution is defined to predict the vector field. To avoid the global application of semantic constraints from disrupting the underlying physical manifold, a medical semantic consistency loss with a time-step truncation mechanism was designed. Only when the time step of the generation process exceeds a set threshold is the cosine similarity between the generated image and the fused text calculated using a pre-trained model and a semantic penalty applied. In the early, high-noise stage, the physical reconstruction loss is mainly used to protect the anatomical skeleton, thereby achieving deep feature decoupling and intent control without compromising the integral stability of ordinary differential equations.

[0026] Based on the above-mentioned inventive concept, the method and system provided by this invention are based on a continuous flow matching framework and construct a novel MIND model (Multimodal Intent-driven Network via Diffusion Transformers). The MIND model combines time-step-aware modulation with multi-scale feature adaptation mechanisms to achieve high-fidelity fusion of medical images. The overall implementation of the method of this invention includes four core stages: intent text generation, latent variable encoding and adaptation, joint sequence processing, and loss optimization based on a truncation mechanism. Figure 1 As shown in the embodiment, an intent-driven multimodal medical image fusion method based on the MIND model is provided, including the following steps: S1, acquire the first modality medical image and the second modality medical image, and generate intent-driven fusion text based on the specified diagnostic intent.

[0027] In this embodiment, a first modality of medical images is acquired. Second modality medical images It utilizes a visual language model to generate intent-driven fusion text based on a specified diagnostic intent. Specifically, the user inputs intent-driven prompts and the aforementioned medical image pairs into the visual language pre-trained large model. The prompts have the following characteristics: constraining the visual language pre-trained large model to be a multimodal medical image expert, used to define the fusion diagnostic intent and describe the ideal visual features of the target image; enabling the model to clearly state, in a rigorous academic medical tone, the high-resolution structural features to be retained from anatomical images and the specific metabolic or physiological signals to be highlighted from functional images, and limiting the output to a preset number of short sentences; using a first general placeholder (<|image_1|>) and a second general placeholder (<|image_2|>) to represent the first and second modal medical images input, respectively. Based on the above input, the parametric prediction model outputs an intent description; a preset standard prefix containing the basic fusion task identifier and image order declaration is concatenated with the intent description to form fusion text used to guide the subsequent generation process. The fusion process can be defined by the following formula: , in, This represents the final fused image. This represents the continuous stream transform mapping of the parameterized prediction model under an ordinary differential equation solver. For first-modality medical images, For second-modality medical images, To integrate text, To obtain from the standard Gaussian distribution The initial noise variable in the sampling.

[0028] S2. Encode and stitch together the first and second modal medical images, and then extract multi-scale spatial features through a multi-scale latent adapter.

[0029] In this embodiment, this step involves image latent variable extraction and multi-scale latent space adaptation. Two medical images are compressed into the latent space using a pre-trained variational autoencoder to obtain a first image latent variable with multiple channels. Second image latent variable The calculation formula is as follows: , , in, This is a pre-trained variational autoencoder.

[0030] like Figure 2 As shown, the two are concatenated along the channel dimension to form the input adapter latent variable. Subsequently, a two-dimensional spatial prior is extracted using a multi-scale latent adapter. Specifically, two two-dimensional residual blocks are used to extract continuous anatomical edges, followed by downsampling through three pooling layers of different lengths to construct a hierarchical feature pyramid containing scales 0, 1, and 2. Then, a two-dimensional convolution is performed to flatten the features and apply layer normalization, accurately mapping the two-dimensional features at each scale to the target sequence dimension. The calculation formula is as follows: , Among them, the The adapter sequence of each scale is , Indicates the flattening operation. Presentation layer normalization operation, The patch size is indicated as follows: Non-overlapping convolution operations, This represents the processing of multi-scale potential adapter networks. Feature extraction operations at each scale.

[0031] S3 decouples multi-scale spatial features into high-frequency and low-frequency sequences, and performs frequency-time step joint affine modulation on the decoupled sequences to generate spatial topology tokens intended for modulation.

[0032] In this embodiment, conventional diffusion Transformers networks employ uniform time-step affine modulation, which cannot adapt to the technical shortcomings of varying sensitivity between high-frequency anatomical features and low-frequency functional metabolic features in medical images at different denoising stages of the ordinary differential equation (ODE) probability flow. Therefore, this embodiment designs a frequency-domain-time-step joint modulation mechanism. Figure 2 ): Specifically, the intended text to be merged is transformed into a merged text embedding through a word segmenter and a text embedding layer. The latent variables of the source image are transformed into visual tokens through block partitioning and linear projection; simultaneously, the dimension-aligned spatial feature sequence of the output of the second-stage multi-scale latent adapter is obtained. and time step embedding Embed the fused text Spatial feature sequence and time step embedding The common input to the cross-modal intent-spatial coordination module proceeds as follows: First, for multi-scale spatial feature sequences Explicit frequency domain decoupling is performed to extract low-frequency sequences containing functional background and smooth metabolic features. and high-frequency sequences containing sharp anatomical edges. : , , in, This is a local average pooling operation.

[0033] Subsequently, the time steps were embedded using a multilayer perceptron (MLP). Perform nonlinear mapping to generate time-step scaling vectors that independently control the high- and low-frequency features. and and global displacement vector : .

[0034] Subsequently, using the scaling vector , With displacement vector The spatial features are subjected to joint affine modulation in the frequency domain and time step and reconstructed into a dynamic query vector. Simultaneously embedding text Perform low-frequency semantic-biased modulation to generate dynamic key vectors Value vector Then it is obtained through conventional linear projection: , , , in, , , These are the corresponding linear projection weight matrices; For the hidden layer feature dimension, For the mapping dimension of queries and keys, The mapping dimension of the value; ⊙ represents the Hadamard product, used to calculate the query vector. transpose of key vector The dot product, after scaling and normalization using the Softmax function, generates a cross-modal multi-dominant semantic relevance weight matrix. : .

[0035] Subsequently, using the weight matrix logarithmic vector Perform weighted aggregation, and output the aggregation result through a projection matrix. Mapping back to feature dimension And input to a feedforward propagation network with gated linear units. The process involves performing a nonlinear mapping, combining it with residual connections, and outputting the final spatial topology tokens for intent modulation. The process is as follows: .

[0036] S4 inputs the concatenated text embedding, time step embedding, target image noise, visual tokens and spatial topology tokens into the diffusion Transformer prediction vector field; at the same time, in the forward propagation of the diffusion Transformer, the time-varying gating coefficients generated at the current time step are used to dynamically inject multi-scale spatial features into the hidden layer of the diffusion Transformer according to the corresponding scale.

[0037] In the embodiment, the text is embedded Time step embedding Target image noise Visual Tokens and Spatial Topology of Intent Modulation The sequences are concatenated to construct a multimodal collaborative input sequence, which is then input into the initial hidden layer of the diffusion Transformer.

[0038] In the adapter feature sequence injection stage, a time-step-aware adaptive gating injection mechanism is designed. Specifically, in the first... During the forward propagation of the hidden layer, the lightweight gating network unique to that hidden layer is utilized. Embedding of continuous time steps Dynamic mapping is performed; specifically, the gating network is constructed by cascading a multilayer perceptron (MLP) and a sigmoid activation function to generate time-varying gating coefficients corresponding to the current denoising progress. : , in, for Activation function This is for multilayer perceptron operation.

[0039] Subsequently, the time-varying gating coefficients were used to assign multi-scale anatomical prior sequences. The aligned spatial feature sequence undergoes adaptive flow control to dynamically adjust the feature injection intensity. Its forward propagation process is reconstructed as follows: , , Among them, in the first In the forward propagation of each hidden layer, This is the output sequence of the previous hidden layer. The default scaling parameters are the base scaling parameters. For the output of the self-attention layer, For the first The output of a multilayer perceptron. Shallow layers handle high-frequency edges, while deep layers handle global semantics, avoiding scale redundancy and boundary ambiguity caused by feature hierarchy disorder.

[0040] Within the continuous flow matching framework, the model is trained to predict the vector field of the ordinary differential equation trajectory connecting pure noise and the true target data distribution, calculated as follows: , , in, For continuous time steps The intermediate state of the trajectory of the ordinary differential equation under the given conditions. For the latent variables of the true target data distribution, Gaussian noise prior; For continuous flow matching loss, To predict the vector field for the diffusion Transformer, For the desired operation, It is the square of the L2 norm.

[0041] S5. Calculate the adaptive dynamic truncation threshold based on the vector field. When the generation time step is greater than the dynamic truncation threshold, activate the medical semantic consistency loss and calculate the physical reconstruction loss. Update the parameters of the diffusion Transformer and the multi-scale potential adapter based on the continuous flow matching loss, physical reconstruction loss and medical semantic consistency loss.

[0042] In this embodiment, within the time step range, the sharp target latent variable of the current denoising step is estimated using the first-order Euler formula. The calculation formula is as follows: , in, The target latent variable is estimated and decoded into an estimated fused image using a variational autoencoder. The cosine similarity is calculated by extracting features from the estimated fused image and the intent text using a multimodal image-text comparison model. , in, For cosine similarity, and These represent the image feature encoder and text feature encoder of the pre-trained contrastive model, respectively.

[0043] An adaptive dynamic truncation mechanism based on the Lipschitz continuity of vector fields is introduced. Existing techniques typically employ fixed empirical truncation thresholds, which cannot adapt to the dynamic changes in medical images of varying complexity during probability flow evolution, easily leading to integral manifold collapse. Therefore, this embodiment calculates the norm of the partial derivative of the current predicted vector field with respect to time in each iteration to construct a dynamic threshold function. : , in, Based on the truncation threshold, This is the sensitivity coefficient. This is a cutoff function that limits the threshold to [0,1]. This mechanism allows the model to perceive the steepness of the ODE trajectory in real time: it automatically raises the threshold when the vector field changes drastically. Delayed semantic intervention; when the trajectory is smooth, the threshold is automatically lowered to accelerate semantic fusion.

[0044] Only when the generation time step is greater than this dynamic threshold And the similarity did not reach the set limit. Only when the time step is less than or equal to the threshold is the medical semantic consistency loss calculated and multiplied by the time step as a weighting coefficient; in the very early high-noise stage, when the time step is less than or equal to the threshold, this semantic loss term is set to zero.

[0045] , in, For medical semantic consistency loss; Set an upper limit for the indicator function. When the time step And the similarity is less than the upper limit. The value is 1 if the condition is met, and 0 otherwise. Based on this, the first-paradigm reconstruction loss and structural similarity loss between the first-modal medical image and the generated image are calculated throughout the entire process: , , in, For physical reconstruction loss, the first image latent variable is used respectively. and source image Calculate the L1 loss and structural consistency loss, where SSIM is the structural similarity calculation function. and For weight hyperparameters, These are the adjustment weights for the semantic loss term; This is the total loss function used to update the network.

[0046] S6. In the inference stage, starting from pure noise, iterative sampling is performed in the vector field predicted by the updated diffusion Transformer to generate the final fused medical image.

[0047] In this embodiment, during the inference phase, given a new input image pair and intent text, the system starts from pure noise and iterates through the vector field predicted by the diffusion Transformer using an ordinary differential equation solver, ultimately outputting a fused medical image that satisfies the diagnostic intent. Through the above specific implementation, this invention achieves leading performance in both subjective quality evaluation and objective quantitative indicators of image fusion.

[0048] Based on the same technical concept, the embodiment also provides an intent-driven multimodal medical image fusion system based on the MIND model, used to implement the aforementioned intent-driven multimodal medical image fusion system based on the MIND model, such as... Figure 3 As shown, it includes: The fusion text generation module is used to acquire a first modality medical image and a second modality medical image, and generate intent-driven fusion text based on the specified diagnostic intent; The multi-scale feature adaptation module is used to encode and stitch together the first modality medical image and the second modality medical image, and then extract multi-scale spatial features through the multi-scale latent adapter; The spatial feature modulation module is used to decouple multi-scale spatial features into high-frequency and low-frequency sequences, and perform frequency-time step joint affine modulation on the decoupled sequences to generate spatial topology tokens intended for modulation. The joint sequence prediction module is used to input the fused text embedding, time step embedding, target image noise, visual tokens and spatial topology tokens into the diffusion Transformer prediction vector field; at the same time, in the forward propagation of the diffusion Transformer, the time-varying gating coefficients generated at the current time step are used to dynamically inject multi-scale spatial features into the hidden layer of the diffusion Transformer according to the corresponding scale. The training module is used to calculate an adaptive dynamic truncation threshold based on the vector field. When the generation time step is greater than the dynamic truncation threshold, the medical semantic consistency loss is activated and the physical reconstruction loss is calculated. The parameters of the diffusion Transformer and the multi-scale latent adapter are updated based on the continuous flow matching loss, the physical reconstruction loss and the medical semantic consistency loss. The generation module, used in the inference phase, starts from pure noise and iteratively samples in the vector field predicted by the updated diffusion Transformer to generate the final fused medical image.

[0049] Based on the same technical concept, the embodiment also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, performs the intent-driven multimodal medical image fusion method based on the MIND model.

[0050] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An intent-driven multimodal medical image fusion method based on the MIND model, characterized in that, Includes the following steps: S1. Acquire the first modality medical image and the second modality medical image, and generate intent-driven fusion text based on the specified diagnostic intent; S2. Encode and stitch together the first modality medical image and the second modality medical image, and then extract multi-scale spatial features through a multi-scale latent adapter; S3. Decouple the multi-scale spatial features into high-frequency and low-frequency sequences, and perform frequency-time step joint affine modulation on the decoupled sequences to generate spatial topology tokens intended for modulation. S4. The text embedding, time step embedding, target image noise, visual tokens and spatial topology tokens are concatenated and input into the diffusion Transformer prediction vector field. At the same time, in the forward propagation of the diffusion Transformer, the time-varying gating coefficients generated at the current time step are used to dynamically inject multi-scale spatial features into the hidden layer of the diffusion Transformer according to the corresponding scale. S5. Calculate the adaptive dynamic truncation threshold based on the vector field. When the generation time step is greater than the dynamic truncation threshold, activate the medical semantic consistency loss and calculate the physical reconstruction loss. Update the parameters of the diffusion Transformer and the multi-scale potential adapter based on the continuous flow matching loss, physical reconstruction loss and medical semantic consistency loss. S6. In the inference stage, starting from pure noise, iterative sampling is performed in the vector field predicted by the updated diffusion Transformer to generate the final fused medical image.

2. The intent-driven multimodal medical image fusion method based on the MIND model according to claim 1, characterized in that, Step 1 specifically includes: The visual language model is input with intent-driven prompts and first and second modal medical images. The prompts include the following constraints: constraining the visual language pre-trained large model to be a multimodal medical image expert, used to define the ideal visual features for fusing diagnostic intent and describing the target image; enabling the visual language pre-trained large model to explicitly state the high-resolution structural features to be retained from anatomical images and the metabolic or physiological signals to be highlighted from functional images in an academic medical tone, and limiting the output to a preset number of short sentences; using a first general placeholder and a second general placeholder to refer to the input first and second modal medical images, respectively. The intent description output by the visual language model is obtained, and a preset standard prefix containing the basic fusion task identifier and image order declaration is concatenated with the intent description to form fused text.

3. The intent-driven multimodal medical image fusion method based on the MIND model according to claim 1, characterized in that, The extraction of multi-scale spatial features via a multi-scale latent adapter as described in S2 includes: Two-dimensional residual blocks are used to extract continuous anatomical edge features of the latent variables after splicing; By downsampling continuous anatomical edge features through pooling layers of different lengths, a hierarchical feature pyramid is constructed to obtain a multi-scale feature map. Embedding convolutions are performed on feature maps at each scale and mapped to the target sequence dimension to obtain multi-scale spatial features.

4. The intent-driven multimodal medical image fusion method based on the MIND model according to claim 1, characterized in that, The decoupling of multi-scale spatial features into high-frequency and low-frequency sequences as described in S3 includes: Local average pooling is performed on the multi-scale spatial features to extract low-frequency sequences; The high-frequency sequence is obtained by subtracting the low-frequency sequence from the multi-scale spatial features.

5. The intent-driven multimodal medical image fusion method based on the MIND model according to claim 1, characterized in that, S3 describes performing frequency-time step joint affine modulation on the decoupled sequence to generate spatial topology tokens intended for modulation, including: The time step embedding is nonlinearly mapped using a multilayer perceptron to generate time step scaling vectors and global displacement vectors that independently control the low-frequency sequence and the high-frequency sequence, respectively. The time step scaling vector and the global displacement vector are used to perform affine modulation on the low-frequency sequence and the high-frequency sequence respectively, and project them into a dynamic query vector. The embedded text is modulated with a bias towards low-frequency semantics and projected into a dynamic key vector, and the value vector is obtained through conventional linear projection. Cross-modal relevance weights are generated based on dynamic query vectors and dynamic key vectors. These cross-modal relevance weights are then used to weight and aggregate value vectors to generate spatial topology tokens for intent modulation.

6. The intent-driven multimodal medical image fusion method based on the MIND model according to claim 1, characterized in that, The time-varying gating coefficients generated at the current time step described in S4, which dynamically inject multi-scale spatial features into the hidden layer of the diffusion Transformer according to the corresponding scale, include: The gating network based on the current hidden layer dynamically maps the embedding of continuous time steps and generates time-varying gating coefficients through Sigmoid activation; Based on the depth of the hidden layer, a corresponding anatomical prior sequence is assigned, and the product of the time-varying gating coefficient and the preset basic scaling parameter is used as the injection intensity. The product of the injection intensity and the anatomical prior sequence is superimposed at the self-attention output of the hidden layer and the output of the multilayer perceptron, respectively. In this process, shallow hidden layers are assigned high-resolution feature sequences to supplement high-frequency details, while deep hidden layers are assigned low-resolution feature sequences to supplement global semantics.

7. The intent-driven multimodal medical image fusion method based on the MIND model according to claim 1, characterized in that, The adaptive dynamic truncation threshold described in S5 is calculated as follows: Based on the basic truncation threshold, the product of the sensitivity coefficient and the L2 norm of the partial derivative of the predicted vector field with respect to time is added, and the result is truncated and restricted to the interval between 0 and 1.

8. The intent-driven multimodal medical image fusion method based on the MIND model according to claim 1, characterized in that, The activation and calculation method for semantic consistency loss in S5 Traditional Chinese Medicine is as follows: Cosine similarity is calculated based on the features of the predicted fused image and the fused text; When the generation time step is greater than the adaptive dynamic truncation threshold and the cosine similarity has not reached the set upper limit, the negative value of the cosine similarity is calculated using the time step as the weight coefficient as the loss value; otherwise, the loss is set to zero.

9. An intent-driven multimodal medical image fusion system based on the MIND model, characterized in that, The method for implementing the intent-driven multimodal medical image fusion based on the MIND model as described in any one of claims 1-8 includes: The fusion text generation module is used to acquire a first modality medical image and a second modality medical image, and generate intent-driven fusion text based on the specified diagnostic intent; The multi-scale feature adaptation module is used to encode and stitch together the first modality medical image and the second modality medical image, and then extract multi-scale spatial features through the multi-scale latent adapter; The spatial feature modulation module is used to decouple multi-scale spatial features into high-frequency and low-frequency sequences, and perform frequency-time step joint affine modulation on the decoupled sequences to generate spatial topology tokens intended for modulation. The joint sequence prediction module is used to input the fused text embedding, time step embedding, target image noise, visual tokens and spatial topology tokens into the diffusion Transformer prediction vector field; at the same time, in the forward propagation of the diffusion Transformer, the time-varying gating coefficients generated at the current time step are used to dynamically inject multi-scale spatial features into the hidden layer of the diffusion Transformer according to the corresponding scale. The training module is used to calculate an adaptive dynamic truncation threshold based on the vector field. When the generation time step is greater than the dynamic truncation threshold, the medical semantic consistency loss is activated and the physical reconstruction loss is calculated. The parameters of the diffusion Transformer and the multi-scale latent adapter are updated based on the continuous flow matching loss, the physical reconstruction loss and the medical semantic consistency loss. The generation module, used in the inference phase, starts from pure noise and iteratively samples in the vector field predicted by the updated diffusion Transformer to generate the final fused medical image.

10. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it performs the intent-driven multimodal medical image fusion method based on the MIND model as described in any one of claims 1-8.

Images