A multi-modal medical image segmentation method based on frequency domain perception fusion

Abstract

The application is suitable for the technical field of image processing, and provides a multi-modal medical image segmentation method based on frequency domain perception fusion, comprising: acquiring a medical image to be segmented and a lesion description text corresponding to the medical image to be segmented; inputting the medical image to be segmented and the lesion description text into an image segmentation model for processing to obtain an image segmentation result; the image segmentation model comprises a text feature extraction network, a global visual feature extraction network, a local visual feature extraction network, a frequency domain perception fusion network and a decoder. The application can improve the segmentation precision of medical images.

Description

A Multimodal Medical Image Segmentation Method Based on Frequency Domain Sensing Fusion Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to a multimodal medical image segmentation method based on frequency domain sensing fusion. Background Technology

[0002] Medical image segmentation is a core technology for assisted diagnosis. With the development of deep learning, purely vision-based methods (such as U-Net and ViT; U-Net is an end-to-end image segmentation model based on convolutional neural networks, and ViT is a visual transformer) have made significant progress. However, when faced with complex clinical scenarios, relying solely on visual information still has insurmountable limitations. On the one hand, medical images often exhibit the characteristics of "different images for the same disease" and "same images for different diseases," with some early lesions visually extremely similar to normal tissue, making them difficult to distinguish based on pixel grayscale alone. On the other hand, in the absence of clinical information (such as "left upper lung" or "distribution along the pleura"), the model is prone to producing false positives in non-target areas, resulting in low segmentation accuracy of medical images. Summary of the Invention

[0003] This application provides a multimodal medical image segmentation method based on frequency domain sensing fusion, which can solve the problem of low segmentation accuracy in medical images.

[0004] This application provides a multimodal medical image segmentation method based on frequency domain sensing fusion, including:

[0005] Obtain the medical image to be segmented, and the corresponding lesion description text;

[0006] The medical image to be segmented and the lesion description text are input into the image segmentation model for processing to obtain the image segmentation result of the medical image to be segmented.

[0007] Image segmentation models include text feature extraction networks, global visual feature extraction networks, local visual feature extraction networks, frequency domain sensing fusion networks, and decoders;

[0008] The global visual feature extraction network includes: a sequence generation module for processing the medical image to be segmented and outputting a visual feature sequence, and a global feature extraction module for processing the visual feature sequence and outputting global visual features.

[0009] The text feature extraction network includes: a text input module for processing lesion description text and visual feature sequences and outputting text input features, and a text feature extraction module for processing text input features and outputting text features;

[0010] Local visual feature extraction networks are used to process visual feature sequences and output local visual features;

[0011] The frequency domain sensing fusion network is used to fuse the outputs of the text feature extraction module, the global feature extraction module, and the local visual feature extraction network to output fused visual features;

[0012] The decoder is used to process text features and fused visual features, and outputs image segmentation results.

[0013] Optionally, the text input module includes a word embedding layer, a visual text feature generation unit, and a concatenation layer. The word embedding layer is used to process the lesion description text and output a hidden representation. The visual text feature generation unit is used to process the visual feature sequence and output visual text features. The concatenation layer is used to concatenate the hidden representation and visual text features and output text input features.

[0014] Optionally, the visual text feature generation unit includes a global average pooling layer, a first MLP network layer, and a second MLP network layer connected in sequence. The input of the global average pooling layer is a visual feature sequence, and the output of the second MLP network layer is the visual text feature.

[0015] Optionally, the text feature extraction module includes multiple text encoding units connected in sequence. Each text encoding unit includes a text Transformer encoding layer, a text low-rank adapter, and a text addition layer. The inputs of the text Transformer encoding layer and the text low-rank adapter are both inputs of the text encoding unit. The outputs of the text Transformer encoding layer and the text low-rank adapter are both connected to the input of the text addition layer. The output of the text addition layer is the output of the text encoding unit.

[0016] The input of the first text encoding unit in a plurality of text encoding units is the text input feature, and the output of the last text encoding unit in a plurality of text encoding units is the text feature.

[0017] Optionally, the global feature extraction module includes multiple image coding units connected in sequence. Each image coding unit includes an image Transformer coding layer, an image low-rank adapter, and an image summing layer. The inputs of the image Transformer coding layer and the image low-rank adapter are both inputs of the image coding unit. The outputs of the image Transformer coding layer and the image low-rank adapter are both connected to the input of the image summing layer. The output of the image summing layer is the output of the image coding unit.

[0018] The input of the first image coding unit in a plurality of image coding units is a sequence of visual features, and the output of the last image coding unit in a plurality of image coding units is global visual features.

[0019] Optionally, the local visual feature extraction network includes multiple convolutional neural network layers connected in sequence;

[0020] The input to the first convolutional neural network layer in a series of convolutional neural network layers is a sequence of visual features, and the output of the last convolutional neural network layer in a series of convolutional neural network layers is local visual features.

[0021] Optionally, the text feature extraction module includes twelve text encoding units connected in sequence, the global feature extraction module includes twelve image encoding units connected in sequence, and the local visual feature extraction network includes three convolutional neural network layers connected in sequence.

[0022] The frequency domain sensing fusion network includes a stitching unit, a first fusion unit for fusing inputs, a second fusion unit for fusing inputs, and a third fusion unit for fusing inputs.

[0023] The stitching unit is used to stitch together the outputs of the first fusion unit, the second fusion unit, and the third fusion unit and output the fused visual features;

[0024] The input to the first fusion unit is the output of the fourth text encoding unit of the text feature extraction module, the output of the fourth image encoding unit of the global feature extraction module, and the output of the first convolutional neural network layer of the local visual feature extraction network.

[0025] The input to the second fusion unit is the output of the eighth text encoding unit of the text feature extraction module, the output of the eighth image encoding unit of the global feature extraction module, and the output of the second convolutional neural network layer of the local visual feature extraction network.

[0026] The input to the third fusion unit is the output of the twelfth text encoding unit of the text feature extraction module, the output of the twelfth image encoding unit of the global feature extraction module, and the output of the third convolutional neural network layer of the local visual feature extraction network.

[0027] Optionally, the first fusion unit, the second fusion unit, and the third fusion unit each include a semantic injection module and a frequency domain fusion module connected in sequence. The semantic injection module includes a cross-attention layer and a residual connection layer connected in sequence, and the output of the residual connection layer is the output of the semantic injection module.

[0028] The input of the cross-attention layer of the first fusion unit is the output of the fourth text encoding unit of the text feature extraction module and the output of the fourth image encoding unit of the global feature extraction module. The output of the fourth image encoding unit of the global feature extraction module is the input of the residual connection layer of the first fusion unit. The output of the first convolutional neural network layer of the local visual feature extraction network is the input of the frequency domain fusion module of the first fusion unit. The output of the frequency domain fusion module of the first fusion unit is the output of the first fusion unit.

[0029] The input of the cross-attention layer of the second fusion unit is the output of the eighth text encoding unit of the text feature extraction module and the output of the eighth image encoding unit of the global feature extraction module. The output of the eighth image encoding unit of the global feature extraction module is the input of the residual connection layer of the second fusion unit. The output of the second convolutional neural network layer of the local visual feature extraction network is the input of the frequency domain fusion module of the second fusion unit. The output of the frequency domain fusion module of the second fusion unit is the output of the second fusion unit.

[0030] The input to the cross-attention layer of the third fusion unit is the output of the twelfth text encoding unit of the text feature extraction module and the output of the twelfth image encoding unit of the global feature extraction module. The output of the twelfth image encoding unit of the global feature extraction module is the input to the residual connection layer of the third fusion unit. The output of the third convolutional neural network layer of the local visual feature extraction network is the input to the frequency domain fusion module of the third fusion unit. The output of the frequency domain fusion module of the third fusion unit is the output of the third fusion unit.

[0031] Optionally, the frequency domain fusion module is used to perform the following steps:

[0032] Haar wavelet decomposition was performed on semantic visual features and convolutional features respectively; semantic visual features are data received from the semantic injection module, and convolutional features are data received from the local visual feature extraction network.

[0033] Frequency domain fusion is performed using the following formula:

[0034] ;

[0035] ;

[0036] ;

[0037] ;

[0038] ;

[0039] in, This indicates the low-frequency fusion result. Low-frequency weights representing semantic visual features , Indicates a gating network. This represents the low-frequency component obtained by Haar wavelet decomposition of semantic visual features. This represents the low-frequency component obtained by Haar wavelet decomposition of convolutional features. This indicates the result of horizontal high-frequency fusion. The high-frequency weights representing semantic visual features , This represents the horizontal high-frequency components obtained by Haar wavelet decomposition of semantic visual features. This represents the horizontal high-frequency components obtained by Haar wavelet decomposition of convolutional features. This indicates the result of vertical high-frequency fusion. Vertical high-frequency weights representing semantic visual features , This represents the vertical high-frequency components obtained by Haar wavelet decomposition of semantic visual features. This represents the vertical high-frequency components obtained by Haar wavelet decomposition of convolutional features. This indicates the result of diagonal high-frequency fusion. Diagonal high-frequency weights representing semantic visual features , This represents the diagonal high-frequency components obtained by Haar wavelet decomposition of semantic visual features. This represents the diagonal high-frequency components obtained by Haar wavelet decomposition of convolutional features. This represents the reconstructed data after inverse wavelet transform. Represents the inverse Haar wavelet transform function;

[0040] The reconstructed data is enhanced using convolutional layers to obtain the output of the frequency domain fusion module.

[0041] Optionally, in the first fusion unit, the semantic visual features are data received from the semantic injection module of the first fusion unit, and the convolutional features are data received from the first convolutional neural network layer of the local visual feature extraction network.

[0042] In the second fusion unit, the semantic visual features are the data received from the semantic injection module of the second fusion unit, and the convolutional features are the data received from the second convolutional neural network layer of the local visual feature extraction network.

[0043] In the third fusion unit, the semantic visual features are the data received from the semantic injection module of the third fusion unit, and the convolutional features are the data received from the third convolutional neural network layer of the local visual feature extraction network.

[0044] The above-mentioned solution in this application has the following beneficial effects:

[0045] In the embodiments of this application, an image segmentation model is designed to process the medical image to be segmented and the corresponding lesion description text to obtain the image segmentation result of the medical image to be segmented. The image segmentation model includes a global visual branch that extracts global visual features from the medical image to be segmented, a local detail branch that extracts local visual features from the medical image to be segmented, and a text-guided branch that obtains image-aware text features based on the medical image to be segmented and the corresponding lesion description text. Furthermore, the image segmentation model can fuse the visual features of the global visual branch and the local detail branch based on the semantics of the text-guided branch to obtain fused visual features. Therefore, when the text features and fused visual features are input into the decoder for processing, the decoder can continuously guide the decoding process of the fused visual features using image-aware text features, outputting a high-precision image segmentation result. This text-feature-guided image segmentation method can significantly improve the segmentation accuracy of medical images compared to existing image segmentation methods that lack clinical information guidance.

[0046] Other beneficial effects of this application will be described in detail in the following detailed description section. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 is a flowchart of a multimodal medical image segmentation method based on frequency domain sensing fusion provided in an embodiment of this application;

[0049] Figure 2 is a schematic diagram of the feature extraction part in an image segmentation model provided in an embodiment of this application. Detailed Implementation

[0050] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0051] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0052] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0053] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0054] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0055] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0056] To address the low segmentation accuracy of current medical images, this application provides a multimodal medical image segmentation method based on frequency domain-aware fusion. This method processes the medical image to be segmented and its corresponding lesion description text using a designed image segmentation model to obtain the segmentation result. The image segmentation model includes a global visual branch extracting global visual features from the medical image to be segmented, a local detail branch extracting local visual features, and a text-guided branch that obtains image-aware text features based on the medical image to be segmented and its corresponding lesion description text. Furthermore, this image segmentation model can fuse the visual features of the global visual branch and the local detail branch based on the semantics of the text-guided branch to obtain fused visual features. Therefore, when the text features and fused visual features are input into the decoder for processing, the decoder can continuously guide the decoding process of the fused visual features using image-aware text features, outputting a high-precision image segmentation result. This text-feature-guided image segmentation method significantly improves the segmentation accuracy of medical images compared to existing image segmentation methods that lack clinical information guidance.

[0057] The following describes the multimodal medical image segmentation method based on frequency domain sensing fusion provided in this application by way of specific embodiments.

[0058] As shown in Figure 1, the multimodal medical image segmentation method based on frequency domain sensing fusion provided in this application includes the following steps:

[0059] Step 11: Obtain the medical image to be segmented and the lesion description text corresponding to the medical image to be segmented.

[0060] The aforementioned medical image to be segmented can be a CT image obtained by scanning a patient's body part using computed tomography (CT) technology. The aforementioned lesion description text is used to describe the patient's lesions, such as a feeling of tightness in the left lung or noting the upper lobe region of the left lung. Generally, the medical image to be segmented and the corresponding lesion description text can be uploaded to the execution terminal of the multimodal medical image segmentation method.

[0061] Step 12: Input the medical image to be segmented and the lesion description text into the image segmentation model for processing to obtain the image segmentation result of the medical image to be segmented.

[0062] The image segmentation result described above is an image of the region where the lesion is located in the medical image to be segmented, also known as a segmented region image. For example, an image of the bilateral lung regions segmented from a medical image to be segmented.

[0063] The image segmentation model described above is a model trained using common training methods, specifically a model trained with a large number of training samples. Each training sample includes a medical image, the corresponding lesion description text, and the segmented region image of that medical image (i.e., the segmentation result).

[0064] The image segmentation model will be illustrated below with specific examples.

[0065] The image segmentation model described above includes a text feature extraction network, a global visual feature extraction network, a local visual feature extraction network, a frequency domain-aware fusion network, and a decoder. The global visual feature extraction network includes a sequence generation module for processing the medical image to be segmented and outputting a visual feature sequence, and a global feature extraction module for processing the visual feature sequence and outputting global visual features. The text feature extraction network includes a text input module for processing the lesion description text and the visual feature sequence and outputting text input features, and a text feature extraction module for processing the text input features and outputting text features. The local visual feature extraction network processes the visual feature sequence and outputs local visual features. The frequency domain-aware fusion network fuses the outputs of the text feature extraction module, the global feature extraction module, and the local visual feature extraction network to output fused visual features. The decoder processes the text features and the fused visual features and outputs the image segmentation result.

[0066] The following is an example of a text feature extraction network.

[0067] In some embodiments of this application, the text input module includes a word embedding layer, a visual text feature generation unit, and a splicing layer.

[0068] The word embedding layer is used to process the lesion description text and output a hidden representation. Specifically, the lesion description text (generally a text sequence for ease of data processing) is processed. This represents the text describing the lesion. This represents the first word in the text sequence. This represents the second word in the text sequence. Represents the first text sequence After processing the input word embedding layer (containing 1 lexical units), the hidden representation can be obtained. , The length of the text sequence. For the hidden layer dimension.

[0069] The visual text feature generation unit processes the visual feature sequence and outputs visual text features. Specifically, the visual text feature generation unit includes a global average pooling layer, a first MLP network layer, and a second MLP network layer connected in sequence. The input to the global average pooling layer is the visual feature sequence, and the output of the second MLP network layer is the visual text features. The global average pooling layer performs a global average pooling operation on the visual feature sequence to obtain a global average pooled representation. Both the first and second MLP network layers are MLP networks, and they constitute an MLP projection layer used to transfer the global average pooling representation. Mapped to text space.

[0070] The concatenation layer is used to concatenate the hidden representations and visual text features to output text input features. Specifically, text input features... , Visual text features, text input features It is a text representation based on visual conditions, and is a diagnostic text embedding expression with image perception capabilities.

[0071] In some embodiments of this application, as shown in Figure 2, the text feature extraction module of the text feature extraction network includes multiple text encoding units connected in sequence. For any text encoding unit, the text encoding unit includes a text Transformer encoding layer (text encoding block in the figure), a text low-rank adapter (text adapter in the figure), and a text addition layer (the "+" to the right of the text adapter and text encoding block in the figure). The input terminals of the text Transformer encoding layer and the text low-rank adapter are both input terminals of the text encoding unit. The output terminals of the text Transformer encoding layer and the text low-rank adapter are both connected to the input terminal of the text addition layer. The output terminal of the text addition layer is the output terminal of the text encoding unit.

[0072] The input of the first text encoding unit among the multiple text encoding units mentioned above is the text input feature, and the output of the last text encoding unit among the multiple text encoding units is the text feature.

[0073] In some specific embodiments of this application, the above-mentioned text Transformer encoding layer is a frozen biomedical pre-trained text encoder (i.e., Transformer encoding layer), the text low-rank adapter is specifically a low-rank adapter (LoRA), and the text addition layer is specifically an addition layer in machine learning.

[0074] As can be seen from the structure of the text feature extraction module, each text Transformer encoding layer introduces a low-rank text adapter, which is used for lightweight adaptation of the query and value projection matrices. For hidden representations... (i.e., text input features) Text low-rank adapter adapted , The projection is calculated as follows:

[0075] ;

[0076] ;

[0077] In the above formula, The Query projection matrix after adaptation with the low-rank text adapter. The Value projection matrix after adaptation by the low-rank text adapter. Pre-trained Query and Value projection matrices for the text Transformer encoding layer (frozen). This is the original weight matrix of the query. This is the original weight matrix for Value; The text is represented by a hidden layer after visual condition enhancement, serving as the input. , For the learnable dimensionality reduction matrix parameters of the query, For the learnable dimensionality reduction matrix parameters of Value, from dimensional projection onto low-rank space dimension; , For the learnable upscaling matrix parameters of the query, For the learnable up-dimensional matrix parameters of Value, the low-rank representation is transformed from... Dimensional projection back dimension; As a low-rank dimension, in this embodiment, we take... , .

[0078] It should be noted that the text Transformer encoding layer uses pre-trained Query and Value projection matrices to apply text input features. The process is performed, and the results are output. The text addition layer adds the incremental features generated by the low-rank text adapter to the output of the text Transformer encoding layer (here, addition refers to the Query and Value projection matrices; that is, adding the Query projection matrix from the text Transformer encoding layer output to the Query projection matrix adapted by the low-rank text adapter, and adding the Value projection matrix from the text Transformer encoding layer output to the Value projection matrix adapted by the low-rank text adapter), and uses the addition result as the input to the next level text encoding unit. It can be understood that if the current text encoding unit is the last text encoding unit, then the addition result is the text feature.

[0079] The following is an example of a global visual feature extraction network.

[0080] In some embodiments of this application, the sequence generation module of the global visual feature extraction network can be a patch embedding layer. (Medical image to be segmented) After inputting into the sequence generation module, the module can extract visual feature sequences containing global context information. . The height of the medical image to be segmented. The width of the medical image to be segmented. The number of patches. , For feature dimensions.

[0081] In some embodiments of this application, as shown in FIG2, the global feature extraction module of the global visual feature extraction network includes multiple image coding units connected in sequence. For any image coding unit, the image coding unit includes an image Transformer coding layer (image coding block in the figure), an image low-rank adapter (image adapter in the figure), and an image addition layer (+ to the right of the image coding block and image adapter in the figure). The input terminals of the image Transformer coding layer and the image low-rank adapter are both input terminals of the image coding unit. The output terminals of the image Transformer coding layer and the image low-rank adapter are both connected to the input terminal of the image addition layer. The output terminal of the image addition layer is the output terminal of the image coding unit.

[0082] The input of the first image coding unit in a plurality of image coding units is a sequence of visual features, and the output of the last image coding unit in a plurality of image coding units is global visual features.

[0083] In some specific embodiments of this application, the above-mentioned image Transformer coding layer is a frozen large-scale pre-trained visual encoder (i.e., Transformer coding layer), the image low-rank adapter is specifically a low-rank adapter (LoRA), preferably a convolutional low-rank adapter, and the image addition layer is specifically an addition layer in machine learning.

[0084] Similar to the text branch, in the global feature extraction module, each image Transformer encoding layer introduces an image low-rank adapter, which is used for lightweight adaptation of the query and value projection matrices. For visual feature sequences... Image low-rank adapter adapted , The projection is calculated as follows:

[0085] ;

[0086] ;

[0087] In the above formula, The Query projection matrix after adapting to the low-rank image adapter. The Value projection matrix after low-rank adapter adaptation for the image. Query and Value projection matrices pre-trained for the image Transformer encoding layer (frozen). This is the original weight matrix of the query. This is the original weight matrix for Value; , For the learnable dimensionality reduction matrix parameters of the query, For the learnable dimensionality reduction matrix parameters of Value, the visual feature sequence from dimensional projection onto low-rank space dimension; , For the learnable upscaling matrix parameters of the query, For the learnable up-dimensional matrix parameters of Value, project the low-rank representation back. dimension; For low-rank dimensions (in this embodiment, take...) , ).

[0088] It should be noted that the image Transformer encoding layer projects the visual feature sequence according to the pre-trained Query and Value projection matrices. The process is performed, and the result is output. The image addition layer adds the incremental features generated by the low-rank image adapter to the output of the image Transformer coding layer (here, addition refers to the Query and Value projection matrices; that is, the Query projection matrix output from the image Transformer coding layer is added to the Query projection matrix adapted by the low-rank image adapter, and the Value projection matrix output from the image Transformer coding layer is added to the Value projection matrix adapted by the low-rank image adapter). The result of this addition is used as the input to the next level image coding unit. It can be understood that if the current image coding unit is the last image coding unit, the result of the addition is the global visual feature.

[0089] The following is an example of a local visual feature extraction network.

[0090] In some embodiments of this application, as shown in FIG2, the local visual feature extraction network includes multiple convolutional neural network layers (convolutional neural networks in the figure) connected in sequence. The input of the first convolutional neural network layer is a sequence of visual features, and the output of the last convolutional neural network layer is the local visual features.

[0091] The convolutional neural network layer can be a convolutional neural network (CNN) used to extract features from the received data. Specifically, it can be a pre-trained ConvNeXt-Tiny, which is an improved convolutional neural network architecture.

[0092] It should be noted that the local visual feature extraction network and the global visual feature extraction network complement each other: the latter focuses on the global semantic context, while the former retains four times the spatial resolution of the latter, capturing local structural details such as lesion boundaries and organ outlines in medical images.

[0093] In some embodiments of this application, the text feature extraction module includes twelve text encoding units connected in sequence, the global feature extraction module includes twelve image encoding units connected in sequence, and the local visual feature extraction network includes three convolutional neural network layers connected in sequence.

[0094] The frequency domain sensing fusion network is illustrated below.

[0095] In some embodiments of this application, the frequency domain sensing fusion network includes a stitching unit, a first fusion unit for fusing inputs, a second fusion unit for fusing inputs, and a third fusion unit for fusing inputs.

[0096] The aforementioned stitching unit is used to stitch together the outputs of the first fusion unit, the second fusion unit, and the third fusion unit to output fused visual features. In some embodiments of this application, the aforementioned stitching unit may specifically be a stitching layer in machine learning.

[0097] The inputs to the first fusion unit are the outputs of the fourth text encoding unit of the text feature extraction module, the fourth image encoding unit of the global feature extraction module, and the output of the first convolutional neural network layer of the local visual feature extraction network.

[0098] The input to the second fusion unit is the output of the eighth text encoding unit of the text feature extraction module, the output of the eighth image encoding unit of the global feature extraction module, and the output of the second convolutional neural network layer of the local visual feature extraction network.

[0099] The input to the third fusion unit is the output of the twelfth text encoding unit of the text feature extraction module, the output of the twelfth image encoding unit of the global feature extraction module, and the output of the third convolutional neural network layer of the local visual feature extraction network.

[0100] In some embodiments of this application, as shown in FIG2, the first fusion unit, the second fusion unit and the third fusion unit all include a semantic injection module (A in FIG2) and a frequency domain fusion module (B in FIG2) connected in sequence. The semantic injection module includes a cross-attention layer and a residual connection layer connected in sequence, and the output end of the residual connection layer is the output end of the semantic injection module.

[0101] The input to the cross-attention layer of the first fusion unit is the output of the fourth text encoding unit of the text feature extraction module and the output of the fourth image encoding unit of the global feature extraction module. The output of the fourth image encoding unit of the global feature extraction module is the input to the residual connection layer of the first fusion unit. The output of the first convolutional neural network layer of the local visual feature extraction network is the input to the frequency domain fusion module of the first fusion unit. The output of the frequency domain fusion module of the first fusion unit is the output of the first fusion unit.

[0102] The input to the cross-attention layer of the second fusion unit is the output of the eighth text encoding unit of the text feature extraction module and the output of the eighth image encoding unit of the global feature extraction module. The output of the eighth image encoding unit of the global feature extraction module is the input to the residual connection layer of the second fusion unit. The output of the second convolutional neural network layer of the local visual feature extraction network is the input to the frequency domain fusion module of the second fusion unit. The output of the frequency domain fusion module of the second fusion unit is the output of the second fusion unit.

[0103] The input to the cross-attention layer of the third fusion unit is the output of the twelfth text encoding unit of the text feature extraction module and the output of the twelfth image encoding unit of the global feature extraction module. The output of the twelfth image encoding unit of the global feature extraction module is the input to the residual connection layer of the third fusion unit. The output of the third convolutional neural network layer of the local visual feature extraction network is the input to the frequency domain fusion module of the third fusion unit. The output of the frequency domain fusion module of the third fusion unit is the output of the third fusion unit.

[0104] It should be noted that the semantic injection module and frequency domain fusion module have the same structure and function in the first, second, and third fusion units; only the input data differs in each fusion unit. Here, we will use the first fusion unit as an example to explain the structure and function of the semantic injection module and frequency domain fusion module.

[0105] In the first fusion unit, the input of the cross-attention layer is the output of the fourth text encoding unit and the output of the fourth image encoding unit, the input of the residual connection layer is the output of the cross-attention layer and the output of the fourth image encoding unit, and the input of the frequency domain fusion module is the output of the residual connection layer and the output of the first convolutional neural network layer.

[0106] The cross-attention layer is used to fuse the output of the fourth text encoding unit and the output of the fourth image encoding unit through cross-attention. The residual connection layer is used to perform residual fusion on the fusion result of the cross-attention layer and the output of the fourth image encoding unit to output semantically injected visual features. The frequency domain fusion module is used to perform frequency domain fusion on the semantically injected visual features and the local features output by the first convolutional neural network layer to obtain feature-enhanced visual features (i.e., fused visual features).

[0107] Specifically, the residual connection layer can be a gated residual connection layer, which processes the input data using the following formula to obtain the semantically injected visual features. :

[0108] ;

[0109] ;

[0110] In the above formula, For learnable gating parameters; This is a layer normalization function to stabilize the training process; It is a feedforward neural network, consisting of two linear transformation layers and the GELU activation function; This is the output of the fourth image coding unit. This is the fusion result of the cross-attention layer.

[0111] Specifically, the frequency domain fusion module is used to perform steps one through three:

[0112] Step 1: Perform Haar wavelet decomposition on both the semantic visual features and convolutional features. The semantic visual features are the data received from the semantic injection module, and the convolutional features are the data received from the local visual feature extraction network. In the first fusion unit, the semantic visual features are the data received from the semantic injection module of the first fusion unit, and the convolutional features are the data received from the first convolutional neural network layer of the local visual feature extraction network. In the second fusion unit, the semantic visual features are the data received from the semantic injection module of the second fusion unit, and the convolutional features are the data received from the second convolutional neural network layer of the local visual feature extraction network. In the third fusion unit, the semantic visual features are the data received from the semantic injection module of the third fusion unit, and the convolutional features are the data received from the third convolutional neural network layer of the local visual feature extraction network.

[0113] Step two, frequency domain fusion is performed using the following formula:

[0114] ;

[0115] ;

[0116] ;

[0117] ;

[0118] ;

[0119] in, This indicates the low-frequency fusion result. Low-frequency weights representing semantic visual features , Indicates a gating network. This represents the low-frequency component obtained by Haar wavelet decomposition of semantic visual features. This represents the low-frequency component obtained by Haar wavelet decomposition of convolutional features. This indicates the result of horizontal high-frequency fusion. The high-frequency weights representing semantic visual features , This represents the horizontal high-frequency components obtained by Haar wavelet decomposition of semantic visual features. This represents the horizontal high-frequency components obtained by Haar wavelet decomposition of convolutional features. This indicates the result of vertical high-frequency fusion. Vertical high-frequency weights representing semantic visual features , This represents the vertical high-frequency components obtained by Haar wavelet decomposition of semantic visual features. This represents the vertical high-frequency components obtained by Haar wavelet decomposition of convolutional features. This indicates the result of diagonal high-frequency fusion. Diagonal high-frequency weights representing semantic visual features , This represents the diagonal high-frequency components obtained by Haar wavelet decomposition of semantic visual features. This represents the diagonal high-frequency components obtained by Haar wavelet decomposition of convolutional features. This represents the reconstructed data after inverse wavelet transform. This represents the inverse Haar wavelet transform function. , , , All of these are learnable parameters. For example, Its initial value is 0.7. , , Its initial value is 0.3.

[0120] Step 3: Use convolutional layers to enhance the features of the reconstructed data to obtain the output of the frequency domain fusion module.

[0121] Specifically, it can be used Convolutional layers enhance the features of the reconstructed data.

[0122] It should be noted that the resolutions of the feature maps output by the first, second, and third convolutional neural network layers in the local visual feature extraction network are respectively... , , Therefore, frequency domain sensing fusion networks can be implemented at three scales ( , , Frequency domain fusion is performed on each of the two domains, and the fusion results are combined and sent to the decoder.

[0123] It is worth mentioning that the frequency domain fusion mechanism differs from the traditional one-time splicing fusion. It utilizes Haar wavelet transform to decompose features into low-frequency and high-frequency components. The low-frequency components are fused primarily by semantic visual features, which excel at global modeling, while the high-frequency components are fused primarily by convolutional features, which excel at detail capture. This fully leverages the inherent complementarity of the two types of encoders in their frequency domain responses. Furthermore, this application utilizes textual modality as a "strong prior condition," directly converting the doctor's lesion description text (such as "left upper lobe of the lung") into a guiding signal for visual attention through a semantic injection module. This suppresses responses from non-target regions during the feature extraction stage.

[0124] In some embodiments of this application, the decoder described above can be a Transformer decoder, which uses a cross-attention mechanism to continuously guide the decoding process of visual features through textual semantics, and finally outputs a high-precision segmented image.

[0125] It is worth mentioning that in this embodiment, the outputs of the 4th, 8th, and 12th text coding units and the 4th, 8th, and 12th image coding units are selected for frequency domain fusion in order to divide the 12 Transformer coding layers into three stages on an average basis, construct a multi-scale feature pyramid, and cooperate with the Transformer decoder to achieve fine segmentation and improve segmentation accuracy.

[0126] The multimodal medical image segmentation method of this application will be explained below with specific examples.

[0127] In this example, a comparison with existing methods is shown in Table 1 on the QaTa-COV19 and MosMedData+ datasets. Both QaTa-COV19 and MosMedData+ datasets are medical image datasets. In Table 1, values ​​are expressed as %, Dice represents the Dice coefficient (a commonly used mathematical metric), and IoU (Intersection over Union) is a widely used core performance evaluation metric. nnUNet is a segmentation framework that only accepts image modality input, LGA is a language-guided method based on a visual transformer, and MMI-Unet is a visual-language fusion segmentation method based on a CNN backbone.

[0128] Table 1. Comparison with existing methods on the QaTa-COV19 and MosMedData+ datasets.

[0129]

[0130] As shown in Table 1, FAF-Net (the method described in this application) was comprehensively compared with three other methods on two publicly available pneumonia segmentation benchmark datasets, QaTa-COV19 and MosMedData+. Experimental results show that FAF-Net achieves state-of-the-art performance across all evaluation metrics on both datasets. Specifically, on the QaTa-COV19 dataset, FAF-Net achieves a Dice coefficient of 91.37% and an IoU of 84.12%, surpassing the current state-of-the-art method MMI-UNet by 0.49% and 0.84%, respectively. On the MosMedData+ dataset, FAF-Net achieves a Dice coefficient of 79.14% and an IoU of 65.49%, respectively, representing improvements of 0.72% and 0.99% over MMI-UNet. Notably, compared to methods that only accept image modal input (such as nnUNet with a Dice coefficient of 80.42%), FAF-Net achieves an absolute improvement of nearly 11% on QaTa-COV19, fully validating the effectiveness of the frequency-domain aware fusion mechanism and text semantic injection strategy. Further analysis reveals that FAF-Net improves the Dice coefficient and IoU on QaTa-COV19 by 6.72% and 7.89% respectively compared to LGA. FAF-Net effectively combines local detail capture capabilities with global structure modeling capabilities through its frequency-domain fusion mechanism. The wavelet transform-based frequency-domain fusion strategy can more effectively integrate complementary features extracted by heterogeneous encoders, avoiding semantic conflicts caused by direct spatial domain fusion, and achieving better results in both local edges and global structure.

[0131] In summary, this application innovatively constructs a multimodal asymmetric three-branch architecture comprising "text guidance - global vision - local texture". Its core feature lies in the asymmetric strategy of "double-frozen base model + fully fine-tuned prior network": simultaneously freezing the large-scale vision base model and medical text encoder to retain general knowledge, and performing domain adaptation only through a lightweight adapter; while simultaneously performing full parameter fine-tuning on the lightweight CNN branch to deeply adapt to the texture features of a specific task.

[0132] Overall, the multimodal medical image segmentation method of this application successfully maps one-dimensional textual semantics to two-dimensional visual space through a carefully designed semantic injection module, achieving deep alignment of image and text features. This avoids the feature space misalignment problem caused by direct splicing and significantly improves the utilization rate of multimodal information. Thanks to the dynamic injection of high-frequency features from CNNs in frequency domain fusion, the method of this application shows significantly better segmentation accuracy than simple ViT or CNN models when dealing with targets with blurred edges, effectively suppressing background noise and sharpening object boundaries. Furthermore, the multimodal asymmetric three-branch architecture gives the model stronger robustness—when data distribution shifts (such as differences in equipment acquisition from different hospitals), the frozen Transformer encoding layer provides a stable semantic baseline, while the fully parameter-tuned CNN provides the ability to quickly adapt to local feature changes. The combination of these two aspects results in better generalization performance on multi-center data. Simultaneously, by freezing the base model, this application fully retains the rich general knowledge acquired during large-scale pre-training, avoiding the catastrophic forgetting and overfitting risks that may result from forcibly fine-tuning a large model on small sample medical data.

[0133] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A multimodal medical image segmentation method based on frequency domain sensing fusion, characterized in that, include: Obtain the medical image to be segmented, and the lesion description text corresponding to the medical image to be segmented; The medical image to be segmented and the lesion description text input image segmentation model are processed to obtain the image segmentation result of the medical image to be segmented. The image segmentation model includes a text feature extraction network, a global visual feature extraction network, a local visual feature extraction network, a frequency domain perception fusion network, and a decoder. The global visual feature extraction network includes a sequence generation module for processing the medical image to be segmented and outputting a visual feature sequence, and a global feature extraction module for processing the visual feature sequence and outputting global visual features. The text feature extraction network includes a text input module for processing the lesion description text and the visual feature sequence and outputting text input features, and a text feature extraction module for processing the text input features and outputting text features. The local visual feature extraction network is used to process the visual feature sequence and output local visual features. The frequency domain perception fusion network is used to fuse the output of the text feature extraction module, the output of the global feature extraction module, and the output of the local visual feature extraction network and output fused visual features. The decoder is used to process the text features... The system processes the fused visual features and outputs the image segmentation result. The text input module includes a word embedding layer, a visual text feature generation unit, and a concatenation layer. The word embedding layer processes the lesion description text and outputs a hidden representation. The visual text feature generation unit processes the visual feature sequence and outputs visual text features. The concatenation layer concatenates the hidden representation and the visual text features and outputs the text input features. The text feature extraction module includes multiple text encoding units connected in sequence. Each text encoding unit includes a text Transformer encoding layer, a text low-rank adapter, and a text addition layer. The inputs of the text Transformer encoding layer and the text low-rank adapter are both inputs of the text encoding unit. The outputs of the text Transformer encoding layer and the text low-rank adapter are both connected to the input of the text addition layer. The output of the text addition layer is the output of the text encoding unit. The input of the first text encoding unit among the multiple text encoding units is the text input feature, and the output of the last text encoding unit among the multiple text encoding units is the text feature.

2. The multimodal medical image segmentation method according to claim 1, characterized in that, The visual text feature generation unit includes a global average pooling layer, a first MLP network layer, and a second MLP network layer connected in sequence. The input of the global average pooling layer is the visual feature sequence, and the output of the second MLP network layer is the visual text feature.

3. The multimodal medical image segmentation method according to claim 1, characterized in that, The global feature extraction module includes multiple image coding units connected in sequence. Each image coding unit includes an image Transformer coding layer, an image low-rank adapter, and an image summing layer. The input terminals of the image Transformer coding layer and the image low-rank adapter are both input terminals of the image coding unit. The output terminals of the image Transformer coding layer and the image low-rank adapter are both connected to the input terminal of the image summing layer. The output terminal of the image summing layer is the output terminal of the image coding unit. The input of the first image coding unit among the multiple image coding units is the visual feature sequence, and the output of the last image coding unit among the multiple image coding units is the global visual feature.

4. The multimodal medical image segmentation method according to claim 3, characterized in that, The local visual feature extraction network includes multiple convolutional neural network layers connected in sequence; the input of the first convolutional neural network layer is the visual feature sequence, and the output of the last convolutional neural network layer is the local visual feature.

5. The multimodal medical image segmentation method according to claim 4, characterized in that, The text feature extraction module includes twelve text encoding units connected in sequence; the global feature extraction module includes twelve image encoding units connected in sequence; the local visual feature extraction network includes three convolutional neural network layers connected in sequence; the frequency domain perception fusion network includes a concatenation unit, a first fusion unit for fusing inputs, a second fusion unit for fusing inputs, and a third fusion unit for fusing inputs; the concatenation unit is used to concatenate the outputs of the first fusion unit, the second fusion unit, and the third fusion unit and output the fused visual features; the input of the first fusion unit is the fourth text of the text feature extraction module. The inputs to the second fusion unit are: the output of the encoding unit, the output of the fourth image encoding unit of the global feature extraction module, and the output of the first convolutional neural network layer of the local visual feature extraction network; the inputs to the third fusion unit are: the output of the encoding unit, the output of the fourth image encoding unit of the global feature extraction module, and the output of the second convolutional neural network layer of the local visual feature extraction network; the inputs to the third fusion unit are: the output of the encoding unit, the output of the fourth image encoding unit of the global feature extraction module, and the output of the third convolutional neural network layer of the local visual feature extraction network; the outputs to the third fusion unit are: the output of the encoding unit, the output of the fourth image encoding unit of the global feature extraction module, and the output of the first convolutional neural network layer of the local visual feature extraction network; the outputs to the second fusion unit are: the output of the encoding unit, the output of the fourth image encoding unit of the global feature extraction module, and the output of the third convolutional neural network layer of the local visual feature extraction network.

6. The multimodal medical image segmentation method according to claim 5, characterized in that, The first fusion unit, the second fusion unit, and the third fusion unit each include a semantic injection module and a frequency domain fusion module connected in sequence. The semantic injection module includes a cross-attention layer and a residual connection layer connected in sequence, with the output of the residual connection layer serving as the output of the semantic injection module. The input of the cross-attention layer of the first fusion unit is the output of the fourth text encoding unit of the text feature extraction module and the output of the fourth image encoding unit of the global feature extraction module. The output of the fourth image encoding unit of the global feature extraction module serves as the input of the residual connection layer of the first fusion unit. The output of the first convolutional neural network layer of the local visual feature extraction network serves as the input of the frequency domain fusion module of the first fusion unit, and the output of the frequency domain fusion module of the first fusion unit is the output of the first fusion unit. The input of the cross-attention layer of the second fusion unit is the output of the eighth text encoding unit of the text feature extraction module and the output of the fourth image encoding unit of the global feature extraction module. The output of the eighth image encoding unit of the global feature extraction module is the input of the residual connection layer of the second fusion unit. The output of the second convolutional neural network layer of the local visual feature extraction network is the input of the frequency domain fusion module of the second fusion unit, and the output of the frequency domain fusion module of the second fusion unit is the output of the second fusion unit. The input of the cross attention layer of the third fusion unit is the output of the twelfth text encoding unit of the text feature extraction module and the output of the twelfth image encoding unit of the global feature extraction module. The output of the twelfth image encoding unit of the global feature extraction module is the input of the residual connection layer of the third fusion unit. The output of the third convolutional neural network layer of the local visual feature extraction network is the input of the frequency domain fusion module of the third fusion unit, and the output of the frequency domain fusion module of the third fusion unit is the output of the third fusion unit.

7. The multimodal medical image segmentation method according to claim 6, characterized in that, The frequency domain fusion module performs the following steps: Haar wavelet decomposition is applied to the semantic visual features and convolutional features respectively; the semantic visual features are data received from the semantic injection module, and the convolutional features are data received from the local visual feature extraction network; frequency domain fusion is performed using the following formula: ； ； ； ； ;in, This indicates the low-frequency fusion result. Low-frequency weights representing semantic visual features ， Indicates a gating network. This represents the low-frequency component obtained by Haar wavelet decomposition of semantic visual features. This represents the low-frequency component obtained by Haar wavelet decomposition of convolutional features. This indicates the result of horizontal high-frequency fusion. The high-frequency weights representing semantic visual features ， This represents the horizontal high-frequency components obtained by Haar wavelet decomposition of semantic visual features. This represents the horizontal high-frequency components obtained by Haar wavelet decomposition of convolutional features. This indicates the result of vertical high-frequency fusion. Vertical high-frequency weights representing semantic visual features ， This represents the vertical high-frequency components obtained by Haar wavelet decomposition of semantic visual features. This represents the vertical high-frequency components obtained by Haar wavelet decomposition of convolutional features. This indicates the result of diagonal high-frequency fusion. Diagonal high-frequency weights representing semantic visual features ， This represents the diagonal high-frequency components obtained by Haar wavelet decomposition of semantic visual features. This represents the diagonal high-frequency components obtained by Haar wavelet decomposition of convolutional features. This represents the reconstructed data after inverse wavelet transform. The Haar wavelet inverse transform function is represented; the reconstructed data is enhanced using convolutional layers to obtain the output of the frequency domain fusion module.

8. The multimodal medical image segmentation method according to claim 7, characterized in that, In the first fusion unit, the semantic visual features are data received from the semantic injection module of the first fusion unit, and the convolutional features are data received from the first convolutional neural network layer of the local visual feature extraction network; in the second fusion unit, the semantic visual features are data received from the semantic injection module of the second fusion unit, and the convolutional features are data received from the second convolutional neural network layer of the local visual feature extraction network; in the third fusion unit, the semantic visual features are data received from the semantic injection module of the third fusion unit, and the convolutional features are data received from the third convolutional neural network layer of the local visual feature extraction network.

Images