Multimodal feature fusion Mamba for automatic generation of medical reports

Abstract

The present invention discloses a multimodal feature fusion Mamba for automatic generation of medical reports, wherein the original medical image is input into a visual feature extraction module to output visual features; the original medical report is input into a text feature extraction module to output text features; the visual features and text features are respectively input into an SSM-Attention visual text feature integration module to output new visual features and text features; the new visual features are input into a Transformer encoder to enhance visual features to extract advanced visual features; the text features and enhanced advanced visual features are input into a multimodal fusion Mamba module to output fusion features, and the fusion features are input into a Transformer decoder to automatically generate a medical report. The SSM-Attention and Multi modal Mamba of the present invention can be independently developed and improved, and it is also convenient to integrate newly developed technical modules, and are not limited to the fusion of medical images and text features, and are applicable to any multimodal feature fusion.

Description

Technical Field

[0001] The present invention belongs to medical image processing and computer vision technology, and specifically relates to a method for automatically generating medical reports by fusing multimodal features with Mamba. Background Art

[0002] Automatic generation of medical imaging reports is a research direction full of challenges and opportunities. Its core lies in using advanced computer technology, especially deep learning technology, to automatically analyze medical images (such as X-rays, CT, MRI, etc.) and extract key information from them to generate comprehensive, accurate and clinically relevant medical reports. This research direction aims to solve the time and energy burden faced by doctors when writing reports, while improving the accuracy and consistency of reports.

[0003] Currently, researchers are exploring the use of convolutional neural networks (CNN) to capture the spatial features of images, recurrent neural networks (RNN, such as LSTM) and Transformer architectures to extract visual features of medical images, so as to gain a deeper understanding of the content of medical images and imitate doctors' diagnostic logic and report writing style to achieve intelligent automatic generation of medical image reports. Most current technologies focus on extracting and integrating visual features from medical images, encoding these features, and then decoding them in sequence, using attention mechanisms to focus on image information throughout the process, and then automatically generating medical reports.

[0004] However, existing similar technologies have several key defects:

[0005] 1. Transformer has a high complexity in integrating visual information:

[0006] The traditional Transformer architecture has a quadratic complexity in self-attention calculation (related to the sequence length), which makes it face serious computational bottlenecks when processing long sequence data (visual features of medical images). Therefore, using the Transformer model when integrating visual information of medical images usually requires large-scale computing resources and is not suitable for use in scenarios when processing high-resolution medical images.

[0007] 2. Insufficient combination of visual and semantic information:

[0008] Most existing MRG (Medical Report Generation) methods usually only focus on the overall features of the image or the contextual information of the text when generating text, while ignoring the synergy between the local area of ​​the image and the semantics of the text. This single attention mechanism limits the expressiveness and generation effect of the model.

[0009] 3. Data annotation increases complexity

[0010] Existing methods usually require training based on a large amount of region-level annotated data and task-specific knowledge, which not only increases the cost of data preparation but also limits the generalization ability of the model. Summary of the invention

[0011] Purpose of the invention: The purpose of the present invention is to solve the deficiencies in the prior art and to provide a method for automatically generating medical reports by using Mamba, a multimodal feature fusion method.

[0012] Technical solution: A multimodal feature fusion Mamba is used in a method for automatically generating a medical report. A medical report and a medical image are input into a medical report automatic generation network, and the method includes the following steps:

[0013] Step 1: Input the original medical image I into the visual feature extraction module based on ResNet101 and output the visual features ;

[0014] Step 2: Input the original medical report into the text feature extraction module based on the BERT model and output the text features ;

[0015] Step 3: The visual features obtained in step 1 And the text features obtained in step 2 They are input into the SSM-Attention visual text feature integration module respectively; the SSM-Attention visual text feature integration module respectively processes the visual features and text features Each feature is processed separately, and the processing process for each type of feature is as follows:

[0016] Step 3.1: Perform linear transformation to obtain two new sequences and ;

[0017] Step 3.2: New sequence It is processed by normalization layer, linear layer, activation function SiLu, convolution layer, state space module SSM and normalization layer in sequence; at the same time, the new sequence It is processed in sequence by the normalization layer, convolution layer, activation function ReLU, self-attention mechanism and normalization layer;

[0018] Step 3.3: Multiply the two sequences obtained after step 3.2 to generate gated features, and then transform them linearly and combine them with the original input visual features. or text features Add and output new visual features and new text features ;

[0019] Step 3: After processing by the SSM-Attention module, the previous one is called t-1, and after three times, t is obtained; The new , The new ;

[0020] Step 4: The visual features obtained in step 3 Input the Transformer encoder and extract high-level visual features through the Transformer encoder;

[0021] Step 5: Convert the text features obtained in step 3 into And the visual features enhanced by the Transformer encoder The multimodal fusion Mamba module includes three layers of multimodal Mamba networks with the same structure; finally, the fusion feature is output ;

[0022] Step 6: Fusion features Input Transformer decoder to automatically generate medical reports.

[0023] In order to obtain the intermediate features and low-level features of the original medical image, the detailed processing method of the visual feature extraction module in step 1 is as follows:

[0024] Step 1.1, perform Patch embedding, divide the original medical image I into small patches through the PatchEmbed class, the size of each patch is patch_size x patch_size, and the number of patches is num_patches = (H / / patch_size) * (W / / patch_size) patches;

[0025] Step 1.2: The first three convolution blocks in the ResNet101 backbone network extract the low-level and mid-level features of each small image block and output a series of feature maps.

[0026] Step 1.3: Reorganize all feature maps to obtain visual features .

[0027] Furthermore, the text feature extraction module in step 2 processes the original medical report as follows:

[0028] Step 2.1: Preprocess the text label text of the original medical report, including word segmentation, stop word removal, and vocabulary construction (word segmentation: split the text label into words or phrases; stop word removal: remove common stop words, such as "de" etc., to reduce noise; vocabulary construction: construct a vocabulary containing all words with a frequency of more than 10 occurrences).

[0029] Step 2.2: Use the BERT model to perform embedding processing on the preprocessed text label: first generate word embeddings with rich context information through a bidirectional Transformer encoder structure, and then convert them into fixed-length vector word embeddings , and then combine the word embedding vectors into sentence embedding vectors through a pooling operation ; these sentence embedding vectors can be used to represent the semantic information of the entire text label;

[0030] Step 2.3: To retain the order information in the text, add a position encoding to each word embedding , and the position encoding enables the model to distinguish the meaning differences of the same vocabulary at different positions. Thus, the text features are obtained , with a size of ; where B is the batch_size, L is the sequence length Length, and D is the embed_dim dimension;

[0031] Text features The expression is as follows:

[0032] ;

[0033] is the word embedding vector of the vocabulary, is the position encoding vector

[0034] Furthermore, to achieve the integration and enhancement of visual features / text features, the processing process of the SSM-Attention visual-text feature integration module is as follows:

[0035] First, perform a linear transformation on the visual features / text features to generate two new sequences and ;

[0036] Then, perform the following different operations on the new sequences and :

[0037] ;

[0038] ;

[0039] In the above formula, refers to the linear transformation operation, refers to layer normalization, refers to one-dimensional convolution, is the activation function, is an active function, refers to the self-attention mechanism operation, is a state-space model;

[0040] Next, Multiply As a gating factor, generate gated features , and then through linear transformation and addition to the original input visual tensor / text features to form the output sequence , the expression is:

[0041]

[0042] In the above formula, refers to element-wise multiplication, Refers to the visual features / text features of the original input.

[0043] Here, the visual features are enhanced in two ways, one is the SSM path, and the other is the Self-attention path. By combining the global dependency modeling capability of SSM and the local feature selection capability of the attention mechanism, the key information in the image can be captured more comprehensively. This not only improves the richness and accuracy of the features, but also reduces information loss when processing complex images, and improves the ability to extract image features and text features.

[0044] Furthermore, the Transformer encoder processes the new visual features as follows:

[0045] Step 4.1, position encoding, that is, adding position encoding to retain the spatial information of the feature;

[0046] Step 4.2: Capture the relationship between features through the multi-head self-attention mechanism;

[0047] Step 4.3: Further process the features through a feed-forward neural network.

[0048] Step 4.4: Gradually extract high-level feature representations through a multi-layer Transformer encoder.

[0049] In order to enhance the complementarity and learning effectiveness of visual and textual features, the visual and textual features are deeply fused and the misalignment of visual and textual features is reduced. and visual features After entering the first layer of multimode Mamba network, output ; With text features After entering the second-layer multimode Mamba network, output the new ; New and text features Enter the third-layer multi-mode Mamba network and output fusion features ;

[0050] The processing method of each layer of the multi-mode Mamba network is as follows: the text features are processed by the normalization layer, convolution layer, and state space module SSM in sequence. At the same time, the visual features enhanced by the Transformer encoder are executed from step 3.1 to step 3.2 to obtain two sequences, and the two sequences are added to the processed text features to obtain new visual features; the new visual features and new text features obtained in this layer are sent to the next layer of the network structure together; the expression is as follows:

[0051] For visual features Processing is as follows:

[0052] ;

[0053] ;

[0054] yes pass Multiply As a gating factor, generate gated features to form an output sequence ; ;

[0055] For text features Processing is as follows:

[0056] ;

[0057]

[0058] ;

[0059] SSM is a state space model. refers to the cross-attention operation, It refers to the addition of two features;

[0060] Finally, the fusion feature is expressed as follows:

[0061] .

[0062] The multimodal fusion Mamba module here is equivalent to a cross-modal optimizer, which maximizes the effect of feature extraction and learning; the three-layer Cross Attention network can effectively reduce the misalignment between visual and language features, which helps the model to more accurately understand the true relationship between images and texts, thereby generating more reliable medical reports and improving the robustness and applicability of the model.

[0063] To fuse the features into a final output, such as generating a text description, the decoder converts the features extracted from the image into a natural language description. The process of the Transformer decoder in step 6 is:

[0064] Step 6.1, input preparation,

[0065] The feature sequence z generated by the encoder has a shape of (N, S, D), where N is the batch size, S is the sequence length (usually the number of image features), and D is the feature dimension;

[0066] Step 6.2, Embedding layer:

[0067] Word embedding: Convert each word in the target sequence into a vector of fixed dimension. Assuming the vocabulary size is V and the embedding dimension is E, the output shape of the embedding layer is (N, T, E);

[0068] Positional encoding: To preserve the position information of the sequence, positional encoding is added to the word embedding. Positional encoding can be in the form of sine and cosine functions or can be learnable parameters;

[0069] Step 6.3, Mask:

[0070] Target sequence mask: In order to avoid seeing future information during the generation process, the target sequence is masked using a mask matrix M. The shape of the mask matrix is ​​(T, T), where the diagonal and the part below it are 1, and the part above is 0;

[0071] Step 6.4, multi-head self-attention mechanism:

[0072] Self-attention: The relationship between each word in the target sequence and itself is calculated through a multi-head self-attention mechanism. The calculation formula is:

[0073]

[0074] Where Q, K and V are query, key and value vectors respectively, and dk is the dimension of the key vector;

[0075] Masked self-attention: Apply a mask matrix M in the self-attention calculation;

[0076] Step 6.5, encoder-decoder attention mechanism:

[0077] Cross-Attention: Through the encoder-decoder attention mechanism, the decoder can focus on the feature sequence generated by the encoder, calculated as:

[0078] Where Qdec is the query vector of the decoder, Kenc and Venc are the key and value vectors of the encoder;

[0079] Step 6.6, feedforward neural network:

[0080] Feedforward network: The features are further processed through a feedforward neural network. The feedforward network usually includes two linear layers and a ReLU activation function;

[0081] Step 6.7, residual connection and layer normalization:

[0082] Residual connection: add the input features to the output features to preserve the original information;

[0083] Layer normalization: Maintain the stability and consistency of features through layer normalization;

[0084] Step 6.8, output layer:

[0085] Linear projection: Mapping features to the dimensions of the vocabulary through a linear layer;

[0086] Softmax: The output is converted into a probability distribution through the softmax function, indicating the probability of each word;

[0087] Target sequence: The target sequence y is the text report we want to generate, with shape (N, T), where T is the maximum length of the report.

[0088] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0089] 1. Reduce complexity when integrating features:

[0090] The SSM-Attention (State Space Attention Module) module of the present invention effectively models global spatial dependencies with linear complexity while maintaining low computational cost and efficient reasoning speed, thereby overcoming some limitations of the traditional Transformer model in visual tasks. It also retains the ability of Attention to select local features.

[0091] 2. Improve model generalization ability

[0092] Existing medical report generation methods usually rely on a large amount of annotated data and task-specific knowledge, which limits the generalization ability of the model. The present invention can more accurately understand the true relationship between images and texts, thereby reducing the dependence on annotated data and external knowledge. This not only improves the accuracy of generated reports, but also enhances the generalization ability of the model, making it perform well when processing unseen data sets and new tasks.

[0093] 3. Accuracy

[0094] In response to the common need to generate long sentences in medical reports, the present invention reduces the deviation of cross-modal data, fully integrates visual and textual features, effectively solves the large errors that may accumulate in the process of generating long sentences, and improves the consistency and accuracy of generated reports.

[0095] 4. Scalability

[0096] The SSM-Attention and Multi modal Mamba of the present invention can be independently developed and improved, and it is also convenient to integrate newly developed technical modules. It is not limited to the fusion of medical images and text features, and is applicable to the fusion of any multimodal features. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0098] Figure 2 Schematic diagram of the SSM-Attention module in the embodiment;

[0099] Figure 3 It is a schematic diagram of the Multi modal Mamba module in the embodiment;

[0100] Figure 4 It is a schematic diagram of the effect of the embodiment. DETAILED DESCRIPTION

[0101] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0102] like Figures 1 to 3 As shown, the multimodal feature fusion Mamba of the present invention is used for the automatic generation method of medical reports, and the medical report and the medical image are input into the automatic generation network of the medical report, which includes the following steps:

[0103] Step 1: Input the original medical image I into the visual feature extraction module based on ResNet101 and output the visual features ;

[0104] Step 2: Input the original medical report into the text feature extraction module based on the BERT model and output the text features ;

[0105] Step 3: The visual features obtained in step 1 And the text features obtained in step 2 They are input into the SSM-Attention visual text feature integration module respectively; the SSM-Attention visual text feature integration module respectively processes the visual features and text features Each feature is processed separately, and the processing process for each type of feature is as follows:

[0106] Step 3.1: Perform linear transformation to obtain two new sequences and ;

[0107] Step 3.2: New sequence It is processed by normalization layer, linear layer, activation function SiLu, convolution layer, state space module SSM and normalization layer in sequence; at the same time, the new sequence It is processed in sequence by the normalization layer, convolution layer, activation function ReLU, self-attention mechanism and normalization layer;

[0108] Step 3.3: Multiply the two sequences obtained after step 3.2 to generate gated features, and then transform them linearly and combine them with the original input visual features. or text features Add and output new visual features and new text features ;

[0109] Step 3 After being processed by the SSM-Attention module, the previous one is called t-1, and t' is obtained after three times; The new , The new ;

[0110] Step 4: The visual features obtained in step 3 Input the Transformer encoder and extract high-level visual features through the Transformer encoder;

[0111] Step 5: Convert the text features obtained in step 3 into And the visual features enhanced by the Transformer encoder The multimodal fusion Mamba module is input together. The multimodal fusion Mamba module includes three layers of multimodal Mamba networks with the same structure. The operation of each multimodal Mamba network is as follows:

[0112] Text Features It is processed by normalization layer, convolution layer and state space module SSM in sequence. At the same time, high-level visual features Execute steps 3.1 to 3.2 to obtain two sequences, and then add the two sequences to the processed text features to obtain new visual features; the new visual features and new text features obtained in this layer are sent to the next layer of network structure together;

[0113] The final output fusion feature ;

[0114] Step 6: Fusion features Input Transformer decoder to automatically generate medical reports.

[0115] In the method for automatically generating medical reports using multimodal feature fusion Mamba in the embodiment, the detailed processing method of the visual feature extraction module in step 1 is as follows:

[0116] Step 1.1, perform Patch embedding, divide the original medical image I into small patches through the PatchEmbed class, the size of each patch is patch_size x patch_size, and the number of patches is num_patches = (H / / patch_size) * (W / / patch_size) patches;

[0117] Step 1.2: The first three convolution blocks in the ResNet101 backbone network extract the low-level and mid-level features of each small image block and output a series of feature maps.

[0118] Step 1.3: Reorganize all feature maps to obtain visual features .

[0119] Embodiment Step 2 The text feature extraction module processes the original medical report as follows:

[0120] Step 2.1: Preprocess the text labels of the original medical reports, including word segmentation, stop word removal, and vocabulary construction;

[0121] Step 2.2: Use the BERT model to embed the preprocessed text tags: first generate word embeddings containing rich contextual information through the bidirectional Transformer encoder structure, and then convert them into fixed-length vector word embeddings. , and then the word embedding vectors are combined into sentence embedding vectors through pooling operations ;

[0122] Step 2.3: To preserve the order information in the text, add a positional encoding to each word embedding Position encoding enables the model to distinguish the difference in meaning of the same word at different positions, thus obtaining text features , the size is ; Where B is batch_size, L is sequence length, and D is embed_dim dimension;

[0123] Text Features The expression is as follows:

[0124] ;

[0125] is the word embedding vector of the vocabulary, is the position encoding vector

[0126] like Figure 2 As shown, the processing process of the SSM-Attention visual text feature integration module in the embodiment is as follows:

[0127] First, the visual features / text features are linearly transformed to generate two new sequences and ;

[0128] Then, for the new sequence and Do one of the following:

[0129]

[0130]

[0131] Next, Multiply As a gating factor, generate gated features , and then through linear transformation and addition to the original input visual tensor / text features to form the output sequence , the expression is:

[0132]

[0133] In the above formula, refers to element-wise multiplication, Refers to the visual features / text features of the original input. Figure 2 SSM-Attention corresponds to Figure 1The state-space attention module in the multimodal Mamba module corresponds to Figure 1 Multimodal fusion in Mamba.

[0134] The process of encoding new visual features by the Transformer encoder is as follows:

[0135] Step 4.1, position encoding, that is, adding position encoding to retain the spatial information of the feature;

[0136] Step 4.2: Capture the relationship between features through the multi-head self-attention mechanism;

[0137] Step 4.3: Further process the features through a feed-forward neural network.

[0138] Step 4.4: Gradually extract high-level feature representations through a multi-layer Transformer encoder.

[0139] like Figure 3 As shown, in the multimodal fusion Mamba module of step 5 of the embodiment,

[0140] After step text features and visual features After entering the first layer of multimode Mamba network, output ; With text features After entering the second-layer multimode Mamba network, output the new ; New and text features Enter the third-layer multi-mode Mamba network and output fusion features ;

[0141] The processing method for each layer of multi-mode Mamba network is:

[0142] For visual features Processing is as follows:

[0143] ;

[0144] ;

[0145] yes pass Multiply As a gating factor, generate gated features to form an output sequence ; ;

[0146] For text features Processing is as follows:

[0147]

[0148]

[0149]

[0150] Finally, the fusion feature is expressed as follows:

[0151] .

[0152] The processing process of the Transformer decoder in step 6 of the embodiment is:

[0153] Step 6.1, input preparation,

[0154] The feature sequence z generated by the encoder has a shape of (N, S, D), where N is the batch size, S is the sequence length (usually the number of image features), and D is the feature dimension;

[0155] Step 6.2, Embedding layer:

[0156] Word embedding: Convert each word in the target sequence into a vector of fixed dimension. Assuming the vocabulary size is V and the embedding dimension is E, the output shape of the embedding layer is (N, T, E);

[0157] Positional encoding: To preserve the position information of the sequence, positional encoding is added to the word embedding. Positional encoding can be in the form of sine and cosine functions or can be learnable parameters;

[0158] Step 6.3, Mask:

[0159] Target sequence mask: In order to avoid seeing future information during the generation process, the target sequence is masked using a mask matrix M. The shape of the mask matrix is ​​(T, T), where the diagonal and the part below it are 1, and the part above is 0;

[0160] Step 6.4, multi-head self-attention mechanism:

[0161] Self-attention: The relationship between each word in the target sequence and itself is calculated through a multi-head self-attention mechanism. The calculation formula is:

[0162]

[0163] Where Q, K and V are query, key and value vectors respectively, and dk is the dimension of the key vector;

[0164] Masked self-attention: Apply a mask matrix M in the self-attention calculation;

[0165] Step 6.5, encoder-decoder attention mechanism:

[0166] Cross-Attention: Through the encoder-decoder attention mechanism, the decoder can focus on the feature sequence generated by the encoder, calculated as:

[0167] Where Qdec is the query vector of the decoder, Kenc and Venc are the key and value vectors of the encoder;

[0168] Step 6.6, feedforward neural network:

[0169] Feedforward network: The features are further processed through a feedforward neural network. The feedforward network usually includes two linear layers and a ReLU activation function;

[0170] Step 6.7, residual connection and layer normalization:

[0171] Residual connection: add the input features to the output features to preserve the original information;

[0172] Layer normalization: Maintain the stability and consistency of features through layer normalization;

[0173] Step 6.8, output layer:

[0174] Linear projection: Mapping features to the dimensions of the vocabulary through a linear layer;

[0175] Softmax: The output is converted into a probability distribution through the softmax function, indicating the probability of each word;

[0176] Target sequence: The target sequence y is the text report we want to generate, with shape (N, T), where T is the maximum length of the report.

[0177] In order to verify the technical effect of the present invention, the technical solution of the present invention is compared with other existing technologies as shown in Table 1 below and the diagram of the generated report.

[0178] Table 1

[0179]

[0180] The meanings of the indicators in Table 1 are as follows:

[0181] BLEU (Bilingual Evaluation Understudy) refers to a machine translation quality evaluation index. The higher the BLEU score, the more similar the generated text is to the reference text. BLEU-1 to BLEU-4 represent the overlap of 1-gram to 4-gram, respectively. As the value of n increases, BLEU-n takes more consideration of the matching of longer segments, thus better reflecting the fluency and coherence of the generated text.

[0182] METEOR (Metric for Evaluation of Translation with Explicit ORdering) is an improvement on BLEU. It not only considers n-gram matching, but also incorporates factors such as stemming, synonym matching, and chunking. METEOR can more comprehensively evaluate the meaning and grammatical correctness of generated text.

[0183] ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation) is an evaluation metric based on the longest common subsequence (LCS), which is mainly used to evaluate the quality of text summaries. The recall rate is estimated by calculating the LCS between the generated summary and the reference summary; ROUGE-L not only considers the matching at the word level, but also the continuity at the sentence level, which is suitable for evaluating whether the generated text can capture the main information points of the original text.

[0184] It can be seen from the data in Table 1 that each index of the present invention is obviously superior to other prior arts.

[0185] like Figure 4 As shown, this embodiment processes a medical image and a real medical report and automatically generates a corresponding report.

Claims

1. A multimodal feature fusion Mamba for automatic generation of medical reports, characterized in that: The medical report and the medical image are input into the medical report automatic generation network, which includes the following steps: Step 1: Input the original medical image I into the visual feature extraction module based on ResNet101 and output the visual features ; Step 2: Input the original medical report into the text feature extraction module based on the BERT model and output the text features ; Step 3: The visual features obtained in step 1 And the text features obtained in step 2 They are input into the SSM-Attention visual text feature integration module respectively; the SSM-Attention visual text feature integration module respectively processes the visual features and text features Each feature is processed separately, and the processing process for each type of feature is as follows: Step 3.1: Perform linear transformation to obtain two new sequences and ; Step 3.2: New sequence It is processed by normalization layer, linear layer, activation function SiLu, convolution layer, state space module SSM and normalization layer in sequence; at the same time, the new sequence It is processed in sequence by the normalization layer, convolution layer, activation function ReLU, self-attention mechanism and normalization layer; Step 3.3: Multiply the two sequences obtained after step 3.2 to generate gated features, and then transform them linearly and combine them with the original input visual features. or text features Add and output new visual features and new text features ; Step 4: The visual features obtained in step 3 Input the Transformer encoder to enhance the visual features through the Transformer encoder; Step 5: Convert the text features obtained in step 3 into And the visual features enhanced by the Transformer encoder Input into the multimodal fusion Mamba module together. In the multimodal fusion Mamba module, the text features and visual features After entering the first layer of multimode Mamba network, output ; With text features After entering the second-layer multimode Mamba network, output the new ; New and text features Enter the third-layer multi-mode Mamba network and output fusion features ; The above multimodal fusion Mamba module includes three layers of multimodal Mamba networks with the same structure. The processing method of each layer of the multimodal Mamba network is as follows: the text features are processed by the normalization layer, the convolution layer, and the state space module SSM in sequence. At the same time, the visual features enhanced by the Transformer encoder are executed from step 3.1 to step 3.2 to obtain two sequences, and the two sequences are added to the processed text features to obtain new visual features; the new visual features and new text features obtained in this layer are sent to the next layer of the network structure together; the expression is as follows: For visual features Processing is as follows: ； ； yes pass Multiply As a gating factor, generate gated features to form an output sequence ; ; For text features Processing is as follows: ； ； ； Finally, the fusion feature is expressed as follows: ； Step 6: Fusion features Input Transformer decoder to automatically generate medical reports.

2. The method for automatically generating medical reports using multimodal feature fusion Mamba according to claim 1, characterized in that: The detailed processing method of the visual feature extraction module in step 1 is as follows: Step 1.1, perform Patch embedding, divide the original medical image I into small patches through the PatchEmbed class, the size of each patch is patch_size x patch_size, and the number of patches is num_patches = (H / / patch_size) * (W / / patch_size) patches; Step 1.2: The first three convolution blocks in the ResNet101 backbone network extract the low-level and mid-level features of each small image block and output a series of feature maps. Step 1.3: Reorganize all feature maps to obtain visual features .

3. The method for automatically generating medical reports using multimodal feature fusion Mamba according to claim 1, characterized in that: The text feature extraction module in step 2 processes the original medical report as follows: Step 2.1: Preprocess the text labels of the original medical reports, including word segmentation, stop word removal, and vocabulary construction; Step 2.2: Use the BERT model to embed the preprocessed text tags: first generate word embeddings containing rich contextual information through the bidirectional Transformer encoder structure, and then convert them into fixed-length vector word embeddings. , and then the word embedding vectors are combined into sentence embedding vectors through pooling operations ; Step 2.3: To preserve the order information in the text, add a positional encoding to each word embedding Position encoding enables the model to distinguish the difference in meaning of the same word at different positions, thus obtaining text features , the size is ; Where B is batch_size, L is sequence length, and D is embed_dim dimension; Text Features The expression is as follows: ； is the word embedding vector of the vocabulary, is the position encoding vector.

4. The method for automatically generating medical reports using multimodal feature fusion Mamba according to claim 1, characterized in that: The processing process of the SSM-Attention visual text feature integration module in step 3 is as follows: First, the visual features / text features are linearly transformed to generate two new sequences and ; Then, for the new sequence and Do one of the following: ； ； Next, Multiply As a gating factor, generate gated features , and then through linear transformation and adding to the original input visual features / text features to form an output sequence, the expression is: ； ； In the above formula, refers to element-wise multiplication, the input refers to the visual features of the original input; the input Refers to the text features of the original input.

5. The method for automatically generating medical reports using multimodal feature fusion Mamba according to claim 1, characterized in that: In step 4, the Transformer encoder generates new visual features after the SSM-Attention module. The encoding process is as follows: Step 4.1, position encoding, that is, adding position encoding to retain the spatial information of the feature; Step 4.2: Capture the relationship between features through the multi-head self-attention mechanism; Step 4.3: Process features through a feedforward neural network. Step 4.4: Gradually extract high-level feature representations through a multi-layer Transformer encoder.

Images