A medical image description method based on a state space model

Abstract

The application discloses a medical image description method based on a state space model, and comprises the following steps: obtaining a target image and a text report; inputting the target image into an image coding model to obtain global information, wherein the image coding model comprises a self-attention Mamba block and a normalization layer; inputting the text report and the global information into a text coding model to obtain a text word probability, wherein the text coding model comprises a self-attention layer, a normalization layer, a cross Mamba block and a full connection layer; and obtaining an image description based on the text word probability. The self-attention Mamba block of the application extracts the features of an abnormal area in the target image through global information modeling, the cross Mamba block optimizes the cross-modal interaction ability between the medical image and the radiology report, and the consistency of the two is enhanced, and the loss function utilizes the sparsity to relieve the problem of unbalanced positive and negative samples.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of automatic radiology image report generation, and particularly relates to a medical image description method based on a state space model. BACKGROUND

[0002] In modern medical systems, quickly and accurately interpreting radiology images such as CT scans and MRIs is crucial for improving the accuracy and efficiency of disease diagnosis. With the rapid increase in the number of medical images, it has become increasingly unrealistic for radiologists to manually write detailed diagnostic reports, especially when faced with urgent medical needs. Using artificial intelligence technology to automatically generate accurate radiology reports not only can significantly reduce the burden on doctors, but also can improve the efficiency and standardization level of the diagnosis process. For doctors in the early stages of training, automatically generated reports can serve as valuable learning tools and auxiliary references, helping them identify subtle abnormalities that are difficult to detect. The development of this technology also means that patients can receive diagnosis results based on accurate data analysis more quickly, thereby speeding up the treatment process.

[0003] Medical image description usually adopts an encoder-decoder architecture. This architecture can encode image information into features and then convert them into textual descriptions through a decoder. The decoder part is mainly divided into two types: autoregressive and non-autoregressive. For medical image description, autoregressive models have become the mainstream choice due to their iterative nature in generating descriptions. In this mode, text generation is performed step by step, with each iteration based on the output of the previous step. This sequential generation method is beneficial for producing more coherent and accurate descriptions. However, autoregressive models also have significant drawbacks. Due to the iterative nature of the generation process, any errors at a generation stage can be accumulated by subsequent steps, leading to the final output deviating from the correct description. This problem is particularly critical in the medical field, as the generation of radiology reports is different from regular natural image description, which usually requires generated sentences to be between 60 and 100 words, much longer than natural image description. In this case, autoregressive decoders are more likely to cause error accumulation, thereby affecting the accuracy of the report.

[0004] To address the above problems, Zhang et al. constructed a graph embedding module to model and learn features of disease discovery and their relationships by integrating a knowledge graph. Then, Zhang et al. proposed a method to combine knowledge graph-guided graph features with visual features to support report generation. Although this method can strengthen the connection between features and text, constructing a knowledge graph for a specific dataset is a complex process. Chen Z et al. designed a relationship memory to retain key information during the generation process, and Xue Y et al. proposed a memory-driven relationship memory aided generation technique. Although this method can effectively simulate the report generation process, the memory-driven mechanism tends to remember the text reports that often appear in the dataset.

[0005] Note the recent progress of state space models (SSMs). Specifically, structured state space sequence models (S4) have become a promising sequence modeling architecture. Based on S4, Mamba applies time-varying parameters to SSMs and proposes an efficient hardware-aware algorithm with a linear complexity global receptive field. These SSM-based models focus on key research problems in natural language processing (NLP) at their inception, especially content-based reasoning and language understanding. Recently, some concurrent methods focus on 2D vision tasks (e.g., classification, segmentation).

[0006] The existing autoregressive decoding has accumulated errors, resulting in large differences in the final output text; the image features of medical reports have small differences, making it difficult for the image encoder to accurately identify the lesions in each image; and the sample imbalance results in a large difference in the number of positive and negative samples, and the confidence information of the prediction result is missing. Therefore, there is an urgent need for a medical image description method based on a state space model. SUMMARY

[0007] To solve the above technical problems, the present application provides a medical image description method based on a state space model. The self-attention Mamba block of the present application extracts the features of the abnormal regions in the target image by modeling the global information, and the cross-Mamba block enhances the consistency between the medical image and the radiology report by optimizing the cross-modal interaction ability between them.

[0008] The present application provides a medical image description method based on a state space model, comprising:

[0009] Obtaining a target image and a text report;

[0010] Inputting the target image into an image encoding model to obtain global information, wherein the image encoding model comprises a self-attention Mamba block and a normalization layer;

[0011] inputting the text report and the global information into a text encoding model to obtain a text word probability, wherein the text encoding model comprises a self-attention layer, a normalization layer, a cross-Mamba block and a full connection layer;

[0012] obtaining an image description based on the text word probability.

[0013] Optionally, before inputting the target image into the image encoding model, the method further comprises:

[0014] projecting the target image into a feature space to obtain a feature vector;

[0015] arranging the feature vector along a spatial dimension to obtain a target feature.

[0016] Optionally, the self-attention Mamba block comprises a first Mamba block, a second Mamba block and a full connection layer.

[0017] The first Mamba block is configured to extract a spatial relationship of the target feature.

[0018] The second Mamba block is configured to process a square of the target feature.

[0019] The full connection layer is configured to fuse the processed features.

[0020] Optionally, inputting the text report and the global information into the text encoding model to obtain the text word probability comprises:

[0021] encoding the text report to obtain a text source sequence;

[0022] inputting the text source sequence into the self-attention layer to obtain a text report hidden sequence;

[0023] fusing the text report hidden sequence and the global information by using the cross-Mamba block to obtain fused features;

[0024] mapping the fused features to obtain the text word probability.

[0025] Optionally, the method of inputting the text source sequence into the self-attention layer to obtain the text report hidden sequence comprises:

[0026] L = MultiHead (Q, K, V) = Concat (head1, head2, …, head h ) W 0

[0027] wherein L is the text report hidden sequence, Q, K and V represent query, key and value matrices, W 0 is a learnable parameter, and head hThe cross-attention mechanism is used.

[0028] Optionally, the cross-Mamba block comprises a self-attention Mamba block, a flattening layer and an attention module.

[0029] The self-attention Mamba block is used for global information modeling of the hidden sequence of the image.

[0030] The flattening layer is used for dimension reduction processing of the hidden sequence of the text report.

[0031] The attention module is used for cross-modal information fusion and interaction of the hidden sequences of the image and the text report.

[0032] Optionally, the text report hidden sequence and the global information are fused by using the cross-Mamba block to obtain a fusion feature, and the method comprises the following steps:

[0033] The global information is processed by using the self-attention Mamba block to obtain an image feature.

[0034] The text report hidden sequence is dimensionally reduced by using the flattening layer to obtain a one-dimensional text feature.

[0035] The image feature and the one-dimensional text feature are merged, and the merged feature is used as a key and a value of the attention module, and cross-attention processing is performed on the text source sequence to obtain the fusion feature.

[0036] Optionally, the fusion feature is mapped to obtain the text word probability, and the method comprises the following steps:

[0037]

[0038] FFN (x) = xW1 + b1 i 、 represents the i-th feature pyramid structure of N decoder structures and a cross-Mamba block, L j , y j respectively represent a hidden state of a text report and a probability of a predicted report, I is a number of layers of an autoregressive decoder, Linear() is a linear mapping layer, mapping is performed into a size of a vocabulary, and Softmax() is a normalization function.

[0039] Compared with the prior art, the present application has the following advantages and technical effects:

[0040] The self-attention Mamba self-mamba block of the application is responsible for global modeling of visual features, while the cross-attention Mamba cross-mamba block is good at cross-modal alignment and disease-related representation learning. By introducing a random mask loss mechanism, the loss function can be further optimized. Experimental results show that the Caption-Mamba framework of the application performs competitively with previous studies on two public datasets. In addition, ablation studies verify the effectiveness of the components of Caption-Mamba. At the same time, illustrative visualizations and discussions are provided to inspire future research directions. BRIEF DESCRIPTION OF DRAWINGS

[0041] The accompanying drawings, which form a part of this application, are intended to provide further understanding of the application and are incorporated herein in their entirety, and together with the description are intended to explain the application. The following drawings are provided to assist in understanding the application and are used to explain the application, but do not constitute an improper limitation on the application. In the drawings:

[0042] Figure 1 is a flowchart of a medical image description method based on a state space model according to an embodiment of the application;

[0043] Figure 2 is an image processed by a Mamba block according to an embodiment of the application, Figure 2 (a) is an original image, Figure 2 (b) is a processed image;

[0044] Figure 3 is a loss change under different training periods according to an embodiment of the application;

[0045] Figure 4 is a graph showing the influence of different learning rates on experimental results on the IU-Xray and COV-CTR datasets according to an embodiment of the application;

[0046] Figure 5 is a graph showing the influence of different random mask coverages on experimental results on the IU-Xray and COV-CTR datasets according to an embodiment of the application. DETAILED DESCRIPTION

[0047] It should be noted that the embodiments and features in the application can be combined with each other without conflict. The application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0048] It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases the steps shown or described herein can be executed in a different order than shown.

[0049] The application proposes a medical image description method based on a state space model, as shown inFigure 1 as shown, specifically comprising the following steps:

[0050] obtaining a target image and a text report;

[0051] inputting the target image into an image encoding model to obtain global information, wherein the image encoding model comprises a self-attention mamba block and a normalization layer;

[0052] inputting the text report and the global information into a text encoding model to obtain text word probability, wherein the text encoding model comprises a self-attention layer, a normalization layer, a cross-mamba block and a fully connected layer;

[0053] obtaining an image description based on the text word probability.

[0054] Specifically, the present application adopts a basic Transformer architecture framework, and introduces a self-mamba block in the image encoder module. The mamba block is crucial for modeling long-distance dependencies. In addition, the present application introduces a cross-mamba block, which is mainly used to study the interaction between image and text modalities. Radiological images are input into the model and converted into the corresponding source sequence X=(x1,x2,..,x n ),x n ∈R d , where x n represents the visual features of the medical image extracted by the visual extractor, and d represents the dimension of the feature vector. The corresponding report forms the target sequence Y=(y1,y2,…,y t ),y t ∈V, where y t is the generated token, t is the length of the generated token, and V is the vocabulary containing all possible tokens.

[0055] Further, before inputting the target image into the image encoding model, it further comprises:

[0056] projecting the target image into a feature space to obtain a feature vector;

[0057] arranging the feature vector along the spatial dimension to obtain a target feature.

[0058] Specifically, the present application uses a visual extraction module to process the input medical image. The feature extractor projects the image into a feature space using a pre-trained ResNet101 model, and arranges these features along the spatial dimension into tokens, denoted as:

[0059]

[0060] where X0 is the initial token set extracted from the image, and x nfor a single feature unit token, for a function to apply the ResNet101 model to an image, converting the image into a series of feature tokens.

[0061] Further, the self-attention mamba block comprises: a first mamba block, a second mamba block, and a fully connected layer;

[0062] The first mamba block is used to extract the spatial relationship of the target feature;

[0063] The second mamba block is used to process the square of the target feature;

[0064] The fully connected layer is used to fuse the processed features.

[0065] Specifically, X0 is input into an image encoder composed of multiple self-mamba blocks. In order to prevent gradient problems, these blocks include residual connections and normalization operations, and perform global information extraction on the input.

[0066]

[0067] Here, represents the i-th self-mamba block for the image encoder structure.

[0068] Inspired by the mamba block, the present application uses self-mamba blocks to extract radiographic image features and model long-range dependencies. First, the principle of mamba implementation needs to be understood, and Table 1 provides a comprehensive overview of the operation. Specifically, the input sequence I l-1 ∈R B×N×C Initial normalization is performed by layer normalization. Subsequently, a multi-layer perceptron (MLP) is used to project the normalized sequence into x∈R B×N×P and z∈R B×N×P . Next, a 1D convolution layer with SiLU activation is applied to process X and produce x'. x' is further projected onto A, B, and C, and converted to discrete versions and The parameter generation process is shown in Algorithm 1, and then the output y is calculated by SSM. Subsequently, y is gated by z and added to the input T l-1 to obtain the output sequence T l . The SSM process is shown in The computational complexity of the mamba block is linear with respect to the sequence length N.

[0069] Table 1

[0070]

[0071]

[0072] As Figure 2 (a), (b) Due to the two-dimensional nature of visual data, the original Mamba module struggles to capture the dependent information in different directions in the image simultaneously during the scanning process, resulting in a limited receptive field and affecting the performance of the model. To address this issue, the self-mamba module uses two mamba modules simultaneously, each processing different input features. One module processes the original image features, and the other processes the square of the original image features. The uniqueness of this design lies in capturing different features and patterns of input data, achieving more comprehensive information extraction. Specifically, the mamba module processing the original image features can extract basic spatial relationships and patterns, while the mamba module processing the square of the image features can capture more complex relationships. After processing by the respective mamba modules, the two features are fused and added through a linear layer, introducing diversified linear changes. This design to some extent overcomes the limitations of the original mamba module, and Table 2 provides a comprehensive overview of the operation.

[0073] Table 2

[0074] Algorithm 2 Self Mamba Block Input: token sequence I l-1 :(B, N, C) Output: token sequence I l : (B, N, C)] / *square of features* / 1: I 2 l-1 : (B, N, C) <- I l-1 *I l-1 ]] / *Algorithm 1 operation* / 2: I' l : (B, N, C) <- MambaBlock(I l-1 ) 3: I 2 l : (B, N, C) <- MambaBlock(I 2 l-1 )]]> 4: I" l : (B, N, C) <- I' l + I 2 l ]]> / *feature mapping* / 5: I l : (B, x, C) <- Linear I (I" l )]]> Return: I l ]]

[0075] Further, the text report and global information are input into the text encoding model to obtain the text word probability, including:

[0076] The text report is encoded to obtain a text source sequence;

[0077] The text source sequence is input into the self-attention layer to obtain a text report hidden sequence;

[0078] The text report hidden sequence and global information are fused using the cross-mamba block to obtain fused features;

[0079] The fused features are mapped to obtain the text word probability.

[0080] Specifically, in the decoding stage, in order to enable the text report Y to express semantics more deeply and prevent the model from learning future semantics, a masked self-attention mechanism is used, and the encoding result is taken as the source sequence of all subsequent modules, expressed as:

[0081] Y = Q = K = V

[0082]

[0083] L = MultiHead (Q, K, V) = Concat (head1, head2, …, head h ) W 0

[0084] where Q, K, V represent query, key, value matrix, and W 0 are learnable parameters. In our work, we choose 8 heads for multi-head attention.

[0085] The modeled feature I and the text report hidden state L j are fused and then passed through the feature pyramid. Finally, the feature is mapped to the vocabulary dimension and then passed through the softmax to express as:

[0086]

[0087] Here, FFN i , represents the i-th feature pyramid structure and cross-mamba block of N decoder structures. L j , y j represent the hidden state of the text report and the probability of the predicted report, respectively, and the length of the inference stage j depends on the length of the predicted text report, which represents the hidden state of the previous word and the predicted probability of the word, respectively. j j

[0088] Further, the cross-mamba block includes: a self-attention mamba block, a flattening layer, and an attention module;

[0089] The self-attention mamba block is used to model the global information of the hidden sequence of the image;

[0090] The flattening layer is used to reduce the dimension of the text report hidden sequence;

[0091] The attention module is used for cross-modal information fusion and interaction of the hidden sequences of the image and the text report.

[0092] Specifically, the present application introduces a novel cross-mamba block, which aims to promote the interaction and fusion of cross-modal features. In this method, the sequence features of the image are first processed by self-mamba, and then the text report processed by the self-attention mechanism is averaged in the second dimension to make its dimension one-dimensional, and the feature is added to the image feature. Next, the combined feature is used as the key and value of cross-attention, and the text report feature processed by the original self-attention is processed by cross-attention. Table 3 details the operation process.

[0093] Specifically, the input image hidden sequence I l-1 ∈R B×N×C follows the steps of algorithm 3, while the report hidden sequence L l-1 ​​∈R B×x×C In the middle, wherein x is less than or equal to 60 for IU-Xray data set, less than or equal to 100 for COV-CTR data set, then, the second dimension average processed text report feature is added to the image sequence through the broadcast mechanism, which can effectively integrate the prior knowledge of the text report into the image sequence, and finally through the cross attention processing with the report hidden sequence L l-1 The cross attention processing with the report hidden sequence L

[0094] Table 3

[0095] Algorithm 3 Cross Modal Mamba Block Input: token sequence I l-1 : (B, N, C), L l-1 : (B, x, C) <!-- 6 -->]]> Output: token sequence L l : (B, x, C)] / *Algorithm 2 operation* / 1: I' l : (B, N, C) <- MambaBlock(I l-1 ) / *flatten operation* / 2: L' l : (B, 1, C) <- Flatten(L l-1 ) / *fusion of prior features* / 3: I" l : (B, N, C) <- I' l + L' l ]]> / *L l-1 as a query;I l as key and value* / ]]> 4: L l : (B, x, C) <- CrossAttention(L l-1 , I l , I l ) Return: L l ]]

[0096] Further, the text report hidden sequence and the global information are fused by using the cross mamba block to obtain the fusion feature, including:

[0097] The global information is processed by using the self-attention mamba block to obtain the image feature;

[0098] The text report hidden sequence is processed by using the flattening layer to obtain the one-dimensional text feature;

[0099] The image feature and the one-dimensional text feature are merged, and the merged feature is used as the key and value of the attention module, and the cross attention processing is performed with the text source sequence to obtain the fusion feature.

[0100] Further, the fusion feature is mapped to obtain the text word probability, and the method is:

[0101]

[0102] Wherein, FFN i 、 Indicates the i-th feature pyramid structure of N decoder structures and the cross mamba block, L j , y j Respectively represent the hidden state of the text report and the probability of the predicted report, I is the number of layers of the autoregressive decoder, Linear() is a linear mapping layer, which is mapped into the size of the vocabulary table, and Softmax() is a normalization function.

[0103] The present application is also inspired by the idea of sparsity, and it is realized that the usual negative log-likelihood loss may be too strict for text generation tasks. Therefore, the present application constructs a customized loss function, which aims to introduce a certain sparsity. Specifically, a random mask is introduced, and part of the loss value is calculated by random selection, so as to achieve the purpose of enhancing the generalization ability and robustness of the model. The loss function is defined as follows:

[0104]

[0105] Here, N represents the batch size, T denotes the length of each sample sequence, rand i,j The random mask, with a value of 1 indicating that the loss of this time step is selected for calculation, and a value of 0 indicating that the loss of this time step is skipped. The output of the model represents the target word y i,j The corresponding log probability.

[0106] This formula calculates the loss by selecting part of the loss value, thereby reducing the model's over-reliance on specific data and improving the model's generalization ability and robustness. As shown in Figure 3 The application of random mask for loss optimization makes the optimization target of each epoch different even if the same sample is encountered at different training epochs. The original method only simply strengthens memory. Our method achieves different optimization targets for each epoch by using random masks, thereby improving the training effect of the model.

[0107] Embodiments

[0108] To evaluate the robustness and generalization ability of Caption-Mamba, as shown in Table 4, two public datasets, IU-Xray and COV-CTR, were used in the experiment. The IU-Xray dataset is a widely used benchmark for evaluating radiology report generation methods. It contains 7470 chest X-ray images and 3955 radiology reports. The dataset is divided into training, validation and test sets in the ratio of 7:1:2. The COV-CTR dataset contains 728 lung CT images and their corresponding Chinese and English diagnosis reports, of which 349 are COVID-19 cases and 379 are non-COVID-19 cases. To ensure fair comparison, the COV-CTR dataset is also divided into training, validation and test sets in the ratio of 7:1:2.

[0109] Table 4

[0110]

[0111] As shown in Table 4, the statistical content includes the number of images, the number of reports and the average length of the reports. First, samples with no reports were excluded. In terms of experimental settings, for the IU-Xray dataset, the maximum length of generated characters was set to 60 characters; for the COV-CTR dataset, the maximum length of generated characters was set to 100 characters. The report texts of the two datasets were uniformly processed, and characters with fewer than three occurrences were automatically filtered out during the training process. In addition, the image preprocessing process was further optimized by standardizing the images, constructing a vocabulary and removing stop words.

[0112] The experiments of this embodiment were performed on an AMD Ryzen 55600G (equipped with Radeon Graphics CPU, 3.9 GHz), 32 GB memory and Nvidia RTX 3090 GPU. For IU-Xray and COV-CTR datasets, ResNet-101 was used as the image feature extractor. The extracted image features were mapped as It is worth noting that for the IU-Xray dataset, the frontal and lateral patient images were processed through the feature extraction network at the same time, and the final mapped features were Through extensive experiments, the optimal parameters were determined. For the image feature extractor of the IU-Xray and COV-CTR datasets, the learning rate range was set between [5e-5, 8e-5, 1e-4, 3e-4, 5e-4, 1e-4], and the optimal parameter was 5e-5. The learning rate range of the model was [5e-5, 8e-5, 1e-4, 3e-4, 5e-4, 1e-4], and the optimal model learning rate was 1e-5. The optimal dropout rate was 0.1.

[0113] This embodiment adopts four commonly used text generation evaluation indicators: BLEU, Rouge-L, METEOR and CIDEr to measure the performance of the model. The BLEU score is based on n-gram precision evaluation, calculates the n-gram overlap between generated text and reference text, and includes a short penalty to prevent overly short sentences. The Rouge-L score is based on the longest common subsequence (LCS) calculation between the generated text and the reference text. METEOR takes into account synonyms and stems when calculating the score, combining precision and recall into an indicator that emphasizes recall. CIDEr is based on consensus, using the term frequency-inverse document frequency (TF-IDF) of n-grams in the generated text and reference text to calculate, and penalizing common n-grams that do not provide meaningful content.

[0114] In reference to previous work, two different approaches were adopted for image processing of the IU-Xray and COV-CTR datasets. The approach included concatenation of visual labels for the IU-Xray dataset, while a single-view approach was used for the COV-CTR dataset. The images were initially resized to (256, 256) and then randomly cropped to (224, 224) during training and center-cropped during inference. During training, random rotation and scaling operations were applied to further augment the dataset. ResNet101 pre-trained on ImageNet was used as the visual feature extractor, generating 7x7 visual labels. A memory-driven Transformer with random initialization was adopted as the backbone of the encoder-decoder module, configured with 1 layer, 8 attention heads, and 512-dimensional hidden states. The visual feature extractor and encoder-decoder were optimized using the Adam optimization algorithm, with learning rate settings of 5e-5 and 1e-4 on the IU-Xray dataset. Similarly, the learning rate settings on the COV-CTR dataset were also set to 5e-5 and 1e-4. In line with previous studies, Beam Search was used as the sampling method to generate reports for the validation and test sets. A beam size of 3 was chosen to balance effectiveness and efficiency. The model was implemented using the PyTorch deep learning framework on an Nvidia RTX3090 GPU.

[0115] The proposed method was compared with previous state-of-the-art (SOTA) models, such as R2Gen, R2GenCMN, KEMMRG, MMKERG, ASGMD, DEKG, MAMRG, MiniGPT-4-RRG, MDAKF, and SERPENT-VLM. Notably, to ensure fairness in comparison, the results of the comparison objects were obtained by comparing with the results of the original papers, as shown in Table 5.

[0116] The method performed well on the IU-Xray and COV-CTR datasets. On the IU-Xray dataset, all evaluation metrics improved by an average of 3.87% compared to the baseline model, with a 5.3% improvement in BLEU-1 performance. Notably, the method outperformed the state-of-the-art models in most metrics. Although the BLEU-1 metric was slightly lower than SERPENT-VLM, the BLEU-4 metric improved by 1.1%, indicating that the generated reports were closer to real reports and the predicted results were more accurate. On the COV-CTR dataset, the method performed best. Except for the CIDEr metric, all evaluation metrics improved by an average of 1.83% compared to the baseline model. This further demonstrated the effectiveness and superiority of the method.

[0117] Table 5

[0118]

[0119]

[0120] Ablation experiments were conducted on IU-Xray and COV-CTR datasets to verify the effectiveness of the key modules proposed in this paper. The results are shown in Table 6.

[0121] In the ablation study of Caption-Mamba model, the effectiveness of each module was first analyzed. On both datasets, the introduction of two modules improved each indicator to varying degrees compared to the baseline model. Figure 5 The visualization results of Caption-Mamba on IU-Xray and COV-CTR datasets are shown. The generated report shows significant consistency with the true report and corresponds to the visualization attention map. Before the cross-mamba module is added, the image area that the model focuses on tends to be inaccurate. By combining the two modules, the model can focus on the important areas of the image significantly (the degree of red represents the intensity of attention). The results show that Caption-Mamba can generate accurate and explainable attention area reports.

[0122] The main ablation study results are listed in Table 6. By adding the self-mamba module, significant improvements can be seen, for example, the BLEU-4 scores on IU-Xray and COV-CTR benchmarks increased from 0.153 to 0.161 and from 0.544 to 0.545, respectively. When further integrating the cross-mamba module, the BLEU-4 scores of these two datasets increased from 0.161 to 0.201 and from 0.545 to 0.555, respectively. In particular, the complete model improved all evaluation indicators on IU-Xray by an average of 18.65%, while the improvement on COV-CTR was not as obvious as IU-Xray, because in the smaller dataset COV-CTR, both the baseline model and the proposed model are easy to learn the corresponding relationship.

[0123] In the ablation study of Caption-Mamba, the effectiveness of the fusion of self-mamba module and cross-mamba module is explored. The fusion of the two modules gets good experimental results. With the stacking of the modules, the overall performance of the model in the IU-Xray dataset gradually improves. The results show that the two main modules proposed are helpful for the generation of medical report task. Compared with the baseline, there is a large improvement in BLEU-1, BLEU-2, BLEU-3 and BLEU-4 scores, and the METEOR evaluation index also has a significant improvement. In the two datasets, the BLEU index is greatly affected by the cross-mamba module. The combination of self-mamba module and cross-mamba module significantly affects the BLEU and METEOR indexes.

[0124] Table 6

[0125]

[0126] As shown in Table 7, the model is improved based on the architecture of 1 layer encoder-decoder, and compared with the baseline of 3 layer encoder-decoder architecture. Taking the IU-Xray dataset as an example (including two visual extractors and the setting of the maximum generated characters being 60), the parameter quantity and size of the model are 66.84M and 1.1GB respectively, while the parameter quantity and size of the baseline model are 78.47M and 1.3GB respectively. It can be seen that the model is smaller than the baseline model in parameter quantity and model size, but the effect is better. As shown in Tables 8 and 9, when trying to increase the number of model layers, it is found that when the number of layers is 2, the parameter quantity and model size are 86.73M and 1.4GB; when the number of layers is 3, the parameter quantity and model size are 106.61M and 1.7GB. With the increase of the number of layers, all evaluation indexes begin to decline. Although the indexes decline is not obvious when the number of layers is 2 and 3, when the number of layers is 2, the parameter quantity and model size are close to the size of the baseline 3 layer model, but the indexes are still better than the baseline model.

[0127] Table 7

[0128]

[0129] From the perspective of model capacity and size, increasing the number of network layers significantly improves the capacity of the model, enabling it to learn and represent more complex features. However, in the experiments, the IU-Xray and COV-CTR datasets themselves are not large, especially the COV-CTR dataset is smaller, and the increased model capacity may not be fully utilized, but instead triggers the overfitting problem. This phenomenon is particularly evident in the experimental results. As the number of layers increases, the scores of evaluation metrics such as BLEU, METEOR, ROUGE-L, and CIDEr all decrease, indicating that although the model becomes more complex, its generalization ability is weakened. This may be due to the model overfitting the noise in the training data and failing to effectively extract useful features.

[0130] Table 8

[0131]

[0132] Table 9

[0133]

[0134] This embodiment describes the results of sensitivity experiments to explore the impact of parameters such as learning rate and random mask coverage on model performance.

[0135] In the parameter analysis of learning rate, different empirical learning rates were set for the model to verify their impact on evaluation metrics. As shown in Figure 4 , on the IU-Xray and COV-CTR datasets, when the learning rate is 1e-4, the indicators as a whole achieve the best results, especially the CIDER indicator. Through these two charts, it can be found that when the learning rate is less than 1e-4, the indicators change little, while when the learning rate gradually increases, the indicators change greatly.

[0136] For the parameter analysis of random mask, six groups of random mask parameters were used to analyze the impact of loss optimization on the model, where 0% represents loss without random mask. As shown in Figure 5 , on the IU-Xray dataset, when the random mask is set to 70%, the model achieves the best results. On the COV-CTR dataset, the results are good and bad when the random mask is set to 70% and 80% and without random mask. Although the results of the method proposed in this paper vary with different parameter selections, the overall performance is better than the current popular medical report generation algorithm.

[0137] cross-mamba performs well in the cross-modal fusion of modalities, but its main improvement is still in the extraction of hidden features of input images. The combination of prior information in the text report for cross-modal fusion is easy to cause the model to rely on prior information, and the generated text report tends to use common words, which cannot deeply mine the deep information of the image. To solve this problem, a compromised solution is adopted to further extract the hidden features of the image before the interaction between the text report features and the image features. In terms of improving the loss function, an attempt is made to mask part of the words for iterative learning, but the effect on the COV-CTR dataset is not ideal, especially for small datasets. Future work may focus on the construction of the loss function. The current loss optimization and evaluation indicators (such as BLEU, METEOR, ROUGE-L, CIDEr) have little relevance, so in the future, reinforcement learning may be combined for optimization, or new loss function construction methods that do not rely on reinforcement learning may be explored.

[0138] In this embodiment, a Caption-Mamba framework for generating medical reports using state space is proposed. The framework aims to improve the accuracy and logical consistency of medical reports. The self-mamba module is responsible for global modeling of visual features, while the cross-mamba module excels in cross-modal alignment and disease-related representation learning. By introducing a random mask loss mechanism, the loss function can be further optimized. Our experimental results show that the Caption-Mamba framework performs competitively with previous studies on two public datasets. In addition, ablation studies verify the effectiveness of the components of the Caption-Mamba framework. At the same time, illustrative visualizations and discussions are provided to inspire future research directions.

[0139] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A medical image description method based on a state space model, characterized by, The method comprises the following steps: obtaining a target image and a text report; inputting the target image into an image encoding model to obtain global information, wherein the image encoding model comprises a self-attention Mamba block and a normalization layer; the method further comprises the following steps before inputting the target image into the image encoding model: projecting the target image into a feature space to obtain a feature vector; arranging the feature vector along a spatial dimension to obtain a target feature; the self-attention Mamba block comprises a first Mamba block, a second Mamba block and a fully connected layer; the first Mamba block is used for extracting the spatial relationship of the target feature; the second Mamba block is used for processing the square of the target feature; the fully connected layer is used for fusing the processed features; inputting the text report and the global information into a text encoding model to obtain a text word probability, wherein the text encoding model comprises a self-attention layer, a normalization layer, a cross-Mamba block and a fully connected layer; the method of inputting the text report and the global information into the text encoding model to obtain the text word probability comprises the following steps: encoding the text report to obtain a text source sequence; inputting the text source sequence into the self-attention layer to obtain a text report hidden sequence; fusing the text report hidden sequence and the global information by using the cross-Mamba block to obtain a fusion feature; mapping the fusion feature to obtain the text word probability; the cross-Mamba block comprises a self-attention Mamba block, a flattening layer and an attention module; the self-attention Mamba block is used for globally modeling the hidden sequence of the image; the flattening layer is used for dimension reduction processing of the text report hidden sequence; the attention module is used for cross-modal information fusion and interaction of the hidden sequences of the image and the text report; the method of fusing the text report hidden sequence and the global information by using the cross-Mamba block to obtain a fusion feature comprises the following steps: processing the global information by using the self-attention Mamba block to obtain an image feature; dimension reduction processing of the text report hidden sequence by using the flattening layer to obtain a one-dimensional text feature; merging the image feature and the one-dimensional text feature, and cross-attention processing of the merged feature as the key and value of the attention module with the text source sequence to obtain the fusion feature; based on the text word probability, obtaining an image description.

2. The method of claim 1, wherein, the method of inputting the text source sequence into the self-attention layer to obtain a text report hidden sequence comprises the following steps: wherein L is a text report hidden sequence, Q, K, V represent query, key, value matrix, are learnable parameters, is a multi-head attention mechanism.

3. The method of claim 1, wherein, the method of mapping the fusion feature to obtain the text word probability comprises the following steps: wherein, represents the N-th feature pyramid structure and cross-mamba block of the N decoder structures, , , respectively represent the hidden state of the text report and the probability of the predicted report, I is the number of layers of the autoregressive decoder, is a linear mapping layer, mapping into the size of the vocabulary, is a normalization function.

Images