A Radiological Report Generation Method Based on Enhanced Dual-Modal Features

Abstract

This application discloses a radiology report generation method based on enhanced bimodal features, belonging to the technical field of radiology report generation methods. In this invention, the Visual Mamba visual encoder enhances visual features to obtain an enhanced visual feature map with horizontal and vertical correlations and two-dimensional spatial information; the report similarity matrix module dynamically learns report similarity information of tokens in text features based on a similarity learning matrix to enhance text features; when the feature enhancement conditional decoder decodes the enhanced feature map obtained from the enhanced visual feature map and the text features output by the report similarity matrix module, it can obtain a predicted radiology report that is closer to the real report. Tests show that the BLEU-1, BLEU-2, BLEU-3, BLEU-4, METEOR, and ROUGE-L indices obtained by this application are all superior.

Description

Technical Field

[0001] This invention belongs to the technical field of radiology report generation methods, specifically relating to a radiology report generation method based on enhanced bimodal features. Background Technology

[0002] A large number of radiological images require radiologists to interpret them manually, resulting in a heavy workload. To alleviate this workload, there is an urgent need for effective radiology report generation methods. Currently, existing radiology report generation methods widely employ the Transformer architecture and improve report quality by promoting cross-modal alignment or incorporating external medical knowledge. However, these methods often suffer from poor enhancement of visual and textual features, making it difficult to obtain predicted radiology reports that closely resemble sample reports. Therefore, this application proposes a radiology report generation method based on enhanced bimodal features. Summary of the Invention

[0003] In view of the defects and shortcomings of the prior art, this application proposes a radiological report generation method based on enhanced dual-modal features.

[0004] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0005] A radiological report generation method based on enhanced bimodal features includes the following steps:

[0006] The X-ray image to be predicted is input into the radiology report generation network model and propagated forward once to obtain the predicted radiology report.

[0007] The radiology report generation network model is based on training a radiology report generation network;

[0008] The radiology report generation network includes a visual feature extractor, a Visual Mamba visual encoder, an embedding layer, a report similarity matrix module, a feature enhancement conditional decoder, a linear layer, and a softmax layer. The report similarity matrix module has T layers, which are connected sequentially. The feature enhancement conditional decoder is an existing feature enhancement conditional decoder. There are three feature enhancement conditional decoders, which are connected sequentially. The output of the third feature enhancement conditional decoder is connected sequentially to the linear layer and the softmax layer.

[0009] The visual feature extractor is used to extract visual features V from the X-ray image to be predicted. i ;

[0010] The Visual Mamba visual encoder is used to process visual features V iEnhancement is performed to obtain enhanced visual feature maps with horizontal correlation, vertical correlation, and two-dimensional spatial information. ;

[0011] The embedding layer is used to perform embedding mapping operations on the radiology report text to obtain its textual features. ;

[0012] The report's similarity matrix module dynamically learns text features based on a similarity learning matrix. The report uses similarity information of the token to enhance text features;

[0013] The output of the similarity matrix module at layer T, after dimensional expansion, is input into three feature enhancement conditional decoders, and the enhanced visual feature maps output by the Visual Mamba visual encoder are then fed into these decoders. The text features are also input into three feature enhancement conditional decoders. The input is fed into the first feature enhancement conditional decoder, which is used to perform decoding operations. The output of the third feature enhancement conditional decoder is processed by the prediction head to obtain the predicted characters. All the predicted characters constitute the radiological report.

[0014] Preferably, the prediction head includes a linear layer and a softmax layer connected in sequence; the output of the third decoder is processed by the linear layer and the softmax layer in the prediction head in sequence, and the probability distribution of each character in the dictionary is output, and the character with the highest probability is obtained, which is the predicted character.

[0015] Preferably, the input of the first feature enhancement conditional decoder is connected to the output of the embedding layer and the output of the Visual Mamba visual encoder, respectively; the input of the second feature enhancement conditional decoder is connected to the output of the first feature enhancement conditional decoder and the output of the Visual Mamba visual encoder, respectively; and the input of the third feature enhancement conditional decoder is connected to the output of the second feature enhancement conditional decoder and the output of the Visual Mamba visual encoder, respectively.

[0016] Preferably, the output of the Visual Mamba visual encoder is connected to the input of the cross-attention layer in the feature enhancement conditional decoder.

[0017] Preferably, the output of the similarity matrix module in layer T, after dimensional expansion, is fed into the first memory-driven conditional normalization layer, the second memory-driven conditional normalization layer, and the third memory-driven conditional normalization layer in the feature enhancement conditional decoder.

[0018] Preferably, the visual feature V output by the visual feature extractor i After inputting the Visual Mamba visual encoder, a feature reshaping operation is first performed, followed by processing the resulting four-dimensional visual feature map with spatial information. Feature segmentation is performed along the channel dimension to obtain the segmented visual feature map. and segmentation visual feature map Segmenting visual feature maps and segmentation visual feature map Input the first linear layer and the second linear layer respectively;

[0019] The first linear layer is connected sequentially to the first two-dimensional convolutional layer and the first Silu activation layer. The first Silu activation layer is then connected to the first feature flattening layer and the second feature flattening layer. After feature segmentation in the channel dimension, the feature map output by the first feature flattening layer is obtained, resulting in a visual feature map that is flattened row by row. Input gating matrix Output gate matrix Visual feature map that flattens out line by line Input gating matrix Output gate matrix The input is fed into the first state space model, which is connected to the fourth Add layer; the feature map output by the second feature flattening layer is segmented along the channel dimension to obtain a visual feature map that is flattened column by column. Input gating matrix Output gate matrix Visual feature diagram of each column flattened Input gating matrix Output gate matrix Input to the second state space model;

[0020] The Visual Mamba visual encoder also includes a fourth linear layer, whose input is the state transition matrix A, and whose output is connected to the input of the first state space model.

[0021] The Visual Mamba visual encoder also includes a fifth linear layer, whose input is also the state transition matrix A. The fifth linear layer is connected to the second state space model. The feature map output by the second state space model is subjected to a feature reshaping operation. The feature map output by the feature reshaping operation is input to the third feature flattening layer. The third feature flattening layer is connected to the fourth Add layer.

[0022] The second linear layer in the Visual Mamba visual encoder is connected in sequence to the second 2D convolutional layer, the second Silu activation layer, and the fourth feature flattening layer; the output of the fourth feature flattening layer and the output of the fourth Add layer are both connected to the input of the Concat layer, and the output of the Concat layer is connected in sequence to the second normalization layer and the third linear layer; the output of the third linear layer in the Visual Mamba visual encoder is connected to the input of the cross-attention layer in the feature enhancement conditional decoder.

[0023] Preferably, the input to the first-level reporting similarity matrix module is text features. The first token and the similarity relation learning matrix M0 are used in the first layer. The first-layer reporting similarity relation matrix module dynamically learns text features based on the similarity relation learning matrix M0. The first token reports similarity information and outputs a similarity learning matrix M1; the second-layer similarity matrix module takes text features as input. The second token and the similarity learning matrix M1 output by the first-layer report similarity matrix module are used to dynamically learn text features based on the similarity learning matrix M1. The second token reports similarity information, outputting a similarity learning matrix M2; and so on; the input to the T-1 layer's reporting similarity matrix module is text features. The similarity learning matrix M output by the similarity matrix module of the (T-1)th token and the (T-2)th layer report. T-2 The T-1 level reporting similarity matrix module is based on the similarity learning matrix M. T-2 Dynamic learning of text features Report similarity information for the (T-1)th token and output the similarity learning matrix M. T-1 The input to the T-th layer reporting similarity matrix module is text features. The similarity learning matrix M output by the similarity matrix module of the T-th token and the (T-1)-th layer report similarity matrix. T-1 The T-th layer reporting similarity matrix module is based on the similarity learning matrix M. T-1 Dynamic learning of text features Report similarity information for the T-th token and output the similarity learning matrix M. T .

[0024] Preferably, the first-layer reporting similarity matrix module includes a first concat layer and a second concat layer, with the input of the first concat layer connected to the output of the embedding layer; the first concat layer has two inputs, one of which is the text features output by the embedding layer. The first token in the feature The other input is the similarity learning matrix M0; the first token in the text features, i.e., the feature... After dimensional expansion, the data is fed into the second Concat layer. The other input to the second Concat layer is the similarity learning matrix M0.

[0025] The first Concat layer connects sequentially to the Mamba2 module, the feature clipping layer, the fifth Add layer, the first normalization layer, the Tanh activation layer, and the first element-wise multiplication unit; the similarity learning matrix M0 also serves as the input to the fifth Add layer;

[0026] The second Concat layer connects sequentially to the sixth linear layer, the ReLU activation layer, and the seventh linear layer; the feature map output by the seventh linear layer is segmented along the channel dimension to obtain the similarity learning matrix. Importance Score i The importance score S of the similarity relation learning matrix M0 f Similarity learning matrix Importance Score i The importance score S of the similarity relation learning matrix M0 f The dimensions are all the same and are half the channel dimension of the feature map output from the seventh linear layer; similarity learning matrix Importance Score i The importance score S of the similarity relation learning matrix M0 f The first and second Sigmoid activation layers are input respectively. The first Sigmoid activation layer is connected to the first element-wise multiplication unit, and the second Sigmoid activation layer is connected to the second element-wise multiplication unit. The similarity learning matrix M0 is also used as the input of the second element-wise multiplication unit. The outputs of the first and second element-wise multiplication units are both connected to the input of the sixth Add layer. The reported similarity matrix M1 output by the sixth Add layer is the output of the first layer's reported similarity matrix module.

[0027] Preferably, training the radiology report generation network includes the following steps: based on the training set and cross-entropy loss. L ce The radiology report generation network is pre-trained to obtain a pre-trained radiology report generation network; based on the training set and multi-index self-reward reinforcement learning loss... L mrl The pre-trained radiology report generation network was fine-tuned to obtain the radiology report generation network model.

[0028] Preferably, the training set is a training set in the IU-Xray dataset or a training set in the IU-Xray dataset.

[0029] Preferably, based on cross-entropy loss L ce The radiology report generation network is pre-trained using the training set from the IU-Xray dataset. Specifically, the steps include: inputting paired images and their corresponding radiology report samples from the training set of the IU-Xray dataset into the radiology report generation network, performing forward propagation, and calculating the cross-entropy loss of the radiology report generation network. L ce and in cross-entropy loss L ce Guided by the algorithm, backpropagation is performed to update the weight parameters of the radiology report generation network, thus completing one epoch of training. After iterating through 50 epochs of training, the pre-training process of the radiology report generation network is completed, resulting in the pre-trained radiology report generation network.

[0030] Preferably, a multi-index self-reward reinforcement learning loss is used. L mrl The pre-trained radiology report generation network was fine-tuned using the training set from the IU-Xray dataset. Specifically, the following steps were performed: paired images and their corresponding radiology report samples from the training set of the IU-Xray dataset were input into the pre-trained radiology report generation network for forward propagation, and the multi-metric self-reward reinforcement learning loss of the radiology report generation network was calculated. L mrl And in multi-indicator self-reward reinforcement learning loss L mrl Guided by the algorithm, backpropagation is performed to update the weight parameters of the pre-trained radiology report generation network, thus completing one epoch of training. After iterating through 30 epochs, the fine-tuning training process of the pre-trained radiology report generation network is completed, resulting in the radiology report generation network model.

[0031] Compared with the prior art, the beneficial technical effects of this application are as follows:

[0032] Because the Visual Mamba visual encoder in this application can effectively enhance visual features, it can obtain enhanced visual feature maps with lateral correlation, vertical correlation, and two-dimensional spatial information. The report's similarity matrix module dynamically learns text features based on a similarity learning matrix. The token reports similarity information, and the similarity information dynamically learned by the similarity learning matrix can effectively enhance the expression of text features; therefore, this enables the feature enhancement conditional decoder to enhance the visual feature map output by the Visual Mamba visual encoder. Furthermore, when the enhanced feature map output by the report similarity matrix module is decoded, a predicted radiology report that is closer to the real report can be obtained. Tests show that the method described in this application achieves better BLEU-1, BLEU-2, BLEU-3, BLEU-4, METEOR, and ROUGE-L scores, regardless of whether the test set is based on the IU-Xray dataset or the MIMIC-CXR dataset. This indicates that the text (i.e., the radiology report) generated by the method described in this application is indeed closer to the actual radiology report text. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the network structure of the radiology report generation network in this application;

[0034] Figure 2 for Figure 1 A schematic diagram illustrating the connection relationships of the similarity matrix module in the middle T-layer report;

[0035] Figure 3 for Figure 1 A schematic diagram of the network structure of the Visual Mamba visual encoder;

[0036] Figure 4 for Figure 2 A schematic diagram of the network structure of the first-layer report similarity matrix module. Detailed Implementation

[0037] Example 1:

[0038] A radiological report generation method based on enhanced bimodal features includes the following steps:

[0039] S1. Construct a radiology report generation network; where the network structure of the radiology report generation network is as follows: Figure 1 As shown;

[0040] In this application, the radiology report generation network includes a visual feature extractor, which is used to extract visual features from the X-ray image to be predicted. Visual features It contains features of all patches in the X-ray image to be predicted; the output of the visual feature extractor is connected to the input of the Visual Mamba visual encoder; the visual feature extractor in this application has the same structure and function as the Visual Extractor in Figure 2 of the paper "Generating Radiology Reports via Memory-driven Transformer";

[0041] In this application, the radiology report generation network further includes three sequentially connected feature enhancement conditional decoders. The input of the first feature enhancement conditional decoder is connected to the output of the embedding layer and the output of the Visual Mamba encoder, respectively. The input of the second feature enhancement conditional decoder is connected to the output of the first feature enhancement conditional decoder and the output of the Visual Mamba encoder, respectively. The input of the third feature enhancement conditional decoder is connected to the output of the second feature enhancement conditional decoder and the output of the Visual Mamba encoder, respectively. In this application, the output of the Visual Mamba encoder is connected to the cross-attention layers in the first, second, and third feature enhancement conditional decoders, respectively. The value matrix V and key matrix K output by the Visual Mamba encoder are fed into the cross-attention layers in the first, second, and third feature enhancement conditional decoders. Furthermore, in this application, the input of the first feature enhancement conditional decoder includes the output of the embedding layer, the output of the Visual Mamba encoder, and the output of the report similarity matrix module at layer T. The input of the second feature enhancement conditional decoder includes the output of the first feature enhancement conditional decoder, the output of the Visual Mamba encoder, and the output of the Visual Mamba encoder at layer T. The inputs to the third feature enhancement conditional decoder include the outputs of the second feature enhancement conditional decoder, the VisualMamba visual encoder, and the T-layer report similarity matrix module.

[0042] The feature-enhanced conditional decoder used in this application has the same structure and function as the DECODER in Figure 2 of the paper "Generating Radiology Reports via Memory-driven Transformer". For example, the masked multi-head attention layer in the feature-enhanced conditional decoder of this application has the same structure and function as the Masked Multi-Head Attention in Figure 2 of the paper "Generating Radiology Reports via Memory-driven Transformer". The first memory-driven conditional normalization layer, the second memory-driven conditional normalization layer, and the third memory-driven conditional normalization layer in the feature-enhanced conditional decoder of this application have the same structure and function as the three bottom-up Memory-driven Conditional LNs in Figure 2 of the paper "Generating Radiology Reports via Memory-driven Transformer". The cross-attention layer in the feature-enhanced conditional decoder of this application has the same structure and function as the Multi-Head Attention in Figure 2 of the paper "Generating Radiology Reports via Memory-driven Transformer". The feedforward fully connected layer in the feature-enhanced conditional decoder of this application has the same structure and function as the Multi-Head Attention in Figure 2 of the paper "Generating Radiology Reports via Memory-driven Transformer". The Feed Forward structure and function are the same as those in Figure 2 of the paper "Reports via Memory-driven Transformer". In addition, the connection method between the embedding layer and the first feature enhancement conditional decoder in this application is also the same as the connection method between OutputEmbedding and the first DECODER in Figure 2 of the paper "Generating Radiology Reports via Memory-driven Transformer".

[0043] The embedding layer in the radiology report generation network of this application is used to perform embedding mapping operations on the radiology report text to obtain the text features of the radiology report text. , Where T is related to text features The sequence lengths are consistent. Representing text features All tokens;

[0044] The output of the third feature-enhanced conditional decoder is connected to the prediction head, which consists of a linear layer and a softmax layer connected in sequence. In other words, the output of the third feature-enhanced conditional decoder is connected to the linear layer and the softmax layer in the prediction head in sequence. The softmax layer outputs the probability distribution of each character in the dictionary and obtains the character with the highest probability. This character is the predicted character. All predicted characters constitute the radiological report. That is, all predicted characters are arranged in the order of output from the prediction head to form the radiological report.

[0045] The output of the embedding layer is also connected to the T-layer reporting similarity matrix module. That is, in this application, the number of layers in the reporting similarity matrix module is related to the text features. The sequence lengths are consistent;

[0046] In this application, the output of the embedded layer is sequentially connected to the T-layer report similarity matrix module. Specifically, the output of the embedded layer is connected to the first-layer report similarity matrix module; the output of the first-layer report similarity matrix module is sequentially connected to the second-layer report similarity matrix module, ..., the T-layer report similarity matrix module. The connection relationship from the first-layer report similarity matrix module to the T-layer report similarity matrix module is as follows: Figure 2 As shown, the feature map output by the similarity matrix module of layer T is dimensionally expanded. The feature map obtained by dimensional expansion is sent to three feature enhancement conditional decoders. Specifically, they are all sent to the first memory-driven conditional normalization layer, the second memory-driven conditional normalization layer, and the third memory-driven conditional normalization layer of the feature enhancement conditional decoder.

[0047] The dimension expansion operation in this application is the same as the Expand method disclosed in the paper "Generating Radiology Reports via Memory-driven Transformer".

[0048] In this application, the network structure of the Visual Mamba visual encoder is as follows: Figure 3 As shown:

[0049] Visual features V output by the visual feature extractor i After inputting the Visual Mamba visual encoder, a feature reshaping operation is first performed, followed by processing the resulting four-dimensional visual feature map with spatial information. Feature segmentation is performed along the channel dimension to obtain the segmented visual feature map. and segmentation visual feature map Segmenting visual feature maps and segmentation visual feature map The dimensions are the same and they are all four-dimensional visual feature maps. Half the size of the channel dimension; segment the visual feature map and segmentation visual feature map Input the first linear layer and the second linear layer respectively;

[0050] The first linear layer is connected sequentially to the first two-dimensional convolutional layer and the first Silu activation layer. The first Silu activation layer is then connected to the first feature flattening layer and the second feature flattening layer. After feature segmentation in the channel dimension, the feature map output by the first feature flattening layer is obtained, resulting in a visual feature map that is flattened row by row. Input gating matrix Output gate matrix Visual feature map that flattens out line by line Input gating matrix Output gate matrix The input is fed into the first state space model, which is connected to the fourth Add layer; the feature map output by the second feature flattening layer is segmented along the channel dimension to obtain a visual feature map that is flattened column by column. Input gating matrix Output gate matrix Visual feature diagram of each column flattened Input gating matrix Output gate matrix Input to the second state space model; visual feature map in this application and visual feature map The channel dimensions are all four-dimensional visual feature maps. The input gating matrix is ​​half the size of the channel dimension. Output gate matrix Input gating matrix and output gating matrix The channel dimensions are consistent with the channel dimensions of SSM parameters B and SSM parameters C disclosed in the paper "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality".

[0051] The Visual Mamba visual encoder also includes a fourth linear layer, whose input is the state transition matrix A, and whose output is connected to the input of the first state space model. The method for generating the state transition matrix A used in this application is exactly the same as the method for generating SSM parameters A disclosed in the paper "Transformers are SSMs: Generalized Models and EfficientAlgorithms Through Structured State Space Duality". The network structures of the first and second state space models used in this application are the same, and are completely consistent with the network structure of the State Space Models in the paper "Transformers are SSMs: Generalized Models and EfficientAlgorithms Through Structured State Space Duality".

[0052] The Visual Mamba visual encoder also includes a fifth linear layer, whose input is also the state transition matrix A. The fifth linear layer is connected to the second state space model. The feature map output by the second state space model is subjected to a feature reshaping operation. The feature map output by the feature reshaping operation is input to the third feature flattening layer. The third feature flattening layer is connected to the fourth Add layer.

[0053] The second linear layer in the Visual Mamba visual encoder is also connected in sequence to the second two-dimensional convolutional layer, the second Silu activation layer, and the fourth feature flattening layer; the output of the fourth feature flattening layer and the output of the fourth Add layer are both connected to the input of the Concat layer, and the output of the Concat layer is connected in sequence to the second normalization layer and the third linear layer; in this application, the kernel size of the first two-dimensional convolutional layer and the second two-dimensional convolutional layer is 1;

[0054] The output of the third linear layer in the Visual Mamba visual encoder is connected to the input of the cross-attention layer in the feature enhancement conditional decoder; the enhanced visual feature map output by the third linear layer in the Visual Mamba visual encoder of this application... V and K are the value matrix and key matrix output by the Visual Mamba visual encoder, respectively.

[0055] Features of the Visual Mamba visual encoder:

[0056] In this application, the Visual Mamba visual encoder is used to extract visual features based on the output of the visual feature extractor. Visual feature enhancement is performed to obtain enhanced visual feature maps with horizontal correlation, vertical correlation, and two-dimensional spatial information. Specifically:

[0057] The visual features Vi output by the visual feature extractor are input into the Visual Mamba visual encoder, where they are first reshaped to obtain a four-dimensional visual feature map with spatial information. Then, the four-dimensional visual feature map containing spatial information was processed. Perform feature segmentation to obtain segmented visual feature maps. and segmentation visual feature map Segmenting visual feature maps and segmentation visual feature map The dimensions are the same and they are all four-dimensional visual feature maps. Half the size of the channel dimension; segment the visual feature map and segmentation visual feature map Input the first linear layer and the second linear layer respectively;

[0058] The first linear layer segments the visual feature map. Perform a high-dimensional projection operation to obtain a visual feature map with higher dimensions. The first two-dimensional convolutional layer provides higher-dimensional visual feature maps. A two-dimensional convolution operation is performed to obtain a feature map with two-dimensional spatial information. The first Silu activation layer performs a non-linear compression operation on the feature map output from the first two-dimensional convolution layer. The feature map output from the first Silu activation layer is then fed to a first feature flattening layer and a second feature flattening layer. The first feature flattening layer flattens the feature map output from the first Silu activation layer row-by-row to flatten it from four-dimensional to three-dimensional. The second feature flattening layer flattens the feature map output from the first Silu activation layer column-by-column to flatten it from four-dimensional to three-dimensional. Feature segmentation is performed on the feature map output from the first feature flattening layer to obtain the input gating matrix. Output gate matrix and visual feature maps Visual feature map The features in the second feature flattening layer are flattened row by row; feature cutting is performed on the feature map output from the second feature flattening layer to obtain the input gating matrix. Output gate matrix and visual feature maps Visual feature map The characteristic of it is that it is flattened out in columns;

[0059] The state transition matrix A is input to the fourth and fifth linear layers respectively. The fourth linear layer performs a linear transformation operation on the state transition matrix A to obtain the state transition matrix. In the fifth linear layer, a linear transformation operation is performed on the state transition matrix A to obtain the state transition matrix. In this application, the parameters of the fourth and fifth linear layers are independent of each other. Therefore, the state transition matrix A of this application can be transformed into a differentiated state transition matrix by performing two independent linear transformation operations through the fourth and fifth linear layers. and Differentiated state transition matrix and The inputs are respectively fed into the first state space model and the second state space model;

[0060] The first state-space model uses the state transition matrix. Input gating matrix Output gate matrix and visual feature maps Selective state-space modeling is performed to obtain feature maps with lateral correlations; the second state-space model is used to transform the state transition matrix. Input gating matrix Output gate matrix and visual feature maps Selective state-space modeling is performed to obtain feature maps with vertical correlations;

[0061] The feature map with vertical correlation output by the second state space model is reshaped into a four-dimensional feature map. The third feature flattening layer flattens the four-dimensional feature map row by row. The feature map output by the third feature flattening layer has the same feature distribution as the feature map output by the first state space model. The fourth Add layer adds the features of the feature map output by the third feature flattening layer and the feature map with horizontal correlation output by the first state space model to obtain a feature map with both horizontal and vertical correlation.

[0062] The second linear layer segments the visual feature map. A linear transformation operation is performed; the second 2D convolutional layer convolves the feature map output by the second linear layer to output a feature map with 2D spatial information; the second Silu activation layer performs a non-linear compression operation on the feature map output by the second 2D convolutional layer; the fourth feature flattening layer flattens the feature map output by the second Silu activation layer row by row to flatten the feature map output by the second Silu activation layer from four-dimensional to three-dimensional, so that the dimension of the feature map output by the fourth feature flattening layer is consistent with that of the feature map output by the fourth Add layer.

[0063] The Concat layer concatenates the feature maps output by the fourth Add layer and the fourth feature flattening layer to obtain a feature map with horizontal correlation, vertical correlation and two-dimensional spatial information.

[0064] In this application, the first state space model and the second state space model lose some two-dimensional spatial information during the selective state space modeling process. The Concat layer concatenates the feature maps output by the fourth Add layer and the fourth feature flattening layer, thus effectively compensating for the lost two-dimensional spatial information using the feature map output by the fourth feature flattening layer, thereby obtaining a feature map with horizontal correlation, vertical correlation, and two-dimensional spatial information. The second normalization layer normalizes the feature map output by the Concat layer, and the third linear layer performs a linear transformation on the feature map output by the second normalization layer, ultimately obtaining an enhanced visual feature map with horizontal correlation, vertical correlation, and two-dimensional spatial information. The enhanced visual feature map obtained in this application Because it contains horizontal and vertical relationships as well as two-dimensional spatial information, the visual features are significantly enhanced.

[0065] In this application, the similarity matrix module dynamically learns text features based on the similarity learning matrix. The report similarity information of the tokens is used, and the report similarity information dynamically learned by the similarity learning matrix can effectively enhance the expression of text features; specifically, the input of the first-layer report similarity matrix module is the text features. Features ( Text features The first token in the text) and the similarity relation learning matrix M0, the first-level reporting similarity relation matrix module dynamically learns text features based on the similarity relation learning matrix M0. The first token (i.e., the feature) The first layer reports similarity relationship information, and the second layer reports similarity relationship matrix M1; the input to the third layer is text features. In ( Text features The second token in the text) and the similarity learning matrix M1 output by the first-layer reporting similarity matrix module, and the second-layer reporting similarity matrix module dynamically learn text features based on the similarity learning matrix M1. The second token (i.e., the feature) The report similarity information is output as a similarity learning matrix M2; and so on; the input of the T-1 layer's report similarity matrix module is text features. In ( Text features The similarity learning matrix M output by the similarity matrix module of layer T-2 (the (T-1)th token) and layer T-2) is obtained. T-2 The T-1 level reporting similarity matrix module is based on the similarity learning matrix M. T-2 Dynamic learning of text features The (T-1)th token (i.e., the feature) The report provides similarity information and outputs a similarity learning matrix M. T-1 The input to the T-th layer reporting similarity matrix module is text features. In ( Text features The similarity learning matrix M output by the similarity matrix module of the (T-1th)th token and the (T-1th)th layer report similarity matrix module. T-1 The T-th layer reporting similarity matrix module is based on the similarity learning matrix M. T-1 Dynamic learning of text features The T-th token (i.e., the feature) The report provides similarity information and outputs a similarity learning matrix M. T ;

[0066] In this application, the generation method of the similarity learning matrix M0 is consistent with the generation method of the Relational Memory Matrix disclosed in the paper "Generating RadiologyReports via Memory-driven Transformer"; in this application, the purpose of setting the similarity learning matrix M0 is to use the similarity learning matrix M0 to learn text features. Similarity information in;

[0067] In this application, taking the first-layer report similarity matrix module as an example, the structure and function of the report similarity matrix module are introduced, and the network structure of the first-layer report similarity matrix module is as follows: Figure 4 As shown:

[0068] The first-layer similarity matrix module includes a first concat layer and a second concat layer. The input of the first concat layer is connected to the output of the embedding layer. The first concat layer has two inputs, one of which is the text features output by the embedding layer. The first token (i.e., feature) in The other input is the similarity learning matrix M0; the first token (i.e., feature) in the text features. After dimensional expansion, the data is fed into the second Concat layer. The other input to the second Concat layer is the similarity learning matrix M0.

[0069] The first Concat layer sequentially connects the Mamba2 module, the feature clipping layer, the fifth Add layer, the first normalization layer, the Tanh activation layer, and the first element-wise multiplication unit; the similarity learning matrix M0 also serves as the input to the fifth Add layer; the Mamba2 module used in this application is completely consistent with the network structure of Mamba2 disclosed in the paper "Transformers are SSMs: Generalized Models and EfficientAlgorithms Through Structured State Space Duality";

[0070] The second Concat layer connects sequentially to the sixth linear layer, the ReLU activation layer, and the seventh linear layer; the feature map output by the seventh linear layer is segmented along the channel dimension to obtain the similarity learning matrix. Importance Score i The importance score S of the similarity relation learning matrix M0 f Similarity learning matrix Importance Score i The importance score S of the similarity relation learning matrix M0 f The dimensions are all the same and are half the channel dimension of the feature map output from the seventh linear layer; similarity learning matrix Importance Score i The importance score S of the similarity relation learning matrix M0 f The first and second Sigmoid activation layers are input respectively. The first Sigmoid activation layer is connected to the first element-wise multiplication unit, and the second Sigmoid activation layer is connected to the second element-wise multiplication unit. The similarity learning matrix M0 is also used as the input of the second element-wise multiplication unit. The outputs of the first and second element-wise multiplication units are both connected to the input of the sixth Add layer. The reported similarity matrix M1 output by the sixth Add layer is the output of the first layer's reported similarity matrix module.

[0071] The function of the first-level report similarity matrix module is essentially to dynamically learn features based on the similarity learning matrix M0 using the first-level report similarity matrix module. The report provides information on similar relationships, specifically:

[0072] The first layer reports the similarity matrix module, where the first Concat layer learns the similarity matrix M0 and the text features output by the embedding layer. The first token in Feature concatenation is performed; the Mamba2 module selectively models the state space of the feature map output from the first Concat layer, realizing the similarity relationship learning matrix M0 and features. Information interaction yields learned features The matrix reporting similarity information; the feature pruning layer learns features from the output of the Mamba2 module. The matrix of similarity information in the report is used for feature cropping, and the cropped features are removed. The similarity learning matrix is ​​obtained. The fifth Add layer learns the similarity matrix from the output of the feature cropping layer. The similarity learning matrix M0 is used to add features; the first normalization layer normalizes the feature map output by the fifth Add layer; the Tanh activation layer performs non-compression operation on the feature map output by the first normalization layer.

[0073] Text features output by the embedding layer The first token (i.e., feature) in Features are obtained after dimensional expansion. ,feature The data is then fed to the second Concat layer, where the features are processed. And similarity relationship learning matrix M0 is used for feature concatenation;

[0074] The sixth linear layer performs a linear transformation operation on the feature map output by the second Concat layer; the ReLU activation layer performs a non-linear compression operation on the feature map output by the sixth linear layer; the seventh linear layer performs a linear transformation operation on the feature map output by the ReLU activation layer; in this application, the sequential connection of the sixth linear layer, the ReLU activation layer, and the seventh linear layer enables this application to obtain a feature map with rich gating information based on the feature map output by the second Concat layer.

[0075] The feature map output from the seventh linear layer is used for feature segmentation to obtain the similarity learning matrix. Importance score Importance scores of similarity relation learning matrix M0 The first Sigmoid layer learns the similarity matrix. Importance score Perform nonlinear compression to assign importance scores Compress the values ​​between 0 and 1 to obtain an importance score. The importance score of the second Sigmoid layer for the similarity relation learning matrix M0 Perform nonlinear compression to assign importance scores Compress the values ​​to between 0 and 1 to obtain an importance score. ;

[0076] The first element-wise multiplication unit performs an element-wise multiplication operation on the feature map output from the Tanh activation layer and the feature map output from the first Sigmoid layer. The first element-wise multiplication unit is based on importance scores. Dynamically preserved features The report contains information on similar relationships;

[0077] The second element-wise multiplication unit performs an element-wise multiplication operation on the similarity learning matrix M0 and the feature map output by the second Sigmoid layer. The second element-wise multiplication unit is based on the importance score. The similarity learning matrix M0 is dynamically preserved; the sixth Add layer performs feature addition on the feature maps output by the first element-wise multiplication unit and the first element-wise multiplication unit, dynamically fusing the features. The similarity relationship information and the similarity relationship learning matrix M0 are reported to obtain the similarity relationship learning matrix M1. The first token reported by the similarity relationship information dynamically learned by the similarity relationship learning matrix M1 can effectively enhance the expression of text features.

[0078] In this application, the linear layer connected to the output of the third feature enhancement conditional decoder is used to perform a linear transformation on the feature map output by the third feature enhancement conditional decoder, while the Softmax layer connected to the linear layer is used to perform a softmax operation on the feature map output by the linear layer to obtain the probability distribution of each character in the dictionary, and to obtain the character with the highest probability, which is the predicted character; all the predicted characters constitute the radiological report. In this application, the dictionary is composed of characters from the real reports in the training set.

[0079] Because the Visual Mamba visual encoder in this application can effectively enhance visual features, it can obtain enhanced visual feature maps with lateral correlation, vertical correlation, and two-dimensional spatial information. The report's similarity matrix module dynamically learns text features based on a similarity learning matrix. The token reports similarity information, and the similarity information dynamically learned by the similarity learning matrix can effectively enhance the expression of text features; therefore, this enables the feature enhancement conditional decoder to enhance the visual feature map output by the Visual Mamba visual encoder. When the enhanced feature map of the text features output by the report similarity matrix module is decoded, a predicted radiological report that is closer to the real report can be obtained. The real report refers to the real report input to the embedding layer. The real report is the real radiological report, which serves as the reference text.

[0080] S2, based on the training set in the IU-Xray dataset, cross-entropy loss L ce Multi-indicator self-reward reinforcement learning loss L mrl The radiology report generation network was pre-trained and then fine-tuned to obtain the radiology report generation network model; specifically...

[0081] Step S2 includes the following steps: based on the training set and cross-entropy loss in the IU-Xray dataset. L ce The radiology report generation network is pre-trained to obtain a pre-trained radiology report generation network; then, it is based on the training set in the IU-Xray dataset and a multi-metric self-reward reinforcement learning loss. L mrl The pre-trained radiology report generation network was fine-tuned to obtain the radiology report generation network model; among which,

[0082] Training set and cross-entropy loss based on the IU-Xray dataset L ce Pre-training the radiology report generation network includes the following steps: inputting paired images and their corresponding radiology report samples from the training set of the IU-Xray dataset into the radiology report generation network, performing forward propagation, and calculating the cross-entropy loss of the radiology report generation network. L ce and in cross-entropy loss L ce Guided by the backpropagation, the weight parameters of the radiology report generation network are updated, which completes one epoch of training. After iterating for 50 epochs, the pre-training process of the radiology report generation network is completed, and the pre-trained radiology report generation network is obtained.

[0083] Training set based on IU-Xray dataset and multi-metric self-reward reinforcement learning loss. L mrlFine-tuning the pre-trained radiology report generation network involves the following steps: inputting paired images and their corresponding radiology report samples from the training set of the IU-Xray dataset into the pre-trained radiology report generation network, performing forward propagation, and calculating the multi-metric self-reward reinforcement learning loss of the radiology report generation network. L mrl And in multi-indicator self-reward reinforcement learning loss L mrl Guided by the algorithm, backpropagation is performed to update the weight parameters of the pre-trained radiology report generation network, thus completing one epoch of training. After iterating through 30 epochs, the fine-tuning training process of the pre-trained radiology report generation network is completed, resulting in the radiology report generation network model.

[0084] In this first embodiment, both the pre-training and fine-tuning training processes use the Adam optimizer to optimize the loss gradient. During pre-training, the learning rate of the visual feature extractor in the radiology report generation network is set to 1e-4, and the learning rate of the trainable parts of the radiology report generation network other than the visual feature extractor is set to 5e-5. During fine-tuning, the learning rate of the visual feature extractor in the pre-trained radiology report generation network is set to 1e-4, and the learning rate of the trainable parts of the pre-trained radiology report generation network other than the visual feature extractor is set to 5e-5. The batch size during pre-training is set to 16, and the batch size during fine-tuning is set to 16.

[0085] In this application, the cross-entropy loss L ce The calculation method is consistent with the calculation method of Cross-Entropy loss disclosed in the paper "TSGET: Two-Stage Global Enhanced Transformer for Automatic Radiology Report Generation";

[0086] In this application, the multi-index self-reward reinforcement learning loss L mrl The design aims to guide the deep alignment of enhanced image and text features to generate more accurate radiological report text; in this application, a multi-index self-reward reinforcement learning loss is used. L mrl The calculation method is shown in equation (1);

[0087] (1)

[0088] In equation (1), N represents the sum of the number of characters in a batch of radiology report samples, where a batch of radiology report samples refers to all radiology report samples of a batch size input to the pre-trained radiology report generation network; B represents the batch size; and T represents the character length of a single radiology report sample input to the pre-trained radiology report generation network. This represents the multi-index reward for the i-th radiology report sample at the t-th character. This represents the predicted probability distribution of the t-th character of the i-th sample in the predicted radiology report obtained during the fine-tuning training process;

[0089] In this application, the multi-index self-reward reinforcement learning loss L mrl The calculation of the multi-indicator reward r depends on multiple NLG indicators, which specifically refer to the BLEU-4 indicator, the METEOR indicator, and the ROUGE-L indicator.

[0090] The calculation process for the multi-index reward r is as follows: All predicted characters obtained by randomly decoding the probability distribution of each character generated during fine-tuning training in the dictionary are used as a sampling report sequence. S sam The greedy report sequence consists of all predicted characters obtained by greedily decoding the probability distribution of each character generated during fine-tuning training in the dictionary. S gre ; will be combined with the sampling report sequence S sam And Greed Report Sequence S gre The corresponding radiological report samples input into the pre-trained radiological report generation network serve as the reference report sequence. S ref ;

[0091] Then, the sampling report sequences were calculated separately. S sam and reference report sequence S ref The sampling report between the BLEU4 index scores B sam METEOR score in sampling report M sam and the ROUGE-L index score in the sampling report R sam Then, the BLEU4 index score in the sampling report will be calculated. B sam METEOR score in sampling report M sam and the ROUGE-L index score in the sampling reportR sam Normalization is performed separately for each indicator to eliminate differences in indicator scales and prevent indicators with large numerical ranges from dominating the calculation of multi-indicator rewards r; then, the normalized sampled reports are used to calculate the BLEU4 indicator scores. B sam METEOR score in sampling report M sam and the ROUGE-L index score in the sampling report R sam The NLG index comprehensive score in the sampling report is obtained by weighting and summing the results in a 4:3:3 ratio. C sam ;

[0092] Then, the greedy report sequence is calculated separately. S gre and reference report sequence S ref Greed Report BLEU4 Index Score B gre Greed Report METEOR score M gre And the ROUGE-L index score in the Greed Report R gre The greed report BLEU4 indicator score B gre Greed Report METEOR score M gre And the ROUGE-L index score in the Greed Report R gre Normalization processing is performed to normalize the BLEU4 score of the greed report. B gre Greed Report METEOR score M gre And the ROUGE-L index score in the Greed Report R gre The Greed Report NLG index is obtained by weighting and summing the results in a 4:3:3 ratio. C gre ;

[0093] Finally, calculate the overall score of the NLG index in the sampling report. C sam Combined score with Greed Report NLG indicator C gre The difference will be used to obtain multi-indicator rewards. ;

[0094] In this application, the multi-index self-reward reinforcement learning loss L mrlThe calculation method for the multi-indicator reward r aims to achieve the following objectives: if the sampled report sequence S sam Superior to Greedy Reporting Sequence S gre If the calculated result of the multi-indicator reward r is greater than 0, then the text generation method of the current random sampling search strategy is encouraged; if the greedy reporting sequence... S gre Superior to sampling report sequence S sam If the calculated result of the multi-index reward r is less than 0, then the text generation method of the current random sampling search strategy will be penalized.

[0095] In this application, the multi-index self-reward reinforcement learning loss L mrl The selection of multiple NLG indicators (i.e., BLEU-4, METEOR, and ROUGE-L indicators) on which the multi-indicator reward r depends is targeted;

[0096] Among them, the BLEU-4 metric focuses on the degree of matching of quadruplets, and the multi-metric self-reward reinforcement learning loss is calculated with the participation of the BLEU-4 metric. L mrl This can effectively constrain the radiology report generation network to generate more accurate quadruplets;

[0097] The METEOR metric comprehensively considers precision and recall, and is a multi-metric self-rewarding reinforcement learning loss calculated with the METEOR metric involved. L mrl It can effectively constrain the radiology report generation network to generate radiology reports that are closer to the radiology report samples in terms of semantic expression flexibility;

[0098] The ROUGE-L metric measures the structural similarity between generated and reference text. Multi-metric self-reward reinforcement learning loss is calculated using the ROUGE-L metric. L mrl It can effectively constrain the radiology report generation network to generate radiology reports that are closer to the radiology report samples in terms of overall structure and coherence at the sentence level;

[0099] S3. Input the X-ray image to be predicted into the radiology report generation network model obtained in step S2, and propagate it forward once to obtain the predicted radiology report.

[0100] Example 2:

[0101] The difference between Example 2 and Example 1 is that in step S2, the training set and cross-entropy loss in the MIMIC-CXR dataset are used. Lce Multi-indicator self-reward reinforcement learning loss L mrl The radiology report generation network was pre-trained and then fine-tuned to obtain the radiology report generation network model; specifically...

[0102] Step S2 includes the following steps: based on the training set and cross-entropy loss in the MIMIC-CXR dataset. L ce The radiology report generation network is pre-trained to obtain a pre-trained radiology report generation network; then, it is based on the training set in the MIMIC-CXR dataset and a multi-metric self-reward reinforcement learning loss. L mrl The pre-trained radiology report generation network was fine-tuned to obtain the radiology report generation network model; among which,

[0103] Training set and cross-entropy loss based on the MIMIC-CXR dataset L ce Pre-training the radiology report generation network includes the following steps: inputting a single image from the training set of the MIMIC-CXR dataset and its corresponding radiology report sample into the radiology report generation network, performing forward propagation, and calculating the cross-entropy loss of the radiology report generation network. L ce and in cross-entropy loss L ce Guided by the backpropagation, the weight parameters of the radiology report generation network are updated, which completes one epoch of training. After iterating for 30 epochs, the pre-training process of the radiology report generation network is completed, and the pre-trained radiology report generation network is obtained.

[0104] Training set based on MIMIC-CXR dataset and multi-metric self-reward reinforcement learning loss. L mrl Fine-tuning the pre-trained radiology report generation network involves the following steps: inputting a single image from the training set of the MIMIC-CXR dataset and its corresponding radiology report sample into the pre-trained radiology report generation network, performing forward propagation, and calculating the multi-metric self-reward reinforcement learning loss of the radiology report generation network. L mrl And in multi-indicator self-reward reinforcement learning loss L mrlGuided by the algorithm, backpropagation is performed to update the weight parameters of the pre-trained radiology report generation network, thus completing one epoch of training. After iterating for 15 epochs, the fine-tuning training process of the pre-trained radiology report generation network is completed, resulting in the radiology report generation network model.

[0105] In this second embodiment, both the pre-training and fine-tuning training processes use the Adam optimizer to optimize the loss gradient. During pre-training, the learning rate of the visual feature extractor in the radiology report generation network is set to 1e-4, and the learning rate of the trainable parts of the radiology report generation network other than the visual feature extractor is set to 5e-5. During fine-tuning, the learning rate of the visual feature extractor in the pre-trained radiology report generation network is set to 1e-4, and the learning rate of the trainable parts of the pre-trained radiology report generation network other than the visual feature extractor is set to 5e-5. The batch size during pre-training is set to 16, and the batch size during fine-tuning is set to 8.

[0106] Test 1:

[0107] To verify the superior performance of the radiology report generation method described in this invention compared to other radiology report generation methods, this application tested the radiology report generation network model obtained in Example 1 using the test set of the IU-Xray dataset. The test results were compared with those obtained by the R2Gen method based on the IU-Xray dataset test set (from "Generating Radiology Reports via Memory-driven Transformer"), the CMN method based on the IU-Xray dataset test set (from "Cross-modal Memory Networks for Radiology Report Generation"), the CA method (from "Contrastive attention for automaticchest x-ray report generation"), the RAMT method based on the IU-Xray dataset test set (from "Semi-supervised medical report generation via graph-guided hybridfeature consistency"), the CMM+RL method based on the IU-Xray dataset test set (from "Reinforced Cross-modal Alignment for Radiology Report Generation"), and the STD method based on the IU-Xray dataset test set. The test results obtained from the test set in the dataset (from "Abnormal-region-aware multi-modal feature fusion for medical report generation") were compared, and the test results are shown in Table 1.

[0108] Table 1. Test results of different methods on the test set of the IU-Xray dataset.

[0109]

[0110] Test 2:

[0111] To verify the superior performance of the radiology report generation method described in this invention compared to other radiology report generation methods, this application tested the radiology report generation network model obtained in Example 2 using the test set of the MIMIC-CXR dataset. The test results were compared with those obtained by the R2Gen method based on the MIMIC-CXR dataset test set (from "Generating Radiology Reports via Memory-driven Transformer"), the CMN method based on the MIMIC-CXR dataset test set (from "Cross-modal MemoryNetworks for Radiology Report Generation"), the CA method based on the MIMIC-CXR dataset test set (from "Contrastive attention for automatic chest x-ray report generation"), the RAMT method based on the MIMIC-CXR dataset test set (from "Semi-supervised medical report generation via graph-guided hybrid featureconsistency"), and the CMM+RL method based on the MIMIC-CXR dataset test set (from "Reinforced Cross-modal Alignment for Radiology Report"). The test results of the generation method and the STD method were compared based on the test set of the MIMIC-CXR dataset (from "Abnormal-region-aware multi-modal feature fusion for medical report generation"). The test results are shown in Table 2.

[0112] Table 2. Test results of different methods on the test set of the MIMIC-CXR dataset.

[0113]

[0114] In Tables 1 and 2, BLEU-1, BLEU-2, BLEU-3, BLEU-4, METEOR, and ROUGE-L are commonly used evaluation metrics for text generation, and the OURS method refers to the radiology report generation method proposed in this application.

[0115] In Tables 1 and 2, the BLEU metric measures the degree of word and phrase matching between the model-generated text and the reference text; a higher value indicates stronger consistency in words and phrases. BLEU-1 to BLEU-4 represent the matching of single characters, bigrams, triples, and quadruples, respectively. The METEOR metric considers both precision and recall; a higher value indicates greater accuracy and flexibility in semantic expression of the generated text. The ROUGE-L metric measures the structural similarity between the generated text and the reference text, calculated based on the longest common subsequence; a higher value indicates that the generated text is closer to the reference text in terms of overall sentence structure and coherence. In summary, these metrics evaluate the quality of the generated text from lexical, semantic, and structural perspectives, providing a multi-faceted reference for assessing the model's generation performance. In Tables 1 and 2, the Ours method represents the radiology report generation method described in this application. In this application, an English word is considered as one character.

[0116] Compared to other existing radiology report generation methods disclosed in this application, the existing STD method demonstrates superior performance regardless of whether the test set is based on the IU-Xray dataset or the MIMIC-CXR dataset. Therefore, this application focuses on comparing the test results of the STD method with those of the radiology report generation method described in this application, as follows:

[0117] As shown in Table 1, when conducting tests based on the test set in the IU-Xray dataset:

[0118] The BLEU-1 index obtained by the radiology report generation method described in this application is 4.77% higher than that obtained by the STD method. This indicates that the radiology report generation method described in this application is more accurate in matching individual characters and can generate more content that is consistent with individual characters of the reference text, thereby improving the generation quality of the text (i.e., radiology report) generated by the method described in this application at the individual character level.

[0119] The BLEU-2 index obtained by the radiology report generation method described in this application is 10.43% higher than that obtained by the STD method. This indicates that the radiology report generation method described in this application is more accurate in matching binary pairs, which also means that the text (i.e., radiology report) generated by the method described in this application has stronger expressive power at the phrase level.

[0120] The BLEU-3 index obtained by the radiology report generation method described in this application is 13.56% higher than that obtained by the STD method. This indicates that the radiology report generation method described in this application is more accurate in matching triples, which also means that the text (i.e., radiology report) generated by the method described in this application has higher phrase matching accuracy.

[0121] The BLEU-4 index obtained by the radiology report generation method described in this application is 13.89% higher than that obtained by the STD method. This indicates that the method described in this application is more accurate in matching quadruplets, which also means that the text (i.e., radiology report) generated by the method described in this application is better in terms of higher-level phrase collocation and semantic consistency.

[0122] The METEOR index obtained by the radiology report generation method described in this application is 5.58% higher than that obtained by the STD method. This indicates that the text (i.e., radiology report) generated by the method described in this application is better in terms of the accuracy and flexibility of semantic expression.

[0123] The ROUGE-L index obtained by the radiology report generation method described in this application is 4.59% higher than that obtained by the STD method. This indicates that the text (i.e., radiology report) generated by the method described in this application is closer to the radiology report sample in terms of overall structure and coherence at the sentence level.

[0124] In summary, based on the test results of BLEU-1, BLEU-2, BLEU-3, BLEU-4, METEOR, and ROUGE-L indicators obtained from the test set in the IU-Xray dataset, the text generated by the method described in this application (i.e., the radiology report) is closer to the radiology report text.

[0125] The MIMIC-CXR dataset used in this application is a large dataset, almost 70 times the size of the IU-Xray dataset, with significantly increased data complexity and label noise compared to the IU-Xray dataset. As shown in Table 2, when this application conducts tests based on the test set of the MIMIC-CXR dataset:

[0126] The BLEU-1 index obtained by the radiology report generation method described in this application is 4.43% higher than that obtained by the STD method. This indicates that when tested on the test set of the MIMIC-CXR dataset (a large dataset), the radiology report generation method described in this application is more accurate in matching individual characters and can generate more content that is consistent with individual characters of the reference text, thereby improving the generation quality of the text (i.e., radiology report) generated by the method described in this application at the individual character level.

[0127] The BLEU-2 index obtained by the radiology report generation method described in this application is 6.81% higher than that obtained by the STD method. This indicates that when tested on the test set of the MIMIC-CXR dataset (a large dataset), the radiology report generation method described in this application is more accurate in matching tuples. This also means that the text (i.e., radiology report) generated by the method described in this application has stronger expressive power at the phrase level.

[0128] The BLEU-3 index obtained by the radiology report generation method described in this application is 10.90% higher than that obtained by the STD method. This indicates that when tested on the test set of the MIMIC-CXR dataset (a large dataset), the radiology report generation method described in this application is more accurate in matching triples, which means that the text (i.e., radiology report) generated by the method described in this application has higher phrase matching accuracy.

[0129] The BLEU-4 index obtained by the radiology report generation method described in this application is 9.91% higher than that obtained by the STD method. This indicates that when tested on the test set of the MIMIC-CXR dataset (a large dataset), the method described in this application is more accurate in matching quadruplets. This also means that the text (i.e., radiology report) generated by the method described in this application is better in terms of higher-level phrase collocation and semantic consistency.

[0130] The radiology report generation method described in this application achieves a 2.63% improvement in the METEOR index compared to the STD method. This indicates that when tested on the test set of the MIMIC-CXR dataset (a large dataset), the text (i.e., radiology report) generated by the method described in this application is superior in terms of the accuracy and flexibility of semantic expression.

[0131] The ROUGE-L index obtained by the radiology report generation method described in this application is 5.49% higher than that obtained by the STD method. This indicates that when tested on the test set of the MIMIC-CXR dataset (a large dataset), the text (i.e., radiology report) generated by the method described in this application is closer to the radiology report sample in terms of overall structure and coherence at the sentence level.

[0132] In summary, based on the test results of BLEU-1, BLEU-2, BLEU-3, BLEU-4, METEOR, and ROUGE-L indicators obtained from the test set in the MIMIC-CXR dataset, the text generated by the method described in this application (i.e., the radiology report) is closer to the radiology report text.

Claims

1. A method for generating radiological reports based on enhanced bimodal features, characterized in that: Includes the following steps: The X-ray image to be predicted is input into the radiology report generation network model and propagated forward once to obtain the predicted radiology report. The model is obtained by training a radiology report generation network; the network includes: A visual feature extractor is used to extract visual features V from the X-ray image to be predicted. i ; The Visual Mamba visual encoder is used to encode visual features V. i Enhancement: Obtaining enhanced visual feature maps with horizontal and vertical correlations and two-dimensional spatial information. ; Embedding layer used to extract textual features from radiology report text ; The report's similarity matrix module dynamically learns text features based on the similarity learning matrix. The report uses similarity information of the token to enhance text features; the module has T layers, which are connected sequentially. The result obtained by dimensional expansion of the output of the module in layer T and the enhanced visual feature map The text features are input into three decoders respectively. The first decoder is input; the three decoders are connected in sequence; the output of the third decoder is passed through the prediction head to obtain the predicted characters, and all the characters constitute the predicted radiological report.

2. The radiological report generation method based on enhanced bimodal features according to claim 1, characterized in that: The input of the first decoder is connected to the output of the embedding layer and the Visual Mamba visual encoder, respectively; the input of the second decoder is connected to the output of the first decoder and the Visual Mamba visual encoder, respectively; the input of the third decoder is connected to the output of the second decoder and the Visual Mamba visual encoder, respectively.

3. The radiological report generation method based on enhanced bimodal features according to claim 1, characterized in that: The results obtained from dimensionality expansion are fed into the first memory-driven conditional normalization layer, the second memory-driven conditional normalization layer, and the third memory-driven conditional normalization layer in the feature enhancement conditional decoder.

4. The radiological report generation method based on enhanced bimodal features according to claim 1, characterized in that: Visual feature V i After inputting the Visual Mamba visual encoder, feature reshaping is first performed to obtain a four-dimensional visual feature map. Then, the feature map Perform feature segmentation to obtain segmented visual feature maps. and Segmenting visual feature maps and Input the first linear layer and the second linear layer respectively; The first linear layer is connected to the first two-dimensional convolutional layer and the first Silu activation layer in sequence. The first Silu activation layer is connected to the first feature flattening layer and the second feature flattening layer respectively. After feature segmentation, the feature map output from the first feature flattening layer is used to obtain a visual feature map that is flattened row by row. Input gating matrix Output gate matrix The feature map The matrix The matrix The input is fed into the first state space model, which is connected to the fourth Add layer; the feature map output from the second feature flattening layer is segmented to obtain a visual feature map that is flattened column by column. Input gating matrix Output gate matrix The feature map The matrix The matrix Input to the second state space model; The encoder also includes a fourth linear layer and a fifth linear layer. The inputs of the fourth and fifth linear layers are both state transition matrices A. The fourth linear layer is connected to the first state space model. The fifth linear layer connects to the second state space model. The output of the second state space model is reshaped and then input to the third feature flattening layer. The third feature flattening layer connects to the fourth Add layer. The second linear layer is also connected in sequence to the second two-dimensional convolutional layer, the second Silu activation layer, and the fourth feature flattening layer; The fourth feature flattening layer and the fourth Add layer are both connected to the Concat layer, which in turn connects to the second normalization layer and the third linear layer. The third linear layer is connected to the cross-attention layer in the decoder.

5. The radiological report generation method based on enhanced bimodal features according to claim 1, characterized in that: The first-level report similarity matrix module dynamically learns text features based on the similarity learning matrix M0. The first token reports similarity information and outputs a similarity learning matrix M1; the second-layer module dynamically learns text features based on the similarity learning matrix M1. The second token reports similarity information, outputting a similarity learning matrix M2; and so on; the T-1 level reporting similarity matrix module is based on the similarity learning matrix M2. T-2 Dynamic learning of text features Report similarity information for the (T-1)th token and output the similarity learning matrix M. T-1 The T-level reporting similarity matrix module is based on the similarity learning matrix M. T-1 Dynamic learning of text features Report similarity information for the T-th token and output the similarity learning matrix M. T .

6. The radiological report generation method based on enhanced bimodal features according to claim 5, characterized in that: The first-layer similarity matrix module includes a first concat layer and a second concat layer. The input of the first concat layer is connected to the output of the embedding layer. The input of the first concat layer includes text features. The first token in the text features is fed into the second Concat layer after dimensional expansion, and the other input is the similarity learning matrix M0. The first token in the text features is fed into the second Concat layer after dimensional expansion, and the similarity learning matrix M0 is input into the second Concat layer. The first Concat layer connects sequentially to the Mamba2 module, the feature clipping layer, the fifth Add layer, the first normalization layer, the Tanh activation layer, and the first element-wise multiplication unit; The similarity learning matrix M0 is input into the fifth Add layer; The second Concat layer connects sequentially to the sixth linear layer, the ReLU activation layer, and the seventh linear layer; the output of the seventh linear layer is used for feature segmentation to obtain the similarity learning matrix. Importance Score i The importance score S of the similarity relation learning matrix M0 f The score S i and the score S f The first and second Sigmoid activation layers are input respectively. The first Sigmoid activation layer is connected to the first element-wise multiplication unit, and the second Sigmoid activation layer is connected to the second element-wise multiplication unit. The similarity learning matrix M0 is input to the second element-wise multiplication unit. Both the first and second element-wise multiplication units are connected to the sixth Add layer. The sixth Add layer outputs the similarity matrix M1.

7. The radiological report generation method based on enhanced bimodal features according to claim 1, characterized in that: Training the radiology report generation network includes the following steps: pre-training the radiology report generation network based on the training set and cross-entropy loss to obtain the pre-trained radiology report generation network; fine-tuning the pre-trained radiology report generation network based on the training set and multi-index self-reward reinforcement learning loss to obtain the radiology report generation network model.

8. The radiological report generation method based on enhanced bimodal features according to claim 7, characterized in that: The training set is either the training set in the IU-Xray dataset or the training set in the IU-Xray dataset.

9. The radiological report generation method based on enhanced bimodal features according to claim 8, characterized in that: The radiology report generation network is pre-trained based on cross-entropy loss and the training set in the IU-Xray dataset. The pre-training process includes the following steps: inputting paired images and their corresponding radiology report samples from the training set of the IU-Xray dataset into the radiology report generation network, performing forward propagation to calculate the cross-entropy loss of the radiology report generation network, performing backpropagation guided by the cross-entropy loss to update the weight parameters of the radiology report generation network, thus completing one epoch of training. After iterating for 50 epochs, the pre-trained radiology report generation network is obtained.

10. A method for generating radiological reports based on enhanced bimodal features according to claim 8, characterized in that: Fine-tuning the pre-trained radiology report generation network based on multi-index self-reward reinforcement learning loss and the training set in the IU-Xray dataset includes the following steps: inputting paired images and their corresponding radiology report samples from the training set of the IU-Xray dataset into the pre-trained radiology report generation network, performing forward propagation to calculate the multi-index self-reward reinforcement learning loss of the radiology report generation network, and performing backpropagation under the guidance of the multi-index self-reward reinforcement learning loss to update the weight parameters of the pre-trained radiology report generation network, thus completing one epoch of training. After iterating for 30 epochs, the radiology report generation network model is obtained.

Images