Radiology report generation method with cross-modal gating fusion enhanced by clinical semantics

Abstract

The application discloses a cross-modal gate fusion radiology report generation method combined with clinical semantic enhancement, and relates to the technical field of medical report generation. In the application, a semantic enhancement decoding unit is used for deeply fusing image sequence representation, historical text embedding and clinical semantic representation, and obtaining a final enhanced decoding representation of multi-modal semantic enhancement; and the final enhanced decoding representation output by the semantic enhancement decoding unit is subjected to report generation, so that a predicted radiology report is obtained. The predicted radiology report generated by the radiology report generation method disclosed in the application is closer to a radiology report sample text in terms of sentence structure and coherence.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of medical report generation, and particularly relates to a cross-modal gating fusion radiology report generation method combined with clinical semantic enhancement. BACKGROUND

[0002] With the wide application of deep learning technology in the field of medical image analysis, radiology report generation based on artificial intelligence has become an important research direction to improve clinical work efficiency and reduce the burden of doctors.

[0003] At present, the existing radiology report generation methods are mostly based on an encoding-decoding architecture, that is, image features are extracted by using a convolutional neural network or a visual Transformer as a visual encoder, and then a corresponding radiology text report is generated by decoding through a long short-term memory network or a Transformer as a sequence decoder. However, the existing radiology report generation methods usually lack the introduction and effective fusion of clinical semantics in the decoding stage, so that the accuracy of the generated radiology report is poor. Therefore, the present application proposes a cross-modal gating fusion radiology report generation method combined with clinical semantic enhancement. SUMMARY

[0004] In view of the problems mentioned in the above background art, the present application proposes a cross-modal gating fusion radiology report generation method combined with clinical semantic enhancement.

[0005] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0006] A cross-modal gating fusion radiology report generation method combined with clinical semantic enhancement comprises the following steps: pre-processing a to-be-predicted X-ray image to obtain a pre-processed to-be-predicted X-ray image, and then inputting the pre-processed to-be-predicted X-ray image into a cross-modal gating fusion neural network model combined with clinical semantic enhancement, and performing forward propagation once to obtain a predicted radiology report.

[0007] The cross-modal gating fusion neural network model combined with clinical semantic enhancement is obtained by training a cross-modal gating fusion neural network combined with clinical semantic enhancement.

[0008] The cross-modal gating fusion neural network combined with clinical semantic enhancement comprises an image feature extractor, an image encoder and a semantic enhancement decoding unit connected in sequence; the input end of the semantic enhancement decoding unit is further connected to the output end of an embedding layer; the cross-modal gating fusion neural network combined with clinical semantic enhancement further comprises a Bio_ClinicalBERT model and a BiomedBERTProjector module connected in sequence, and the output end of the BiomedBERTProjector module is connected to the semantic enhancement decoding unit.

[0009] The image feature extractor extracts high-level semantic features based on the input image, and the image encoder obtains image sequence representation based on the high-level semantic features; the embedding layer obtains historical text embedding based on the radiology report sample corresponding to the paired image; the Bio_ClinicalBERT model obtains original clinical semantic features based on the radiology report sample corresponding to the paired image, and the BiomedBERTProjector module obtains clinical semantic representation based on the original clinical semantic features;

[0010] The semantic enhancement decoding unit is used for deep fusion of the image sequence representation, the historical text embedding and the clinical semantic representation, to obtain a final enhanced decoding representation of multi-modal semantic enhancement; and the final enhanced decoding representation output by the semantic enhancement decoding unit is used for report generation, to obtain a predicted radiology report.

[0011] Preferably, the preprocessing comprises the following steps: sequentially performing image size normalization processing and center cropping processing on the to-be-predicted X-ray image, to obtain a preprocessed chest X-ray image.

[0012] Preferably, the semantic enhancement decoding unit comprises a first semantic enhancement decoding module, a second semantic enhancement decoding module, a third semantic enhancement decoding module, a fourth semantic enhancement decoding module, a fifth semantic enhancement decoding module and a sixth semantic enhancement decoding module connected in sequence, and the first semantic enhancement decoding module to the sixth semantic enhancement decoding module are of the same structure; the input ends of the first semantic enhancement module are connected with the output ends of the embedding layer, the image encoder and the BiomedBERTProjector module respectively, the input ends of the second semantic enhancement module to the sixth semantic enhancement module are connected with the output end of the image encoder, and the input ends of the second semantic enhancement module to the sixth semantic enhancement module are also connected with the output end of the BiomedBERTProjector module.

[0013] Preferably, the first semantic enhancement decoding module comprises a self-attention layer, a cross-modal attention layer, a gated neural network, a feedforward neural network, a residual connection layer and a layer normalization layer; the gated neural network comprises a feature concatenation unit, a weight generation subnetwork and a feature fusion unit connected in sequence; the output ends of the self-attention layer are connected with the cross-modal attention layer and the feature fusion unit respectively, the output ends of the cross-modal attention layer are connected with the feature concatenation unit and the feature fusion unit respectively; the feedforward neural network, the residual connection layer and the layer normalization layer are connected in sequence.

[0014] Preferably, the self-attention layer in the first semantic enhancement decoding module is connected with the output end of the embedding layer, the cross-modal attention layer is connected with the output end of the image encoder, and the feature splicing unit and the feature fusion unit are both connected with the output end of the BiomedBERTProjector module.

[0015] The self-attention layer in the second semantic enhancement decoding module is connected with the output end of the layer normalization layer in the first semantic enhancement decoding module, the cross-modal attention layer is connected with the output end of the image encoder, and the feature splicing unit and the feature fusion unit are both connected with the output end of the BiomedBERTProjector module.

[0016] The self-attention layer in the third semantic enhancement decoding module is connected with the output end of the layer normalization layer in the second semantic enhancement decoding module, the cross-modal attention layer is connected with the output end of the image encoder, and the feature splicing unit and the feature fusion unit are both connected with the output end of the BiomedBERTProjector module.

[0017] The self-attention layer in the fourth semantic enhancement decoding module is connected with the output end of the layer normalization layer in the third semantic enhancement decoding module, the cross-modal attention layer is connected with the output end of the image encoder, and the feature splicing unit and the feature fusion unit are both connected with the output end of the BiomedBERTProjector module.

[0018] The self-attention layer in the fifth semantic enhancement decoding module is connected with the output end of the layer normalization layer in the fourth semantic enhancement decoding module, the cross-modal attention layer is connected with the output end of the image encoder, and the feature splicing unit and the feature fusion unit are both connected with the output end of the BiomedBERTProjector module.

[0019] The self-attention layer in the sixth semantic enhancement decoding module is connected with the output end of the layer normalization layer in the fifth semantic enhancement decoding module, the cross-modal attention layer is connected with the output end of the image encoder, and the feature splicing unit and the feature fusion unit are both connected with the output end of the BiomedBERTProjector module.

[0020] Preferably, in the first semantic enhancement decoding module, the feature splicing unit is used for dimension checking, linear transformation and tensor splicing on the image semantic features and the clinical semantic representation, realizing cross-modal information integration, and obtaining a joint feature tensor fused with image information and clinical semantic information; the weight generation sub-network performs linear mapping and nonlinear compression mapping based on the joint feature tensor output by the feature splicing unit, and obtains three-dimensional weight coefficients; the feature fusion unit is used for slice processing on the three-dimensional weight coefficients, respectively obtaining a clinical semantic weight tensor, an image semantic weight tensor and a text context weight tensor, and performing Hadamard product operation on the text context features and the corresponding text context weight tensor, the image semantic features and the corresponding image semantic weight tensor, and the clinical semantic representation and the corresponding clinical semantic weight tensor, respectively, to obtain a clinical semantic weighted tensor, an image semantic weighted tensor and a text context weighted tensor, and then performing element-by-element weighted summation on the clinical semantic weighted tensor, the image semantic weighted tensor and the text context weighted tensor to obtain a fusion feature tensor with rich clinical semantics, image semantics and text semantics.

[0021] Preferably, the feature splicing unit comprises a linear transformation layer; in the first semantic enhancement decoding module, the feature splicing unit of the gating neural network first compares the image semantic features, the clinical semantic representation and the dimension size of the semantic enhancement decoding unit, and judges whether the image semantic features and the clinical semantic representation are consistent with the dimension size of the semantic enhancement decoding unit.

[0022] When the dimension size of the image semantic features and the clinical semantic representation is consistent with the dimension size of the semantic enhancement decoding unit, the image semantic features and the clinical semantic representation are tensor-spliced to obtain a joint feature tensor;

[0023] When the dimensions of the clinical semantic representation and the semantic enhancement decoding unit are the same, but the dimensions of the image semantic features and the semantic enhancement decoding unit are different, the image semantic features are linearly transformed using a linear transformation layer to make their dimensions the same as those of the semantic enhancement decoding unit. When the dimensions of the image semantic features and the semantic enhancement decoding unit are the same, but the dimensions of the clinical semantic representation and the semantic enhancement decoding unit are different, the clinical semantic representation is linearly transformed using a linear transformation layer to make its dimensions the same as those of the semantic enhancement decoding unit. Then, the image semantic features and the clinical semantic representation are tensor concatenated to obtain a joint feature tensor. When the dimensions of both the image semantic features and the clinical semantic representation are different from those of the semantic enhancement decoding unit, both the image semantic features and the clinical semantic representation are linearly transformed using a linear transformation layer to make their dimensions the same as those of the semantic enhancement decoding unit. Then, the image semantic features and the clinical semantic representation are tensor concatenated to obtain a joint feature tensor that integrates image information and clinical semantic information.

[0024] Preferably, the weight generation subnetwork of the gated neural network in the first semantic enhancement decoding module includes a linear transformation layer and a sigmoid activation function layer connected in sequence. The linear transformation layer performs a linear mapping on the joint feature tensor output by the feature concatenation unit, and the sigmoid activation function layer normalizes the output of the linear transformation layer to the [0,1] interval through nonlinear compression mapping to obtain a three-dimensional weight tensor, which is the three-dimensional weight coefficient.

[0025] Preferably, the feature fusion unit includes a first Hadamard product unit, a second Hadamard product unit, and a third Hadamard product unit, and the outputs of the three Hadamard product units are all connected to the input of the weighted summation unit.

[0026] Among them, in the feature fusion unit:

[0027] First, the three-dimensional weight coefficients (i.e., the three-dimensional weight tensor) are sliced ​​to obtain the clinical semantic weight tensor, the image semantic weight tensor, and the text context weight tensor, respectively.

[0028] Then, the clinical semantic representation and the clinical semantic weight tensor are transmitted to a first Hadamard product unit, the image semantic feature and the image semantic weight tensor are transmitted to a second Hadamard product unit, and the text context feature and the text context weight tensor are transmitted to a third Hadamard product unit; the first Hadamard product unit is configured to perform Hadamard product operation on the clinical semantic representation and the clinical semantic weight tensor to generate a corresponding clinical semantic weighted tensor; the second Hadamard product unit is configured to perform Hadamard product operation on the image semantic feature and the image semantic weight tensor to generate an image semantic weighted tensor; and the third Hadamard product unit is configured to perform Hadamard product operation on the text context feature and the text context weight tensor to generate a text context weighted tensor.

[0029] Finally, the weighted sum unit performs element-wise weighted summation on the clinical semantic weighted tensor, the image semantic weighted tensor and the text context weighted tensor to obtain a fusion feature tensor with rich clinical semantic, image semantic and text semantic.

[0030] Preferably, the cross-modal gated fusion neural network combined with clinical semantic enhancement is trained to obtain a cross-modal gated fusion neural network model combined with clinical semantic enhancement, including the following steps: training the cross-modal gated fusion neural network combined with clinical semantic enhancement based on the preprocessed chest X-ray image, the report text, the radiology report sample corresponding to the paired image contained in the training set of the IU-XRay data set and the total loss of the radiology report generation network to obtain the cross-modal gated fusion neural network model combined with clinical semantic enhancement.

[0031] Preferably, the cross-modal gated fusion neural network combined with clinical semantic enhancement is trained, including the following steps: inputting the preprocessed chest X-ray image, the radiology report sample corresponding to the paired image contained in the training set of the IU-XRay data set (the preprocessed chest X-ray image is obtained based on the paired image) and the report text (the report text is obtained by sequentially performing word segmentation processing and encoding processing on the radiology report sample corresponding to the paired image) into the radiology report generation network, performing forward propagation, calculating the total loss L of the radiology report generation network, and performing back propagation under the guidance of the total loss L to update the weight parameters of the radiology report generation network, completing an epoch of training process, and after 50 epochs of training process, the training of the radiology report generation network is completed, and a radiology report generation network model is obtained.

[0032] Compared with the prior art, the application has the following beneficial technical effects:

[0033] In the present application, the semantic enhancement decoding unit can deeply fuse the historical text embedding, image sequence representation and clinical semantic representation (the clinical semantic representation contains clinical semantic information) and perform semantic enhancement decoding to obtain a multi-modal semantic enhanced final enhanced decoding representation, and then generate a predicted radiology report based on the multi-modal semantic enhanced final enhanced decoding representation; the above setting effectively overcomes the problem that the existing method lacks effective fusion of clinical semantics in the real medical scene in the decoding stage, resulting in poor accuracy of the generated radiology report. The predicted radiology report obtained by the method described in the present application has stronger semantic generation accuracy at the level of one tuple, two tuples, three tuples and four tuples, and is closer to the radiology report sample text in terms of sentence structure and coherence, which also indicates that the radiology report generated by the method described in the present application has better accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 Network structure diagram of the cross-modal gating fusion neural network combined with clinical semantic enhancement;

[0035] Figure 2 Network structure diagram of the BiomedBERTProjector module in the present application; Figure 1

[0036] Network structure diagram of the gating neural network in the present application. Figure 3 Figure 1 Network structure diagram of the semantic enhancement decoding unit in the present application.

[0037] Figure 4 Network structure diagram of the first semantic enhancement decoding module in the present application.

[0038] Figure 5 Network structure diagram of the gating neural network in the present application. Figure 4 DETAILED DESCRIPTION

[0039] In order to more clearly introduce the purpose, technical scheme and advantages of the present application, the following will be further described in combination with the drawings and examples.

[0040] A cross-modal gating fusion radiology report generation method combined with clinical semantic enhancement, comprising the following steps:

[0041] ​​S1, pre-process the paired images in the training set of the IU-XRay dataset to obtain pre-processed chest X-ray images; in this application, the paired images in the training set are referred to as original chest X-ray images; the pre-processing process includes sequentially performing image size normalization processing and center cropping processing on the original chest X-ray images to obtain pre-processed chest X-ray images; in this application, after the original chest X-ray images are subjected to image size normalization processing, the size of the chest X-ray image is 256x256; after the center cropping processing, the size of the chest X-ray image is 224x224.

[0042] S2, constructing a cross-modal gating fusion neural network enhanced by combining clinical semantics;

[0043] The cross-modal gating fusion neural network enhanced by combining clinical semantics, as shown in Figure 1 The cross-modal gating fusion neural network enhanced by combining clinical semantics, as shown in

[0044] In this application, the input of the image feature extractor is the pre-processed chest X-ray image, and the image feature extractor is used to extract high-level semantic features from the pre-processed chest X-ray image to obtain a high-dimensional feature tensor with rich semantic representation; wherein the image feature extractor used in this application has the same structure and function as the ResNet-101 network disclosed in the paper “Deep Residual Learning for Image Recognition”;

[0045] In this application, the image encoder obtains an image sequence representation containing global context semantics, local region context semantic association features and image anatomy features by modeling the relevance and context dependency between image regions layer by layer; in this application, the image encoder has the same structure and function as the Encoder disclosed in the paper “Generating Radiology Reports via Memory-driven Transformer”;

[0046] In this application, the inputs of the semantic enhancement decoding unit include historical text embedding, image sequence representation output by the image encoder, and clinical semantic representation output by the BiomedBERTProjector module. In this application, the semantic enhancement decoding unit is used to perform deep fusion and semantic enhancement decoding on historical text embedding, image sequence representation, and clinical semantic representation to obtain the final enhanced decoding representation of multimodal semantic enhancement. The historical text embedding is obtained by performing embedding mapping operations on radiology report samples (which refer to radiology report samples corresponding to paired images in the training set of the IU-XRay dataset) using an embedding layer.

[0047] The Bio_ClinicalBERT model is used for feature extraction and hidden state aggregation of report text, outputting original clinical semantic features. The report text is obtained by sequentially segmenting and encoding radiology report samples (which also refer to radiology report samples corresponding to paired images in the training set of the IU-XRay dataset). The Bio_ClinicalBERT model used in this application is the publicly available Bio_ClinicalBERT model from https: / / huggingface.co / emilyalsentzer / Bio_ClinicalBERT. In this application, the BiomedBERTProjector module, such as... Figure 2 As shown, it includes two fully connected layers and one ReLU activation layer. The two fully connected layers will be referred to below as the first fully connected layer and the second fully connected layer, as follows. Figure 2 As shown, the first fully connected layer, the ReLU activation layer, and the second fully connected layer are connected sequentially. In the BiomedBERTProjector module, the first fully connected layer is used to perform a linear transformation on the original clinical semantic features output by the Bio_ClinicalBERT model to obtain a higher-dimensional feature representation. The ReLU activation layer is used to perform a non-linear activation operation on the feature representation output by the first fully connected layer, introducing non-linear mapping capabilities to enhance feature expression. The second fully connected layer is used to perform linear dimensionality reduction on the features output by the ReLU activation layer to obtain a clinical semantic representation consistent with the dimension of the semantic enhancement decoding unit.

[0048] The structure of the semantic enhancement decoding unit in this application is as follows: Figure 3The semantic enhancement decoding unit includes six semantic enhancement decoding modules, the six semantic enhancement decoding modules are identical in structure, the six semantic enhancement decoding modules include a first semantic enhancement decoding module, a second semantic enhancement decoding module, a third semantic enhancement decoding module, a fourth semantic enhancement decoding module, a fifth semantic enhancement decoding module, and a sixth semantic enhancement decoding module; the first semantic enhancement decoding module to the sixth semantic enhancement decoding module are connected in sequence, the input end of the first semantic enhancement module is connected with the output end of the embedding layer, the image encoder, and the BiomedBERTProjector module, the input end of the second semantic enhancement module to the sixth semantic enhancement module is connected with the output end of the image encoder, and the input end of the second semantic enhancement module to the sixth semantic enhancement module is also connected with the output end of the BiomedBERTProjector module; in the application, the first semantic enhancement decoding module is used for fusing and semantically enhancing decoding of the image sequence representation, the historical text embedding, and the clinical semantic representation; the second semantic enhancement decoding module is used for fusing and semantically enhancing decoding of the image sequence representation, the clinical semantic representation, and the enhanced feature representation output by the first semantic enhancement decoding module; the third semantic enhancement decoding module is used for fusing and semantically enhancing decoding of the image sequence representation, the clinical semantic representation, and the enhanced feature representation output by the second semantic enhancement decoding module; the fourth semantic enhancement decoding module is used for fusing and semantically enhancing decoding of the image sequence representation, the clinical semantic representation, and the enhanced feature representation output by the third semantic enhancement decoding module; the fifth semantic enhancement decoding module is used for fusing and semantically enhancing decoding of the image sequence representation, the clinical semantic representation, and the enhanced feature representation output by the fourth semantic enhancement decoding module; and the sixth semantic enhancement decoding module is used for fusing and semantically enhancing decoding of the image sequence representation, the clinical semantic representation, and the enhanced feature representation output by the fifth semantic enhancement decoding module, to obtain a final enhanced decoding representation of multi-modal semantic enhancement.

[0049] In the application, the semantic enhancement decoding module includes a self-attention layer, a cross-modal attention layer, a gated neural network, a feedforward neural network, a residual connection layer, and a layer normalization layer; the gated neural network includes a feature splicing unit, a weight generation subnetwork, and a feature fusion unit connected in sequence.

[0050] The self-attention layer in the first semantic enhancement decoding module is connected with the output end of the embedding layer, the cross-modal attention layer is connected with the output end of the image encoder, and the feature splicing unit is connected with the output end of the BiomedBERTProjector module.

[0051] The self-attention layer in the second semantic enhancement decoding module is connected with the output end of the layer normalization layer in the first semantic enhancement decoding module, the cross-modal attention layer is connected with the output end of the image encoder, and the feature splicing unit is connected with the output end of the BiomedBERTProjector module.

[0052] The self-attention layer in the third semantic enhancement decoding module is connected with the output end of the layer normalization layer in the second semantic enhancement decoding module, the cross-modal attention layer is connected with the output end of the image encoder, and the feature splicing unit is connected with the output end of the BiomedBERTProjector module.

[0053] The self-attention layer in the fourth semantic enhancement decoding module is connected with the output end of the layer normalization layer in the third semantic enhancement decoding module, the cross-modal attention layer is connected with the output end of the image encoder, and the feature splicing unit is connected with the output end of the BiomedBERTProjector module.

[0054] The self-attention layer in the fifth semantic enhancement decoding module is connected with the output end of the layer normalization layer in the fourth semantic enhancement decoding module, the cross-modal attention layer is connected with the output end of the image encoder, and the feature splicing unit is connected with the output end of the BiomedBERTProjector module.

[0055] The self-attention layer in the sixth semantic enhancement decoding module is connected with the output end of the layer normalization layer in the fifth semantic enhancement decoding module, the cross-modal attention layer is connected with the output end of the image encoder, and the feature splicing unit is connected with the output end of the BiomedBERTProjector module.

[0056] Taking the first semantic enhancement decoding module as an example, the function of the first semantic enhancement decoding module is introduced, and the structure and input of the first semantic enhancement decoding module are as shown in Figure 4 The function of the first semantic enhancement decoding module is as follows:

[0057] In this application, the input of the self-attention layer in the first semantic enhancement decoding module is the historical text embedding, the output of the self-attention layer is the text context feature, and the output end of the self-attention layer is connected with the input end of the feature fusion unit.

[0058] The input of the cross-modal attention layer in the first semantic enhancement decoding module is the image sequence representation output by the image encoder and the text context feature output by the self-attention layer, the output of the cross-modal attention layer is the image semantic feature, the input end of the cross-modal attention layer is connected with the output end of the image encoder and the output end of the self-attention layer, and the output end of the cross-modal attention layer is respectively connected with the input end of the feature splicing unit and the input end of the feature fusion unit.

[0059] The output of the BiomedBERTProjector module is a clinical semantic representation, and the output end of the BiomedBERTProjector module is connected with the input end of the feature splicing unit and the input end of the feature fusion unit, respectively;

[0060] The output end of the gating neural network is connected with the feedforward neural network, the residual connection layer and the layer normalization layer in sequence, and the output end of the gating neural network is the output end of the feature fusion unit in the gating neural network;

[0061] In the present application, the self-attention layer extracts the dependency relationship between the generated words in the historical text embedding through the multi-head self-attention mechanism, and obtains text context features with rich context information; the cross-modal attention layer is used to perform dot product operation on the image sequence representation (i.e. key vector K) output by the image encoder and the text context features (i.e. query vector Q) output by the self-attention layer, to obtain attention scores, and then normalize the calculated attention scores to obtain attention weights, and then perform weighted summation on the attention weights and the numerical vectors V of the image sequence representation to obtain image semantic features related to the text context;

[0062] The self-attention layer used in the present application has the same structure and function as the self-attention layers in the decoder disclosed in the paper “Attention is all you need”; the cross-modal attention layer used in the present application has the same structure and function as the encoder-decoder attention disclosed in the paper “Attention is all you need”;

[0063] The feature splicing unit is used to perform dimension checking, linear transformation and tensor splicing on the image semantic features and the clinical semantic representation output by the BiomedBERTProjector module, to realize multi-modal information fusion, and obtain a joint feature tensor fused with image information and clinical semantic information, the dimension size of the joint feature tensor being consistent with the dimension size of the semantic enhancement decoding unit; the weight generation subnetwork is used to perform linear mapping and nonlinear compression mapping on the joint feature tensor output based on the feature splicing unit to obtain three-dimensional weight coefficients;

[0064] The feature fusion unit is configured to slice the three-dimensional weight coefficients to obtain a clinical semantic weight tensor, an image semantic weight tensor and a text context weight tensor, and perform Hadamard product operation on the text context feature and the corresponding text context weight tensor, the image semantic feature and the corresponding image semantic weight tensor, and the clinical semantic representation and the corresponding clinical semantic weight tensor, to obtain a clinical semantic weighted tensor, an image semantic weighted tensor and a text context weighted tensor, and then perform element-by-element weighted summation on the clinical semantic weighted tensor, the image semantic weighted tensor and the text context weighted tensor to obtain a fusion feature tensor with rich clinical semantics, image semantics and text semantics.

[0065] The feedforward neural network is configured to perform nonlinear transformation on the fusion feature tensor output by the feature fusion unit to obtain a high-dimensional semantic feature representation; the residual connection layer is configured to perform residual connection on the high-dimensional semantic feature representation output by the feedforward neural network to obtain a residual feature tensor; and the layer normalization layer performs layer normalization processing on the residual feature tensor output by the residual connection layer, thereby completing the semantic enhancement decoding process of the first semantic enhancement decoding module to obtain an enhanced feature representation; wherein the enhanced feature representation is the output result of the first semantic enhancement decoding module after fusion and semantic enhancement decoding of the image sequence representation, the historical text embedding and the clinical semantic representation.

[0066] The feedforward neural network used in the present application has the same structure and function as the fully connected feed-forward network disclosed in the paper Attention is all you need.

[0067] In the present application, the structure and input of the gating neural network in the first semantic enhancement decoding module are taken as an example, as shown in Figure 5 The function of the gating neural network in the first semantic enhancement decoding module is as follows:

[0068] In the present application, the input of the gating neural network in the first semantic enhancement decoding module is the text context feature, the image semantic feature and the clinical semantic representation; the text context feature, the image semantic feature and the clinical semantic representation are all feature tensors.

[0069] In the feature concatenation unit of the gating neural network in the first semantic enhancement decoding module, the dimension sizes of the image semantic feature, the clinical semantic representation and the semantic enhancement decoding unit are compared first to determine whether the image semantic feature and the clinical semantic representation are consistent with the dimension size of the semantic enhancement decoding unit;

[0070] When the dimension sizes of the image semantic feature and the clinical semantic representation are consistent with the dimension size of the semantic enhancement decoding unit, the image semantic feature and the clinical semantic representation are tensor-spliced to obtain a joint feature tensor;

[0071] When the dimension size of the clinical semantic representation is consistent with the dimension size of the semantic enhancement decoding unit and the dimension size of the image semantic feature is inconsistent with the dimension size of the semantic enhancement decoding unit, the image semantic feature is linearly transformed by a linear transformation layer to make the dimension size consistent with the dimension size of the semantic enhancement decoding unit; when the dimension size of the image semantic feature is consistent with the dimension size of the semantic enhancement decoding unit and the dimension size of the clinical semantic representation is inconsistent with the dimension size of the semantic enhancement decoding unit, the clinical semantic representation is linearly transformed by a linear transformation layer to make the dimension size consistent with the dimension size of the semantic enhancement decoding unit, and then the image semantic feature and the clinical semantic representation are tensor-spliced to obtain a joint feature tensor; when the dimension size of the image semantic feature and the dimension size of the clinical semantic representation are both inconsistent with the dimension size of the semantic enhancement decoding unit, the image semantic feature and the clinical semantic representation are linearly transformed by a linear transformation layer to make the dimension sizes consistent with the dimension size of the semantic enhancement decoding unit, and then the image semantic feature and the clinical semantic representation are tensor-spliced to obtain a joint feature tensor that fuses image information and clinical semantic information;

[0072] In this application, tensor splicing refers to that after the dimension size of the image semantic feature and the dimension size of the clinical semantic representation are consistent with the dimension size of the semantic enhancement decoding unit, the two feature tensors are concatenated along the feature dimension to obtain a joint feature tensor;

[0073] The weight generation subnetwork of the gating neural network in the first semantic enhancement decoding module includes linear transformation layers and Sigmoid activation function layers connected in sequence, wherein the linear transformation layers perform linear mapping on the joint feature tensor output by the feature splicing unit, and the Sigmoid activation function layers perform non-linear compression mapping on the output results of the linear transformation layers, and normalize them to the interval [0, 1] to obtain a three-dimensional weight tensor; wherein the form of the three-dimensional weight tensor is (B, T, 3), B represents the batch size, that is, the number of images simultaneously loaded by the clinical semantic enhancement cross-modal gating fusion neural network in the embodiment, B = 16, that is, the clinical semantic enhancement cross-modal gating fusion neural network simultaneously loads 16 images, T represents the sequence length of the historical text embedding, and the number 3 in the form of the three-dimensional weight tensor represents three features, including clinical semantic features, image semantic features and text context features; the three-dimensional weight tensor is referred to as a three-dimensional weight coefficient in the application;

[0074] Structure of the feature fusion unit:

[0075] The feature fusion unit includes three Hadamard product units and a weighted summation unit, the three Hadamard product units are referred to as a first Hadamard product unit, a second Hadamard product unit and a third Hadamard product unit, and the output ends of the three Hadamard product units are connected with the input end of the weighted summation unit;

[0076] Function of the feature fusion unit:

[0077] The three-dimensional weight coefficient (i.e. the three-dimensional weight tensor) is subjected to slicing processing to obtain a clinical semantic weight tensor, an image semantic weight tensor and a text context weight tensor, respectively;

[0078] The clinical semantic representation and the clinical semantic weight tensor are transmitted to the first Hadamard product unit, the image semantic feature and the image semantic weight tensor are transmitted to the second Hadamard product unit, and the text context feature and the text context weight tensor are transmitted to the third Hadamard product unit;

[0079] The first Hadamard product unit is used for performing Hadamard product operation on the clinical semantic representation and the clinical semantic weight tensor to generate a corresponding clinical semantic weighted tensor; the second Hadamard product unit is used for performing Hadamard product operation on the image semantic feature and the image semantic weight tensor to generate an image semantic weighted tensor; and the third Hadamard product unit is used for performing Hadamard product operation on the text context feature and the text context weight tensor to generate a text context weighted tensor;

[0080] The weighted summation unit element-wise weighted sums the clinical semantic weighted tensor, the image semantic weighted tensor and the text context weighted tensor to obtain a fusion feature tensor with rich clinical semantics, image semantics and text semantics;

[0081] In the present application, the final enhanced decoding representation output by the semantic enhancement decoding unit is used for report generation to obtain a predicted radiology report. In the present application, the report generation method is the same as the autoregressive generation method disclosed in the paper “Generating Radiology Reports via Memory-driven Transformer”.

[0082] S2, based on the pre-processed chest X-ray image, the report text, the radiology report samples corresponding to the paired images contained in the training set of the IU-XRay data set and the total loss of the radiology report generation network, the cross-modal gating fusion neural network combined with clinical semantic enhancement is trained to obtain a cross-modal gating fusion neural network model combined with clinical semantic enhancement. Specifically:

[0083] The pre-processed chest X-ray image, the radiology report samples corresponding to the paired images contained in the training set of the IU-XRay data set (the pre-processed chest X-ray image is obtained based on the paired image) and the report text (the report text is obtained by sequentially performing word segmentation processing and encoding processing on the radiology report samples corresponding to the paired images) are input into the radiology report generation network for forward propagation, the total loss L of the radiology report generation network is calculated, and the total loss L is guided to perform back propagation, update the weight parameters of the radiology report generation network, complete an epoch training process, and after 50 epoch training processes, the training of the radiology report generation network is completed, and the radiology report generation network model is obtained. In the present application, the paired images and their corresponding radiology report samples contained in the training set of the IU-XRay data set are input into the radiology report generation network to train the radiology report generation network, which is a conventional training method. For example, the paper “Multimodal recurrent model with attention for automated radiology report generation” discloses such a training method.

[0084] In the present application, the total loss includes a negative log-likelihood loss and an L2 regularization loss; wherein the total loss and the relationship between the negative log-likelihood loss and the L2 regularization loss is shown in equation (1):

[0085] (1)

[0086] In equation (1), This represents the negative log-likelihood loss. Let λ represent the L2 regularization loss, and λ represent the L2 regularization coefficient; where, in this application, the negative log-likelihood loss... The calculation method is the same as that published in the paper "R2Gen-Mamba: A Selective State Space Model for Radiology Report Generation" (1), which calculates the minimization of the negative log-likelihood loss. The calculation method is consistent with that of the L2 regularization loss; the calculation method of L2 regularization loss is consistent with the calculation method of L2 regularization disclosed in Equation (1) of the paper "Generalization Bound Regularization: A Unified Perspective for Understanding Weight Decay Regularization". In this application, L2 regularization is introduced as a parameter constraint operation to constrain the learnable weight matrix and bias vector of the linear transformation layer in the weight generation subnetwork.

[0087] In this application, the dimension d_model of the cross-modal gated fusion neural network combined with clinical semantic enhancement is set to 512, and the number of self-attention heads n_head in the self-attention layer is set to 8; the optimizer is the Adam optimizer, and the initial learning rate of the image feature extractor is set to 5e. -5 The initial learning rate of all trainable parts in the cross-modal gated fusion neural network combined with clinical semantic enhancement, excluding the image feature extractor, is set to e. -4 Furthermore, in this embodiment, during the training process of the cross-modal gated fusion neural network combined with clinical semantic enhancement, a linear warm-up strategy is first used for adjustment. When the number of warm-up steps (i.e., the number of training steps) reaches a preset warm-up step threshold (in this embodiment, the preset warm-up step threshold is set to 1000), a cosine annealing learning rate scheduling is then adopted. This application uses a beam search strategy for inference testing. Specifically, in terms of inference testing, a beam search strategy is adopted, and the beam size is set to 3.

[0088] S4, the to-be-predicted X-ray image (the to-be-predicted X-ray image refers to an X-ray image to be generated radiology report) is preprocessed according to the preprocessing step described in step S1 to obtain a preprocessed to-be-predicted X-ray image; then, the preprocessed to-be-predicted X-ray image is input into the cross-modal gating fusion neural network model combined with clinical semantic enhancement obtained in step S3, and is forward propagated once, that is, after the preprocessed to-be-predicted X-ray image is input into the cross-modal gating fusion neural network model combined with clinical semantic enhancement obtained in step S3, it is processed in turn by the image feature extractor, the image encoder, and the semantic enhancement decoding unit, and then the report is generated to obtain the predicted radiology report.

[0089] Test 1:

[0090] In order to verify the superiority of the radiology report generation method described in the present application compared with other radiology report generation methods, the present application uses the test set in the IU-XRay dataset to test the radiology report generation method described in the present application and four existing radiology report generation methods, COATT method (from the paper “On the Automatic Generation of Medical Imaging Reports”), HRGR method (from the paper “Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation”), R2Gen method (from the paper “Generating Radiology Reports via Memory-driven Transformer”), and R2GenCMN method (from the paper “Cross-modal Memory Networks for Radiology Report Generation”), and the test results are shown in Table 1. The test process is as follows:

[0091] First, the paired images in the test set in the IU-XRay dataset are processed using the preprocessing step described in step S1 to obtain test images;

[0092] Then, the neural networks involved in the above four existing radiology report generation methods are pre-trained; wherein the pre-training method of the neural networks involved in the above four existing radiology report generation methods is the same as the training method of the cross-modal gating fusion neural network combined with clinical semantic enhancement in step S3 of the present application;

[0093] Finally, the test images are directly input into the cross-modal gating fusion neural network model with clinical semantic enhancement obtained by the method described in the present application and the neural network model obtained after pre-training of the four existing radiology report generation methods described above, respectively, to obtain test results, as shown in Table 1.

[0094] Table 1 Test results of different methods on the test set of the IU-XRay dataset

[0095]

[0096] Test 2:

[0097] In order to verify the superiority of the radiology report generation method described in the present application compared with other radiology report generation methods, the test set in the MIMIC-CXR dataset is also used to test the radiology report generation method described in the present application and five existing radiology report generation methods, i.e. the ST method (from the paper “Show and tell: A neural image caption generator”), the ADAATT method (from the paper “Knowing when to look: Adaptive attention via a visual sentinel for image captioning”), the CA method (from the paper “Contrastive Attention for Automatic Chest X-ray Report Generation”), the R2GenRL method (from the paper “Reinforced Cross-modal Alignment for Radiology Report Generation”), and the R2GenMamba method (from the paper “R2Gen-Mamba: A Selective State Space Model for Radiology Report Generation”), and the test results are shown in Table 2. The difference between the test process in Test 2 and the test process in Test 1 is that in Test 2, the paired images in the test set in the MIMIC-CXR dataset are processed using the preprocessing step described in step S1 to obtain test images.

[0098] Table 2 Test results of different methods on the test set of the MIMIC-CXR dataset

[0099]

[0100] In Table 1 and Table 2, BLEU-1, BLEU-2, BLEU-3, BLEU-4, ROUGE-L are commonly used evaluation indicators in text generation, and Ours refers to the radiology report generation method proposed in the present application. Among them, the BLEU series indicators (the BLEU series indicators refer to BLEU-1 indicator, BLEU-2 indicator, BLEU-3 indicator and BLEU-4 indicator) are used to measure the degree of word matching between the generated text and the reference text, and the higher the value, the stronger the consistency of the generated text in vocabulary and phrase; BLEU-1 to BLEU-4 respectively represent the matching degree of single character, bigram, trigram and quadruple, and the level gradually advances, the higher the values of BLEU-1, BLEU-2, BLEU-3 and BLEU-4, the better the accurate matching degree of the generated text and the reference text. The ROUGE-L indicator is used to measure the similarity of the generated text and the reference text in structure, which is calculated based on the longest common subsequence, and the higher the value, the closer the generated text is to the reference text in overall structure and coherence at the sentence level.

[0101] From Table 1, it can be known that when the test set in the IU-XRay dataset is tested, the test effect of the existing R2GenCMN method is better than that of the other existing radiology report generation methods, and therefore, the test results of the R2GenCMN method and the test results of the radiology report generation method described in the present application are compared, and the specific analysis is as follows:

[0102] From Table 1, it can be seen that when the test set in the IU-XRay dataset is tested, the performance of the method described in the present application is better than that of the R2GenCMN method in all indicators shown in Table 1: the BLEU-1 obtained by the method described in the present application is 0.485, the BLEU-2 is 0.317, the BLEU-3 is 0.230, the BLEU-4 is 0.178, and the ROUGE-L is 0.385; wherein, the improvement of the BLEU-1 indicator shows that the semantic generation accuracy of the method described in the present application at the single character level is stronger, which can generate more content consistent with the single character of the reference text, and improves the generation quality at the single character level; the improvement of the BLEU-2 indicator shows that the language construction ability of the method described in the present application at the two-element group (i.e. two-element phrase combination) is enhanced, which can generate more accurate phrase combinations, and enhances the expression quality of the text (i.e. radiology report) at the phrase level; the improvement of the BLEU-3 indicator shows that the semantic generation accuracy of the method described in the present application at the three-element group level is stronger, and the generated text (i.e. radiology report) is more coherent in grammar and semantics; the improvement of the BLEU-4 indicator shows that the semantic generation accuracy of the method described in the present application at the four-element group level is stronger, and the generated text (i.e. radiology report) has a stronger advantage in higher-level phrase collocation and semantic consistency, further improving the quality of the generated radiology report; the improvement of the ROUGE-L indicator further shows that the method of the present application is closer to the radiology report sample text in terms of sentence structure and semantic coherence, and can better maintain the overall logic and semantic integrity of the radiology report.

[0103] The MIMIC-CXR dataset used in the present application is a large dataset, which is almost 70 times the size of the IU-XRay dataset, and the data complexity and label noise are significantly increased compared with the IU-XRay dataset. From Table 2, it can be seen that when the test set in the MIMIC-CXR dataset is tested, the performance of the method described in the present application is better than that of the R2GenMamba method in all indicators shown in Table 2: the BLEU-1 obtained by the method described in the present application is 0.398, the BLEU-2 is 0.249, the BLEU-3 is 0.167, the BLEU-4 is 0.117, and the ROUGE-L is 0.290;

[0104] Obviously, the BLEU-1 indicator obtained by the radiology report generation method described in the present application is improved by 13.07% compared with the BLEU-1 indicator obtained by the R2GenMamba method, which shows that when the test set in the MIMIC-CXR dataset (large dataset) is tested, the semantic generation accuracy of the method described in the present application at the single character level is stronger, which can generate more content consistent with the single character of the reference text, and improves the generation quality at the single character level;

[0105] The BLEU-2 index obtained by the radiology report generation method described in the present application is improved by 12.16% compared to the BLEU-2 index obtained by the R2GenMamba method, which indicates that when tested on the test set of the MIMIC-CXR dataset (large dataset), the language construction ability of the method described in the present application on the binary group (i.e., the combination of binary phrases) is enhanced, and more accurate phrase combinations can be generated, enhancing the expression quality of the text (i.e., the radiology report) at the phrase level;

[0106] The BLEU-3 index obtained by the radiology report generation method described in the present application is improved by 9.87% compared to the BLEU-3 index obtained by the R2GenMamba method, which indicates that when tested on the test set of the MIMIC-CXR dataset (large dataset), the semantic generation accuracy of the method described in the present application at the ternary group level is stronger, and the generated text (i.e., the radiology report) is more coherent in grammar and semantics;

[0107] The BLEU-4 index obtained by the radiology report generation method described in the present application is improved by 6.36% compared to the BLEU-4 index obtained by the R2GenMamba method, which indicates that when tested on the test set of the MIMIC-CXR dataset (large dataset), the method described in the present application can have stronger semantic generation accuracy at the quaternary group level, and the generated text (i.e., the radiology report) has stronger advantages in higher-level phrase collocation and semantic consistency, further improving the quality of the generated radiology report;

[0108] The ROUGE-L index obtained by the radiology report generation method described in the present application is improved by 2.1% compared to the ROUGE-L index obtained by the R2GenMamba method, which indicates that when tested on the test set of the MIMIC-CXR dataset (large dataset), the method described in the present application is closer to the radiology report sample text in terms of sentence structure and semantic coherence, and can better maintain the overall logic and semantic integrity of the radiology report.

[0109] In summary, the radiology report generation method described in the present application performs well in single character, binary group, ternary group, and quaternary group matching, improving the quality of the generated report; the text (i.e., the predicted radiology report) generated by the radiology report generation method described in the present application is closer to the radiology report sample text in terms of sentence structure and coherence; especially when tested based on the MIMIC-CXR dataset (large dataset), the effect is more obvious.

Claims

1. A cross-modality gated fusion radiology report generation method enhanced with clinical semantics, characterized in that: The method comprises the following steps: The to-be-predicted X-ray image is input into the cross-modal gating fusion neural network model combined with clinical semantic enhancement after preprocessing, and is forward propagated once to obtain a predicted radiology report; The model is obtained based on training of the cross-modal gating fusion neural network combined with clinical semantic enhancement; The network comprises an embedding layer, an image feature extractor, an image encoder and a semantic enhancement decoding unit connected in sequence, and further comprises a Bio_ClinicalBERT model and a BiomedBERTProjector module connected in sequence, and the embedding layer and the BiomedBERTProjector module are connected to the semantic enhancement decoding unit; The image encoder is used to acquire image sequence representation from the features output by the image feature extractor; the embedding layer and the Bio_ClinicalBERT model are used to acquire historical text embedding and original clinical semantic features based on radiology report samples; and the BiomedBERTProjector module is used to acquire clinical semantic representation based on the original clinical semantic features; The semantic enhancement decoding unit is used to deeply fuse the image sequence representation, the historical text embedding and the clinical semantic representation to obtain final enhanced decoding representation of multi-modal semantic enhancement. The final enhanced decoding representation output by the semantic enhancement decoding unit is used for report generation to obtain the predicted radiology report. The BiomedBERTProjector module comprises two fully connected layers and a ReLU activation layer, and the first fully connected layer, the ReLU activation layer and the second fully connected layer are connected in sequence.

2. The method of claim 1, wherein the method is enhanced by clinical semantics. The preprocessing comprises the following steps: The to-be-predicted X-ray image is subjected to image size normalization processing and center cropping processing in sequence to obtain a preprocessed chest X-ray image.

3. The method of claim 1, wherein the method further comprises: The semantic enhancement decoding unit comprises first to sixth semantic enhancement decoding modules connected in sequence, and the first to sixth semantic enhancement decoding modules have the same structure; the embedding layer, the image encoder and the BiomedBERTProjector module are connected to the first semantic enhancement module; the image encoder is further connected to the second to sixth semantic enhancement modules; and the BiomedBERTProjector module is further connected to the second to sixth semantic enhancement modules.

4. The method of claim 3, wherein the method further comprises: The first semantic enhancement decoding module comprises a self-attention layer, a cross-modal attention layer, a gating neural network, a feedforward neural network, a residual connection layer and a layer normalization layer; the gating neural network comprises a feature splicing unit, a weight generation subnetwork and a feature fusion unit connected in sequence; the self-attention layer is connected to the cross-modal attention layer and the feature fusion unit; the cross-modal attention layer is connected to the feature splicing unit and the feature fusion unit; and the feedforward neural network, the residual connection layer and the layer normalization layer are connected in sequence.

5. The method of claim 4, wherein the method further comprises: The self-attention layer in the first semantic enhancement decoding module is connected to the output end of the embedding layer, the cross-modal attention layer is connected to the output end of the image encoder, and the feature splicing unit and the feature fusion unit are connected to the output end of the BiomedBERTProjector module. The self-attention layer in the second semantic enhancement decoding module is connected with the output end of the layer normalization layer in the first semantic enhancement decoding module, the cross-modal attention layer is connected with the output end of the image encoder, and the feature splicing unit and the feature fusion unit are both connected with the output end of the BiomedBERTProjector module; The self-attention layer in the third semantic enhancement decoding module is connected with the output end of the layer normalization layer in the second semantic enhancement decoding module, the cross-modal attention layer is connected with the output end of the image encoder, and the feature splicing unit and the feature fusion unit are both connected with the output end of the BiomedBERTProjector module; The self-attention layer in the fourth semantic enhancement decoding module is connected with the output end of the layer normalization layer in the third semantic enhancement decoding module, the cross-modal attention layer is connected with the output end of the image encoder, and the feature splicing unit and the feature fusion unit are both connected with the output end of the BiomedBERTProjector module; The self-attention layer in the fifth semantic enhancement decoding module is connected with the output end of the layer normalization layer in the fourth semantic enhancement decoding module, the cross-modal attention layer is connected with the output end of the image encoder, and the feature splicing unit and the feature fusion unit are both connected with the output end of the BiomedBERTProjector module; The self-attention layer in the sixth semantic enhancement decoding module is connected with the output end of the layer normalization layer in the fifth semantic enhancement decoding module, the cross-modal attention layer is connected with the output end of the image encoder, and the feature splicing unit and the feature fusion unit are both connected with the output end of the BiomedBERTProjector module.

6. The cross-modality gated fusion radiology report generation method enhanced with clinical semantics according to claim 4, characterized in that: In the first semantic enhancement decoding module, the feature splicing unit is used for dimensional verification, linear transformation and tensor splicing of the image semantic features and the clinical semantic representation, so as to obtain a joint feature tensor fused with image and clinical semantic information; the weight generation sub-network performs linear mapping and nonlinear compression mapping based on the joint feature tensor output by the feature splicing unit, so as to obtain three-dimensional weight coefficients; The feature fusion unit is used for slice processing of the three-dimensional weight coefficients, so as to obtain a clinical semantic weight tensor, an image semantic weight tensor and a text context weight tensor, respectively, and perform Hadamard product operation on the text context feature and its corresponding text context weight tensor, the image semantic feature and its corresponding image semantic weight tensor, and the clinical semantic representation and its corresponding clinical semantic weight tensor, respectively, so as to obtain a clinical semantic weighted tensor, an image semantic weighted tensor and a text context weighted tensor, and then perform element-by-element weighted summation on the clinical semantic weighted tensor, the image semantic weighted tensor and the text context weighted tensor, so as to obtain a fusion feature tensor with rich clinical semantics, image semantics and text semantics.

7. The method of claim 4, wherein the method further comprises: The feature splicing unit comprises a linear transformation layer; the feature splicing unit of the gating neural network in the first semantic enhancement decoding module first compares the dimensions of the image semantic features, the clinical semantic representation and the semantic enhancement decoding unit, and judges whether the dimensions of the image semantic features and the clinical semantic representation are consistent with the dimension of the semantic enhancement decoding unit; When the dimensions of the image semantic features and the clinical semantic representation are consistent with the dimension of the semantic enhancement decoding unit, the image semantic features and the clinical semantic representation are tensor spliced to obtain a joint feature tensor; When the dimensions of the image semantic features and / or the clinical semantic representation are inconsistent with the dimension of the semantic enhancement decoding unit, the dimensions of the inconsistent image semantic features and / or the clinical semantic representation are linearly transformed by the linear transformation layer to make the dimensions consistent with the dimension of the semantic enhancement decoding unit; and then tensor splicing is performed to obtain a joint feature tensor fused with image and clinical semantic information.

8. The method of claim 4, wherein the method is enhanced by clinical semantics. The weight generation subnetwork of the gating neural network in the first semantic enhancement decoding module comprises a linear transformation layer and a Sigmoid activation function layer connected in sequence, wherein the linear transformation layer linearly maps the joint feature tensor output by the feature splicing unit, and the Sigmoid activation function layer non-linearly compresses and maps the output result of the linear transformation layer to normalize it to the [0, 1] interval to obtain a three-dimensional weight tensor, which is a three-dimensional weight coefficient.

9. The method of claim 4, wherein the method is enhanced by clinical semantics. The feature fusion unit comprises a first Hadamard product unit to a third Hadamard product unit, and the three Hadamard product units are connected with a weighted summation unit; In the feature fusion unit: First, the three-dimensional weight coefficients are sliced to obtain a clinical semantic weight tensor, an image semantic weight tensor and a text context weight tensor, respectively; Then, the first Hadamard product unit performs Hadamard product operation on the clinical semantic representation and the clinical semantic weight tensor to generate a clinical semantic weighted tensor; the second Hadamard product unit performs Hadamard product operation on the image semantic features and the image semantic weight tensor to generate an image semantic weighted tensor; and the third Hadamard product unit performs Hadamard product operation on the text context features and the text context weight tensor to generate a text context weighted tensor; Finally, the weighted summation unit performs element-wise weighted summation on the clinical semantic weighted tensor, the image semantic weighted tensor and the text context weighted tensor to obtain a fusion feature tensor with rich clinical semantic, image semantic and text semantic.

10. The method of claim 1, wherein: The training of the cross-modal gating fusion neural network combined with clinical semantic enhancement to obtain the cross-modal gating fusion neural network model combined with clinical semantic enhancement comprises the following steps: training the cross-modal gating fusion neural network combined with clinical semantic enhancement based on the preprocessed chest X-ray images, the report texts, the radiology report samples corresponding to the paired images contained in the training set of the IU-XRay data set and the total loss of the radiology report generation network to obtain the cross-modal gating fusion neural network model combined with clinical semantic enhancement.

Images