Radiology report generation method based on semantic alignment

Abstract

The invention discloses a radiology report generation method based on semantic alignment, and belongs to the technical field of radiology report generation. The radiology report generation network constructed by the invention comprises a multi-scale feature extraction module, a cross-modal semantic alignment module and a Transform report generation module which are connected in sequence; the multi-scale feature extraction module comprises a convolutional neural network and a multi-scale local sparse attention module which are connected in sequence; the cross-modal semantic alignment module comprises an embedded layer, a multi-head attention module and a gating fusion module, wherein the multi-head attention module and the gating fusion module are connected with the embedded layer. The input end of the multi-head attention module and the input end of the gating fusion module are connected with the output end of the multi-scale local sparse attention module. The predicted radiology report generated by the radiology report generation method is good in quality.

Description

Technical Field

[0001] The present invention belongs to the technical field of radiology report generation, and in particular relates to a radiology report generation method based on semantic alignment. Background Art

[0002] In modern medical diagnosis, radiology reports are an important basis for doctors to make diagnosis and treatment decisions. However, the traditional way to generate radiology reports is to write diagnostic reports after manually interpreting medical images. This method is very time-consuming and labor-intensive, and may even lead to inaccurate radiology reports. Therefore, the method of generating radiology reports based on neural networks has attracted the attention of more and more researchers. However, radiology report generation is a very challenging task. Unlike traditional image description tasks, this task requires a detailed and professional description of radiology images (i.e., X-ray images). Moreover, the semantic relationship between professional terms in radiology reports and their corresponding image areas is often more complex.

[0003] At present, the methods for generating radiology reports based on neural networks widely adopt the encoder-decoder architecture. Some existing methods for generating radiology reports based on neural networks often focus on learning single-modal features and ignore the importance of cross-modal interaction. Although there are some methods for generating radiology reports based on cross-modal alignment in the prior art, the radiology reports generated by these methods are generally of poor quality, such as poor accuracy in semantic expression and overall expression. To this end, this application designs a method for generating radiology reports based on semantic alignment. Summary of the invention

[0004] In view of the defects and shortcomings in the above-mentioned prior art, the present application proposes a radiology report generation method based on semantic alignment.

[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme: A radiology report generation method based on semantic alignment comprises the following steps: S1. Construct a radiology report generation network, which includes a multi-scale feature extraction module, a cross-modal semantic alignment module and a Transformer report generation module connected in sequence; the multi-scale feature extraction module includes a convolutional neural network and a multi-scale local sparse attention module connected in sequence; the cross-modal semantic alignment module includes an embedding layer and a multi-head attention module and a gated fusion module connected to the output end of the embedding layer, and the input ends of the multi-head attention module and the gated fusion module are also connected to the output end of the multi-scale local sparse attention module in the multi-scale feature extraction module; In this application, a convolutional neural network is used to extract the features of all patches in the radiological image and obtain the image region feature map. ; Multi-scale local sparse attention module for enhancing image region feature maps The features of each patch in , obtain a unimodal image embedding containing multi-scale semantic information E (I) ; The embedding layer is used to perform embedding mapping operations on the radiology report samples to obtain unimodal text embedding E (R) ; The multi-head attention module is based on a learnable matrix M and unimodal embedding E All vectors of Calculate the similarity between the two, and then select only the learnable matrix The most similar top embedding E in vectors for information interaction, and then based on the selected learnable matrix Center front The vectors are weighted and feature fused to complete the attention calculation and obtain a single-modal fusion representation containing cross-modal information. ; Among them, when the unimodal embedding E Embedding for unimodal images E (I) When , the obtained single-modal fusion representation containing cross-modal information It is a single-modality fusion image representation , when the unimodal embedding E Embedding for unimodal images E (R) When , the obtained single-modal fusion representation containing cross-modal information It is a single-modality fusion image representation ; The gated fusion module is used to embed the single modality into E and single modality fusion image representation The final unimodal feature F is obtained by fusion. When the gated fusion module embeds the unimodal image E (I) and single modality fusion representation Fusion to obtain the final image features , when the gated fusion module embeds the unimodal text E (R) and unimodal text representations Fusion to obtain the final image features ; S2. Total loss of building a radiology report generation network : This application will predict all the characters in the order in which they are obtained. As the predicted report sequence generated by the radiology report generation network model, the character sequence The corresponding radiology report sample sequence is , predict report sequence And the corresponding radiology report sample sequence The cross entropy loss between And multi-label contrast loss The loss constitutes the total loss of the radiology report generation network ; S3, training the radiology report generation network based on the training set and the total loss of the radiology report generation network to obtain a radiology report generation network model; S4. Input the radiological image for which the radiological report is to be generated into the radiological report generation network model, forward propagate once, and obtain the predicted radiological report.

[0006] Preferably, in step S1, the convolutional neural network is a pre-trained DenseNet121 network, a DenseNet121 network, a DenseNet network or a ResNet101 network.

[0007] Preferably, in step S1, the multi-scale local sparse attention module includes four linear layers, wherein the inputs of three linear layers are image region feature maps , image region feature map After linear transformation through three linear layers, we get query, key and value respectively. Then, we divide query, key and value into several attention heads respectively. Then, according to the number of attention heads, Split the query into indivual Subvectors, split the key into indivual Subvectors, splitting values ​​into indivual sub-vector; and then through the sliding window operation from Subvector and Select some patches in the sub-vector to calculate the scaled dot product attention and get the weighted output ; Then, all the output The Concat layer is used for splicing, and then the feature fusion is performed through the last linear layer to obtain the image region features containing rich multi-scale semantic information. , and then, the image region features containing multi-scale semantic information Flattening in the spatial dimension to obtain a unimodal image embedding containing rich multi-scale semantic information .

[0008] Preferably, in step S1, a sliding window operation is performed from Subvector and Selecting some patches in the sub-vector for calculating the scaled dot product attention includes the following steps: Subvector and In the sub-vector, each patch is centered, and the patches around it (i.e. the patch at the center) are selected by setting the hyperparameters of the sliding window operation, namely the convolution kernel size k and the expansion rate r, and a new key is obtained. and the new value ; In this application, different attention heads are set with different expansion rates, and different attention heads can obtain the semantic information of radiological images at different scales; then, calculate Subvectors and new keys The similarity between them is then divided by the scaling factor to obtain the scaled similarity. The scaled similarity value is converted into an attention weight through a softmax operation; then, the attention weight is combined with the new value Multiply them together to get the weighted output , the weighted output That is, the image features containing local semantic information are used to calculate the scaled dot product attention.

[0009] Preferably, in step S1, the gated fusion module includes two linear layers, the input end of the first linear layer is respectively connected to the output end of the multi-scale local sparse attention module and the output end of the embedding layer, the input end of the second linear layer is connected to the output end of the multi-head attention module, the output ends of the two linear layers are both connected to the first Add layer, the output end of the first Add layer is connected to the Sigmoid activation layer, the output end of the Sigmoid activation layer is respectively connected to the first element-by-element multiplication module and the second element-by-element multiplication module, and the input end of the first element-by-element multiplication module is also connected to the input end of the first linear layer; the input end of the first linear layer and the input end of the second linear layer are both connected to the input end of the second Add layer, and the output end of the second Add layer is sequentially connected to the tanh activation layer and the second element-by-element multiplication module; the output end of the first element-by-element multiplication module and the input end of the second linear layer are both connected to the input end of the third Add layer, and the output end of the third Add layer and the output end of the second element-by-element multiplication module are both connected to the fourth Add layer.

[0010] Preferably, in step S1, the Transformer report generation module includes a coding unit, a decoding unit and a prediction head connected in sequence, and the coding unit and the decoding unit are also connected to the output end of the gated fusion module in the cross-modal semantic alignment module; wherein the coding unit is used to output the final image features of the gated fusion module. Encode and get the encoded image representationL ; The decoding unit is used to represent the image L And the final text features output by the current gated fusion module Perform a decoding operation and output a decoded representation; the prediction head is used to perform a linear transformation on the decoded representation output by the decoding unit and apply a Softmax operation to output the character with the highest probability, which is the predicted next character.

[0011] Preferably, in step S1, the encoding unit includes three sequentially connected Transformer encoder layers, each Transformer encoder layer includes a multi-head self-attention mechanism and a feedforward neural network, and the parameters of each Transformer encoder layer are set to 8 attention heads and 512-dimensional hidden states.

[0012] Preferably, in step S1, the decoding unit includes three sequentially connected Transformer decoder layers, each Transformer decoder layer includes two parts: a multi-head self-attention mechanism and a feedforward neural network, and the parameters of each Transformer decoder layer are set to 8 attention heads and 512-dimensional hidden states.

[0013] Preferably, in step S1, the prediction head includes a linear layer and a Softmax layer connected in sequence.

[0014] Preferably, in step S2, the multi-label contrast loss is calculated When performing the training, first, use the CheXpert tool to generate a pseudo label for each X-ray image-text pair sample used for training. Since each X-ray image-text pair sample may contain multiple disease labels at the same time, the pseudo label set in this application is represented as a one-hot vector of multiple labels. The current X-ray image sample is used as the image anchor point, and then the sample with the highest cosine similarity to the anchor point is selected from the text samples with completely different labels from the current X-ray image sample as the text negative sample. Then, the current text sample (the current text sample and the X-ray image sample in the previous text belong to an X-ray image-text pair sample) is used as the text anchor point, and then the sample with the highest cosine similarity to the anchor point is selected from the X-ray image samples with completely different labels from the current text sample as the image negative sample; then the multi-label contrast loss is calculated based on the determined negative samples.

[0015] Preferably, in step S3, the images in the training set and their corresponding radiology report samples are input into the radiology report generation network, forward propagation is performed, and the total loss of the radiology report generation network is calculated. , and in the total loss Back propagation is performed under the guidance of , the weight parameters of the radiology report generation network are updated, the training process of one epoch is completed, and the iterative training is repeated to obtain the radiology report generation network model.

[0016] Preferably, in step S3, the training set is a training set in the IU-Xray dataset or a training set in the MIMIC-CXR dataset; when the radiology report generation network is trained based on the training set in the IU-Xray dataset, the paired images of the IU-Xray dataset are input into the radiology report generation network, and μ in the most similar first μ vectors is set to 64; when the radiology report generation network is trained based on the training set in the MIMIC-CXR dataset, a single image of the MIMIC-CXR dataset is input into the radiology report generation network, and μ in the most similar first μ vectors is set to 128.

[0017] Compared with the prior art, the beneficial technical effects of this application are: The multi-scale local sparse attention module constructed in this application can effectively enhance the features of each patch in the image region feature map and obtain a single-modal image embedding containing multi-scale semantic information. E (I) ; The multi-head attention module constructed in this application introduces a shared learnable matrix M to process the embeddings from the two modalities separately (i.e., unimodal text embedding E (R) and unimodal image embedding E (I)), which enables the radiology report generation network described in this application to use the multi-head attention module to realize cross-modal information interaction, and then use the gated fusion module to obtain single-modal features with fused multi-modal information; in addition, the multi-label contrast loss constructed in this application can also establish a closer semantic connection between the features of the two modalities, image features and text features, and realize supervised cross-modal information fusion and semantic alignment. Through testing, it can be known that the radiology report generation method described in the present application shows stronger accuracy in the matching of single characters, can generate more content consistent with single characters of the reference text, and improves the generation quality at the single character level; the radiology report generation method described in the present application also performs better in bigram matching, can generate more precise phrase combinations, and enhances the expressiveness of the text at the phrase level; the radiology report generation method described in the present application is optimized in triple matching, and the generated text is more coherent in grammar and semantics, reflecting higher phrase matching accuracy; the method of the present application performs well in quadruple matching, and the generated text has stronger advantages in higher-level phrase matching and semantic consistency, further improving the quality of generated reports; the text generated by the radiology report generation method described in the present application is better in accuracy and flexibility of semantic expression and overall expression; the radiology report text generated by the radiology report generation method described in the present application is closer to the radiology report sample text in terms of sentence structure and coherence, and can better maintain the overall logic and semantic integrity of the radiology report; in summary, the quality of the report generated by the radiology report generation method described in the present application is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of the network structure of a radiology report generation network of the present invention; Figure 2 for Figure 1 Schematic diagram of the network structure of the medium-scale local sparse attention module; Figure 3 for Figure 1 Schematic diagram of the network structure of the gated fusion module. DETAILED DESCRIPTION

[0019] Embodiment 1: A radiology report generation method based on semantic alignment comprises the following steps: S1. Construct an end-to-end radiology report generation network. The network structure of the radiology report generation network is as follows: Figure 1As shown, the radiology report generation network includes a multi-scale feature extraction module, a cross-modal semantic alignment module and a Transformer report generation module connected in sequence; wherein the multi-scale feature extraction module includes a convolutional neural network and a multi-scale local sparse attention module connected in sequence; in the present application, the cross-modal semantic alignment module includes an embedding layer and a multi-head attention module and a gated fusion module connected to the output end of the embedding layer, and the input ends of the multi-head attention module and the gated fusion module are also connected to the output end of the multi-scale local sparse attention module in the multi-scale feature extraction module; in the present application, the Transformer report generation module includes an encoding unit, a decoding unit and a prediction head connected in sequence, and the encoding unit and the decoding unit are also connected to the output end of the gated fusion module in the cross-modal semantic alignment module; the encoding unit includes three Transformer encoder layers connected in sequence, and the decoding unit includes three Transformer decoder layers connected in sequence.

[0020] In this application, the convolutional neural network in the multi-scale feature extraction module is used to extract the radiological image (in this embodiment, the X-ray image) ) to obtain the image region feature map , image region feature map X-ray images are included The convolutional neural network in this application can be a pre-trained DenseNet121 network, a DenseNet121 network, a DenseNet network or a ResNet101 network. In this embodiment, a pre-trained DenseNet121 network is used. The pre-trained DenseNet121 network has the same structure and function as the DenseNet121 network disclosed in the paper "Densely connected convolutional networks". In this application, the multi-scale local sparse attention module is used to perform image region feature mapping on the output of the convolutional neural network. Each patch in is a query, and in a window centered on each patch, key regions (e.g., anatomical regions) are sparsely selected as keys and values ​​by setting the dilation rate r and the convolution kernel size k, and then self-attention is calculated on these key regions to obtain a unimodal image embedding E (I) In this application, key areas (e.g., anatomical areas) are sparsely selected as keys and values ​​in a window centered on each patch by setting the dilation rate r and the convolution kernel size k, and then self-attention is calculated on these key areas, so that a multi-scale local sparse attention module can calculate self-attention on key areas of different sizes in radiological images.

[0021] In this application, the embedding layer in the cross-modal semantic alignment module is used to perform an embedding mapping operation on the radiology report sample to obtain the text features of the radiology report sample and obtain a unimodal text embedding E (R) ; The multi-head attention module is first based on a learnable matrix M and unimodal image embedding E (I) All vectors of Calculate the similarity between the two, and then select only the learnable matrix Medium and Unimodal Image Embedding Most similar to vectors for information interaction, and then based on the selected learnable matrix Center front The vectors are weighted and feature fused to complete the attention calculation and obtain a single-modal fusion image representation containing cross-modal information. ; The multi-head attention module can also be based on the calculation of the learnable matrix M and unimodal text embeddings E (R) All vectors of Calculate the similarity between the two, and then select only the learnable matrix Medium and Unimodal Text Embedding Most similar to vectors for information interaction, and then based on the selected learnable matrix Center front The vectors are weighted and feature fused to complete the attention calculation and obtain a single-modal fusion image representation containing cross-modal information. ; Single-modal image embedding in this application E (I) and unimodal text embeddings E (R) The feature dimension of M A randomly initialized 2048× d The matrix of d Represents unimodal image embedding E (I) The characteristic dimension of The multi-head attention module used in this application has the same structure as the multi-head self-attention mechanism module in the decoder part of the Transformer structure in the paper "Attention is all you need", but the multi-head attention module in this application and the multi-head self-attention mechanism module in the decoder part of the Transformer structure in the paper "Attention is all you need" have completely different functions. Specifically: the multi-head attention mechanism module in the decoder part of the Transformer structure in the paper "Attention is all you need" inputs a sequence of a single modality, calculates the attention score through linear transformation of the query, key and value, and summarizes the input information weightedly according to the attention score, thereby capturing the dependencies within the sequence; while the multi-head attention module in this application introduces a shared learnable matrix M to separately process the embeddings from the two modalities (i.e., unimodal text embeddings and unimodal image embedding ), which enables the radiology report generation network described in this application to use the multi-head attention module to realize cross-modal information interaction, and then use the gated fusion module to obtain single-modal features with fused multi-modal information; in addition, when the multi-head attention module in this application performs attention calculation, since only the learnable matrix is ​​selected in this application Medium and Unimodal Text Embedding The most similar vectors are used for information interaction, and only learnable matrices are selected Medium and Unimodal Image Embedding The most similar vectors for information exchange instead of using a learnable matrix All vectors in the above setting can effectively ensure that the matrix can be learned Some basic vectors in the matrix space will not be over-updated, which effectively avoids the problem of overfitting or excessive attention to certain features caused by the interaction between vectors at all positions; The gated fusion module is used to embed single-modality images into E (I) and single modality fusion image representation Fusion to obtain the final image features ; The gated fusion module is also used to embed single-modality images into E (R) and unimodal fusion text representation Fusion to obtain the final text features .

[0022] In this application, the encoding unit in the Transformer report generation module is used to generate the final image features output by the gated fusion module in the cross-modal alignment module. (Final image features The final image features including N patches are expressed as , ,in, is the final image feature of the Nth patch) to obtain the encoded image representation L ( L=l 1 ,l 2 ,l 3 ,...,l N ,in, l N represents the image representation of the Nth patch); The decoding unit is used to represent the encoded image L And the final text features output by the current gated fusion module (Final text features The final text features including T-1 characters, the final text features ,in, Decode the final text feature of the T-1th character) and output the decoded representation; The prediction head is used to perform a linear transformation on the decoded representation output by the decoding unit and apply a Softmax operation to obtain the probability distribution of each character in the dictionary (in this application, the dictionary is composed of characters in real reports in the training set), and output the character with the highest probability, which is the next predicted character. In this application, the prediction head includes a linear layer and a Softmax layer connected in sequence.

[0023] In this embodiment, the next character is predicted based on all the characters in the current radiology report using the radiology report generation network described in this application. The initial content of the radiology report is a preset sentence marker character. That is to say, at the beginning, the embedding layer of the radiology report generation network of this application is used to obtain the embedding features of the preset sentence marker character, and then the embedding features of the preset sentence marker character are input into the multi-head attention module. After the next character is obtained using the radiology report generation network of this application, the embedding features of the preset sentence marker character and the next character of the preset sentence marker character in the current radiology report are input into the multi-head attention module, and the radiology report generation network of this application is used to obtain the next character of the next character. The cycle is repeated until the preset end character is obtained to obtain the predicted radiology report corresponding to the X-ray image.

[0024] In this application, the network structure of the multi-scale local sparse attention module is as follows: Figure 2 As shown, it includes four linear layers, of which the input of three linear layers is the image region feature map , image region feature map After linear transformation of three linear layers, we get query, key and value respectively. The calculation method of query, key and value is shown in formula (1).

[0025] (1) In formula (1), Q represents query, K represents key, and V represents value. , , All represent trainable parameters, Represents the image region feature map.

[0026] In order to obtain the semantic information of different regions of radiological images, this application further divides the query, key and value into several attention heads, and then, according to the number of attention heads Split the query into indivual Sub-vector, split the key into indivual Sub-vectors, splitting the value into indivual subvector, Subvector, Subvectors and The characteristic dimensions of the sub-vectors become ,in, express Subvector, Subvectors and The characteristic dimension of the sub-vector, Subvector, Subvectors and The feature dimensions of the sub-vectors are of the same size, Represents the feature dimensions of query, key, and value. The feature dimensions of query, key, and value are consistent in size. In this application, To be An integer that is divisible.

[0027] Then, in order to calculate the local sparse self-attention, this application uses a sliding window operation from Subvector and Part of the patch is selected from the sub-vector to calculate the scaled dot product attention. Specifically: Since images usually have spatial structures, in this application, Subvector and In the sub-vector, each patch is centered, and the patches around it (i.e. the patch at the center) are selected by setting the hyperparameters of the sliding window operation, namely the convolution kernel size k and the expansion rate r, to obtain the new key and the new value ; Different dilation rates are set for different attention heads in this application. In this embodiment, the number of attention heads is four, and the dilation rates r of the four attention heads are set to 1, 2, 3 and 4 respectively, and k is uniformly set to 3, so that different attention heads can obtain semantic information of radiological images at different scales; Then, calculate Subvectors and new keys The similarity between them is then divided by the scaling factor to obtain the scaled similarity. The scaled similarity value is converted into an attention weight through a softmax operation; then, the attention weight is combined with the new value Multiply them together to get the weighted output , as shown in formula (2), the weighted output That is, it is an image feature that contains local semantic information.

[0028] (2) In formula (2), T represents the transposition operation, express Subvector, Subvectors and The characteristic dimension of the sub-vector, Subvector, Subvectors and The feature dimensions of the sub-vectors have the same size.

[0029] In this application, first calculate Subvectors and new keys The similarity between them is then divided by the scaling factor to obtain the scaled similarity. The scaled similarity value is converted into an attention weight through a softmax operation; then, the attention weight is combined with the new value Multiplying, the above method realizes the calculation of scaled dot product attention.

[0030] Since different attention heads are set with different expansion rates in this application, different attention heads can obtain semantic information of radiological images at different scales. The Concat layer is used for splicing, and then the last linear layer is used for feature fusion. The above method can effectively enhance the features of each patch. After the last linear layer is used for feature fusion, the image region features are obtained. That is, the image region features containing rich multi-scale semantic information, and then, the image region features containing multi-scale semantic information Flatten in the spatial dimension (spatial dimension size is H×W) to get the unimodal image embedding ,in N Indicates the number of patches. N The size is , Represents unimodal image embedding The characteristic dimension of The feature dimension size represented by The feature dimensions represented are equal. Since the image region features is an image region feature containing rich multi-scale semantic information. Therefore, Flattened unimodal image embedding It also contains rich multi-scale semantic information, that is, single-modal image embedding Embedding unimodal images with rich multi-scale semantic information.

[0031] In this application, unimodal embedding After entering the multi-head attention module, the multi-head attention module first obtains the attention score in the attention operation ; Then, through the attention score Finding the learnable matrix Medium and Unimodal Embedding E The most similar preceding vector, we can get a new attention score and a new value embedded , the new attention score A * The calculation method is as shown in formula (3): (3) In formula (3), Represents unimodal embedding, i.e. unimodal embedding Can represent unimodal image embeddings E (I) Or unimodal text embedding E (R) , , All represent trainable parameters, T represents the transposition operation, dRepresents unimodal embedding The characteristic dimension of New value embedding The calculation method is as shown in formula (4): (4) In formula (4), represents the learnable matrix, represents trainable parameters; Based on the new attention score A * And the new value embedding Calculate attention and obtain single-modal fusion representation , as shown in formula (5): (5) In formula (5), the single modality fusion representation Represents single modality fusion image representation Or unimodal fusion text representation When the new attention score A * It is based on unimodal image embedding E (I) Calculated, then based on the new attention score A * Computed single-modal fusion representation It represents the single modality fusion image representation ; When the new attention score A * It is based on unimodal image embedding E (R) Calculated, then based on the new attention score A * Computed single-modal fusion representation It represents the unimodal fusion text representation .

[0032] In this application, the network structure of the gated fusion module is as follows: Figure 3 As shown, Figure 3 middle, represents an element-by-element multiplication module. The gated fusion module includes two linear layers. The input end of the first linear layer is respectively connected to the output end of the multi-scale local sparse attention module and the output end of the embedding layer. The input end of the second linear layer is connected to the output end of the multi-head attention module. The output ends of the two linear layers are both connected to the first Add layer. The output end of the first Add layer is connected to the Sigmoid activation layer. The output end of the Sigmoid activation layer is respectively connected to the first element-by-element multiplication module and the second element-by-element multiplication module. The input end of the first element-by-element multiplication module is also connected to the input end of the first linear layer; the input end of the first linear layer and the input end of the second linear layer are both connected to the input end of the second Add layer, and the output end of the second Add layer is sequentially connected to the tanh activation layer and the second element-by-element multiplication module; the output end of the first element-by-element multiplication module and the input end of the second linear layer are both connected to the input end of the third Add layer, and the output end of the third Add layer and the output end of the second element-by-element multiplication module are both connected to the fourth Add layer.

[0033] In this application, the functions of each submodule in the gated fusion module are: In the gated fusion module, the first linear layer is used to embed the input unimodal E (Unimodal embedding E Image embedding output by the multi-scale local sparse attention module E (I) Or the text embedding output by the embedding layer E (R) ) performs a linear transformation operation; the second linear layer is used to represent the single-modal fusion output of the multi-head attention module (Single modality fusion representation Single modality fusion image representation Or unimodal fusion text representation ) performs a linear transformation operation; the setting of the first linear layer and the second linear layer can facilitate single-mode embedding E and single modality fusion representation Mapping to the same space makes it easier for the first Add layer to add features to achieve information integration; the first Add layer is used to add the features output by the first linear layer and the features output by the second linear layer to obtain a single-mode feature with fused multi-modal information; the Sigmoid activation layer is used to perform nonlinear compression operations on the features output by the first Add layer to obtain weights of different features; the first element-by-element multiplication module is used to embed the single-mode E The weights of the Sigmoid activation layer are multiplied element by element to obtain a single-modal feature with weighted fusion information characteristics; the second Add layer is used to represent the single-modal fusion And unimodal embedding EDirectly add features; the tanh activation layer is used to perform nonlinear transformation operations on the features output by the second Add layer to obtain unimodal features; the second element-by-element multiplication module is used to perform element-by-element multiplication of the unimodal features output by the tanh activation layer and the weights output by the Sigmoid activation layer to obtain the unimodal features after weight adjustment; the third Add layer is used to embed the unimodal E The fourth Add layer is used to add the features output by the second Add layer and the features output by the third Add layer to obtain the final single-modal features of deep information interaction. F .

[0034] In this application, since the learnable matrix M must be selected to be consistent with the unimodal text embedding The most similar Vectors are used for information interaction, and a learnable matrix should also be selected Medium and Unimodal Image Embedding The most similar Vectors are used for information interaction. During the training of the radiology report generation network, under the supervision of the total loss and the update of the network parameters, the learnable matrix M contains the cross-modal information of the unimodal text embedding and the unimodal image embedding. Therefore, the multi-head attention module is based on the learnable matrix M and the unimodal image embedding. The obtained single modality fusion image representation It contains cross-modal information and single-modal fusion image representation. and image embedding E (I) After the linear transformation operation of the linear layer, the first Add layer is used to add the features, and the obtained unimodal features are unimodal features with fused multimodal information; In this application, the final single-mode feature F The calculation method is as shown in formula (6): (6) In formula (6), F Represents the final unimodal feature, the final unimodal feature F Can represent the final image features Or the final text feature , Represents unimodal embedding, unimodal embedding Can represent unimodal image embeddings E (I) Or unimodal text embedding E (R) , Represents single-modal fusion representation, single-modal fusion representation Can represent the final image features Or the final text feature , represents element-wise multiplication, and tanh represents the activation function. E Embedding for unimodal images E (I) , Single modality fusion representation Single modality fusion image representation When , the gated fusion module outputs the final image features ; When unimodal embedding E Embedding for unimodal text E (R) , Single modality fusion representation Fusion of text representations for unimodal When , the gated fusion module outputs the final text feature .

[0035] In formula (6), G represents the weight of the output of the Sigmoid activation layer. The calculation method of G is shown in formula (7): (7) In formula (7), is the sigmoid function, Represents single-modal fusion representation, single-modal fusion representation Can represent the final image features Or the final text feature , Represents unimodal embedding, unimodal embedding Can represent unimodal image embeddings E (I) Or unimodal text embedding E (R) , and Both represent trainable parameters; in formula (10), the unimodal embedding E Embedding for unimodal images E (I) When , the single modality fusion representation Single modality fusion image representation ; When unimodal embedding E Embedding for unimodal text E (R) When , the single modality fusion representation Unimodal fusion text representation .

[0036] In the present application, the encoding unit includes three sequentially connected Transformer encoder layers, each Transformer encoder layer includes a multi-head self-attention mechanism and a feedforward neural network, and the parameters of each Transformer encoder layer are set to 8 attention heads and 512-dimensional hidden states; the decoding unit includes three sequentially connected Transformer decoder layers, each Transformer decoder layer includes two parts: a multi-head self-attention mechanism and a feedforward neural network, and the parameters of each Transformer decoder layer are set to 8 attention heads and 512-dimensional hidden states.

[0037] S2. Total loss of building a radiology report generation network : This application will predict all the characters in the order in which they are obtained. As the predicted report sequence generated by the radiology report generation network model, the corresponding radiology report sample sequence is , predict report sequence And the corresponding radiology report sample sequence The cross entropy loss between And multi-label contrast loss The loss constitutes the total loss of the radiology report generation network , total loss , cross entropy loss And multi-label contrast loss The relationship between them is shown in formula (8): (8) The multi-label contrast loss constructed in this application , aims to establish a closer semantic connection between the features of the two modalities, image features and text features, and realize supervised cross-modal information fusion and semantic alignment. In this application, multi-label contrast loss The calculation method of is shown in formula (9);

[0038] (9) In formula (9), represents the cosine similarity, is the boundary value, which is 0.7 in this embodiment. The final image features representing the image anchor points, The final image features representing the negative samples of the image, The final text feature representing the text anchor, Represents the final text features of text negative samples.

[0039] This application calculates multi-label contrast loss When performing the training, first, use the CheXpert tool (from "Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison.") to generate pseudo labels for each X-ray image-text pair sample used for training. Since each X-ray image-text pair sample may contain multiple disease labels at the same time, the pseudo labels set in this application are represented as one-hot vectors of multiple labels. The current X-ray image sample is used as the image anchor point, and then the sample with the highest cosine similarity to the anchor point is selected from the text samples with completely different labels from the current X-ray image sample as the text negative sample. Then, the current text sample (the current text sample and the X-ray image sample in the previous text belong to an X-ray image-text pair sample) is used as the text anchor point, and then the sample with the highest cosine similarity to the anchor point is selected from the X-ray image samples with completely different labels from the current text sample as the image negative sample. Then, based on the determined negative samples, the multi-label contrast loss is calculated using formula (9).

[0040] In the present application formula (9), the cross entropy loss The calculation method is as shown in formula (10): (10) In formula (10), Indicates the number of characters in the report. represents the i-th character in the radiology report sample, Represents the i-th character predicted by the model.

[0041] S3. Train the radiology report generation network based on the training set in the IU-Xray dataset and the total loss of the radiology report generation network to obtain a radiology report generation network model; specifically: The paired images of the IU-Xray dataset and their corresponding radiology report samples are input into the radiology report generation network for forward propagation, and the total loss of the radiology report generation network is calculated. , and in the total loss Under the guidance of , back propagation is performed to update the weight parameters of the radiology report generation network, and the training process of one epoch is completed. After iterating the training process of 50 epochs, the training of the radiology report generation network is completed, and the radiology report generation network model is obtained. In this application, the paired images of the IU-Xray dataset and their corresponding radiology report samples are input into the radiology report generation network to train the radiology report generation network. This is a conventional training method. For example, the paper "Cross-modal memory networks for radiology report generation" discloses this training method.

[0042] In this embodiment, the training process of the radiology report generation network uses the Adam optimizer to optimize the loss gradient. During the training process, the learning rate of the pre-trained DenseNet121 network in the radiology report generation network is set to 1e -4 The learning rate of all trainable parts of the radiology report generation network except the pre-trained DenseNet121 network is set to 5e -4 , the μ in the first μ most similar vectors is set to 64.

[0043] S4. Input the X-ray image for which the radiology report is to be generated into the radiology report generation network model obtained in step S3, and forward propagate once to obtain a predicted radiology report.

[0044] Embodiment 2: The difference between the second embodiment and the first embodiment is that in step S3, the radiology report generation network is trained based on the training set of the MIMIC-CXR data set and the total loss of the radiology report generation network to obtain a radiology report generation network model; specifically: Input a single image of the MIMIC-CXR dataset and its corresponding radiology report sample into the radiology report generation network for forward propagation and calculate the total loss of the radiology report generation network. , and in the total loss Back propagation is performed under the guidance of , the weight parameters of the radiology report generation network are updated, and an epoch training process is completed. After 30 epochs of training process, the training of the radiology report generation network is completed, and the radiology report generation network model is obtained. In this application, a single image of the MIMIC-CXR dataset and its corresponding radiology report sample are input into the radiology report generation network to train the radiology report generation network, which is also a conventional training method. For example, the paper "Cross-modal memory networks for radiology report generation" discloses this training method.

[0045] In this embodiment, the training process of the radiology report generation network uses the Adam optimizer to optimize the loss gradient. During the training process, the learning rate of the pre-trained DenseNet121 network in the radiology report generation network is set to 5e -5 The learning rate of all trainable parts of the radiology report generation network except the pre-trained DenseNet121 network is set to 1e -4 , the μ in the first μ most similar vectors is set to 128.

[0046] Test 1: In order to verify the excellent effect of the radiology report generation method of the present invention compared with other radiology report generation methods, this application uses the test set in the IU-Xray data set and the radiology report generation network model obtained in Example 1 to test six existing radiology report generation methods, including the CoATT method (from "On the automatic generation of medical imaging reports"), the HRGR method (from "Hybrid retrieval-generation reinforced agent formedical image report generation"), the CMAS-RL method (from "Show, describe and conclude: On exploiting the structure information of chest X-ray reports"), the SENTSAT+KG method (from "When radiology report generation meets knowledge graph"), the CMCL method (from "Competence-based multimodal curriculum learning for medical report generation") and the R2GenCMN method (from "Cross-modal memory networks for radiology report generation"), and the test results are shown in Table 1.

[0047] Table 1 Test results of different methods on the test set in the IU-Xray dataset and the test set in the MIMIC-CXR dataset

[0048] Test 2: In order to verify the excellent effect of the radiology report generation method of the present invention compared with other radiology report generation methods, this application uses the test set in the MIMIC-CXR dataset and the radiology report generation network model obtained in Example 2 to test six existing radiology report generation methods, including the ST method (from "A hierarchical approach for generating descriptive image paragraphs"), the ATT2IN method (from "Self-critical sequence training for image captioning"), the ADAATT method (from "Knowing when to look: Adaptive attention via avisual sentinel for image captioning"), the TopDown method (from "Bottom-up and top-down attention for image captioning and visual question answering"), the CMCL method (from "Competence-based multimodal curriculum learning for medical report generation") and the R2GenCMN method (from "Cross-modal memory networks for radiology report generation"), and the test results are shown in Table 2.

[0049] Table 2 Test results of different methods on the test set in the MIMIC-CXR dataset and the test set in the MIMIC-CXR dataset

[0050] In Tables 1 and 2, BLEU-1, BLEU-2, BLEU-3, BLEU-4, ROUGE-L and METEOR are commonly used evaluation indicators for measuring text generation, and the OURS method refers to the radiology report generation method proposed in this application.

[0051] In Tables 1 and 2, the BLEU index is used to measure the degree of vocabulary matching between the model-generated text and the reference text. The higher the value, the stronger the consistency of the generated text in vocabulary and phrases; BLEU-1 to BLEU-4 represent the matching of single characters, bigrams, trigrams, and quadruple groups, respectively. The levels are gradually progressive, and the higher the value, the better the precise matching degree between the generated text and the reference text. The METEOR index comprehensively considers precision and recall, can capture the similarity between the generated text and the reference text at the semantic level, and also introduces factors such as synonym matching and word order. The higher the value, the higher the accuracy and flexibility of the generated text in semantic expression. The ROUGE-L index measures the structural similarity between the generated text and the reference text. It is calculated based on the longest common subsequence. The higher the value, the closer the generated text is to the reference text in terms of overall structure and coherence at the sentence level. Overall, these indicators evaluate the quality of the generated text from the lexical, semantic, and structural levels, respectively, and provide a multi-angle reference for evaluating the generation effect of the model. Ours represents the method for generating radiology reports based on semantic alignment described in this application. In this application, an English word is regarded as a character.

[0052] Compared with other existing radiology report generation methods disclosed in this application, the existing R2GENCMN method is closer to the radiology report generation method described in this application. Therefore, this application focuses on comparing the test results of the R2GENCMN method with the test results of the radiology report generation method described in this application, as follows: As can be seen from Table 1, when testing based on the test set in the IU-Xray dataset: The BLEU-1 index obtained by the radiology report generation method described in the present application is 8.46% higher than the BLEU-1 index obtained by the R2GENCMN method, which shows that the radiology report generation method described in the present application shows stronger accuracy in matching single characters, can generate more content consistent with single characters of the reference text, and improves the generation quality at the single character level; The BLEU-2 index obtained by the radiology report generation method described in the present application is 14.97% higher than the BLEU-2 index obtained by the R2GENCMN method, which shows that the radiology report generation method described in the present application performs better in bigram matching, can generate more accurate phrase combinations, and enhances the expressive power of the text (i.e., the radiology report) at the phrase level; The BLEU-3 index obtained by the radiology report generation method described in this application is 15.38% higher than the BLEU-3 index obtained by the R2GENCMN method, which shows that the radiology report generation method described in this application is optimized in triple matching, and the generated text (i.e., radiology report) is more coherent in syntax and semantics, reflecting a higher phrase matching accuracy; The BLEU-4 index obtained by the radiology report generation method described in this application is 10.97% higher than the BLEU-4 index obtained by the R2GENCMN method, which shows that the method of this application performs well in quadruple matching, and the generated text (i.e., radiology report) has a stronger advantage in higher-level phrase collocation and semantic consistency, further improving the quality of the generated report; The METEOR index obtained by the radiology report generation method described in the present application is 19.79% higher than the METEOR index obtained by the R2GENCMN method, which indicates that the text (i.e., radiology report) generated by the radiology report generation method described in the present application is more accurate and flexible in semantic expression and overall expression; The ROUGE-L index obtained by the radiology report generation method described in the present application is 6.98% higher than the ROUGE-L index obtained by the R2GENCMN method, which indicates that the text (i.e., radiology report) generated by the radiology report generation method described in the present application is closer to the radiology report sample text in terms of sentence structure and coherence, and can better maintain the overall logic and semantic integrity of the radiology report.

[0053] The MIMIC-CXR dataset used in this application is a large dataset, which is almost 70 times larger than the IU-Xray dataset. The data complexity and label noise are significantly increased compared with the IU-Xray dataset. As can be seen from Table 2, when this application is tested based on the test set in the MIMIC-CXR dataset, The BLEU-1 index obtained by the radiology report generation method described in the present application is 0.88% higher than the BLEU-1 index obtained by the R2GENCMN method, which means that when tested on the test set in the MIMIC-CXR dataset (large dataset), the radiology report generation method described in the present application shows stronger accuracy in matching single characters, and can generate more content consistent with single characters of the reference text, thus improving the generation quality at the single character level; The BLEU-2 index obtained by the radiology report generation method described in the present application is 1.88% higher than the BLEU-2 index obtained by the R2GENCMN method, which shows that when tested on the test set in the MIMIC-CXR dataset (large dataset), the radiology report generation method described in the present application performs better in bigram matching, can generate more accurate phrase combinations, and enhances the expressive power of the text (i.e., radiology report) at the phrase level; The BLEU-3 index obtained by the radiology report generation method described in the present application is 2.78% higher than the BLEU-3 index obtained by the R2GENCMN method, which indicates that when tested on the test set in the MIMIC-CXR dataset (large dataset), the radiology report generation method described in the present application is optimized in triple matching, and the generated text (i.e., radiology report) is more coherent in syntax and semantics, reflecting a higher phrase matching accuracy; The BLEU-4 index obtained by the radiology report generation method described in this application is 3.85% higher than the BLEU-4 index obtained by the R2GENCMN method, which shows that when tested on the test set in the MIMIC-CXR dataset (large dataset), the method of this application performs well in quadruple matching, and the generated text (i.e., radiology report) has a stronger advantage in higher-level phrase collocation and semantic consistency, further improving the quality of the generated report; The METEOR index obtained by the radiology report generation method described in the present application is 4.44% higher than that obtained by the R2GENCMN method, which indicates that when tested on the test set in the MIMIC-CXR dataset (large dataset), the text (i.e., radiology report) generated by the radiology report generation method described in the present application is better in accuracy and flexibility in semantic expression and overall expression; The ROUGE-L index obtained by the radiology report generation method described in the present application is 2.53% higher than the ROUGE-L index obtained by the R2GENCMN method. This shows that when tested on the test set in the MIMIC-CXR dataset (large dataset), the text (i.e., radiology report) generated by the radiology report generation method described in the present application is closer to the radiology report sample text in terms of sentence structure and coherence, and can better maintain the overall logic and semantic integrity of the radiology report.

Claims

1. A method for generating radiology reports based on semantic alignment, characterized in that: The following steps are involved: S1. Constructing a radiology report generation network, the radiology report generation network includes a multi-scale feature extraction module, a cross-modal semantic alignment module and a Transformer report generation module connected in sequence; the multi-scale feature extraction module includes a convolutional neural network and a multi-scale local sparse attention module connected in sequence; The cross-modal semantic alignment module includes an embedding layer and a multi-head attention module and a gated fusion module connected to the embedding layer, and the input ends of the multi-head attention module and the gated fusion module are also connected to the output end of the multi-scale local sparse attention module; Convolutional neural network is used to extract the features of all patches in radiological images and obtain image region feature maps. ; Multi-scale local sparse attention module for enhancing image region feature maps The features of each patch in , obtain a unimodal image embedding containing multi-scale semantic information E (I) ; The embedding layer is used to perform embedding mapping operations on the radiology report samples to obtain unimodal text embedding E (R) ; Multi-head attention module based on learnable matrix M and unimodal embedding E Perform attention calculation to obtain a unimodal fusion representation containing cross-modal information , the unimodal embedding E Embedding for unimodal images E (I) Or unimodal image embedding E (R) , the single modality fusion representation Single modality fusion image representation Or single modality fusion image representation ; The gated fusion module is used to embed the single modality into E and single modality fusion image representation Fusion is performed to obtain the final unimodal feature F, which is the final image feature Or the final image features ; S2. Total loss of building a radiology report generation network , where the prediction report sequence And the corresponding radiology report sample sequence The cross entropy loss between And multi-label contrast loss The loss constitutes the total loss of the radiology report generation network ; S3, training the radiology report generation network based on the training set and the total loss of the radiology report generation network to obtain a radiology report generation network model; S4. Input the radiological image for which the radiological report is to be generated into the radiological report generation network model, forward propagate once, and obtain the predicted radiological report.

2. The method for generating radiology reports based on semantic alignment according to claim 1, characterized in that: In step S1, the multi-scale local sparse attention module includes four linear layers, and the inputs of three linear layers are image region feature maps , image region feature map After linear transformation through three linear layers, we get query, key and value respectively. Then, we divide query, key and value into several attention heads respectively. Then, according to the number of attention heads, Split the query into indivual Subvectors, split the key into indivual Subvectors, splitting values ​​into indivual sub-vector; and then through the sliding window operation from Subvector and Select some patches in the sub-vector to calculate the scaled dot product attention and get the weighted output ; Then, all the output The Concat layer is used for splicing, and then the feature fusion is performed through the last linear layer to obtain the image region features containing rich multi-scale semantic information. , and then, the image region features containing multi-scale semantic information Flattening in the spatial dimension to obtain a unimodal image embedding containing rich multi-scale semantic information .

3. The method for generating radiology reports based on semantic alignment according to claim 2, characterized in that: In step S1, a sliding window operation is performed from Subvector and Selecting some patches in the sub-vector for calculating the scaled dot product attention includes the following steps: Subvector and In the sub-vector, each patch is centered and the surrounding patches are selected by setting the hyperparameters of the sliding window operation, namely the convolution kernel size k and the expansion rate r, and a new key is obtained. and the new value ; Different attention heads are set with different expansion rates; then, calculate Subvectors and new keys The similarity between them is then divided by the scaling factor to obtain the scaled similarity. The scaled similarity value is converted into an attention weight through a softmax operation; then, the attention weight is combined with the new value Multiply them together to get the weighted output , the weighted output That is, the image features containing local semantic information are used to calculate the scaled dot product attention.

4. The method for generating radiology reports based on semantic alignment according to claim 1, characterized in that: In step S1, the gated fusion module includes two linear layers, the input end of the first linear layer is respectively connected to the output end of the multi-scale local sparse attention module and the output end of the embedding layer, the input end of the second linear layer is connected to the output end of the multi-head attention module, the output ends of the two linear layers are both connected to the first Add layer, the output end of the first Add layer is connected to the Sigmoid activation layer, the output end of the Sigmoid activation layer is respectively connected to the first element-by-element multiplication module and the second element-by-element multiplication module, and the input end of the first element-by-element multiplication module is also connected to the input end of the first linear layer; the input end of the first linear layer and the input end of the second linear layer are both connected to the input end of the second Add layer, and the output end of the second Add layer is sequentially connected to the tanh activation layer and the second element-by-element multiplication module; the output end of the first element-by-element multiplication module and the input end of the second linear layer are both connected to the input end of the third Add layer, and the output end of the third Add layer and the output end of the second element-by-element multiplication module are both connected to the fourth Add layer.

5. The method for generating radiology reports based on semantic alignment according to claim 1, characterized in that: In step S1, the Transformer report generation module includes a coding unit, a decoding unit and a prediction head connected in sequence, and the coding unit and the decoding unit are also connected to the output end of the gated fusion module in the cross-modal semantic alignment module; wherein the coding unit is used to output the final image features of the gated fusion module. Encode to obtain the encoded image representation L; the decoding unit is used to decode the image representation L and the final text features output by the current gated fusion module Perform a decoding operation and output a decoded representation; the prediction head is used to perform a linear transformation on the decoded representation output by the decoding unit and apply a Softmax operation to output the character with the highest probability, which is the predicted next character.

6. The method for generating radiology reports based on semantic alignment according to claim 5, characterized in that: In step S1, the encoding unit includes three sequentially connected Transformer encoder layers, each of which contains a multi-head self-attention mechanism and a feedforward neural network. The parameters of each Transformer encoder layer are set to 8 attention heads and 512-dimensional hidden states.

7. The method for generating radiology reports based on semantic alignment according to claim 5, characterized in that: In step S1, the decoding unit includes three sequentially connected Transformer decoder layers, each of which includes two parts: a multi-head self-attention mechanism and a feedforward neural network. The parameters of each Transformer decoder layer are set to 8 attention heads and 512-dimensional hidden states.

8. The method for generating radiology reports based on semantic alignment according to claim 1, characterized in that: In step S2, the multi-label contrast loss is calculated When training, first, use the CheXpert tool to generate pseudo labels for each X-ray image-text pair sample used for training. Since each X-ray image-text pair sample may contain multiple disease labels at the same time, the pseudo labels set are represented as one-hot vectors of multiple labels. The current X-ray image sample is used as the image anchor point, and then the sample with the highest cosine similarity to the anchor point is selected from the text samples with completely different labels from the current X-ray image sample as the text negative sample. Then, the current text sample is used as the text anchor point, and then the sample with the highest cosine similarity to the anchor point is selected from the X-ray image samples with completely different labels from the current text sample as the image negative sample; then the multi-label contrast loss is calculated based on the determined negative samples.

9. The method for generating radiology reports based on semantic alignment according to claim 1, characterized in that: In step S3, the images in the training set and their corresponding radiology report samples are input into the radiology report generation network for forward propagation, and the total loss of the radiology report generation network is calculated. , and in the total loss Back propagation is performed under the guidance of , the weight parameters of the radiology report generation network are updated, the training process of one epoch is completed, and the iterative training is repeated to obtain the radiology report generation network model.

10. The method for generating radiology reports based on semantic alignment according to claim 9, characterized in that: In step S3, the training set is the training set in the IU-Xray dataset or the training set in the MIMIC-CXR dataset; when the radiology report generation network is trained based on the training set in the IU-Xray dataset, the paired images of the IU-Xray dataset are input into the radiology report generation network, and μ in the most similar first μ vectors is set to 64; when the radiology report generation network is trained based on the training set in the MIMIC-CXR dataset, a single image of the MIMIC-CXR dataset is input into the radiology report generation network, and μ in the most similar first μ vectors is set to 128.

Images