Chest DR imaging report generation method based on visual language feature comparison

Abstract

The present invention discloses a method for generating chest DR image reports based on visual language feature comparison. First, chest DR images containing lesion and non-lesion instances and corresponding reports are obtained to form anchor instances, positive instances, and negative instances of the training data set; then, the convolution sequence features of each instance image are extracted through a convolutional neural network to obtain visual coding features; then, the anchor instances, positive instances, and negative instances are encoded separately to obtain language coding features; further, the visual coding features and language coding features of the anchor instances, positive instances, and negative instances containing lesion instances and non-lesion instances are compared to optimize the network; finally, in the testing phase, the report can be generated by simply inputting the visual coding features of the image into a text generation decoder. The present invention can effectively enhance the model's representation of lesions by comparing the visual features and language features between different instances, thereby improving the performance of lesion instance report generation.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a method for generating a chest DR image report based on visual language feature comparison. Background Art

[0002] As a low-cost diagnostic imaging tool for chest diseases, chest DR imaging provides doctors with an important diagnostic basis. As the number of DR images continues to grow, the use of artificial intelligence (AI) to assist in the writing of DR imaging reports is expected to improve the efficiency of radiologists in writing reports.

[0003] In recent years, deep learning-based image captioning models have demonstrated strong performance in various image captioning tasks. The chest DR imaging report generation task can be considered an image captioning task in the medical field. However, the widespread imbalance in data distribution in medical imaging poses a challenge to the performance of deep learning models. Specifically, the number of lesion-free cases acquired daily is often far greater than the number of lesion cases. This data imbalance results in sentences describing normal and lesion-free conditions dominating the DR imaging report generation dataset, while sentences describing lesions are relatively rare. This imbalance in data distribution will cause deep learning models to tend to generate reports describing normal conditions with no significant lesions.

[0004] However, in practical applications, accurate identification and description of lesion information is often more important. Therefore, how to alleviate the problem of uneven data distribution and improve the performance of deep learning models in generating reports based on lesion instances is an important issue that needs to be addressed. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the existing technology and propose a chest DR imaging report generation method based on visual and language feature comparison. By comparing the visual and language features between lesion instances and non-lesion instances, effective visual and language expressions of lesion features are learned, thereby alleviating the problem of uneven data distribution and improving the performance of lesion instance report generation.

[0006] In order to solve the above technical problems, the technical solution of the present invention is:

[0007] A chest DR imaging report generation method based on visual language feature comparison, the specific steps are as follows:

[0008] Step 1: Construct different types of training data. The first type uses non-lesion instances as anchor instances and positive instances, and lesion instances as negative instances. The other type uses lesion instances with the same medical subject heading as anchor instances and positive instances, and non-lesion instances as negative instances. Each instance contains a chest DR image I and a reference report R.

[0009] Step 2: Perform image and text augmentation on all instances constructed in Step 1 to obtain more training data. Image augmentation includes random cropping, and text augmentation includes random synonym replacement and random deletion of non-lesion description sentences.

[0010] Step 3: Input the DR image into the convolutional neural network to extract the convolution feature map of the image, and expand it to obtain the convolution sequence feature. Among them, N is the length of the convolution sequence feature, d model is the characteristic dimension of the model;

[0011] Step 4: Concatenate the convolution sequence features with the special CLS class vector to obtain the sequence features And input the self-attention encoding network encoding to obtain the image visual encoding features Among them, the self-attention encoding network includes a multi-head attention network and a forward propagation linear network. The convolution sequence feature E passes through the multi-head attention network and is then input into the forward propagation linear network to obtain the visual encoding feature C:

[0012] Step 5: Transform the visual encoding features of the anchor instance C = {c, v1, v2, ..., v N} Input the Transformer-based text generation decoder to generate the corresponding report text, where the text generation decoder contains a masked self-attention network and a cross-attention network. The visual encoding feature C and the report text R generated before the current time t are combined. t-1 Common input to the text generation decoder output hidden state H = {h1, h2, ..., h t-1}; Then, take the hidden state h at the adjacent time t-1 t-1 The report text output r at the current time t is obtained through linear mapping and softmax operation t ;

[0013] Step 6: Input the generation report of the anchor instance and the reference reports of the positive and negative instances into the language feature encoder to obtain the corresponding language encoding features. The language feature encoder is a pre-trained BERT-based language feature encoder.

[0014] Step 7: Compare the visual coding feature comparison vectors c corresponding to the CLS class vectors in the visual coding features C of the anchor instance, positive instance, and negative instance to shorten the distance between the positive instance and the anchor instance and increase the distance between the anchor instance and the negative instance.

[0015] Step 8: Compare the language encoding feature vectors of the anchor instance, positive instance, and negative instance to narrow the distance between the positive instance and the anchor instance and widen the distance between the anchor instance and the negative instance.

[0016] Step 9: Use the two sets of data from the training dataset in step 1 for joint training, and compare the visual and language features between the lesion instances and the non-lesion instances to optimize the network parameters; specifically, use the cross entropy loss function, visual feature contrast loss function, and language feature contrast loss function generated by the report to jointly optimize the model;

[0017] Step 10. During the test, it is only necessary to input the visual coding features corresponding to the chest DR image to be tested into the text generation decoder to generate text through an autoregressive text generation method, and then concatenate the text generated before the end character to obtain a complete report.

[0018] The present invention has the following characteristics and beneficial effects:

[0019] Using the above technical solution, the present invention constructs two different sets of positive, anchor, and negative data containing lesion and non-lesion instances for model training. The lesion and non-lesion instances are then compared using visual and linguistic features, narrowing the distance between the anchor and positive instances and widening the distance between the anchor and negative instances. This approach alleviates the imbalanced distribution of lesion and non-lesion samples in report generation datasets. Experiments conducted on two public datasets demonstrate that by comparing lesion and non-lesion instances, the chest DR imaging report generation method based on visual and linguistic feature comparison can achieve better lesion feature representation, thereby better generating reports for lesion instances and providing doctors with more effective and accurate diagnostic evidence. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 It is a schematic flow diagram of the method of the present invention;

[0022] Figure 2 It is a detailed framework diagram of the method model of the present invention;

[0023] Figure 3 are two sets of training examples constructed by the method of the present invention;

[0024] Figure 4 This is a demonstration of the effect of the method of the present invention on the IU X-Ray public dataset.

[0025] Figure 5 This is a demonstration of the effect of the method of the present invention on the MIMIC-CXR public dataset. DETAILED DESCRIPTION

[0026] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0027] The present invention provides a chest DR image report generation method based on visual language feature comparison, such as Figure 1 As shown in the figure, the visual and linguistic features between lesion instances and non-lesion instances are compared to enhance the representation of lesion features, alleviate the problem of data imbalance, and improve the performance of lesion instance report generation. First, chest DR images containing lesion and non-lesion instances and their corresponding reports are obtained to form the anchor instances, positive instances, and negative instances of the training dataset. Then, the convolutional sequence features of each instance image are extracted through a convolutional neural network, and self-attention encoding is performed after concatenating the class vectors to obtain the visual encoding features. Subsequently, the visual encoding features of the anchor instance are input into the text generation decoder to generate the report, and they are encoded to obtain the language encoding features for generating the anchor instance report. At the same time, the positive and negative instance reports are encoded to obtain the language encoding features. Furthermore, the visual encoding features and language encoding features of the anchor instances, positive instances, and negative instances containing lesion instances and non-lesion instances are compared to optimize the network. Finally, in the testing phase, only the visual encoding features of the image need to be input into the text generation decoder to generate the report.

[0028] Specific implementation steps include Figure 2 As shown:

[0029] Step 1: Construct different types of training data: Figure 3 As shown in (a), the first type uses non-lesioned instances as anchor instances and positive instances, and uses lesioned instances as negative instances; Figure 3 As shown in (b), another type of data uses lesion instances with the same medical subject heading as anchor instances and positive instances, and non-lesion instances as negative instances; each instance contains a chest DR image I and a reference report R;

[0030] Step 2: Perform image and text augmentation on all instances constructed in Step 1 to obtain more training data. Image augmentation includes random cropping, and text augmentation includes random synonym replacement and random deletion of non-lesion description sentences.

[0031] Step 3: Input the DR image into the convolutional neural network to extract the convolution feature map of the image, and expand it to obtain the convolution sequence feature. The specific formula is as follows:

[0032] X={x1,x2,…,x N}=Flatten(CNN(I)) Formula (1)

[0033] Among them, CNN represents convolutional neural network, Flatten represents the flattening operation of feature map, N is the length of convolution sequence feature, d model is the characteristic dimension of the model;

[0034] Step 4: Concatenate the convolution sequence features with the special CLS class vector to obtain the sequence features And input the self-attention encoding network to encode and obtain the visual encoding features of the instance image Among them, the self-attention encoding network consists of a multi-head attention network and a forward propagation linear network. The formula is as follows:

[0035] C={c,v1,...,v N}=FFN(MHA(E,E,E)) Formula (2)

[0036] Here, MHA stands for Multi-Head Attention Network, FFN stands for Feedforward Linear Network, and both MHA and FFN are followed by Residual Connection and Drop-out operations. C represents the visual encoding feature contrast vector corresponding to the CLS class vector. MHA consists of h attention heads and can be expressed as follows:

[0037] MHA(Q,K,V)=[head1;...;head h ]W O Formula (3)

[0038] head i =Attention(QW i Q , KW i K , VW i V ) Formula (4)

[0039]

[0040] in, Represents the i-th attention head i The mapping matrix, Represents the mapping matrix of the multi-head attention layer output, d k Indicates the dimensions of the query matrix Q and key-value matrix K, d v Represents the dimension of the value matrix V. The forward propagation linear layer FFN can be expressed as the following formula:

[0041] FFN(x)=max(0,xW1+b1)W2+b2 Formula (6)

[0042] Among them, W1 and W2 represent linear mapping matrices, and b1 and b2 represent biases.

[0043] Step 5: Transform the visual encoding features of the anchor instance C = {c, v1, ..., v N} Input the Transformer-based text generation decoder to generate the corresponding report text; Among them, the text generation decoder contains a masked self-attention network and a cross-attention network, and the masked attention network Layer mask It can be expressed as:

[0044] Layer mask (x)=LayerNorm(MHA(x,x,x)) Formula (7)

[0045] Cross-Attention Network Layer cross It can be expressed as:

[0046] Layer cross (x, y)=FFN(LayerNorm(MHA(x, y, y))) Formula (8)

[0047] Among them, LayerNorm represents the layer normalization operation, and the MHA and FFN in the self-attention network and the cross-attention network are followed by the residual connection operation and the Drop-out operation. The text generation decoder Decoder can be expressed as:

[0048] Decoder(x, y) = Layer cross (Layer mask (x), y) formula (9)

[0049] The hidden state output by the text generation decoder can be expressed as the following formula:

[0050] H={h1, h2, ..., h t-1}=Decoder(C,R t-1 ) Formula (10)

[0051] Then, take the hidden state h at the adjacent time t-1 t-1 , through linear mapping and softmax operation, the report text output r at the current time t is obtained t , the specific formula is as follows:

[0052] r t ~p t =softmax(h t-1 W p +b p ) Formula (11)

[0053] Among them, W p represents the linear mapping matrix, b prepresents the bias, r t Indicates the word in the report generated at the current moment, p t Represents the text word distribution output at the moment;

[0054] Step 6: Input the generation report of the anchor instance and the reference reports of the positive instance and the negative instance into the language feature encoder to obtain the corresponding language coding features; wherein the language coding feature comparison vector output by the language feature encoder can be expressed as the following formula:

[0055] l a =LanguageEncoder(R a ) Formula (12)

[0056] l p =LanguageEncoder(R p ) Formula (13)

[0057] l n =LanguageEncoder(R n ) Formula (14)

[0058] Among them, l a Represents the language encoding feature contrast vector of the anchor instance, l p Represents the language encoding feature contrast vector of the positive instance, l n is the language encoding feature comparison vector of the negative instance. LanguageEncoder represents the pre-trained language feature encoder;

[0059] Step 7: Compare the visual coding feature comparison vectors c corresponding to the CLS class vectors in the visual coding features of the anchor instance, positive instance, and negative instance to shorten the distance between the positive instance and the anchor instance and increase the distance between the anchor instance and the negative instance. The specific formula is as follows:

[0060]

[0061] Among them, c a Represents the visual encoding feature contrast vector of the anchor instance, c p Represents the visual encoding feature contrast vector of the positive instance, c n is the visual encoding feature contrast vector of the negative instance. m v Indicates the boundary value of visual feature contrast. vision represents the visual feature contrast loss function;

[0062] Step 8: Compare the language encoding features of the anchor instance, positive instance, and negative instance to narrow the distance between the positive instance and the anchor instance and widen the distance between the anchor instance and the negative instance. The specific formula is as follows:

[0063]

[0064] Among them, l a Represents the language encoding feature contrast vector of the anchor instance, l p The language encoding feature representing the positive instance, l n is the language encoding feature comparison vector of the negative instance. m l Indicates the boundary value of language feature comparison. language represents the language feature contrast loss function;

[0065] Step (9) uses the two sets of data in the training data set of step 1 for joint training, and compares the visual features and language features between the lesion instance and the non-lesion instance to optimize the network parameters; the cross entropy loss function, the visual feature contrast loss function and the language feature contrast loss function are used as the optimization function for model training:

[0066] l all =l generation +l language +l vision Formula (17)

[0067]

[0068] Among them, l all Represents the overall loss function of model training. generation The cross entropy loss function generated by the report, M represents the sequence length of the report, l language represents the language feature contrast loss function, l vision represents the visual feature contrast loss function.

[0069] Step 10. During the test, it is only necessary to input the visual coding features corresponding to the chest DR image to be tested into the text generation decoder to generate text through an autoregressive text generation method, and then concatenate the text generated before the end character to obtain a complete report.

[0070] To measure the effectiveness of the method of the present invention, qualitative and quantitative experiments were conducted on two public datasets, IU X-Ray and MIMIC-CXR. The qualitative experimental results on the two datasets are shown as follows: Figure 4 and Figure 5 As shown. Figure 5(b) The present invention can generate "The patient is in a state after median sternotomy and coronary artery bypass grafting. The heart size is slightly enlarged. The contours of the mediastinum and hilum are unchanged. The pulmonary vessels are not congested. Patchy opacities appear at the lung bases, which may reflect areas of atelectasis. There is no focal consolidation, pleural effusion or pneumothorax. There are mild degenerative changes in the thoracic spine." This is consistent with the description of the lesions covered in the reference report written by the doctor: "The patient is in a state after median sternotomy, CABG, and vascular stent placement. The heart size is slightly enlarged but stable. The contours of the mediastinum and hilum are normal, and the pulmonary vessels are not congested. The lungs are clear, without focal consolidation. No pleural effusion or pneumothorax is observed. No acute bone abnormalities are observed." This phenomenon shows that the method of the present invention can effectively improve the accuracy of the generation of corresponding lesion reports.

[0071] In terms of quantitative analysis, the quantitative comparison experimental results of the method of the present invention with the existing DR imaging report generation method on two existing public datasets IU X-Ray and MIMIC-CXR are shown in Tables 1 and 2.

[0072] Table 1 Performance comparison of the IU X-Ray dataset (the best results are marked in bold)

[0073] method BLEU-3 BLEU-4 METEOR ROUGE-L CIDEr Show-Tell 0.108 0.078 0.157 0.307 0.197 Transformer 0.147 0.136 0.168 0.317 0.310 CNN-Transformer 0.179 0.135 0.164 0.342 - <![CDATA[M 2 Transformer]]> 0.205 0.152 0.176 0.353 - HRGR 0.208 0.151 - 0.322 0.343 The present invention 0.246 0.178 0.214 0.366 0.422

[0074] Table 2 Performance comparison of MIMIC-CXR dataset (the best results are marked in bold)

[0075] method BLEU-3 BLEU-4 METEOR ROUGE-L CIDEr Show-Tell 0.125 0.088 0.122 0.256 0.096 Transformer 0.140 0.092 0.129 0.267 0.134 RACHET 0.139 0.099 0.136 0.280 0.140 CMN 0.148 0.106 0.142 0.278 - CCRG 0.139 0.100 0.133 0.274 0.136 The present invention 0.151 0.111 0.140 0.286 0.182

[0076] As can be seen from Tables 1 and 2, on two public chest DR image datasets, under various text generation evaluation indicators, the method proposed in the present invention is improved compared with the existing methods, indicating that the method of the present invention can effectively improve the report generation performance.

[0077] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It will be apparent to those skilled in the art that various changes, modifications, substitutions, and variations of these embodiments, including components, without departing from the principles and spirit of the present invention are still within the scope of protection of the present invention.

Claims

1. A chest DR image report generation method based on visual language feature comparison, characterized in that: The steps include: Step 1: Construct a dataset, wherein the dataset includes a chest DR image I and a reference report R. The dataset defines anchor instances, positive instances, and negative instances based on lesion instances. Step 2: Perform image and text augmentation on the dataset to obtain more training data; Step 3: Input the DR image into the convolutional neural network to extract the convolution feature map of the image, and expand it to obtain the convolution sequence feature X; Step 4: Concatenate the convolution sequence feature with the CLS class vector to obtain the sequence feature E and input it into the self-attention encoding network to obtain the anchor instance image visual encoding feature C; Step 5: Input the anchor instance image visual encoding feature C into the Transformer-based text generation decoder to generate the corresponding report text; Step 6: Input the generation report of the anchor instance and the reference reports of the positive instance and the negative instance into the language feature encoder to obtain the corresponding language encoding features; Step 7: Compare the visual coding feature vectors c corresponding to the CLS class vectors in the visual coding features of the anchor instance, positive instance, and negative instance to shorten the distance between the positive instance and the anchor instance and increase the distance between the anchor instance and the negative instance. In step 7, the distance between the anchor instance and the negative instance is increased. The specific formula is as follows: Among them, c a Represents the visual encoding feature contrast vector of the anchor instance, c p Represents the visual encoding feature contrast vector of the positive instance, c n is the visual encoding feature contrast vector of the negative instance, m v is the boundary value of visual feature comparison, represents the visual feature contrast loss function; Step 8: Compare the language encoding features of the anchor instance, positive instance, and negative instance to narrow the distance between the positive instance and the anchor instance and widen the distance between the anchor instance and the negative instance; In step 8, the distance between the anchor instance and the negative instance is increased. The specific formula is as follows: Among them, l a Represents the language encoding feature contrast vector of the anchor instance, l p The language encoding feature representing the positive instance, l n is the language encoding feature comparison vector of the negative instance, m l Indicates the boundary value of language feature comparison, represents the language feature contrast loss function; Step 9: Perform joint training using the training data set from step 1, and use the optimization function trained using the cross entropy loss function, the visual feature contrast loss function, and the language feature contrast loss function; The optimization function expression is as follows: in, Represents the overall loss function of model training; The cross entropy loss function generated for the report, M represents the sequence length of the report, represents the language feature contrast loss function, represents the visual feature contrast loss function; Step 10. During the test, it is only necessary to input the visual coding features corresponding to the chest DR image to be tested into the text generation decoder to generate text through an autoregressive text generation method, and then concatenate the generated text before the end character to obtain a complete report.

2. The chest DR image report generation method based on visual language feature comparison according to claim 1 is characterized in that: The dataset defines anchor instances, positive instances, and negative instances according to two types of lesion instances, thereby obtaining two sets of training data. The first type uses non-lesion instances as anchor instances and positive instances, and lesion instances as negative instances; the other type uses lesion instances with the same medical subject terms as anchor instances and positive instances, and non-lesion instances as negative instances.

3. The chest DR image report generation method based on visual language feature comparison according to claim 1 is characterized in that: The image augmentation method includes random cropping, and the text augmentation method includes random synonym replacement of reports and random deletion of non-lesion description sentences.

4. The chest DR image report generation method based on visual language feature comparison according to claim 1, characterized in that: The self-attention encoding network consists of a multi-head attention network and a forward propagation linear network, and the formula is as follows: C={c,v1,v2,...,v N }=FFN(MHA(E,E,E)) Among them, MHA represents the multi-head attention network, FFN represents the forward propagation linear network, MHA and FFN are followed by residual connection operation and Drop-out operation; c represents the visual encoding feature vector corresponding to the CLS class vector.

5. The chest DR image report generation method based on visual language feature comparison according to claim 4 is characterized in that: The multi-head attention network MHA consists of h attention heads, which are expressed as follows: MHA(Q,K,V)=[head1;...;head h ]W o in, Represents the i-th attention head i The mapping matrix, Represents the mapping matrix of the multi-head attention layer output, d k Indicates the dimensions of the query matrix Q and key-value matrix K, d v Represents the dimension of the value matrix V.

6. The chest DR image report generation method based on visual language feature comparison according to claim 4, characterized in that: The forward propagation linear network FFN is expressed as follows: FFN(x)=max(0,xW1+b1)W2+b2, where W1 and W2 represent linear mapping matrices, and b1 and b2 represent biases.

7. The chest DR image report generation method based on visual language feature comparison according to claim 1, characterized in that: The text generation decoder contains a masked self-attention network and a cross-attention network.

8. The chest DR image report generation method based on visual language feature comparison according to claim 1 is characterized in that: The text generation decoder Decoder is expressed as: Decoder(x,y)=Layer cross (Layer mask (x),y) Among them, Layer cross For the cross attention network, Layer mask is a masked attention network, where x and y represent the text sequence and image visual encoding features of the decoder input respectively; The output of the text generation decoder is the hidden state of the text feature, which is expressed as follows: H={h1,h2,...,h t-1 }=Decoder(C,R t-1 )。 9. The chest DR image report generation method based on visual language feature comparison according to claim 8, characterized in that: In step 5, the hidden state h at the adjacent time t-1 is taken t-1 , through linear mapping and softmax operation, the report text output r at the current time t is obtained t , the specific formula is as follows: r t ~p t =softmax(h t-1 W p +b p ) Among them, W p represents the linear mapping matrix, b p represents the bias, r t Indicates the word in the report generated at the current moment, p t Represents the text word distribution output at a certain moment.

10. The chest DR image report generation method based on visual language feature comparison according to claim 2, characterized in that: In step 9, during training, two sets of training data obtained according to two types of lesion instances are used for joint training, and the visual features and language features between the lesion instances and the non-lesion instances are compared to optimize the network parameters.

Images