Chest DR image report generation method based on visual language feature comparison
Through the visual language feature comparison method, the problem of insufficient lesion recognition performance caused by data imbalance in the production of chest DR image reports is solved, and more accurate lesion report generation is achieved, improving the performance and diagnostic basis of report generation. Validity.
Patent Information
- Application Number
- CN202411045347.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-01
AI Technical Summary
In the generation of chest DR image reports, deep learning models tend to generate reports that describe normal and no significant lesions due to uneven data distribution, resulting in insufficient lesion identification and description performance.
By constructing different types of training data, including lesion instances and lesion-free instances, using visual language feature comparison method, learning visual and linguistic expression of lesion features, using convolutional neural networks and Transformer to generate decoders to generate reports, and optimizing network parameters to mitigate Data imbalance problem.
The performance of the generation of lesion case reports is improved, the generated reports are more accurate and effective, providing doctors with a better diagnostic basis, and the experimental results show performance that is better than existing methods on the public data set.
Smart Images

Figure CN119069063A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of medical image processing, and in particular to a method for generating a chest DR image report based on visual language feature comparison. Background Art
[0002] As a low-cost diagnostic image for chest diseases, chest DR images provide doctors with important diagnostic evidence. Currently, the number of DR images is growing, so the use of artificial intelligence technology to assist in the writing of DR image reports is expected to improve the efficiency of radiologists in writing reports.
[0003] In recent years, deep learning-based image captioning models have demonstrated strong performance in different image captioning tasks. The chest DR imaging report generation task can be seen as an image captioning task in the medical field. However, the common problem of data imbalance in medical imaging poses a challenge to the performance of deep learning models. Specifically, the number of lesion-free cases obtained daily is often far greater than that of lesion cases. This data imbalance causes sentences describing normal lesion-free conditions to dominate the DR imaging report generation dataset, while sentences involving lesion descriptions are relatively rare. This imbalance in data distribution will cause deep learning models to tend to generate reports describing normal conditions with no significant lesions.
[0004] However, in practical applications, accurate identification and description of lesion information is often more important. Therefore, how to alleviate the problem of uneven data distribution and improve the performance of deep learning models in generating reports for lesion instances is an important issue that needs to be solved urgently. Summary of the invention
[0005] The purpose of the present invention is to address the deficiencies of the prior art and propose a chest DR image report generation method based on visual and language feature comparison. By comparing the visual and language features between lesion instances and non-lesion instances, effective visual and language expressions of lesion features are learned, thereby alleviating the problem of uneven data distribution and improving the performance of lesion instance report generation.
[0006] In order to solve the above technical problems, the technical solution of the present invention is:
[0007] A chest DR image report generation method based on visual language feature comparison, the specific steps are as follows:
[0008] Step 1: Construct different types of training data: the first type uses non-lesion instances as anchor instances and positive instances, and uses lesion instances as negative instances; the other type of data uses lesion instances with the same medical subject terms as anchor instances and positive instances, and uses non-lesion instances as negative instances; each instance contains a chest DR image I and a reference report R;
[0009] Step 2: Perform image augmentation and text augmentation on all instances constructed in step 1 to obtain more training data. Image augmentation includes random cropping, and text augmentation includes random synonym replacement of reports and random deletion of description sentences without lesions.
[0010] Step 3: Input the DR image into the convolutional neural network to extract the convolution feature map of the image, and expand it to obtain the convolution sequence feature. Among them, N is the length of the convolution sequence feature, d model is the characteristic dimension of the model;
[0011] Step 4: Concatenate the convolution sequence features with the special CLS class vector to obtain the sequence features And input the self-attention encoding network encoding to obtain the image visual encoding features Among them, the self-attention encoding network includes a multi-head attention network and a forward propagation linear network. The convolution sequence feature E passes through the multi-head attention network and then is input into the forward propagation linear network to obtain the visual encoding feature C:
[0012] Step 5: Convert the visual encoding feature C of the anchor instance to 1 , v 2 , ..., v N} Input the Transformer-based text generation decoder to generate the corresponding report text, where the text generation decoder contains a masked self-attention network and a cross-attention network. The visual encoding feature C and the report text R generated before the current time t are combined. t-1 The common input is used to generate the decoder output hidden state H = {h 1 ,h 2 , ..., h t-1}; Then, take the hidden state h at the adjacent time t-1 t-1 The report text output r at the current time t is obtained through linear mapping and softmax operation t ;
[0013] Step 6: Input the generation report of the anchor instance and the reference reports of the positive instance and the negative instance into the language feature encoder to obtain the corresponding language encoding features. The language feature encoder is a pre-trained BERT-based language feature encoder;
[0014] Step 7: Compare the visual coding feature comparison vectors c corresponding to the CLS class vectors in the visual coding features C of the anchor instance, the positive instance, and the negative instance, so as to shorten the distance between the positive instance and the anchor instance and increase the distance between the anchor instance and the negative instance;
[0015] Step 8: Compare the language encoding feature comparison vectors of the anchor instance, the positive instance, and the negative instance to shorten the distance between the positive instance and the anchor instance and increase the distance between the anchor instance and the negative instance.
[0016] Step 9: Use the two sets of data in the training data set of step 1 for joint training, compare the visual features and language features between the lesion instance and the non-lesion instance to optimize the network parameters; specifically, use the cross entropy loss function, visual feature contrast loss function and language feature contrast loss function generated by the report to jointly optimize the model;
[0017] Step 10. During the test, it is only necessary to input the visual coding features corresponding to the chest DR image to be tested into the text generation decoder to generate text through an autoregressive text generation method, and then concatenate the text generated before the end character to obtain a complete report.
[0018] The present invention has the following characteristics and beneficial effects:
[0019] By adopting the above technical solution, the present invention constructs two different sets of positive examples, anchor examples and negative example data containing lesion examples and non-lesion examples for model training, compares the visual features and language features of the lesion examples with the non-lesion examples, shortens the distance between the anchor example and the positive example and widens the distance between the anchor example and the negative example, so as to alleviate the problem of uneven distribution of lesion and non-lesion samples in the report generation dataset. Experiments conducted on two public datasets show that by comparing lesion examples and non-lesion examples, the chest DR image report generation method based on visual language feature comparison can achieve better lesion feature representation, thereby better generating reports for lesion examples, thereby providing doctors with more effective and accurate diagnostic basis. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0021] Figure 1 It is a schematic flow diagram of the method of the present invention;
[0022] Figure 2 It is a detailed framework diagram of the method model of the present invention;
[0023] Figure 3 are two groups of training examples constructed by the method of the present invention;
[0024] Figure 4This is a demonstration of the effect of the method of the present invention on the IU X-Ray public dataset.
[0025] Figure 5 This is a demonstration of the effect of the method of the present invention on the MIMIC-CXR public dataset. DETAILED DESCRIPTION
[0026] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.
[0027] The present invention provides a method for generating chest DR image reports based on visual language feature comparison, such as Figure 1 As shown in the figure, the visual features and language features between lesion instances and non-lesion instances are compared, so as to enhance the representation of lesion features, alleviate the problem of uneven data distribution, and improve the performance of lesion instance report generation. First, chest DR images containing lesions and non-lesion instances and corresponding reports are obtained to form anchor instances, positive instances and negative instances of the training data set; then, the convolution sequence features of each instance image are extracted through a convolutional neural network, and self-attention encoding is performed after concatenating the class vector to obtain the visual encoding features; then, the visual encoding features of the anchor instance are input into the text generation decoder to generate a report, and they are encoded to obtain the language encoding features for generating the anchor instance report; at the same time, the positive instance and negative instance reports are encoded to obtain the language encoding features; further, the visual encoding features and language encoding features of the anchor instances, positive instances and negative instances containing lesion instances and non-lesion instances are compared to optimize the network; finally, in the test phase, only the visual encoding features of the image need to be input into the text generation decoder to generate the report.
[0028] The specific implementation steps are as follows: Figure 2 As shown:
[0029] Step 1: Construct different types of training data: Figure 3 As shown in (a), the first type uses non-lesion instances as anchor instances and positive instances, and uses lesion instances as negative instances; Figure 3 As shown in (b), another type of data uses lesion instances with the same medical subject terms as anchor instances and positive instances, and non-lesion instances as negative instances; each instance contains a chest DR image I and a reference report R;
[0030] Step 2: Perform image augmentation and text augmentation on all instances constructed in step 1 to obtain more training data. Image augmentation includes random cropping, and text augmentation includes random synonym replacement of reports and random deletion of description sentences without lesions.
[0031] Step 3: Input the DR image into the convolutional neural network to extract the convolution feature map of the image, and expand it to obtain the convolution sequence feature. The specific formula is as follows:
[0032] X={x 1 , x 2 , …, x N}=Flatten(CNN(I)) Formula (1)
[0033] Among them, CNN represents convolutional neural network, Flatten represents the flattening operation of feature map, N is the length of convolution sequence feature, d model is the characteristic dimension of the model;
[0034] Step 4: Concatenate the convolution sequence features with the special CLS class vector to obtain the sequence features And input the self-attention encoding network for encoding to obtain the visual encoding features of the instance image Among them, the self-attention encoding network consists of a multi-head attention network and a forward propagation linear network, and the formula is as follows:
[0035] C={c,v 1 , ..., v N}=FFN(MHA(E,E,E)) Formula (2)
[0036] Among them, MHA represents multi-head attention network, FFN represents forward propagation linear network, MHA and FFN are followed by residual connection operation and Drop-out operation; C represents the visual encoding feature comparison vector corresponding to the CLS class vector. MHA consists of h attention heads and can be expressed as the following formula:
[0037] MHA(Q, K, V) = [head 1 ;...;head h ]W O Formula (3)
[0038] head i =Attention(QW i Q , KW i K , VW i V ) Formula (4)
[0039]
[0040] in, Represents the i-th attention head i The mapping matrix, Represents the mapping matrix of the multi-head attention layer output, d k represents the dimensions of the query matrix Q and the key-value matrix K, d vRepresents the dimension of the value matrix V. The forward propagation linear layer FFN can be expressed as the following formula:
[0041] FFN(x)=max(0,xW 1 +b 1 )W 2 +b 2 Formula (6)
[0042] Among them, W 1 and W 2 represents the linear mapping matrix, b 1 and b 2 Indicates bias.
[0043] Step 5: Convert the visual encoding feature C of the anchor instance to 1 , ..., v N} Input the Transformer-based text generation decoder to generate the corresponding report text; the text generation decoder contains a masked self-attention network and a cross-attention network, and the masked attention network Layer mask It can be expressed as:
[0044] Layer mask (x)=LayerNorm(MHA(x,x,x)) Formula (7)
[0045] Cross-Attention Network Layer cross It can be expressed as:
[0046] Layer cross (x, y)=FFN(LayerNorm(MHA(x, y, y))) Formula (8)
[0047] Among them, LayerNorm represents the layer normalization operation, and the MHA and FFN in the self-attention network and the cross-attention network are followed by the residual connection operation and the Drop-out operation. The text generation decoder Decoder can be expressed as:
[0048] Decoder(x, y) = Layer cross (Layer mask (x), y) Formula (9)
[0049] The hidden state output by the text generation decoder can be expressed as the following formula:
[0050] H={h 1 ,h 2 , ..., h t-1}=Decoder(C,R t-1 ) Formula (10)
[0051] Then, take the hidden state h at the adjacent time t-1 t-1 , through linear mapping and softmax operation, we get the report text output r at the current time t t , the specific formula is as follows:
[0052] r t ~p t =softmax(h t-1 W p +b p ) Formula (11)
[0053] Among them, W p represents the linear mapping matrix, b p represents the bias, r t Indicates the words in the report generated at the current moment, p t Represents the text word distribution output at the moment;
[0054] Step 6: Input the generation report of the anchor instance and the reference reports of the positive instance and the negative instance into the language feature encoder to obtain the corresponding language coding features; wherein the language coding feature comparison vector output by the language feature encoder can be expressed as the following formula:
[0055] l a = LanguageEncoder(R a ) Formula (12)
[0056] l p = LanguageEncoder(R p ) Formula (13)
[0057] l n = LanguageEncoder(R n ) Formula (14)
[0058] Among them, l a Represents the language encoding feature contrast vector of the anchor instance, l p Represents the language encoding feature contrast vector of the positive instance, l n is the language encoding feature comparison vector of the negative instance. LanguageEncoder represents the pre-trained language feature encoder;
[0059] Step 7: Compare the visual coding feature comparison vector c corresponding to the CLS class vector in the visual coding features of the anchor instance, the positive instance, and the negative instance to shorten the distance between the positive instance and the anchor instance and increase the distance between the anchor instance and the negative instance. The specific formula is as follows:
[0060]
[0061] Among them, c a Represents the visual encoding feature contrast vector of the anchor instance, c p Represents the visual encoding feature contrast vector of the positive instance, c n is the visual encoding feature contrast vector of the negative instance. v Indicates the boundary value of visual feature contrast. vision represents the visual feature contrast loss function;
[0062] Step 8: Compare the language encoding features of the anchor instance, positive instance, and negative instance to narrow the distance between the positive instance and the anchor instance and widen the distance between the anchor instance and the negative instance. The specific formula is as follows:
[0063]
[0064] Among them, l a Represents the language encoding feature contrast vector of the anchor instance, l p The language encoding feature representing the positive instance, l n is the language encoding feature comparison vector of the negative instance. l Indicates the boundary value for comparing language features. language represents the language feature contrast loss function;
[0065] Step (9), using the two sets of data in the training data set of step 1 for joint training, comparing the visual features and language features between the lesion instance and the non-lesion instance to optimize the network parameters; using the cross entropy loss function, the visual feature contrast loss function and the language feature contrast loss function as the optimization function for model training:
[0066] l all = l generation +l language +l vision Formula (17)
[0067]
[0068] Among them, l all Represents the overall loss function of model training. generation is the cross entropy loss function generated by the report, M represents the sequence length of the report, l language represents the language feature contrast loss function, l vision represents the visual feature contrast loss function.
[0069] Step 10. During the test, it is only necessary to input the visual coding features corresponding to the chest DR image to be tested into the text generation decoder to generate text through an autoregressive text generation method, and then concatenate the text generated before the end character to obtain a complete report.
[0070] To measure the effectiveness of the method of the present invention, qualitative and quantitative experiments were conducted on two public datasets, IU X-Ray and MIMIC-CXR. The qualitative experimental results on the two datasets are shown in Figure 2. Figure 4 and Figure 5 As shown. Figure 5 (b), the present invention can generate "The patient is in a state after median sternotomy and coronary artery bypass grafting. The heart size is slightly enlarged. The contours of the mediastinum and hilum are unchanged. The pulmonary vessels are not congested. Patchy opacities appear at the lung bases, which may reflect areas of atelectasis. There is no focal consolidation, pleural effusion or pneumothorax. There are mild degenerative changes in the thoracic spine", which is consistent with the description of the lesions covered in the reference report written by the doctor: "The patient is in a state after median sternotomy, CABG, and vascular stent placement. The heart size is slightly enlarged but stable. The contours of the mediastinum and hilum are normal, and the pulmonary vessels are not congested. The lungs are clear, without focal consolidation. No pleural effusion or pneumothorax is found. No acute bone abnormalities are found." This phenomenon shows that the method of the present invention can effectively improve the accuracy of the generation of the corresponding lesion report.
[0071] In terms of quantitative analysis, the quantitative comparison experimental results of the method of the present invention with the existing DR imaging report generation method on two existing public data sets IU X-Ray and MIMIC-CXR are shown in Tables 1 and 2.
[0072] Table 1 Performance comparison of IU X-Ray dataset (the best results are marked in bold)
[0073] method BLEU-3 BLEU-4 METEOR ROUGE-L CIDEr Show-Tell 0.108 0.078 0.157 0.307 0.197 Transformer 0.147 0.136 0.168 0.317 0.310 CNN-Transformer 0.179 0.135 0.164 0.342 - <![CDATA[M 2 Transformer]]> 0.205 0.152 0.176 0.353 - HRGR 0.208 0.151 - 0.322 0.343 The present invention 0.246 0.178 0.214 0.366 0.422
[0074] Table 2 Performance comparison of MIMIC-CXR dataset (the best results are marked in bold)
[0075] method BLEU-3 BLEU-4 METEOR ROUGE-L CIDEr Show-Tell 0.125 0.088 0.122 0.256 0.096 Transformer 0.140 0.092 0.129 0.267 0.134 RACHET 0.139 0.099 0.136 0.280 0.140 CMN 0.148 0.106 0.142 0.278 - CCRG 0.139 0.100 0.133 0.274 0.136 The present invention 0.151 0.111 0.140 0.286 0.182
[0076] It can be seen from Tables 1 and 2 that on two public chest DR image datasets, under various text generation evaluation indicators, the method proposed in the present invention is improved compared with the existing methods, indicating that the method of the present invention can effectively improve the report generation performance.
[0077] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments including components are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.
Claims
1. A method for generating chest DR image report based on visual language feature comparison, characterized in that: The steps include: Step 1: construct a data set, wherein the data set includes a chest DR image I and a reference report R, and the data set defines an anchor instance, a positive instance, and a negative instance according to a lesion instance; Step 2: Perform image augmentation and text augmentation on the dataset to obtain more training data; Step 3: Input the DR image into the convolutional neural network to extract the convolution feature map of the image, and expand it to obtain the convolution sequence feature X; Step 4: Concatenate the convolution sequence feature with the CLS class vector to obtain the sequence feature E and input it into the self-attention encoding network to obtain the anchor instance image visual encoding feature C; Step 5: Input the anchor instance image visual encoding feature C into the Transformer-based text generation decoder to generate the corresponding report text; Step 6: Input the generation report of the anchor instance and the reference reports of the positive instance and the negative instance into the language feature encoder to obtain the corresponding language encoding features; Step 7: Compare the visual coding feature vectors c corresponding to the CLS class vectors in the visual coding features of the anchor instance, the positive instance, and the negative instance to shorten the distance between the positive instance and the anchor instance. Step 8: Compare the language encoding features of the anchor instance, the positive instance, and the negative instance to narrow the distance between the positive instance and the anchor instance and widen the distance between the anchor instance and the negative instance; Step 9: Use the training data set in step 1 for joint training, and use the optimization function trained by the cross entropy loss function, the visual feature contrast loss function, and the language feature contrast loss function; Step 10. During the test, it is only necessary to input the visual coding features corresponding to the chest DR image to be tested into the text generation decoder to generate text through an autoregressive text generation method, and then concatenate the generated text before the end character to obtain a complete report.
2. The chest DR image report generation method based on visual language feature comparison according to claim 1 is characterized in that: The dataset defines anchor instances, positive instances and negative instances according to two types of lesion instances, thereby obtaining two groups of training data. The first type uses non-lesion instances as anchor instances and positive instances, and uses lesion instances as negative instances; the other type uses lesion instances with the same medical subject terms as anchor instances and positive instances, and uses non-lesion instances as negative instances.
3. The chest DR image report generation method based on visual language feature comparison according to claim 1 is characterized in that: The image augmentation method includes random cropping, and the text augmentation method includes random synonym replacement of reports and random deletion of non-lesion description sentences.
4. The chest DR image report generation method based on visual language feature comparison according to claim 1 is characterized in that: The self-attention encoding network consists of a multi-head attention network and a forward propagation linear network, and the formula is as follows: C={c,v1,v2,…,v N }=FFN(MHA(E,E,E)) Among them, MHA represents the multi-head attention network, FFN represents the forward propagation linear network, MHA and FFN are followed by residual connection operation and Drop-out operation; c represents the visual encoding feature vector corresponding to the CLS class vector.
5. The method for generating chest DR image report based on visual language feature comparison according to claim 4, characterized in that: The multi-head attention network MHA consists of h attention heads, expressed as follows: MHA(A,K,V)=[head1;...;head h ]W o head i =Attention(QW i Q ,KW i K ,VW i V ) in, represents the i-th attention head i The mapping matrix, Represents the mapping matrix of the multi-head attention layer output, d k represents the dimensions of the query matrix Q and the key-value matrix K, d v Represents the dimensions of the value matrix V.
6. The method for generating chest DR image report based on visual language feature comparison according to claim 3, characterized in that: The forward propagation linear network FFN is expressed as follows: FFN(x)=max(0,xW1+b1)W2+b2 Among them, W1 and W2 represent linear mapping matrices, and b1 and b2 represent biases.
7. The method for generating chest DR image report based on visual language feature comparison according to claim 1, characterized in that: The text generation decoder contains a masked self-attention network and a cross-attention network.
8. The method for generating chest DR image report based on visual language feature comparison according to claim 1, characterized in that: The text generation decoder Decoder is expressed as: Decoder(x,y)=Layer cross (Layer mask (x),y) Among them, Layer cross For the cross attention network, Layer mask is a masked attention network, x and y represent the text sequence and image visual encoding features of the decoder input respectively. The output of the text generation decoder is the hidden state of the text feature, expressed as follows: H={h1,h2,...,h t-1 }=Decoder(C,R t-1 )。 9. The method for generating chest DR image report based on visual language feature comparison according to claim 8, characterized in that: In step 5, take the hidden state h at the adjacent time t-1 t-1 , through linear mapping and softmax operation, we get the report text output r at the current time t t , the specific formula is as follows: r t ~p t =softmax(h t-1 W p +b p ) Among them, W p represents the linear mapping matrix, b p represents the bias, r t Indicates the words in the report generated at the current moment, p t Represents the text word distribution output at a certain moment.
10. The method for generating chest DR image report based on visual language feature comparison according to claim 1, characterized in that: In step 7, the distance between the anchor instance and the negative instance is increased. The specific formula is as follows: Among them, c a Represents the visual encoding feature contrast vector of the anchor instance, c p Represents the visual encoding feature contrast vector of the positive instance, c n is the visual encoding feature contrast vector of the negative instance, m v is the boundary value of visual feature comparison, represents the visual feature contrast loss function.
11. The chest DR image report generation method based on visual language feature comparison according to claim 1, characterized in that: In step 8, the distance between the anchor instance and the negative instance is increased. The specific formula is as follows: Among them, l a Represents the language encoding feature contrast vector of the anchor instance, l p The language encoding feature representing the positive instance, l n is the language encoding feature comparison vector of the negative instance, m l Indicates the boundary value of language feature comparison, Represents the language feature contrast loss function.
12. The chest DR image report generation method based on visual language feature comparison according to claim 2, characterized in that: In step 9, during training, two groups of training data obtained according to two types of lesion instances are used for joint training, and the visual features and language features between the lesion instances and the non-lesion instances are compared to optimize the network parameters.
13. The chest DR image report generation method based on visual language feature comparison according to claim 11, characterized in that: The optimization function expression is as follows: in, represents the overall loss function of model training, is the cross entropy loss function generated by the report, M represents the sequence length of the report, represents the language feature contrast loss function, represents the visual feature contrast loss function.
Citation Information
Patent Citations
Machine generated text detection method and system based on linguistic feature comparative learning
CN115081437A
Medical image retrieval network training method, medical image retrieval network application method and electronic equipment
CN115757844A
Medical image report automatic generation method based on memory ability learning
CN116364227A
Unsupervised pre-training method and system for strengthening vector retrieval capability
CN118014045A
Combined vision and language learning models for automated medical reports generation
US20230386646A1
