X-ray chest radiography report generation method fusing spatial information and joint loss optimization

Abstract

The invention discloses an X-ray chest radiograph report generation method fusing spatial information and joint loss optimization. The method comprises the following steps: constructing an image feature extractor, constructing a spatial information matrix learning layer, constructing a disease classifier, constructing a convolution block attention module, and constructing a codec. The text report loss and the disease classification loss are taken as the joint loss, and the generation of the report is guided by using the disease classification result, so that the accuracy of generating the report is improved; learning rich spatial information existing in the feature map by adding a spatial information matrix; an accurate feature map space attention relation is learned by adding a convolution block attention module.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic generation of medical text reports, and in particular to a method for generating an X-ray chest film report by integrating spatial information and joint loss optimization. Background Art

[0002] As one of the most common medical images, chest X-rays are an important tool for doctors to diagnose lung, heart and other related diseases, and are widely used in medical clinical diagnosis and treatment. Traditional chest X-ray diagnosis requires doctors to read the chest X-ray carefully and accurately describe the abnormalities shown in the X-ray image before giving the corresponding diagnosis results. However, this method has problems such as high time cost and strong subjectivity. Therefore, how to quickly and accurately diagnose chest X-rays has become a topic of common concern in the medical and computer science communities.

[0003] Automatic diagnosis report generation is a research task that requires the combination of multiple cutting-edge research fields such as computer vision and natural language processing. It mainly uses the features extracted from the image to predict the words corresponding to each time step, thereby constructing the entire encoding and decoding model framework. However, the inventors of this application have found through research that the current method of generating chest X-ray diagnosis reports using deep learning has the following shortcomings: (1) It is difficult to learn the accurate spatial attention relationship of feature maps by paying attention to the patches of feature maps; (2) There is rich spatial information in feature maps, and after Transformer divides the feature maps into patches, the rich spatial information may be lost; (3) Only using the generated report to calculate the loss may cause the model to tend to generate normal reports instead of abnormal reports. Summary of the invention

[0004] The purpose of the present invention is to provide a method for generating an X-ray chest film report integrating spatial information and joint loss optimization, so as to solve the problems in the prior art mentioned in the above background technology.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for generating an X-ray chest film report integrating spatial information and joint loss optimization comprises the following steps:

[0007] S1. Extract visual features from an X-ray chest film using an image feature extractor, the image feature extractor being composed of a deep convolutional network, including first to fourth densely connected layer groups and first to third transition layers, the first transition layer being located after the first densely connected layer group, the second transition layer being located after the second densely connected layer group, the third transition layer being located after the third densely connected layer group, and the fourth densely connected layer group being located after the third transition layer;

[0008] S2, a matrix containing the spatial information of the feature map is generated through the spatial information matrix learning layer, which consists of a self-attention layer, a feedforward layer, and a convolutional layer;

[0009] S3, the convolutional block attention module is used to learn the attention relationship between each feature, which consists of channel attention and spatial attention;

[0010] S4, the codec is used to generate text reports. The codec consists of ten standard transformer encoding layers, ten standard transformer decoding layers, a fully connected layer, and a softmax regression layer;

[0011] S5. Use the generated report and disease classification results to calculate the loss and optimize the network model.

[0012] Preferably, the image feature extractor in step S1 is used to extract visual features in the chest X-ray, and the image feature extractor is composed of DenseNet.

[0013] Preferably, in step S5, the network model parameters are first initialized using the Xavier method, and then the network objective function is optimized using the Adam algorithm, and the network model parameters are updated using the error back propagation algorithm; the data set MIMIC-CXR is divided into a training set, a validation set, and a test set in a ratio of 7:1:2.

[0014] Preferably, the number of attention heads in the multi-head self-attention and multi-head cross-attention modules in step S3 is 8.

[0015] Technical effects and advantages of the present invention: Compared with the prior art, the present invention provides a method for generating an X-ray chest film report by integrating spatial information and joint loss optimization, which has the following advantages:

[0016] 1. Improve accuracy: The present invention uses the text report loss and the disease classification loss as a joint loss, expands the disease classification results into a matrix and splices them into the feature map, and uses the disease classification results to guide the generation of the report, thereby improving the accuracy of the generated report;

[0017] 2. The present invention learns the rich spatial information existing in the feature map by adding a spatial information matrix;

[0018] 3. Learn accurate feature map spatial attention relations by adding a convolutional block attention module. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a network structure diagram of the present invention.

[0020] Figure 2This is a structural diagram of the image feature extractor of the present invention.

[0021] Figure 3 This is a diagram of the spatial information matrix learning layer structure of the present invention.

[0022] Figure 4 This is a structural diagram of the disease classifier of the present invention.

[0023] Figure 5 This is a structural diagram of the codec of the present invention.

[0024] Figure 6 This is the parameter table of the feature extractor of the present invention.

[0025] Figure 7 This is the parameter table of the spatial information matrix learning layer of the present invention.

[0026] Figure 8 This is the parameter table of the disease classifier of the present invention.

[0027] Fig. 9 This is the parameter table of the convolution block attention module of the present invention. DETAILED DESCRIPTION

[0028] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. The specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0029] The present invention provides Figure 1-9 The method for generating an X-ray chest film report by integrating spatial information and joint loss optimization is shown, comprising the following steps:

[0030] S1. Extract visual features from an X-ray chest film using an image feature extractor, the image feature extractor being composed of a deep convolutional network, including first to fourth densely connected layer groups and first to third transition layers, the first transition layer being located after the first densely connected layer group, the second transition layer being located after the second densely connected layer group, the third transition layer being located after the third densely connected layer group, and the fourth densely connected layer group being located after the third transition layer;

[0031] S2, a matrix containing the spatial information of the feature map is generated through the spatial information matrix learning layer, which consists of a self-attention layer, a feedforward layer, and a convolutional layer;

[0032] S3, the convolutional block attention module is used to learn the attention relationship between each feature, which consists of channel attention and spatial attention;

[0033] S4, the codec is used to generate text reports. The codec consists of ten standard transformer encoding layers, ten standard transformer decoding layers, a fully connected layer, and a softmax regression layer;

[0034] S5. Use the generated report and disease classification results to calculate the loss and optimize the network model.

[0035] The image feature extractor in step S1 is used to extract visual features in the chest X-ray, and the image feature extractor is composed of DenseNet121.

[0036] In step S5, the Xavier method is first used to initialize the network model parameters, and then the Adam algorithm is used to optimize the network objective function, and the error back propagation algorithm is used to update the network model parameters; the disease classification result is input into the fully connected layer to obtain a matrix that can be spliced ​​with the feature map at the channel level to guide the generation of the report; the data set MIMIC-CXR is divided into a training set, a validation set, and a test set in a ratio of 7:1:2.

[0037] The number of attention heads in the multi-head self-attention and multi-head cross-attention modules in step S3 is 8.

[0038] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating an X-ray chest film report by integrating spatial information and joint loss optimization, characterized in that: The steps include: S1. Extract visual features from an X-ray chest film using an image feature extractor, the image feature extractor being composed of a deep convolutional network, including first to fourth densely connected layer groups and first to third transition layers, the first transition layer being located after the first densely connected layer group, the second transition layer being located after the second densely connected layer group, the third transition layer being located after the third densely connected layer group, and the fourth densely connected layer group being located after the third transition layer; S2, a matrix containing the spatial information of the feature map is generated through the spatial information matrix learning layer, which consists of a self-attention layer, a feedforward layer, and a convolutional layer; S3, the convolutional block attention module is used to learn the attention relationship between each feature, which consists of channel attention and spatial attention; S4, the codec is used to generate text reports. The codec consists of ten standard transformer encoding layers, ten standard transformer decoding layers, a fully connected layer, and a softmax regression layer; S5. Use the generated report and disease classification results to calculate the loss and optimize the network model.

2. The method for generating an X-ray chest film report integrating spatial information and joint loss optimization according to claim 1, characterized in that: The image feature extractor in step S1 is used to extract visual features in the chest X-ray, and the image feature extractor is composed of DenseNet.

3. The method for generating an X-ray chest film report integrating spatial information and joint loss optimization according to claim 1, characterized in that: In step S5, the Xavier method is first used to initialize the network model parameters, and then the Adam algorithm is used to optimize the network objective function, and the error back propagation algorithm is used to update the network model parameters; the data set MIMIC-CXR is divided into a training set, a validation set and a test set in a ratio of 7:1:

2.

4. The method for generating an X-ray chest film report integrating spatial information and joint loss optimization according to claim 1, characterized in that: The number of attention heads in the multi-head self-attention and multi-head cross-attention modules in step S3 is 8.

Images