Medical image report generation method based on fine-grained optimal visual angle constraint

Abstract

The invention relates to the technologies of deep learning, natural language processing, cross-modal alignment, image subtitles and the like, and discloses a medical image report generation method based on fine-grained optimal view constraint. According to the method, the most suitable visual angle of the medical image is determined by analyzing the position relation and visual features of all objects in the medical image. Then, taking the optimal view angle as a prefix code, fusing position information and visual features of objects corresponding to sentences, and filtering an object set by calculating probability scores corresponding to the objects and the sentences; and on this basis, a combined object sub-image is reconstructed by using the filtered object set, a sentence of a current time step is generated according to a previous sentence and a visual feature corresponding to the current sentence, whether the sentence is the last sentence is determined, and finally a medical image text report is obtained. According to the technical scheme, a gap between vision and language in the field of medical image report generation is filled up, and an innovative method is provided for solving the problem.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and specifically relates to technologies such as deep learning, natural language processing, cross-modal alignment, image subtitles, and more particularly to a method for generating a medical imaging report based on fine-grained optimal viewing angle constraints. Background Art

[0002] In contemporary medical practice, medical imaging has become a key component in diagnosing patient diseases and formulating treatment plans. Medical imaging technology can reveal the internal structure of the human body and provide reliable visual evidence for diseases that cannot be observed from the outside. The efficient use of these technologies depends largely on the doctor's accurate interpretation of the imaging results and the detailed recording of relevant clinical information, that is, the preparation of medical imaging reports. These reports provide important information for the patient's medical history and play a vital role in subsequent treatment decisions. However, the preparation of high-quality medical imaging reports is a time-consuming and professionally demanding task. Faced with the increasing amount of medical imaging data, the workload of radiologists has gradually increased, and the pressure of report writing has also increased accordingly. In addition, the uneven distribution of medical resources around the world has led to a shortage of professional radiologists in resource-poor areas, and the interpretation of medical imaging reports may be delayed, which further increases the risk to patients.

[0003] With the deepening application of artificial intelligence technology, especially machine learning and deep learning in the medical field, automated medical imaging report generation is gradually showing its broad prospects. This study on automatic generation of medical imaging reports aims to use machine learning and deep learning algorithms to learn and identify disease patterns from complex imaging data, and then automatically write structured and information-rich medical imaging reports. This study helps to reduce the work of radiologists and give them more time to focus on clinical decision-making and patient care. In addition, automatically generated reports can maintain consistency in format, reduce the impact of individual differences among doctors, and help improve the quality of reports. The development of medical imaging report generation technology has extremely important practical significance and provides new impetus and direction for the development of modern medical diagnosis and treatment.

[0004] Generating text descriptions from traditional images and videos has been extensively studied, such as image captioning and video summarization. Recently, encoder-decoder architectures based on deep learning have shown excellent performance in the field of image description. Inspired by the success of image captioning tasks, most existing medical report generation methods adopt an encoder-decoder framework to generate free text reports. However, unlike conventional image captioning tasks that focus on object recognition and brief explanation of context, medical report generation usually involves complex medical terminology and clinical knowledge, which requires the model to have a deeper level of semantic understanding and expertise. In addition, accuracy is crucial in medical reports, and any wrong information or omissions may lead to misdiagnosis or missed diagnosis, bringing unpredictable risks to patients. Although encoder-decoder models based on deep learning have achieved remarkable results in image captioning tasks, research in the field of medical report generation still faces the challenge of difficult connection between the visual content of medical images and professional clinical terminology. Summary of the invention

[0005] The current medical image report generation method still has defects in establishing a connection between visual content and language expression, making it difficult to automatically generate detailed and accurate reports from medical images. The present invention aims to solve the above problems of the prior art. The present invention proposes a medical image report generation method based on fine-grained optimal view constraints, the purpose of which is to solve the problems of heterogeneous gap and semantic gap between medical images and text reports in existing methods.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0007] The present invention provides a method for generating a medical imaging report based on fine-grained optimal viewing angle constraints, which comprises the following steps:

[0008] 1) Construct an encoder-decoder architecture. When constructing a new medical imaging report generation model, a fine-grained optimal view constraint method is additionally introduced on the basis of the encoder-decoder framework.

[0009] 2) Obtain a data set. Obtain a medical data set containing real data, including multiple sets of chest X-ray images and corresponding medical reports. Then, the data set will be appropriately divided to meet the needs of the actual task, and the data will be divided into a training set, a validation set, and a test set.

[0010] 3) Use the pre-trained FasterR-CNN model to obtain the location information of K objects in the medical image r = {r1,...,r k} and visual features v = {v1,...,v k}.

[0011] 4) Obtain the best viewing angle. For the i-th object, its position information and visual features are connected to form the comprehensive visual feature v of the object. i '. Define five perspectives: "from left to right", "from right to left", "from top to bottom", "from bottom to top" and "from primary to secondary", using v' = {v1',...,v' k}Calculate the predicted probability sp of each perspective and select the optimal perspective.

[0012] 5) The information fusion network completes the visual feature fusion task in the sentence, where the optimal view is used as the prefix code. The position information r and visual features v of K objects in the medical image are processed by the Transformer decoder to obtain an overall pool. The position information sequence cr of object i is taken out from the overall pool i ={cr i,1 ,...,cr i,l} and visual feature sequence cv i ={cv i,1 ,...,cv i,l}Used to calculate fusion visual features Specifically, cr i and cv i Each of them passes through a fully connected layer and then takes the average to obtain the composite position information and feature vector of l objects. Subsequently, these two components are connected to generate the sentence t i Fusion visual features

[0013] 6) Select the object corresponding to the sentence. Fusion of visual features After Transformer processing, each object is selected as sentence t i The probability p of the corresponding object i , and set the threshold to filter the object to get the sentence t i The corresponding object collection.

[0014] 7) Joint object strategy based on position information sequence cr i The corresponding object regions are selected, the image information of these object regions is retained, and other positions are filled with zeros to form a new joint object sub-image.

[0015] 8) Joint object sub-image feature extraction: The visual features of the joint object sub-image are used as the input of the report generation module.

[0016] 9) Report generation. The report generation module generates reports based on the previous sentences {t1,...,t i-1} and the visual features corresponding to the current sentence to generate the sentence t at the current time step i , and determine ti Is it the last sentence of the description, and finally get a text report.

[0017] Specifically, the data sets used in step 2) are the IU-Xray data set and the MIMIC-CXR data set. In addition, for the IU-Xray data set, the data segmentation method of the previous work R2Gen was followed, and the data was divided into training, validation and test sets in a ratio of 7:1:2 to ensure that there was no overlap between patients. For the MIMIC-CXR data set, the data segmentation provided by the official was used.

[0018] Furthermore, in step 3), when encoding the medical image, two medical images (usually a chest X-ray image from a frontal perspective and a chest X-ray image from a side perspective) are input, and the features of the two medical images containing different information are fused together to capture more medical information.

[0019] Furthermore, in step 6), the threshold in the threshold filter object is set to 0.5.

[0020] Furthermore, in step 8) of the present invention, ResNet101 is used as a visual feature extractor.

[0021] Furthermore, in step 9) of the present invention, Transformer is used as a text feature extractor.

[0022] Furthermore, the medical imaging report generation method based on fine-grained optimal view constraints proposed in the present invention uses a variety of natural language generation (NLG) metrics and clinical effect metrics as evaluation indicators, including BLEU-n, CIDEr and ROUGE-L scores.

[0023] Furthermore, the present invention proposes a medical imaging report generation method based on fine-grained optimal view constraints, which uses a variety of natural language generation (NLG) metrics and clinical effect metrics as evaluation indicators, including BLEU-n, CIDEr and ROUGE-L scores, to obtain the model dimension, encoder-decoder dimension and hyperparameter value.

[0024] In summary, due to the adoption of the above technical scheme, the advantages and positive effects of the present invention are as follows: the present invention aims at the problem that it is difficult to accurately convey the hierarchical and spatial information of images in previous medical image report generation methods, and provides a medical image report generation method based on fine-grained optimal perspective constraints, which solves the problem that it is difficult to accurately convey the hierarchical and spatial information of images to a certain extent. The present invention introduces a Transformer encoder as an optimal perspective prediction module, and determines the most suitable perspective for the medical image by analyzing the positional relationship and visual features of all objects in the medical image. Subsequently, an information fusion and object selection strategy is proposed, which uses the optimal perspective as a prefix code, fuses the positional information and visual features of the corresponding objects of the sentence, and filters the object set by calculating the probability score. Finally, the joint object sub-image is reconstructed to generate a medical image text report, which improves the quality and accuracy of medical report generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0026] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0027] Figure 2 It is a schematic diagram of the model framework of the present invention.

[0028] Specific implementation method

[0029] The present invention provides a method for generating a medical image report based on fine-grained optimal viewing angle constraints, and the described embodiments are only part of the embodiments of the present invention.

[0030] The flow chart of the method is as follows Figure 1 First, FasterR-CNN is used to extract the features of all objects in the image. After that, the optimal perspective prediction module is used to analyze the positional relationship and visual features of all objects in the medical image to determine the most suitable perspective for the medical image. Subsequently, the information fusion network uses the optimal perspective as a prefix encoding to complete the visual feature fusion task in the sentence. At the same time, the probability score is used to filter the set of objects corresponding to the sentence. Then, the joint sub-image is reconstructed by the selected objects, and its visual features are extracted using ResNet101 as the input of the report generation module. Finally, the report generation module generates fine-grained image captions. It is divided into the following steps:

[0031] Step 1: Get a dataset. Get a medical dataset containing real data, including multiple sets of chest X-ray images and their corresponding medical reports. Then, the dataset will be appropriately divided to meet the needs of the actual task, and the data will be divided into a training set, a validation set, and a test set.

[0032] Step 2: Obtain the location information and visual features of all objects. Use the pre-trained FasterR-CNN model to obtain the location information of K objects in the medical image r = {r1,...,r k} and visual features v = {v1,...,v k}.

[0033] Step 3: Get the best viewing angle. For the i-th object, its position information and visual features are connected to form the comprehensive visual feature v of the object. i '. Define five perspectives: "from left to right", "from right to left", "from top to bottom", "from bottom to top" and "from primary to secondary", using v' = {v1',...,v' k}Calculate the predicted probability sp of each perspective and select the best perspective.

[0034] Step 4: The object feature fusion network completes the visual feature fusion task in the sentence. The position information r and visual features v of the K objects in the medical image are processed by the Transformer decoder to obtain an overall pool. The position information sequence cr of object i is taken out from the overall pool. i ={cr i,1 ,...,cr i,l} and visual feature sequence cv i ={cv i,1 ,...,cv i,l}Used to calculate fusion visual features Specifically, cr i and cv i Each of them passes through a fully connected layer and then takes the average to obtain the composite position information and feature vector of l objects. Subsequently, these two components are connected to generate the sentence t i Fusion visual features

[0035] Step 5: Select the object corresponding to the sentence. Fusion of visual features Each object is selected as sentence t after the best view and customized view are processed by Transformer i The probability p of the corresponding object i , and set the threshold to filter the object to get the sentence t i The corresponding object collection.

[0036] Step 6: Joint object strategy based on position information sequence cr i The corresponding object regions are selected, the image information of these object regions is retained, and other positions are filled with zeros to form a new joint object sub-image.

[0037] Step 7: Joint object sub-image feature extraction: The visual features of the joint object sub-image are used as the input of the report generation module.

[0038] Step 8: Report generation. The report generation module adopts two strategies: VIRTEX and SCST with repetition penalty. According to the previous sentence {t1,...,t i-1} and the visual features corresponding to the current sentence to generate the sentence t at the current time step i , and determine t i Is it the last sentence of the description, and finally get a text report.

[0039] The method for generating a medical imaging report based on fine-grained optimal viewing angle constraints is characterized in that: the data sets used in step 1) include the IU-Xray data set and the MIMIC-CXR data set. In addition, for the IU-Xray data set, the data segmentation method of the previous work R2Gen is followed, and the data is divided into training, validation, and test sets in a ratio of 7:1:2 to ensure that there is no overlap between patients. For the MIMIC-CXR data set, the data segmentation provided by the official was used.

[0040] The method for generating a medical image report based on fine-grained optimal viewing angle constraints is characterized in that: in step 2), when encoding the medical image, two medical images (usually a chest X-ray image from a frontal perspective and a chest X-ray image from a side perspective) are input, and the features of the two medical images containing different information are fused together to capture more medical information.

[0041] The method for generating a medical image report based on fine-grained optimal viewing angle constraint is characterized in that: step 3) using the comprehensive visual feature v′={v1′,...,v′ k}Calculate the predicted probability sp for each perspective as follows:

[0042] v i '=Concate([v i ,r i ])

[0043] h=Encoder(E cls ,v1',...,v' k )

[0044] sp = softmax(Wh cls +b)

[0045] Among them, Concate([,]) is a function that combines visual features and position information, E cls represents the embedding corresponding to the classification vector, h cls is the output corresponding to the classification vector. In addition, W∈R h*5 And b∈R 5 , where h represents the dimension of the hidden layer and 5 represents the number of view categories.

[0046] The method for generating a medical image report based on fine-grained optimal viewing angle constraints is characterized in that: step 4) the position information cr of object i i And the visual feature cv i Used to construct fused visual features The construction method is as follows:

[0047]

[0048] Among them; represents connection, And D r =8,D r '=256,D v =2048,D v '=256, l represents the number of objects processed.

[0049] The method for generating a medical imaging report based on fine-grained optimal viewing angle constraints is characterized in that: step 5) each object is selected as a sentence t i The probability p of the corresponding object i The calculation is as follows:

[0050]

[0051] p i =softmax(Wh i +n)

[0052] The left side of “|” is the input of the Transformer encoder, and the right side is the input of the Transformer decoder. pref Indicates prefix encoding. D h is the Transformer hidden layer dimension, n∈R 2 .

[0053] The method for generating a medical imaging report based on fine-grained optimal viewing angle constraints is characterized in that: in step 6), the threshold in the threshold filtering object is set to 0.5.

[0054] The method for generating a medical image report based on fine-grained optimal viewing angle constraints is characterized in that the visual feature extractor used in step 8) is ResNet101.

[0055] The method for generating a medical imaging report based on fine-grained optimal viewing angle constraints is characterized by: Step 9) VIRTEX is a strategy for enhancing text generation using visual features, learning how to map visual elements in an image to a natural language description.

[0056] The method for generating a medical imaging report based on fine-grained optimal view constraints is characterized in that: Step 9) SCST (Self-Critical Sequence Training) with repetition penalty is an algorithm for training a generative model under a reinforcement learning framework. SCST combines the advantages of standard policy gradient training and MLE (maximum likelihood estimation) training, and optimizes the score of the model generation sequence in a self-critical manner.

[0057] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for generating medical image reports based on fine-grained optimal viewing angle constraints, characterized in that: The following steps are involved: 1) Obtain a medical dataset containing real data, including multiple sets of chest X-ray images and corresponding medical reports. Then, the dataset will be appropriately divided to meet the needs of the actual task, and the data will be divided into a training set, a validation set, and a test set; 2) Use the pre-trained FasterR-CNN model to obtain the location information of K objects in the medical image r = {r1,...,r k } and visual features v = {v1,...,v k }; 3) For the i-th object, its position information and visual features are connected to form the comprehensive visual feature v of the object i '. Define five perspectives: "from left to right", "from right to left", "from top to bottom", "from bottom to top" and "from primary to secondary", using v' = {v1',...,v' k }Calculate the predicted probability sp of each perspective and select the optimal perspective; 4) The location information r and visual features v of K objects in the medical image are processed by the Transformer decoder to obtain an overall pool. The location information sequence cr of object i is taken out from the overall pool. i ={cr i,1 ,...,cr i,l } and visual feature sequence cv i ={cv i,1 ,...,cv i,l }Used to calculate fusion visual features Specifically, cr i and cv i Each of them passes through a fully connected layer and then takes the average to obtain the composite position information and feature vector of l objects. Subsequently, these two components are connected to generate the sentence t i Fusion visual features 5) Select the object corresponding to the sentence. Fusion of visual features After Transformer processing, each object is selected as sentence t i The probability p of the corresponding object i , and set the threshold to filter the object to get the sentence t i The corresponding object collection; 6) Joint object strategy based on position information sequence cr i Select the corresponding object regions, retain the image information of these object regions, and fill other positions with zeros to form a new joint object sub-image; 7) using the visual features of the joint object sub-image as input to the report generation module; 8) The report generation module adopts two strategies: VIRTEX and SCST with repetition penalty. According to the previous sentence {t1,...,t i-1 } and the visual features corresponding to the current sentence to generate the sentence t at the current time step i , and determine t i Is it the last sentence of the description, and finally get a text report.

2. The method for generating a medical imaging report based on fine-grained optimal viewing angle constraints according to claim 1, characterized in that: When constructing a new medical imaging report generation model, a fine-grained optimal view constraint method was introduced based on the encoder-decoder framework.

3. The method for generating a medical image report based on fine-grained optimal viewing angle constraints according to claim 1, characterized in that: The fine-grained optimal view constraint method uses an optimal view prediction module to determine the most appropriate view of the medical image by analyzing the positional relationship and visual features of all objects in the medical image. The information fusion and object selection strategy uses the optimal view as a prefix code, fuses the position information and visual features of the corresponding objects in the sentence, and filters the object set by calculating the probability score. Finally, the joint object sub-image is reconstructed to generate a medical image text report, which improves the quality and accuracy of the generated medical report.

4. The method for generating a medical image report based on fine-grained optimal viewing angle constraints according to claim 1, characterized in that: The data sets used in the fine-grained optimal view constraint method are the IU-Xray data set and the MIMIC-CXR data set. In addition, for the IU-Xray data set, the data segmentation method of the previous work R2Gen was followed, and the data was divided into training, validation, and test sets in a ratio of 7:1:2 to ensure that there is no overlap between patients. For the MIMIC-CXR data set, the data segmentation provided by the official was used.

5. The method for generating a medical imaging report based on fine-grained optimal viewing angle constraints according to claim 1, characterized in that: The joint object sub-image visual encoder uses ResNet-101.

6. The method for generating a medical imaging report based on fine-grained optimal viewing angle constraints according to claim 1, characterized in that: The threshold value in the threshold filter object is set to 0.

5.

7. The method for generating a medical image report based on fine-grained optimal viewing angle constraints according to claim 1, characterized in that: When Faster R-CNN is used to encode medical images, two medical images (usually a chest X-ray image from a frontal perspective and a chest X-ray image from a side perspective) are input, and the features of the two medical images containing different information are fused together to capture more medical information.

8. The method for generating a medical image report based on fine-grained optimal viewing angle constraints according to claim 1, characterized in that: The text feature extractor uses a Transformer structure.

9. The method for generating a medical image report based on fine-grained optimal viewing angle constraints according to claim 1, characterized in that: The fine-grained optimal view constraint method adopts multiple natural language generation (NLG) metrics and clinical effect metrics as evaluation indicators, including BLEU-n, CIDEr and ROUGE-L scores.

Images