Chest radiation image report generation method and device, electronic equipment and storage medium

Abstract

The invention provides a chest radiation image report generation method and device, electronic equipment and a storage medium. The method comprises the following steps: obtaining visual feature images of a plurality of anatomical region categories corresponding to a chest radiographic image by using an anatomical region detector; identifying the visual feature map according to an anatomical region selection module and an anomaly classification module, and determining a text generation region in the visual feature map; performing contrast attention enhancement on the text generation area to obtain a residual visual feature corresponding to the text generation area; the preset language model and the residual visual features are utilized to obtain multiple texts corresponding to the text generation region, the multiple texts are utilized to generate the chest radiographic image report, and the complex task of generating the complete chest radiographic image report is effectively decomposed by paying attention to the local anatomical region features of the chest image. And the accuracy and the consistency of the generated text are improved. And meanwhile, text generation is combined with a local anatomical region, so that the interpretability and the safety of the report are enhanced.

Description

Technical Field

[0001] The present disclosure relates to the field of deep learning technology, and in particular to a method, device, electronic device and storage medium for generating a chest radiography report. Background Art

[0002] As a common medical imaging method, chest radiography (i.e. chest X-ray) plays a key role in diagnosing chest diseases. However, with the surge in medical imaging data and the shortage of radiologists, traditional manual interpretation and report writing methods face huge challenges. To address this problem, the technology of automatically generating radiology reports has emerged.

[0003] At present, the relevant technology mainly combines the convolutional neural network (CNN) in deep learning for image feature extraction and the recurrent neural network (RNN) or its variants (such as Transformer) for text generation to automatically generate radiology reports for chest X-rays. However, this solution faces multiple challenges: 1) The report structure is complex and requires accurate description of medical observations in multiple anatomical regions, making it difficult to generate complete and accurate reports; 2) The data is seriously biased, with normal areas dominating the generated content, and the model easily ignores subtle abnormal areas, resulting in incorrect descriptions; 3) Interpretability and security are critical, and opaque report generation systems are difficult to gain the trust of the medical community. Summary of the invention

[0004] The present invention provides a chest radiography report generation method, device, electronic device and storage medium.

[0005] According to a first aspect of the present disclosure, a method for generating a chest radiography report is provided, the method comprising: using an anatomical region detector to obtain visual feature images of multiple anatomical region categories corresponding to a chest radiography; identifying the visual feature map based on an anatomical region selection module and an abnormality classification module to determine a text generation region in the visual feature map; performing contrast attention enhancement on the text generation region to obtain a residual visual feature corresponding to the text generation region; using a preset language model and the residual visual feature to obtain a plurality of texts corresponding to the text generation region, and using the plurality of texts to generate a chest radiography report.

[0006] In some embodiments of the present disclosure, an anatomical region detector is used to obtain visual feature images of multiple anatomical region categories corresponding to a chest radiograph, including: using the anatomical region detector to determine a potential anatomical region corresponding to the chest radiograph; mapping the potential anatomical region onto a backbone feature map, and extracting multiple anatomical region feature maps of a preset size, wherein the backbone feature map is obtained by extracting features of the chest radiograph by a backbone network; classifying the multiple anatomical region feature maps according to an anatomical region classifier in the anatomical region detector to obtain an anatomical region category corresponding to each anatomical region feature map; and determining a visual feature map corresponding to a target anatomical region category in the anatomical region category.

[0007] In some embodiments of the present disclosure, determining the visual feature map corresponding to the target anatomical region category in the anatomical region category includes: determining the anatomical region category with the highest score in each anatomical region feature map, and using the anatomical region category with the highest score as the target anatomical region category in each anatomical region feature map; using the anatomical region feature map with the highest score in the target anatomical region category in all anatomical region feature maps as the target anatomical region feature map corresponding to the target anatomical region category; performing a linear transformation on each target anatomical region feature map to obtain a visual feature map corresponding to each target anatomical region category. .

[0008] In some embodiments of the present disclosure, contrast attention enhancement is performed on a text generation area to obtain a residual visual feature corresponding to the text generation area, including: using an aggregated attention mechanism to adjust the weight of a normal area corresponding to the text generation area, where the normal area is obtained from a pre-constructed normal area feature pool; determining common information between a normal area having a weight greater than a preset value and the text generation area; clearing the common information from the text generation area to obtain contrast information between the text generation area and the normal area, and using the contrast information as a residual visual feature.

[0009] In some embodiments of the present disclosure, a preset language model and residual visual features are used to obtain multiple texts corresponding to a text generation area, and a chest radiography report is generated using the multiple texts, including: concatenating a feature map of the text generation area with the residual visual features to obtain contrast-enhanced regional visual features; using a pseudo self-attention mechanism to input the regional visual features into the self-attention mechanism of a preset language model to obtain multiple texts corresponding to the text generation area; performing similarity processing on the multiple texts corresponding to the text generation area to obtain multiple target texts, and generating a chest radiography report using the multiple target texts.

[0010] In some embodiments of the present disclosure, similarity processing is performed on multiple texts corresponding to a text generation area to obtain multiple target texts, and a chest radiography report is generated using the multiple target texts, including: determining the similarity of multiple texts corresponding to each feature map in the text generation area, and taking texts with a similarity greater than a preset similarity as similar texts; deleting texts in the similar texts whose text length is less than or equal to a preset text length to obtain multiple target texts after deletion in the anatomical feature map, and generating a chest radiography report using the multiple target texts.

[0011] In some embodiments of the present disclosure, an aggregated attention mechanism is used to adjust the weight of the normal area corresponding to the text generation area. Previously, the method included: randomly extracting a normal image from the training data set images; using an anatomical region detector to determine the anatomical region feature maps corresponding to a preset number of anatomical region categories in each normal image; and aggregating the anatomical region feature maps corresponding to each anatomical region category to construct a normal region feature pool corresponding to a preset number of anatomical region categories.

[0012] According to a second aspect of the present disclosure, there is provided a chest radiographic image report generating device, the device comprising:

[0013] A classification unit, used for obtaining visual feature images of multiple anatomical region categories corresponding to the chest radiographic image by using an anatomical region detector;

[0014] A recognition unit, used for recognizing the visual feature map based on the anatomical region selection module and the abnormality classification module, and determining the text generation region in the visual feature map;

[0015] A contrastive attention enhancement unit is used to perform contrastive attention enhancement on the text generation area to obtain the residual visual features corresponding to the text generation area;

[0016] The report generating unit is used to obtain a plurality of texts corresponding to the text generating area by using a preset language model and residual visual features, and to generate a chest radiography report by using the plurality of texts.

[0017] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0018] at least one processor; and

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0021] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0022] The chest radiography report generation method, device, electronic device and storage medium provided by the present disclosure obtain visual feature images of multiple anatomical region categories corresponding to the chest radiography by using an anatomical region detector; identify the visual feature map according to the anatomical region selection module and the abnormal classification module, and determine the text generation region in the visual feature map; perform contrast attention enhancement on the text generation region to obtain the residual visual features corresponding to the text generation region; use a preset language model and residual visual features to obtain multiple texts corresponding to the text generation region, and use the multiple texts to generate a chest radiography report, realizing a contrast attention mechanism based on anatomical regions, and by focusing on the local anatomical region features of the chest image, effectively decomposing the complex task of generating a complete chest radiography report, and improving the accuracy and consistency of the generated text. At the same time, by combining text generation with local anatomical regions, the interpretability and security of the report are enhanced.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0025] Figure 1 A schematic diagram of a process for generating a chest radiographic image report provided by an embodiment of the present disclosure;

[0026] Figure 2 A schematic diagram of a flow chart of another method for generating a chest radiographic image report provided by an embodiment of the present disclosure;

[0027] Figure 3 A schematic diagram of a flow chart of another method for generating a chest radiographic image report provided by an embodiment of the present disclosure;

[0028] Figure 4 A schematic diagram of a flow chart of another method for generating a chest radiographic image report provided by an embodiment of the present disclosure;

[0029] Figure 5 A schematic diagram of a deep neural network structure of a specific chest radiography report generation method provided by an embodiment of the present disclosure;

[0030] Figure 6A flowchart of a specific chest radiographic image report generation method provided by an embodiment of the present disclosure;

[0031] Figure 7 A schematic diagram of the structure of a chest radiographic image report generating device provided by an embodiment of the present disclosure;

[0032] Figure 8 A schematic block diagram of an example electronic device 800 provided for an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0034] As an important tool in medical diagnosis, chest X-rays can reveal abnormalities in chest structures such as the lungs, heart, and ribs, and are crucial for the early detection and treatment of diseases. However, with the advancement of medical technology and the increase in the number of patients, the traditional method of manually interpreting and writing radiology reports has been unable to meet clinical needs. On the one hand, the massive amount of image data that radiologists need to process increases their workload; on the other hand, the shortage of professional radiologists further exacerbates this problem. Therefore, the automated generation of radiology reports has become an urgent research direction, aiming to assist doctors through artificial intelligence and machine learning technologies to improve diagnostic efficiency and accuracy.

[0035] Currently, the technology for automatically generating radiology reports in related technologies is mainly based on deep learning models, especially the combination of convolutional neural networks (CNN) and recurrent neural networks (RNN) or their variants (such as LSTM, Transformer).

[0036] Although relevant technologies have made some progress in automatically generating radiology reports, there are still many challenges: First, a radiology report consists of multiple sentences, each of which describes a specific medical observation of a specific anatomical area. It is very difficult to generate a report that fully describes medical facts and accurately diagnoses clinically. In addition, there are serious data biases in chest X-ray images and reports. The vast majority of a chest X-ray image is normal, and the descriptions corresponding to these normal areas dominate the final generated radiology report. Abnormal areas account for a very small proportion of chest X-ray images, and the lesions are subtle and difficult to detect. Learning-based models often ignore abnormal areas and generate incorrect descriptions. Finally, explainability and security are crucial in the medical field, and a highly accurate but opaque report generation system may not be adopted.

[0037] In order to solve the problems in the related art, the chest radiography report generation method proposed in the present disclosure generates a chest radiography report by using a contrastive attention mechanism based on anatomical regions. A text description of a specific anatomical region is generated based on the local visual features of the chest anatomical region rather than the image-level visual features of the entire chest X-ray, and finally the descriptions of these anatomical regions are combined into a complete report. A report generation framework based on local anatomical regions is implemented, which decomposes the difficult task of generating a complete and consistent report from the entire image into a simple task of generating a series of short and consistent sentences for isolated anatomical regions. At the same time, the report generated by this method visually associates each sentence with the predicted bounding box around the anatomical region, thereby introducing a high degree of interpretability to the report generation process. Through visual association, radiologists can easily verify the correctness of each generated sentence, which increases the interpretability and credibility of the model predictions.

[0038] The following describes the chest radiography report generation method, device, electronic device and storage medium of the embodiments of the present disclosure with reference to the accompanying drawings.

[0039] Figure 1 The following is a flow chart of a method for generating a chest radiographic image report provided by an embodiment of the present disclosure. Figure 1 As shown, the method includes:

[0040] Step 101: using an anatomical region detector, obtain visual feature images of multiple anatomical region categories corresponding to the chest radiographic image.

[0041] In some embodiments, the present disclosure uses Faster R-CNN as a chest X-ray image anatomical region detector, whose backbone network is ResNet-50 pre-trained on ImageNet. Faster R-CNN includes a region proposal network (RPN), which generates a proposed region (i.e., a bounding box of a potential anatomical region) based on a backbone feature map extracted from an input image by the backbone network.

[0042] The present invention can be based on an anatomical region detector, which first obtains multiple anatomical region feature maps corresponding to a chest radiography image; then uses an anatomical region classifier in the anatomical region detector to classify the multiple anatomical region feature maps to obtain the anatomical region category corresponding to each anatomical region feature map; and finally uses the anatomical region category to determine the visual feature map.

[0043] Among them, in the present disclosure, the number of multiple anatomical region categories can be specifically 29, and can also be adjusted according to actual conditions, which is not limited in the embodiments of the present disclosure.

[0044] Step 102: Based on the anatomical region selection module and the abnormality classification module, the visual feature map is identified to determine the text generation area in the visual feature map.

[0045] In some embodiments, after obtaining the visual feature map, the anatomical region selection module and the abnormality classification module may identify the visual feature map simultaneously or sequentially, depending on actual circumstances and is not limited in the embodiments disclosed herein.

[0046] Both the anatomical region selection module and the anomaly classification module are binary classifiers composed of multilayer perceptrons (MLP), which gradually reduce the input features to an output logit, which is used to predict whether a region is selected to generate text (sentence) and whether it is abnormal.

[0047] Anatomical region selection refers to which of the detected anatomical regions are selected for report generation, i.e., generating text. The anatomical region selection module is trained with a training set, and after training, inference is performed to determine which region is selected.

[0048] Since radiology reports focus on describing abnormal contents in chest radiographs, the abnormality classification module detects whether each detected anatomical region is abnormal (referring to lesions). If it is abnormal, then this region is selected to generate text. The anomaly detector is also trained on the training set, and after training, it is used to determine whether a region is abnormal.

[0049] Step 103: Perform contrast attention enhancement on the text generation area to obtain the residual visual features corresponding to the text generation area.

[0050] In some embodiments, contrast attention enhancement is to compare the anatomical region image features of the input text generation area with the normal image features of the area and identify the differences, with the aim of making it easier for the model to learn and identify abnormal region content and mitigate the impact of data bias. The present disclosure can first adjust the weights of different anatomical regions in the normal region feature pool through an aggregated attention mechanism, and select the closest normal region features. Then, through differential attention, contrast information is learned, and the residual actual features are matched for report generation.

[0051] Step 104: Utilize the preset language model and the residual visual features to obtain a plurality of texts corresponding to the text generation area, and utilize the plurality of texts to generate a chest radiography report.

[0052] In some embodiments, the present disclosure may adopt the GPT-2 Medium model with 355M parameters fine-tuned on PubMed as the preset language model for text generation.

[0053] Since the final radiology report (i.e., chest radiography report) is formed by connecting texts generated by the selected anatomical regions, the present disclosure can connect multiple texts generated by the preset language model to obtain the final chest radiography report.

[0054] In summary, the technical solution provided by the present disclosure uses an anatomical region detector to obtain visual feature images of multiple anatomical region categories corresponding to chest radiographs; identifies the visual feature map based on the anatomical region selection module and the abnormality classification module to determine the text generation region in the visual feature map; performs contrast attention enhancement on the text generation region to obtain the residual visual features corresponding to the text generation region; uses a preset language model and residual visual features to obtain multiple texts corresponding to the text generation region, and uses the multiple texts to generate a chest radiograph report, thereby realizing an anatomical region-based contrast attention mechanism, and by focusing on the local anatomical region features of the chest image, effectively decomposes the complex task of generating a complete chest radiograph report, and improves the accuracy and consistency of the generated text. At the same time, by combining text generation with local anatomical regions, the interpretability and security of the report are enhanced.

[0055] As a possible implementation, Figure 2 The flowchart of another method for generating a chest radiography report is shown. Based on the above embodiment, the specific process of obtaining visual feature images of multiple anatomical region categories corresponding to the chest radiography using an anatomical region detector includes the following steps:

[0056] Step 201: using an anatomical region detector, determine a potential anatomical region corresponding to a chest radiographic image.

[0057] In some embodiments, the chest radiograph to be detected is input into a trained anatomical region detector. The backbone network of the anatomical region detector first extracts features from the chest radiograph to generate a backbone feature map. The anatomical region detector generates potential anatomical regions based on the backbone feature map and the region proposal network, and the potential anatomical regions refer to proposed regions (bounding boxes) that may include anatomical regions.

[0058] Step 202: Map the potential anatomical region onto the backbone feature map, and extract multiple anatomical region feature maps of a preset size. The backbone feature map is obtained by extracting features from the chest radiography image through the backbone network.

[0059] In some embodiments, before mapping the potential anatomical region to the backbone feature map, the present disclosure may use a pre-trained backbone network (such as ResNet, VGG, etc.) to extract features from the chest radiography image. This backbone network generates a backbone feature map that encodes rich information in the original image, but the size is usually much smaller than the original image to facilitate subsequent processing.

[0060] The region of interest (ROI) pooling layer maps each potential anatomical region to the backbone feature map. For each potential anatomical region, the ROI pooling layer finds the corresponding region on the feature map and maps it to a uniform preset size (such as spatial size 7x7). This ensures that regardless of the size of the original region proposal, the resulting anatomical region feature map has the same size, which is convenient for subsequent processing.

[0061] Step 203: Classify the multiple anatomical region feature maps according to the anatomical region classifier in the anatomical region detector to obtain the anatomical region category corresponding to each anatomical region feature map.

[0062] In some embodiments, after being processed by the ROI pooling layer, the obtained anatomical region feature maps are classified according to the standard process of FasterR-CNN. Specifically, these anatomical region feature maps are fed into a series of fully connected layers (or convolutional layers plus global average pooling), and then an anatomical region classifier (such as a softmax classifier) ​​is used to predict the anatomical region category (a total of 29 categories) to which each anatomical region feature map belongs.

[0063] Step 204: Determine a visual feature map corresponding to a target anatomical region category in the anatomical region category.

[0064] In some embodiments, in order to obtain the regional visual features of each anatomical region, the present disclosure determines the anatomical region feature map with the "highest score" for each anatomical region category, that is, the target anatomical region feature map. For each target anatomical region feature map, 2D average pooling is applied in the spatial dimension to further reduce the size of the feature map while retaining key information. Then, the pooled feature map is converted into a fixed-length feature vector through a linear transformation (such as a fully connected layer), which represents the visual feature map of the target anatomical region category.

[0065] Specifically, the present invention can determine the anatomical region category with the highest score in each anatomical region feature map, and use the anatomical region category with the highest score as the target anatomical region category for each anatomical region feature map; use the anatomical region feature map with the highest score among the target anatomical region categories in all anatomical region feature maps as the target anatomical region feature map corresponding to the target anatomical region category; perform a linear transformation on each target anatomical region feature map to obtain a visual feature map corresponding to each target anatomical region category.

[0066] It is worth noting that if an anatomical region category does not reach the highest score in at least one anatomical region feature map, the anatomical region category is considered as an undetected category. The undetected category will be automatically deselected by the region selection module.

[0067] In summary, by utilizing the anatomical region detector, multiple local anatomical region features are effectively extracted and classified from chest radiographic images, thereby improving the accuracy of chest radiographic image recognition.

[0068] As a possible implementation, Figure 3 The flowchart of another method for generating a chest radiology report is shown. Based on the above embodiment, the specific process of performing contrast attention enhancement on the text generation area to obtain the residual visual features corresponding to the text generation area includes the following steps:

[0069] Step 301: Using the aggregated attention mechanism, adjust the weight of the normal area corresponding to the text generation area, where the normal area is obtained from a pre-constructed normal area feature pool.

[0070] In some embodiments, the weight of the normal area corresponding to the text generation area is adjusted using an aggregated attention mechanism. Previously, the present disclosure also includes: randomly extracting normal images from the training data set images; using an anatomical region detector to determine the anatomical region feature maps corresponding to a preset number of anatomical region categories in each normal image; and collecting the anatomical region feature maps corresponding to each anatomical region category to construct a normal region feature pool corresponding to a preset number of anatomical region categories. In other words, the present disclosure can first collect a normal chest radiographic image pool of size 1000, and these normal images are randomly extracted from the training data set images. For each normal image, the anatomical region detector is used to extract the detection boxes and anatomical region features corresponding to 29 anatomical regions. Then, the anatomical region feature maps corresponding to each anatomical region category are collected together to form 29 normal region feature pools for the anatomical region, each feature pool having 1000 normal features of the region for subsequent contrast attention enhancement.

[0071] For each anatomical region in the text generation region, the images in the normal region feature pool are normal, and there is no ranking order between them. Therefore, when capturing contrast information, all normal anatomical regions in the anatomical region category are treated equally. However, some normal anatomical region images are noisy (for example, the orientation information and rotation angle are different from the input image region). If these images are directly compared with the anatomical region feature map of the input text generation region, it will hinder the contrast attention mechanism from effectively capturing the accurate abnormal region. Therefore, the aggregate attention mechanism increases the weight of normal regions of the same anatomical region category close to the current text generation region through dot product attention, while reducing the weight of normal regions of the same anatomical region category not close to the current text generation region. This method gives priority to normal regions that are closer to the current text generation region and filters out normal regions with different appearances, thereby enhancing the effect of the contrast process.

[0072] Step 302: Determine common information between a normal region having a weight greater than a preset value and a text generation region.

[0073] Step 303: Clear common information from the text generation area, obtain contrast information between the text generation area and the normal area, and use the contrast information as residual visual features.

[0074] In some embodiments, in order to learn the contrast information between the input text generation region and the closest normal region, the present disclosure first attempts to capture their similarity, i.e., common information, and then removes this part of the similarity from the input text generation region to obtain the contrast information. In this way, the residual visual features of the input text generation region are regarded as capturing the contrast information of the difference attributes between the text generation region and the normal region.

[0075] In summary, the present invention strengthens the feature weights of normal regions by aggregating attention and combines differential attention to learn contrast information, thereby effectively improving the accuracy and pertinence of report generation.

[0076] As a possible implementation, Figure 4 The flowchart of another method for generating a chest radiography report is shown. Based on the above embodiment, a preset language model and residual visual features are used to obtain multiple texts corresponding to the text generation area, and the specific process of generating a chest radiography report using the multiple texts includes the following steps:

[0077] Step 401: Cascading the feature map of the text generation area with the residual visual feature to obtain contrast-enhanced regional visual features.

[0078] In some embodiments, when generating a sentence corresponding to a text generation region, the present disclosure first cascades a feature map of the input text generation region with the residual visual feature to obtain a final contrast-enhanced regional visual feature.

[0079] Step 402: Using a pseudo self-attention mechanism, the regional visual features are input into the self-attention mechanism of a preset language model to obtain multiple texts corresponding to the text generation area.

[0080] In some embodiments, the present disclosure uses pseudo self-attention to directly inject the final regional visual features into the self-attention mechanism of the model. In this way, the text generation process is conditionally generated based on the regional visual features, thereby obtaining multiple texts corresponding to the text generation area.

[0081] Step 403: performing similarity processing on the multiple texts corresponding to the text generation area to obtain multiple target texts, and using the multiple target texts to generate a chest radiology image report.

[0082] In some embodiments, if the lesion spans multiple regions or multiple anatomically similar regions, similar sentences may be generated for these regions, resulting in repeated sentences in the generated report. To remove these duplications, the present disclosure may use BERTScore to determine similarity, and delete shorter sentences and retain longer sentences. This is because longer sentences tend to contain more clinically relevant information, for example, "The cardiac diaphragm contour and hilar contour are normal." and "The cardiac diaphragm contour is normal." These two sentences are very similar, but the longer sentence contains more medically relevant information.

[0083] Specifically, the present disclosure can determine the similarity of multiple texts corresponding to each feature map in the text generation area, and take the texts with similarity greater than the preset similarity as similar texts; delete the texts with text length less than or equal to the preset text length in the similar texts to obtain multiple target texts after deletion in the anatomical feature map, and use the multiple target texts to generate a chest radiography report.

[0084] In summary, the present disclosure improves the accuracy and interpretability of chest radiography reports by combining residual visual features to generate text, and further improves the accuracy by performing similarity processing on the text when generating the report.

[0085] As a possible implementation method, based on the above embodiment, Figure 5 As shown, the present disclosure provides a schematic diagram of a deep neural network structure of a specific chest radiology image report generation method, and as shown Figure 6 As shown, the present disclosure provides a flowchart of a specific method for generating a chest radiology image report.

[0086] In some embodiments, reference Figure 5 , Figure 6The present disclosure uses an anatomical region detector to determine the potential anatomical region corresponding to a chest radiographic image; maps the potential anatomical region on a backbone feature map, extracts multiple anatomical region feature maps of a preset size, and the backbone feature map is obtained by extracting features of the chest radiographic image through a backbone network; classifies multiple anatomical region feature maps according to an anatomical region classifier in the anatomical region detector (for example, classifies multiple anatomical region feature maps according to 29 anatomical region categories), and obtains the anatomical region category corresponding to each anatomical region feature map; determines the anatomical region category with the highest score in each anatomical region feature map, and uses the anatomical region category with the highest score as the target anatomical region category of each anatomical region feature map; uses the anatomical region feature map with the highest score in the target anatomical region category in all anatomical region feature maps as the target anatomical region feature map corresponding to the target anatomical region category; performs a linear transformation on each target anatomical region feature map to obtain a visual feature map corresponding to each target anatomical region category. Then, the visual feature map is identified according to the anatomical region selection module and the abnormal classification module, and the text generation area in the visual feature map is determined (the anatomical region feature is selected, i.e., the area used to generate sentences). Afterwards, the weight of the normal region corresponding to the text generation region is adjusted by using the aggregate attention mechanism. The normal region is obtained from the pre-constructed normal region feature pool; the common information between the normal region and the text generation region whose weight is greater than the preset value is determined; the common information is removed from the text generation region to obtain the contrast information between the text generation region and the normal region, and the contrast information is used as the residual visual feature. The feature map of the text generation region is cascaded with the residual visual feature to obtain the contrast enhanced regional visual feature. Finally, the pseudo self-attention mechanism is used to input the regional visual feature into the self-attention mechanism of the preset language model to obtain multiple texts corresponding to the text generation region; the multiple texts corresponding to the text generation region are processed for similarity to obtain multiple target texts, and the multiple target texts are used to generate a chest radiology report.

[0087] Among them, the construction of the normal region feature pool (normal anatomical region feature pool or normal image feature pool based on anatomical region) specifically includes: randomly extracting normal images from the training data set images; using an anatomical region detector to determine the anatomical region feature maps corresponding to a preset number of anatomical region categories in each normal image; and collecting the anatomical region feature maps corresponding to each anatomical region category to construct a normal region feature pool corresponding to a preset number of anatomical region categories.

[0088] It is important to note that the training of the model proposed in the present disclosure includes three steps. First, only the anatomical region detector is trained, then the region detector is trained in combination with two binary classifiers (anatomical region selection and abnormality classification), and finally the complete model is trained end-to-end when all parameters are trainable. In order to train the preset language model, the present disclosure can only use regional visual features with corresponding reference sentences, and assume that the region selection module can correctly select these regions where sentences need to be generated during testing. For regions containing multiple sentences, these sentences will be connected so that the model can learn to predict multiple sentences in this case.

[0089] For the above training process, the overall training loss is defined as:

[0090] L=λ 区域检测 L 区域检测 +λ 区域选择 L 区域选择 +λ 异常分类 L 异常分类 +λ 语言模型 L 语言模型

[0091] Among them, the anatomical region detector loss L 区域检测 is the loss of the Faster R-CNN target detector. Region selection loss L 区域选择 and the anomaly classification loss L 异常分类 is the binary cross entropy loss of the two binary classifiers. The language model loss L 语言模型 is the cross entropy loss of the language model. i is the weight parameter.

[0092] Corresponding to the above-mentioned chest radiography report generation method, the present invention also provides a chest radiography report generation device. Since the device embodiment of the present invention corresponds to the above-mentioned method embodiment, the details not disclosed in the device embodiment can be referred to the above-mentioned method embodiment, and will not be repeated in the present invention.

[0093] Figure 7 A schematic diagram of the structure of a chest radiographic image report generating device provided by an embodiment of the present disclosure is shown in FIG. Figure 7 As shown, the device comprises:

[0094] The classification unit 710 is used to obtain visual feature images of multiple anatomical region categories corresponding to the chest radiographic image using an anatomical region detector;

[0095] The recognition unit 720 is used to recognize the visual feature map based on the anatomical region selection module and the abnormality classification module, and determine the text generation area in the visual feature map;

[0096] A contrast attention enhancement unit 730 is used to perform contrast attention enhancement on the text generation area to obtain residual visual features corresponding to the text generation area;

[0097] The report generating unit 740 is used to obtain a plurality of texts corresponding to the text generating area by using a preset language model and residual visual features, and generate a chest radiography report by using the plurality of texts.

[0098] In some embodiments of the present disclosure, the classification unit 710 is used to: determine the potential anatomical region corresponding to the chest radiography image using an anatomical region detector; map the potential anatomical region onto a backbone feature map, and extract multiple anatomical region feature maps of a preset size, where the backbone feature map is obtained by extracting features from the chest radiography image using a backbone network; classify the multiple anatomical region feature maps according to an anatomical region classifier in the anatomical region detector to obtain an anatomical region category corresponding to each anatomical region feature map; and determine a visual feature map corresponding to a target anatomical region category in the anatomical region category.

[0099] In some embodiments of the present disclosure, the classification unit 710 is used to: determine the anatomical region category with the highest score in each anatomical region feature map, and use the anatomical region category with the highest score as the target anatomical region category of each anatomical region feature map; use the anatomical region feature map with the highest score in the target anatomical region category in all anatomical region feature maps as the target anatomical region feature map corresponding to the target anatomical region category; perform linear transformation on each target anatomical region feature map to obtain a visual feature map corresponding to each target anatomical region category.

[0100] In some embodiments of the present disclosure, the contrast attention enhancement unit 730 is used to: use the aggregated attention mechanism to adjust the weight of the normal area corresponding to the text generation area, where the normal area is obtained from a pre-constructed normal area feature pool; determine the common information between the normal area whose weight is greater than a preset value and the text generation area; clear the common information from the text generation area, obtain the contrast information between the text generation area and the normal area, and use the contrast information as the residual visual feature.

[0101] In some embodiments of the present disclosure, the report generation unit 740 is used to: concatenate the feature map of the text generation area with the residual visual feature to obtain contrast-enhanced regional visual features; use a pseudo self-attention mechanism to input the regional visual features into the self-attention mechanism of a preset language model to obtain multiple texts corresponding to the text generation area; perform similarity processing on the multiple texts corresponding to the text generation area to obtain multiple target texts, and use the multiple target texts to generate a chest radiography report.

[0102] In some embodiments of the present disclosure, the report generation unit 740 is used to: determine the similarity of multiple texts corresponding to each feature map in the text generation area, and use texts with similarity greater than a preset similarity as similar texts; delete texts with a text length less than or equal to a preset text length in the similar texts to obtain multiple target texts after deletion in the anatomical feature map, and use the multiple target texts to generate a chest radiography report.

[0103] In some embodiments of the present disclosure, the device also includes: a construction module for randomly extracting normal images from the training data set images before using the aggregated attention mechanism to adjust the weights of the normal areas corresponding to the text generation areas; using an anatomical region detector to determine anatomical region feature maps corresponding to a preset number of anatomical region categories in each normal image; and aggregating the anatomical region feature maps corresponding to each anatomical region category to construct a normal region feature pool corresponding to a preset number of anatomical region categories.

[0104] It should be noted that the above explanation of the method embodiment is also applicable to the device of this embodiment, and the principle is the same, which is not limited in this embodiment.

[0105] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0106] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0107] like Figure 8As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 802 or a computer program loaded from a storage unit 808 to a RAM (Random Access Memory) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An I / O (Input / Output) interface 805 is also connected to the bus 804.

[0108] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0109] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as a chest radiography report generation method. For example, in some embodiments, the chest radiography report generation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the aforementioned chest radiography report generating method in any other appropriate manner (eg, by means of firmware).

[0110] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0111] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0112] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0113] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0114] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0115] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0116] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0117] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0118] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating a chest radiographic image report, characterized in that: The method comprises: Using an anatomical region detector, visual feature images of multiple anatomical region categories corresponding to the chest radiographic image are obtained; According to the anatomical region selection module and the abnormality classification module, the visual feature map is identified to determine the text generation area in the visual feature map; Performing contrast attention enhancement on the text generation area to obtain residual visual features corresponding to the text generation area; A preset language model and the residual visual features are used to obtain a plurality of texts corresponding to the text generation area, and a chest radiography report is generated using the plurality of texts.

2. The method according to claim 1, characterized in that: The method of obtaining visual feature images of multiple anatomical region categories corresponding to the chest radiographic image by using the anatomical region detector includes: Using an anatomical region detector, determining potential anatomical regions corresponding to the chest radiographic image; Mapping the potential anatomical region on a backbone feature map, extracting a plurality of anatomical region feature maps of a preset size, wherein the backbone feature map is obtained by extracting features of the chest radiographic image through a backbone network; Classifying the plurality of anatomical region feature maps according to the anatomical region classifier in the anatomical region detector to obtain an anatomical region category corresponding to each anatomical region feature map; Determine a visual feature map corresponding to a target anatomical region category in the anatomical region categories.

3. The method according to claim 2, characterized in that The step of determining a visual feature map corresponding to a target anatomical region category in the anatomical region category comprises: Determine the anatomical region category with the highest score in each anatomical region feature map, and use the anatomical region category with the highest score as the target anatomical region category in each anatomical region feature map; The anatomical region feature map with the highest score in all anatomical region feature maps of the target anatomical region category is used as the target anatomical region feature map corresponding to the target anatomical region category; Each target anatomical region feature map is linearly transformed to obtain a visual feature map corresponding to each target anatomical region category.

4. The method according to claim 1, characterized in that: The contrasting attention enhancement is performed on the text generation area to obtain the residual visual features corresponding to the text generation area, including: Using an aggregated attention mechanism, adjusting the weight of a normal region corresponding to the text generation region, wherein the normal region is obtained from a pre-constructed normal region feature pool; Determine common information between a normal area having a weight greater than a preset value and the text generation area; The common information is removed from the text generation region to obtain contrast information between the text generation region and the normal region, and the contrast information is used as a residual visual feature.

5. The method according to claim 1, characterized in that The method of using the preset language model and the residual visual features to obtain a plurality of texts corresponding to the text generation area, and using the plurality of texts to generate a chest radiology image report includes: Cascading the feature map of the text generation area with the residual visual feature to obtain a contrast-enhanced regional visual feature; Using a pseudo self-attention mechanism, the visual features of the region are input into a self-attention mechanism of a preset language model to obtain a plurality of texts corresponding to the text generation region; A plurality of texts corresponding to the text generation area are processed for similarity to obtain a plurality of target texts, and a chest radiography report is generated using the plurality of target texts.

6. The method according to claim 5, characterized in that The performing similarity processing on the multiple texts corresponding to the text generation area to obtain multiple target texts, and using the multiple target texts to generate a chest radiology image report includes: Determine the similarity of multiple texts corresponding to each feature graph in the text generation area, and take the texts with similarity greater than a preset similarity as similar texts; The texts whose length is less than or equal to the preset text length are deleted from the similar texts to obtain a plurality of target texts after being deleted from the anatomical feature map, and a chest radiography report is generated using the plurality of target texts.

7. The method according to claim 4, characterized in that Before the method uses the aggregate attention mechanism to adjust the weight of the normal area corresponding to the text generation area, the method includes: Randomly extract normal images from the training dataset images; Determine, by using the anatomical region detector, anatomical region feature maps corresponding to a preset number of anatomical region categories in each normal image; The anatomical region feature maps corresponding to each anatomical region category are collected to construct a normal region feature pool corresponding to a preset number of anatomical region categories.

8. A chest radiographic image report generating device, characterized in that: The device comprises: A classification unit, used for obtaining visual feature images of multiple anatomical region categories corresponding to the chest radiographic image by using an anatomical region detector; A recognition unit, configured to recognize the visual feature map based on the anatomical region selection module and the abnormality classification module, and determine a text generation region in the visual feature map; A contrast attention enhancement unit, used for performing contrast attention enhancement on the text generation area to obtain a residual visual feature corresponding to the text generation area; The report generating unit is used to obtain a plurality of texts corresponding to the text generating area by using a preset language model and the residual visual features, and generate a chest radiography report by using the plurality of texts.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

Images