Region-level retrieval enhanced multi-view chest radiograph report generation method

Abstract

The invention discloses a region-level retrieval enhanced multi-view chest radiography report generation method. According to the method, pairwise cross attention fusion is carried out on multi-view chest radiograph image features, and feature mapping is carried out to serve as a visual prompt for inputting a large language model. A chest radiograph anatomical region-level knowledge base containing a plurality of semantic entities is established, and each entity contains region features, region tags, entity description and lesion tags of an anatomical region. Different anatomical areas are obtained by adopting a detection algorithm for an input chest radiograph image, and text descriptions of entities corresponding to the related anatomical areas are retrieved from a knowledge base to serve as text prompts of a large language model. And finally, the large language model generates a corresponding diagnosis report by adopting an autoregression generation mode based on visual prompt, text prompt and a report generation instruction. According to the method, feature fusion is effectively carried out on the multi-view chest radiograph images, the prior knowledge of the chest radiograph images in the anatomical area level is effectively fused into the report generation process, and the method is suitable for the field of chest radiograph report generation.

Description

Technical Field

[0001] The present invention relates to the fields of multi-view image feature fusion, retrieval enhancement generation and medical image analysis, and in particular to a method for generating a multi-view chest X-ray report with regional-level retrieval enhancement. Background Art

[0002] Automatic generation of chest X-ray reports aims to automatically analyze the pathological features in chest X-rays through intelligent algorithms, automatically generate diagnostic text to describe the lesions in chest X-ray images, provide instant reference information for radiologists to assist in writing reports, and reduce the risk of missed diagnosis and misdiagnosis. With the rapid development of large language models (LLMs), it has become a trend to integrate them into the automatic generation process of chest X-ray image reports. However, most current automatic generation methods of chest X-ray reports are still limited to the visual information analysis of a single chest X-ray image, ignoring the importance of comprehensive diagnosis of multi-view chest X-rays in clinical practice. Combining chest X-ray information from multiple perspectives can more accurately locate lesions, such as combining frontal and lateral chest X-ray images to accurately locate the specific location of lung nodules. In addition, most existing chest X-ray report generation methods are based on end-to-end training methods, directly generating diagnostic text from the overall visual information of the image, and not analyzing each anatomical region in the chest X-ray at the regional level. At the same time, existing models do not utilize existing semantic knowledge of anatomical regions of chest X-rays. In order to improve the accuracy and interpretability of chest X-ray image diagnosis, it is necessary to fuse the multi-view chest X-ray image features to obtain a more effective visual representation. At the same time, it is also necessary to introduce the existing regional-level chest X-ray anatomical region semantic knowledge. Summary of the invention

[0003] The present invention proposes a method for generating a multi-view chest X-ray report with enhanced regional-level retrieval. Specifically, the present invention adopts a large language model (LLM) as a text generation decoder, fuses the multi-view chest X-ray image features as the visual cues of the large language model, and uses the retrieved regional-level entity descriptions as the text cues of the large language model. First, the present invention proposes to perform pairwise cross-attention fusion on the multi-view chest X-ray image features to effectively fuse the visual features of the multi-view chest X-ray images. Then, the visual features are further mapped to provide effective visual context information, i.e., visual cues, for the large language model. Secondly, a chest X-ray anatomical region-level knowledge base containing 11 anatomical regions is established, each anatomical region contains 3,000 entities, and each entity contains the regional features, regional labels, entity descriptions, and lesion labels of an anatomical region.

[0004] A method for generating a multi-view chest radiograph report with enhanced regional retrieval, the specific steps of which are as follows:

[0005] 1. A method for generating a multi-view chest X-ray report with enhanced regional retrieval, characterized in that a large language model is used as a text generation decoder; visual features of multi-view chest X-rays are fused as visual prompts of the large language model, and a pre-built chest X-ray anatomical regional knowledge base is retrieved, and the relevant descriptions of each anatomical region of the input chest X-ray obtained by retrieval are used as text prompts of the large language model;

[0006] 2. According to the method for generating a multi-view chest X-ray report with enhanced regional retrieval as described in claim 1, the specific steps are as follows:

[0007] Step (1), select a chest radiograph containing multi-view images {I1, I2, ..., I N The samples of} and their diagnosis reports R constitute the training data set; the multi-view chest radiographs include three perspectives: PA (posteroanterior), AP (anteroposterior) and Lateral (lateral);

[0008] Step (2), select samples from the Chest ImaGenome dataset and construct a chest X-ray anatomical region-level knowledge base; divide the different regions of each chest X-ray into 11 anatomical regions: right lung, right hilar structure, right lung apex, left lung, left hilar structure, left lung apex, trachea, spine, mediastinum, heart shadow and abdomen; each anatomical region contains N S semantic entities, each entity is recorded as:

[0009] entity i = {v i ,l i ,s i ,a i}

[0010] Among them, entity i represents the i-th entity, v i is the regional feature of the i-th entity encoded by the visual encoder MedCLIP model, l i is the region label of the i-th entity, s i is the entity description of the regional lesion of the i-th entity, a i is the normal or abnormal lesion label of the i-th entity;

[0011] Step (3), using the SwinTransformer model to extract the sequence features of multi-view chest radiographs, the specific calculation formula is as follows:

[0012]

[0013] Among them, E n Represents the nth image I in the multi-view chest radiograph nThe image sequence features encoded and output by the SwinTransformer model, represents the mth image sequence feature of the chest X-ray image at the nth viewing angle, and is the length of the image sequence feature;

[0014] Step (4), the sequence features of the multi-view chest X-ray images are input into the fusion network for pairwise cross-attention feature fusion to obtain the fused visual features. The specific formula of the fusion process is:

[0015]

[0016] in, Represents the fused visual features; d model represents the feature dimension, M is the length of the visual feature, N is the number of multi-view chest radiographs, and MHA is the multi-head attention operation;

[0017] Step (5), use the multi-layer perceptron MLP model to map the visual feature S, and use the mapping result as the visual prompt Prompt of the large language model vision ;

[0018] Step (6), pre-training a Fast R-CNN detector on the Chest ImaGenome dataset and fixing the parameters to detect various anatomical regions of the input multi-view chest X-ray images;

[0019] Step (7), using the pre-trained MedCLIP model to encode the visual features of each anatomical region to extract the visual features of each anatomical region;

[0020] Step (8): for each anatomical region of the input chest X-ray, a corresponding entity is retrieved from the pre-built chest X-ray anatomical region-level knowledge base; the text descriptions of the entities retrieved for each anatomical region of the input chest X-ray are concatenated and then used as text prompts of the large language model:

[0021] Prompt text =[{l1:s1};{l2:S2};…;{l F :s F}]

[0022] Among them, Prompt text It is a text prompt containing the region label and its related entity description. [;] indicates a concatenation operation; {l i :s i} i is the region label of the i-th entity, s i is the entity description of the regional lesion of the i-th entity; F represents the total number of entities contained in the text prompt;

[0023] Step (9), concatenating the visual prompt obtained in step (5), the text prompt obtained in step (8) and the preset report generation instruction prompt, and inputting the concatenated result into the pre-trained large language model to generate a diagnosis report; the generation process is expressed as the following autoregressive equation:

[0024]

[0025] Among them, R represents the complete diagnosis report, the length is L, p represents the probability distribution, p θ represents the probability distribution of the output based on the parameter θ, r t is the tth word in the diagnosis report R, R 0:t-1 Represents the report sequence before the tth word; Prompt vision Indicates visual prompts, Promote txet Indicates a text prompt, H indicates a preset report generation instruction prompt;

[0026] Step (10), freeze the Fast RCNN detection model, MedCLIP model and large language model, and train the SwinTransformer model, fusion network and MLP mapping layer in the following manner; specifically, the optimized loss function is as follows:

[0027]

[0028] in, is the cross entropy loss function, p represents the probability distribution, represents the input multi-view image, and H represents the report generation instruction prompt;

[0029] Step (11), during the test, the multi-view chest X-ray images to be tested are processed to obtain visual prompts and text prompts, and combined with the report generation instruction prompts, they are input into the large language model, and the words at all times are generated by autoregression and spliced ​​to obtain the final output diagnosis report.

[0030] Further, the specific implementation of step (8) is as follows:

[0031] 8-1. Perform region detection on the input chest X-ray and record the detected anatomical region as query region I q , input the MedCLIP model encoding to obtain the query feature v of the query area q =Φ vis (I q ), Φ vis represents the MedCLIP model;

[0032] 8-2. From the constructed chest radiograph anatomical region-level knowledge base, all entities with the same region label as the query region are screened out; for the anatomical regions corresponding to the screened entities, the MedCLIP model encoding is used to extract their feature vectors v j =Φ vis (I j ), indicating that the jth entity corresponds to image I j regional characteristics;

[0033] 8-3. Calculate query feature v q The feature vector v corresponding to the entity in the knowledge base that has the same region label as the query region j The cosine similarity of is as follows:

[0034]

[0035] Among them, N S Represents the number of entities in the knowledge base that have the same region label as the query region;

[0036] 8-4. Select the first K entities with the highest similarity according to the cosine similarity score to initialize the candidate entity set. The corresponding regional feature set is {v1,v2,…,v K};

[0037] 8-5. Aggregate the abnormal attributes of the candidate entity set to determine whether the query area is abnormal; if the number of abnormal candidate entities is greater than or equal to the normal entities, the query area is classified as abnormal, and the abnormal candidate entity with the highest similarity is selected as the retrieved entity; otherwise, the query area is considered normal, and the normal candidate entity with the highest similarity is selected as the retrieved entity;

[0038] 8-6. Repeat the above steps to stitch together the entity text descriptions retrieved from each anatomical region of the input chest X-ray as text prompts for the large language model.

[0039] The beneficial effects of the present invention are as follows:

[0040] In order to simulate the process of doctors examining each anatomical region in turn during clinical diagnosis, the present invention proposes to retrieve entities from a pre-constructed knowledge base, use the text descriptions corresponding to the entities as text prompts for the large language model, and provide regional anatomical region-level knowledge for chest X-ray report generation, which significantly improves the performance of report generation. A large number of experiments on public data sets have shown that this method has achieved the current best performance. The present invention can effectively fuse features of multi-view chest X-ray images, and can effectively integrate anatomical region-level chest X-ray image prior knowledge into the report generation process, and is suitable for the field of chest X-ray report generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flow chart of the present invention;

[0042] Figure 2 It is a schematic diagram of the overall framework of the present invention;

[0043] Figure 3 To demonstrate the effects of the present invention;

[0044] Figure 4 This is a diagram of the entity retrieval effect of the method of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0046] like Figure 1 As shown, the present invention provides a method for generating a multi-view chest X-ray report with enhanced regional-level retrieval, adopts a large language model as a text generation decoder, fuses visual features of multi-view chest X-rays as visual prompts of the large language model, and retrieves descriptions of anatomical regions related to the input chest X-rays as text prompts of the large language model. First, the present invention performs cross-attention fusion on multi-view chest X-ray image features in pairs, and performs feature mapping as visual prompts for input to the large language model. Then, the present invention establishes a chest X-ray anatomical region-level knowledge base containing multiple semantic entities, each entity containing regional features, regional labels, entity descriptions, and lesion labels of the anatomical region. Subsequently, a detection algorithm is used to obtain different anatomical regions for the input chest X-ray images, and text descriptions of entities corresponding to the relevant anatomical regions are retrieved from the knowledge base as text prompts of the large language model. Finally, the large language model generates a corresponding diagnostic report using an autoregressive generation method based on visual prompts, text prompts, and report generation instructions.

[0047] The specific implementation steps are as follows: Figure 2 As shown:

[0048] Step (1), select a chest radiograph containing multi-view images {I1, I2, ..., I N The samples of} and their diagnosis reports R constitute the training data set.

[0049] Step (2): Select samples from the Chest ImaGenome dataset to construct a chest X-ray anatomical region-level knowledge base, and divide the different regions of each chest X-ray into 11 anatomical regions, each of which contains 3,000 semantic entities.

[0050] Step (3), using the SwinTransformer model to extract the sequence features of multi-view chest radiographs, the specific calculation formula is as follows:

[0051]

[0052] Among them, En Represents the nth image I in the multi-view chest radiograph n The image sequence features encoded and output by the SwinTransformer model, Represents the mth image sequence feature of the nth viewing angle chest X-ray image, which is the length of the image sequence feature.

[0053] Step (4), input the sequence features of the multi-view chest X-ray images into the fusion network for pairwise cross-attention feature fusion to obtain the fused visual features. The specific formula of the fusion process is:

[0054]

[0055] in, Represents the fused visual features; d model represents the feature dimension, M is the length of the visual feature, N is the number of multi-view chest X-ray images, and MHA is the multi-head attention operation.

[0056] Step (5) uses a multi-layer perceptron (MLP) model to further map the visual features as a visual prompt for the large language model. vision ;

[0057] Step (6), pre-training a Fast R-CNN detector on the Chest ImaGenome dataset and fixing the parameters to detect various anatomical regions of the input multi-view chest X-ray images;

[0058] Step (7), using the pre-trained MedCLIP model to perform visual feature encoding on each anatomical region to extract the visual features of each region;

[0059] Step (8), for each anatomical region of the input chest X-ray, searching in a pre-constructed chest X-ray anatomical region-level knowledge base to obtain a corresponding entity;

[0060] Step (9), concatenate the text descriptions of the entities retrieved from each anatomical region of the input chest X-ray as text prompts for the large language model:

[0061] Prompt text =[{l1:s1};{l2:s2};…;{l F :s F}]

[0062] Among them, Prompt text It is a text prompt containing the region label and its related entity description. [;] indicates a concatenation operation; {l i :s i} i is the region label of the i-th entity, si is the entity description of the regional lesion of the i-th entity; F represents the total number of entities contained in the text prompt.

[0063] Step (10): concatenate the visual prompt obtained in step (5), the text prompt obtained in step (9), and the report generation instruction prompt and input them into the pre-trained large language model, and generate a diagnostic report using an autoregressive generation method.

[0064] Step (11), freeze the Fast RCNN detection model, MedCLIP model and large language model, and train the model's visual encoder SwinTransformer, fusion network and MLP mapping layer.

[0065] Step (12), during the test, the multi-view chest X-ray images to be tested are processed in the above steps to obtain visual prompts and text prompts, which are combined with report generation instruction prompts and input into the large language model. The words at all times are generated by autoregression and spliced ​​to obtain the final output diagnosis report.

[0066] To test the performance of the method of the present invention, qualitative and quantitative experiments were conducted on the public dataset MIMIC-CXR. The qualitative experimental results are as follows: Figure 3 As shown. Figure 3 In part (a), the present invention accurately generates "PA and lateral views of the chest are provided. The leads of the left-sided dual-chamber pacemaker terminate in the expected positions of the right atrium and right ventricle. The lungs are overinflated and the diaphragm is flat, suggesting chronic obstructive pulmonary disease. There are no focal consolidation, pleural effusion, or pneumothorax. The cardiac silhouette is slightly enlarged.", which is almost the same as the lesions covered in the reference report written by the doctor: "The leads of the left-sided dual-chamber pacemaker terminate in the right atrium and right ventricle. The cardiac silhouette is slightly enlarged and stable. The aorta remains tortuous and diffusely calcified. The pulmonary vessels are normal. The lungs are still overinflated. No pleural effusion or pneumothorax is seen. There is mild atelectasis at the lung bases, but no areas of focal consolidation. There are no acute bone abnormalities." In addition, Figure 4 The corresponding regions retrieved by the present invention when generating a report are shown. It can be seen that the method of the present invention can accurately retrieve relevant semantic entities from the pre-constructed chest X-ray anatomical region-level knowledge base, providing prior knowledge for chest X-ray report generation.

[0067] In terms of quantitative analysis, the method of the present invention is compared with the report generation method based on the encoder-decoder structure in recent years on the public data set MIMIC-CXR. The results of the quantitative comparison experiments are shown in Tables 1 and 2. Table 1 shows the performance comparison of the method of the present invention with the existing methods under the evaluation index of natural language text generation. Table 2 shows the performance comparison of the method of the present invention with the existing methods under the evaluation index of lesion accuracy, and the average values ​​of the diagnostic accuracy Precision, recall rate Recall and F1 for 14 categories are taken. The 14 categories include "atelectasis", "cardiomegaly", "consolidation", "edema", "pleural effusion", "pleural other lesions", "pneumonia", "pneumothorax", "enlarged cardiomediastinum", "lung opacity", "lung lesion", "fracture", "support devices" and "no lesion".

[0068] In Table 1, the method of the present invention achieved the best performance in BLEU3, BLEU4 and METEOR, among which BLEU3, BLEU4 and METEOR reached 0.185, 0.135 and 0.164 respectively, which were significantly better than other comparison methods, especially BLEU4 index was 5.5% higher than the suboptimal comparison method. Under the evaluation of index ROUGE-L, the method of the present invention also achieved relatively high performance. In Table 2, the method of the present invention achieved the best performance in the three accuracy evaluation indicators of Precision, Recall and F1, which shows that the method of the present invention can accurately describe the lesions in chest X-rays. In particular, the Precision, Recall and F1 of this method reached 0.536, 0.405 and 0.461 respectively, which were significantly better than other comparison methods, especially Precision was 10.5% higher than the suboptimal method, and F1 was 7.7% higher than the suboptimal method.

[0069] Furthermore, Table 3 shows the ablation experimental results of the method of the present invention, including the baseline method, the baseline method + multi-view fusion, the baseline method + entity retrieval and the method of the present invention. Under various natural language text generation indicators and lesion accuracy indicators, it can be seen that the method of the present invention can effectively improve the performance of report generation by combining multi-view feature fusion and entity semantic retrieval. In general, multi-view feature fusion and entity retrieval can improve the accuracy of report generation to a certain extent.

[0070] Table 1 Natural language text generation indicators of the MIMIC-CXR dataset (the best results are marked in bold, METrans. represents the METransformer model)

[0071] method years BLEU-3 BLEU-4 METEOR ROUGE-L R2Gen 2020 0.145 0.103 0.142 0.277 CMN 2021 0.148 0.106 0.142 0.278 CMM 2022 0.155 0.109 0.151 0.287 GSK 2022 0.156 0.115 - 0.284 DCL 2023 - 0.109 0.150 0.284 METrans. 2023 0.169 0.124 0.152 0.291 R2GenGPT 2023 0.179 0.128 0.160 0.287 RAMT 2024 0.157 0.113 0.153 0.284 FMVP 2024 0.156 0.108 0.150 0.284 MA 2024 0.162 0.115 0.151 0.274 Method of the present invention 2024 0.185 0.135 0.164 0.286

[0072] Table 2 Performance comparison of lesion accuracy indicators of MIMIC-CXR dataset (the best results are marked in bold)

[0073]

[0074]

[0075] Table 3 Ablation experiments on the MIMIC-CXR dataset (the best results are marked in bold)

[0076] Evaluation indicators Baseline Method +Multi-view fusion +Entity Retrieval Method of the present invention BLEU-1 0.397 0.412 0.406 0.419 BLEU-2 0.250 0.263 0.262 0.268 BLEU-3 0.172 0.181 0.183 0.185 BLEU-4 0.125 0.131 0.134 0.135 METEOR 0.155 0.163 0.161 0.164 ROUGE-L 0.283 0.292 0.291 0.286 Precision 0.476 0.522 0.530 0.536 Recall 0.322 0.405 0.392 0.405 F1 0.384 0.456 0.451 0.461

[0077] Finally, it should be noted that the purpose of publishing the embodiments is to help further understand the present invention, but those skilled in the art can understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection claimed by the present invention shall be subject to the scope defined in the claims.

[0078] Those skilled in the art will appreciate that the embodiments in the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt a complete hardware embodiment, a complete software embodiment, or a form of an embodiment combining software and hardware. Moreover, the present application may adopt a form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0079] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0080] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0081] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0082] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.

[0083] Obviously, those skilled in the art can make various changes and modifications to the embodiments in the present application without departing from the spirit and scope of the embodiments in the present application. Thus, if these modifications and variations of the embodiments in the present application fall within the scope of the claims and their equivalents in the embodiments of the present application, the embodiments of the present application are also intended to include these modifications and variations.

Claims

1. A method for generating a multi-view chest X-ray report with enhanced regional retrieval, characterized in that: A large language model is used as a text generation decoder. The visual features of multi-view chest X-rays are fused as visual cues of the large language model, and the anatomical region-level knowledge base of the chest X-rays is retrieved from a pre-built knowledge base. The relevant descriptions of the anatomical regions of the input chest X-rays are used as text cues of the large language model.

2. The method for generating a multi-view chest X-ray report with enhanced regional retrieval according to claim 1, characterized in that: The specific steps are as follows: Step (1), select a chest radiograph containing multi-view images {I1, I2, ..., I N The samples of} and their diagnostic reports R constitute the training data set; the multi-view chest radiographs include three view angles: posterior-anterior PA, anteroposterior AP and lateral Lateral; Step (2), select samples from the Chest ImaGenome dataset and construct a chest X-ray anatomical region-level knowledge base; divide different regions of each chest X-ray into 11 anatomical regions: right lung, right hilar structure, right lung apex, left lung, left hilar structure, left lung apex, trachea, spine, mediastinum, heart shadow and abdomen; Step (3), using the SwinTransformer model to extract sequence features of multi-view chest X-ray images; Step (4), inputting the sequence features of the multi-view chest X-ray images into the fusion network to perform pairwise cross-attention feature fusion to obtain fused visual features; Step (5), use the multi-layer perceptron MLP model to map the visual feature S, and use the mapping result as the visual prompt Prompt of the large language model vision ; Step (6), pre-training a Fast R-CNN detector on the Chest ImaGenome dataset and fixing the parameters to detect various anatomical regions of the input multi-view chest X-ray images; Step (7), using the pre-trained MedCLIP model to perform visual feature encoding on each anatomical region to extract the visual features of each anatomical region; Step (8): for each anatomical region of the input chest X-ray, a corresponding entity is retrieved from the pre-built chest X-ray anatomical region-level knowledge base; the text descriptions of the entities retrieved for each anatomical region of the input chest X-ray are concatenated and then used as text prompts of the large language model: Step (9), splicing the visual prompt obtained in step (5), the text prompt obtained in step (8) and the preset report generation instruction prompt, and inputting the splicing result into the pre-trained large language model to generate a diagnosis report; Step (10), freeze the Fast RCNN detection model, the MedCLIP model and the large language model, and train the SwinTransformer model, the fusion network and the MLP mapping layer in the following manner; Step (11), during the test, the multi-view chest X-ray images to be tested are processed to obtain visual prompts and text prompts, and combined with the report generation instruction prompts, they are input into the large language model, and the words at all times are generated by autoregression and spliced ​​to obtain the final output diagnosis report.

3. The method for generating a multi-view chest X-ray report with enhanced regional retrieval according to claim 2, characterized in that: Step (2) Each anatomical region contains N S semantic entities, each entity is recorded as: entity i ={v i ,l i ,s i ,a i } Among them, entity i represents the i-th entity, v i is the regional feature of the i-th entity encoded by the visual encoder MedCLIP model, l i is the region label of the i-th entity, s i is the entity description of the regional lesion of the i-th entity, a i is the normal or abnormal lesion label of the i-th entity.

4. The method for generating a multi-view chest X-ray report with enhanced regional retrieval according to claim 3, characterized in that: The specific calculation formula of the sequence characteristics in step (3) is as follows: Among them, E n Represents the nth image I in the multi-view chest radiograph n The image sequence features encoded and output by the SwinTransformer model, It represents the mth image sequence feature of the chest X-ray image at the nth viewing angle, and M is the length of the image sequence feature.

5. The method for generating a multi-view chest X-ray report with enhanced regional retrieval according to claim 4, characterized in that: The specific formula of the fusion process in step (4) is: in, Represents the fused visual features; d model represents the feature dimension, M is the length of the visual feature, N is the number of multi-view chest X-ray images, and MHA is the multi-head attention operation.

6. The method for generating a multi-view chest X-ray report with enhanced regional retrieval according to claim 5, characterized in that: The specific implementation of step (8) is as follows: 8-1. Perform region detection on the input chest X-ray and record the detected anatomical region as query region I q , input the MedCLIP model encoding to obtain the query feature v of the query area q =Φ vis (I q ), Φ vis represents the MedCLIP model; 8-2. From the constructed chest radiograph anatomical region-level knowledge base, all entities with the same region label as the query region are screened out; for the anatomical regions corresponding to the screened entities, the MedCLIP model encoding is used to extract their feature vectors v j =Φ vis (I j ), indicating that the jth entity corresponds to image I j regional characteristics; 8-3. Calculate query feature v q The feature vector v corresponding to the entity in the knowledge base that has the same region label as the query region j The cosine similarity of is as follows: Among them, N S Represents the number of entities in the knowledge base that have the same region label as the query region; 8-4. Select the first K entities with the highest similarity according to the cosine similarity score to initialize the candidate entity set. The corresponding regional feature set is {v1,v2,…,v K }; 8-5. Aggregate the abnormal attributes of the candidate entity set to determine whether the query area is abnormal; if the number of abnormal candidate entities is greater than or equal to the normal entities, the query area is classified as abnormal, and the abnormal candidate entity with the highest similarity is selected as the retrieved entity; otherwise, the query area is considered normal, and the normal candidate entity with the highest similarity is selected as the retrieved entity; 8-6. Repeat the above steps to stitch together the entity text descriptions retrieved from each anatomical region of the input chest X-ray as text prompts for the large language model.

7. The method for generating a multi-view chest X-ray report with enhanced regional retrieval according to claim 6, characterized in that: The text prompt of step (8) is calculated as follows: Prompt text =[{l1:s1};{l2:s2};…;{l F :s F }] Among them, Prompt text It is a text prompt containing the region label and its related entity description. [;] indicates a concatenation operation; {l i :s i } i is the region label of the i-th entity, s i is the entity description of the regional lesion of the i-th entity; F represents the total number of entities contained in the text prompt.

8. The method for generating a multi-view chest X-ray report with enhanced regional retrieval according to claim 7, characterized in that: The generation process of step (9) is expressed as the following autoregressive equation: Among them, R represents the complete diagnosis report, the length is L, p represents the probability distribution, p θ represents the probability distribution of the output based on the parameter θ, r t is the tth word in the diagnosis report R, R 0:t-1 Represents the report sequence before the tth word; Prompt vision Indicates a visual prompt. text Indicates a text prompt, and H indicates a preset report generation instruction prompt.

9. The method for generating a multi-view chest X-ray report with enhanced regional retrieval according to claim 8, characterized in that: The loss function of step (10) training optimization is as follows: in, is the cross entropy loss function, p represents the probability distribution, {I1,I2,…,I N } represents the input multi-view image, and H represents the report generation instruction prompt.

Images