Focus perception and domain knowledge collaborative chest radiation report generation method

Through the method of synergistic lesion perception and domain knowledge, combining imaging visual characteristics and radiological expertise, more accurate medical imaging reports are generated, solving the problem of low reporting accuracy in the existing technology and achieving high-quality diagnosis in remote areas .

CN119092034APending Publication Date: 2024-12-06KUNMING UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411109346.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The prior art is difficult to effectively improve the accuracy of medical imaging reports, especially in remote areas with scarce resources, which makes it difficult for patients to obtain timely and accurate diagnosis.

Method used

A method of synergistic lesion perception and domain knowledge is adopted to generate more accurate radiological diagnostic reports through a multi-headed attention mechanism combining imaging visual characteristics and radiological expertise. Specific steps include specific feature extraction, supplementary knowledge embedding and knowledge-enhancing multi-head attention mechanism, fusion of image visual features and professional knowledge, and transferring it to the Transformer decoder to generate reports.

Benefits of technology

The accuracy and quality of medical report generation has been significantly improved, and experimental results show that reports generated on IU X-ray and MIMIC-CXR datasets are superior to current mainstream methods in terms of effectiveness and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119092034A_ABST
    Figure CN119092034A_ABST
Patent Text Reader

Abstract

The invention relates to a focus perception and domain knowledge collaborative chest radiation report generation method, and belongs to the fields of natural language processing, computer vision and cross-modal fusion. Comprising the following steps: firstly, extracting lesion area features and global visual features in a specific feature extraction module; secondly, constructing a knowledge graph by utilizing a supplementary knowledge embedding module; secondly, a multi-head attention mechanism is enhanced by introducing knowledge, and visual features of the radiation image and supplementary knowledge are effectively combined; finally, in a report generation module, a Transform decoder is adopted, and an accurate and efficient radiology diagnosis report is generated by connecting the extracted visual features and supplementing knowledge. Experimental results show that the method can better capture image fine-grained lesion visual features through lesion area feature perception and radiology knowledge guidance, and the quality of a generated report is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a chest radiology report generation method based on the collaboration of lesion perception and domain knowledge, and belongs to the fields of natural language processing, computer vision, and cross-modal fusion. Background Art

[0002] Chest disease is one of the main factors threatening health. Chest X-ray examination has become a common medical imaging examination due to its advantages such as low cost, fast imaging and low radiation. The lack of medical resources in remote areas makes it difficult for patients to obtain timely and accurate diagnosis. Therefore, a system that automatically generates high-quality medical imaging reports is imminent.

[0003] Existing methods are mainly based on deep learning and can be divided into the following three categories: (1) Retrieval-based report generation model: Li et al. achieved efficient report generation by retrieving similar cases and optimizing them with reinforcement learning. Yang et al. proposed a hierarchical retrieval mechanism to improve the accuracy of report generation. Tanveer et al. used fine-grained label learning to generate reports with more clinical value. (2) Hybrid report generation model: Yin et al. generated structured radiology reports based on RNN. Wang et al. proposed Tmrgm, which combined the template method with the multi-attention mechanism to generate more detailed reports. Li et al. used a hierarchical decision process to retrieve matching report templates and generate the final report. (3) Generative report generation model: Wang et al. proposed a text-image embedding network that integrates a multi-level attention model. Liu et al. proposed a domain-aware automatic report generation method. Chen et al. designed a relational memory and introduced conditional layer normalization to improve the content richness of long report generation. Tanida et al. used Faster R-CNN to generate abnormal image regions and generate descriptive sentences. Summary of the invention

[0004] The problem to be solved by the present invention is: the present invention relates to a chest radiology report generation method that cooperates with lesion perception and domain knowledge, which is used to effectively improve the accuracy of medical report generation.

[0005] The technical solution of the present invention is: a method for generating chest radiology reports that collaborates with lesion perception and domain knowledge. First, the lesion area in the image is identified, the lesion features and the image with disease category annotations are obtained, and the lesion area feature vector is combined with the image visual features through a multi-head attention mechanism to obtain fine-grained image features. Secondly, a knowledge graph of prior knowledge and supplementary knowledge is constructed to integrate radiological expertise into the report generation model. Finally, the image visual features are effectively combined with prior knowledge and supplementary knowledge to generate a more accurate radiology diagnosis report. Experimental results show that this method can effectively improve the accuracy of medical report generation;

[0006] The method specifically includes the following:

[0007] Step 1: In the specific feature extraction module, extract the lesion area features and global visual features.

[0008] Step 2: In the supplementary knowledge embedding module, construct the knowledge graph corresponding to the supplementary knowledge.

[0009] Step 3: Introduce the knowledge-enhanced multi-head attention mechanism (KG-Att) for multimodal feature fusion. Effectively combine the visual features of the radiological image in step 1 with the supplementary knowledge in step 2.

[0010] Step 4: In the report generation module, the fused features obtained in step 3 are passed into the Transformer decoder to generate the final radiology diagnosis report.

[0011] Furthermore, the specific feature extraction module in step 1 specifically includes the following:

[0012] The input image Img is processed by the MLLA multi-disease detection module and the Vision Transformer global visual feature extraction module (Vit) to extract the lesion area feature F of the image. M and the global visual feature F v , and use the feature F of the lesion area M To adjust the global visual feature F v The weight of the lesion area is used to better focus on the lesion area. M and the global visual feature F v Through the multi-head attention mechanism GMSA, the global visual features of the lesion area are obtained. The calculation formula is as follows:

[0013] F v =Vit(Img)

[0014] F M =AvgPool(MLLP(Img))

[0015]

[0016] GMSA=Concat(head1,...,head h )W o

[0017]

[0018] Among them, AvgPool(·) represents the average pooling operation, MLLP(·) represents the linear mapping; Attention(·) represents the attention mechanism, Q M , K v , V v Indicates FM , F v The matrix form query, key and value obtained by linear transformation, is a scaling factor used to stabilize the Softmax operation; headi represents the attention mechanism of the i-th head, represents the weight parameters of the query, key, and value of different heads; Concat(·) represents the concatenation operation of multiple attention heads, W o Represented as trainable weights; is the eigenvector F v Dimension, the global visual feature F of the lesion area is obtained through GMSA I .

[0019] Furthermore, the supplementary knowledge embedding module in step 2 specifically includes the following:

[0020] The construction and embedding of the supplementary knowledge graph is divided into two parts: (1) obtaining imaging disease labels from the MLLA multi-disease detection model; (2) aligning disease labels with entities in RadGraph, constructing a multi-hop subgraph, and mining supplementary knowledge.

[0021] For the input image, the disease label distribution T is first obtained through the MLLA pre-training model. i , and its calculation formula is as follows:

[0022] Ti=MLLP(Img i)

[0023] in N t Represents the total number of disease labels. In this paper, N t is 14, i represents the number of disease labels in the current image, and MLLP represents the chest multi-disease detection model, which includes a positioning image with disease label annotations and bounding boxes, as well as lesion area features.

[0024] Secondly, the disease label T i Align with entities in RadGraph. First, BERT is used to align disease labels T i The entities in RadGraph are converted into corresponding word embedding vectors. Then, the disease label encoding vector is evaluated by calculating the cosine similarity. With each entity encoding vector in RadGraph The similarity between them is calculated to get the similarity score. The entities in RadGraph are sorted according to the similarity score and the top-k entity formulas are selected. The calculation formula is as follows:

[0025]

[0026] Among them, ClinicalBERT Encode (·) is the text encoding function of the Clinical BERT model, Cos(·) is the cosine similarity calculation function, and · represents the vector dot product.

[0027] Finally, according to entity E i Construct a multi-hop subgraph to mine complementary knowledge. For each entity e i k ∈E i , taking it as the starting node, using the personalized pagerank (PPR) algorithm, through m rounds of iterations, each round of iteration is called a hop, to visit its neighbor nodes and related connections. In each hop, some neighbor nodes are selected according to the degree (number of connections) of the node to control the size of the subgraph, and these nodes are added to the subgraph. Repeat this process and gradually build a subgraph containing multi-hop neighbors. The calculation formula is as follows:

[0028]

[0029] E subgraph =E subgraph ∪{(e i k ,u)|e i k ∈V subgraph,

[0030] u∈SampleNeighbors(e i k , PPR(e i k , α, γ))}

[0031]

[0032] G a =(V subgraph , E subgraph )

[0033] Among them, e i k represents the starting node, γ represents the target node (initialized as the starting node), α represents the random jump probability, PPR(e i k , α, γ) represents the probability distribution of random walk from the starting node to the target node, δ represents that when the starting node is equal to the target node, δ=1, N(e i k ) represents the starting node e i k The set of neighbor nodes, d out(u) represents the out-degree of node u, E subgraph is the edge set of the subgraph, V subgraph is the node set of the subgraph, SampleNeighbors(e i k , PPR(e i k , α, γ)) represents the probability distribution of node e i k The neighbor nodes of G are sampled, n is the size of the entity set, m is the number of hops, which is set to 3 in this paper. a Represents the constructed supplementary knowledge graph.

[0034] The supplementary knowledge graph is embedded in the graph using the graph embedding model RotatE to represent entities and relationships as vectors rotated on the complex plane to capture the information in the graph, thereby obtaining the entity embedding and relationship embedding of the supplementary knowledge. The calculation formula is as follows:

[0035] {E ae , E ar = RotatE(G subgraph )

[0036] Where E ae and E ar Entity embedding and relation embedding representing complementary knowledge respectively.

[0037] Furthermore, the knowledge-enhanced multi-head attention mechanism (KG-Att) module in step 3 specifically includes the following:

[0038] The knowledge-enhanced multi-head attention mechanism (KG-Att) is introduced to perform multimodal feature fusion. First, the aggregate relation embedding r is constructed from the complementary knowledge. ik ,Since there may be more than one relationship between two entities, this paper uses the average value as the aggregated relationship embedding, and its calculation formula is as follows:

[0039] r ij =AvgPool(E r (i,j) )

[0040] E r (i,j) = Ralations(e i , e j )

[0041] Where E r =E ar is the relational embedding of the acquired supplementary knowledge; AvgPool(·) refers to the average pooling function; e i , e kThey represent the source entity and target entity in the supplementary knowledge graph respectively; Ralations(·) represents all the relations between the source entity and the target entity.

[0042] Secondly, the relationship deviation calculation formula is as follows:

[0043] φ(r)=AvgPool(rW R )

[0044] Where W R represents the learnable parameters of the linear projection; φ(r) represents the aggregate relation embedding of entities in the supplementary knowledge graph.

[0045] Finally, the image visual feature F I and entity embedding E in the knowledge graph ae and relation embedding E ar , input knowledge-enhanced multi-head attention, and output knowledge-enhanced visual representation. The calculation formula is as follows:

[0046]

[0047] KEMHA(Q,K,V,r)=[head1,...,head h ]W o

[0048]

[0049] C a =KEMHA(F I , E ae , E ae ,φ(ar))

[0050] where Q, K, V represent query, key and value matrices; d k represents the embedding dimension; φi(r) represents the relation deviation of the i-th head; W i Q , and W o represents the learnable parameter of the linear mapping; φ(ar) represents the aggregation relation bias of the supplementary knowledge graph; E ae represents the embedding of supplementary knowledge graph entities; C a Representation complements knowledge-enhanced visual representations.

[0051] Furthermore, the report generation module in step 4 specifically includes the following:

[0052] The present invention adopts the decoder of Transformer to generate a radiology diagnosis report by connecting the extracted visual features and supplementary knowledge.

[0053] W t =Decoder(concat(F I , C a ), W <t ), W0=0

[0054] Among them, W t represents the word generated by the current decoder, W <t represents the word generated before time t, W0=0 means that the initial state of the generation process is set to 0, and concat(·) represents the concatenation operation.

[0055] The beneficial effects of the present invention are:

[0056] This method can better capture the fine-grained visual features of image lesions through lesion area feature perception and radiological knowledge guidance, thus improving the quality of generated reports.

[0057] The present invention conducted ablation experiments on two public medical imaging datasets, IU X-ray and MIMIC-CXR-II, and compared the results with the current method. The experimental results show that the description of the chest imaging diagnosis report generated by the present invention is superior to the current mainstream in terms of effectiveness and accuracy.

[0058] method. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is the overall framework of the chest imaging report generation method based on lesion perception and radiological knowledge guidance of the present invention;

[0060] Figure 2 This is a structural diagram of the specific feature extraction module of the present invention;

[0061] Figure 3 The structure diagram of the multi-head attention network for knowledge enhancement of the present invention;

[0062] Figure 4 Comparison of visualization results of the present invention DETAILED DESCRIPTION

[0063] Embodiment 1, as Figure 1-Figure 4 As shown, a chest radiology report generation method based on lesion perception and domain knowledge collaboration is provided, and the specific steps of the method are as follows:

[0064] Step 1: In the specific feature extraction module, extract the lesion area features and global visual features.

[0065] Step 2: In the supplementary knowledge embedding module, construct the knowledge graph corresponding to the supplementary knowledge.

[0066] Step 3: Introduce the knowledge-enhanced multi-head attention mechanism (KG-Att) for multimodal feature fusion. Effectively combine the visual features of the radiological image in step 1 with the supplementary knowledge in step 2.

[0067] Step 4: In the report generation module, the fused features obtained in step 3 are passed into the Transformer decoder to generate the final radiology diagnosis report.

[0068] Furthermore, the specific feature extraction module in step 1 specifically includes the following:

[0069] The input image Img is processed by the MLLA multi-disease detection module and the Vision Transformer global visual feature extraction module (Vit) to extract the lesion area feature F of the image. M and the global visual feature F v , and use the feature F of the lesion area M To adjust the global visual feature F v The weight of the lesion area is used to better focus on the lesion area. M and the global visual feature F v Through the multi-head attention mechanism GMSA, the global visual features of the lesion area are obtained. The calculation formula is as follows:

[0070] F v =Vit(Img)

[0071] F M =AvgPool(MLLP(Img))

[0072]

[0073] GMSA=Concat(head1,...,head h )W o

[0074]

[0075] Among them, AvgPool(·) represents the average pooling operation, MLLP(·) represents the linear mapping; Attention(·) represents the attention mechanism, Q M , K v , V v Indicates F M , F v The matrix form query, key and value obtained by linear transformation, is a scaling factor used to stabilize the Softmax operation; head i represents the attention mechanism of the i-th head, represents the weight parameters of the query, key, and value of different heads; Concat(·) represents the concatenation operation of multiple attention heads, W o Represented as trainable weights; is the eigenvector F v Dimension, the global visual feature F of the lesion area is obtained through GMSA I .

[0076] Furthermore, the supplementary knowledge embedding module in step 2 specifically includes the following:

[0077] The construction and embedding of the supplementary knowledge graph is divided into two parts: (1) obtaining imaging disease labels from the MLLA multi-disease detection model; (2) aligning disease labels with entities in RadGraph, constructing a multi-hop subgraph, and mining supplementary knowledge.

[0078] For the input image, the disease label distribution T is first obtained through the MLLA pre-training model. i , and its calculation formula is as follows:

[0079] T i =MLLP(Img i )

[0080] in N t Represents the total number of disease labels. In this paper, N t is 14, i represents the number of disease labels in the current image, and MLLP represents the chest multi-disease detection model, which includes a positioning image with disease label annotations and bounding boxes, as well as lesion area features.

[0081] Secondly, the disease label T i Align with entities in RadGraph. First, BERT is used to align disease labels T i The entities in RadGraph are converted into corresponding word embedding vectors. Then, the disease label encoding vector is evaluated by calculating the cosine similarity. With each entity encoding vector in RadGraph The similarity between them is calculated to get the similarity score. The entities in RadGraph are sorted according to the similarity score and the top-k entity formulas are selected. The calculation formula is as follows:

[0082]

[0083] Among them, ClinicalBERT Encode (·) is the text encoding function of the Clinical BERT model, Cos(·) is the cosine similarity calculation function, and · represents the vector dot product.

[0084] Finally, according to entity E i Construct a multi-hop subgraph to mine complementary knowledge. For each entity e i k ∈E i , taking it as the starting node, using the personalized pagerank (PPR) algorithm, through m rounds of iterations, each round of iteration is called a hop, to visit its neighbor nodes and related connections. In each hop, some neighbor nodes are selected according to the degree (number of connections) of the node to control the size of the subgraph, and these nodes are added to the subgraph. Repeat this process and gradually build a subgraph containing multi-hop neighbors. The calculation formula is as follows:

[0085]

[0086] E subgraph =E subgraph ∪{(e i k ,u)|e i k ∈V subgraph ,

[0087] u∈SampleNeighbors(e i k , PPR(e i k , α, γ))}

[0088]

[0089] G a =(V subgraph , E subgraph )

[0090] Among them, e i k represents the starting node, γ represents the target node (initialized as the starting node), α represents the random jump probability, PPR(e i k , α, γ) represents the probability distribution of random walk from the starting node to the target node, δ represents that when the starting node is equal to the target node, δ=1, N(e i k ) represents the starting node e i k The set of neighbor nodes, d out (u) represents the out-degree of node u, E subgraph is the edge set of the subgraph, V subgraph is the node set of the subgraph, SampleNeighbors(e ik , PPR(e i k , α, γ)) represents the probability distribution of node e i k The neighbor nodes of G are sampled, n is the size of the entity set, m is the number of hops, which is set to 3 in this paper. a Represents the constructed supplementary knowledge graph.

[0091] The supplementary knowledge graph is embedded in the graph using the graph embedding model RotatE to represent entities and relationships as vectors rotated on the complex plane to capture the information in the graph, thereby obtaining the entity embedding and relationship embedding of the supplementary knowledge. The calculation formula is as follows:

[0092] {E ae , E ar = RotatE(G subgraph )

[0093] Where E ae and E ar Entity embedding and relation embedding representing complementary knowledge respectively.

[0094] Furthermore, the knowledge-enhanced multi-head attention mechanism (KG-Att) module in step 3 specifically includes the following:

[0095] The knowledge-enhanced multi-head attention mechanism (KG-Att) is introduced to perform multimodal feature fusion. First, the aggregate relation embedding r is constructed from the complementary knowledge. ij ,Since there may be more than one relationship between two entities, this paper uses the average value as the aggregated relationship embedding, and its calculation formula is as follows:

[0096] r ij =AvgPool(E r (i,j) )

[0097] E r (i,j) = Ralations(e i , e j )

[0098] Where E r =E ar is the relational embedding of the acquired supplementary knowledge; AvgPool(·) refers to the average pooling function; e i , e j They represent the source entity and target entity in the supplementary knowledge graph respectively; Ralations(·) represents all the relations between the source entity and the target entity.

[0099] Secondly, the relationship deviation calculation formula is as follows:

[0100] φ(r)=AvgPool(rW R )

[0101] Where W R represents the learnable parameters of the linear projection; φ(r) represents the aggregate relation embedding of entities in the supplementary knowledge graph.

[0102] Finally, the image visual feature F I and entity embedding E in the knowledge graph ae and relation embedding E ar , input knowledge-enhanced multi-head attention, and output knowledge-enhanced visual representation. The calculation formula is as follows:

[0103]

[0104] KEMHA(Q,K,V,r)=[head1,...,head h ]W o

[0105]

[0106] C a =KEMHA(F I , E ae , E ae ,φ(ar))

[0107] where Q, K, V represent query, key and value matrices; d k represents the embedding dimension; φi(r) represents the relation deviation of the i-th head; W i Q , and W o represents the learnable parameter of the linear mapping; φ(ar) represents the aggregation relation bias of the supplementary knowledge graph; E ae represents the embedding of supplementary knowledge graph entities; C a Representation complements knowledge-enhanced visual representations.

[0108] Furthermore, the report generation module in step 4 specifically includes the following:

[0109] This paper adopts the decoder of Transformer to generate radiology diagnosis reports by connecting the extracted visual features and supplementary knowledge.

[0110] W t =Decoder(concat(F I , C a ), W <t ), W0=0

[0111] Among them, W t represents the word generated by the current decoder, W <t represents the word generated before time t, W0=0 means that the initial state of the generation process is set to 0, and concat(·) represents the concatenation operation.

[0112] Furthermore, in order to verify the effect of the present invention, the trained model was applied to the IU X-ray and MIMIC-CXR medical report datasets for ablation experiments. The IUX-Ray dataset contains 7470 sets of chest radiographs and 3955 sets of diagnostic reports, which are divided into training, validation, and test sets in a ratio of 7:1:2. MIMIC-CXR contains 227,835 sets of medical images, and one report corresponds to one or more images, usually a front view and a side view. This paper extracts data from one report corresponding to one image, including 29,521 images and corresponding reports. The dataset is divided according to official standards, with a ratio of 6:2:2. The environment configuration of this experiment GPU: NVIDIA RTX 3080 single GPU: operating system: Ubuntu 1 8.04.3; optimizer: AdamW: machine learning framework: PyTorch.

[0113] In the present invention, a chest radiology report generation method based on lesion perception and domain knowledge is provided, and the overall structure is as follows: Figure 1 First, the lesion area features and global visual features are extracted using a specific feature extraction module, as shown in Figure 2 As shown; secondly, combined with the knowledge graph, feature fusion is performed through the knowledge-enhanced multi-head attention mechanism, as shown Figure 3 As shown; finally, the fused features are input into the Transformer decoder to generate a more accurate radiology diagnosis report.

[0114] In order to verify the effectiveness of the chest imaging report generation method based on lesion perception and domain knowledge proposed in the present invention, a series of experiments were conducted on the public datasets IU X-ray and MIMIC-CXR to compare and verify the effectiveness of the method of the present invention. The experimental results are shown in Table 1 and Table 2:

[0115] Table 1 Comparison of test results of different models on IU X-ray dataset

[0116]

[0117] The results in Table 1 show the comparison results between the present invention and the mainstream medical report generation model in recent years on the IU X-ray dataset. This method can better capture the fine-grained lesion visual features of the image by utilizing the lesion region feature perception and domain knowledge collaboration, thereby improving the quality of the generated report. As shown in the table above, this method effectively improves the accuracy of the medical report generation model.

[0118] Table 2 Comparison of test results of different models on the MIMIC-CXR dataset

[0119]

[0120]

[0121] The results in Table 2 show that in the experiment of the MIMIC-CXR dataset, representative mainstream medical report generation models in recent years were also selected for comparison. The results show that the present invention effectively improves the accuracy of medical report generation.

[0122] In order to verify the effectiveness of each module in the model method of this chapter, the present invention conducted a series of ablation experiments on the IU X-ray and MIMIC-CXR datasets. The experimental results are shown in Table 3:

[0123] Table 3 Ablation experiment results

[0124]

[0125] As shown in Table 3, by perceiving the lesion region features extracted by the MLLA model, the model improved the BLEU-4 and ROUGE-L scores on the two datasets, respectively, and enhanced the accuracy of the report. In addition, after incorporating supplementary knowledge, the model significantly improved both the BLEU and ROUGE-L indicators, proving that combining professional knowledge can significantly improve the quality of automatically generated radiology reports.

[0126] In order to more intuitively demonstrate the effectiveness of the chest imaging report generated by the model in this paper, the text report generated by the trained model based on the radiological image is compared with the true value. Figure 4 As shown;

[0127] When normal images were input, the report generated by the model not only confirmed that the lungs were clear, but also excluded abnormalities such as focal airway consolidation, pleural effusion, or pneumothorax, which was more detailed than the original report. The description of heart size and mediastinal contour was consistent with the real report, but the model was more specific in describing the heart, emphasizing the degree of "significant" enlargement. When abnormal images were input, the model accurately located and described myocardial hypertrophy and aortic dilatation, enhancing the interpretability of the diagnosis. Although the original report did not mention the condition of the lungs, the model supplemented the description of clear lungs and no effusion or pneumothorax, ensuring the completeness of the report. Both reports mentioned atherosclerotic calcifications in the thoracic aorta, verifying the accuracy and attention to detail of the model.

[0128] In summary, the generated report maintains the accuracy of the original benchmark, adds a description of the lung condition, and provides more detailed information on the heart, thoracic aorta, and spine. By focusing on the fine-grained features of the lesion area, the model can more accurately capture the abnormal areas in the image, and the generated report is more comprehensive and accurate. In addition, by incorporating radiological knowledge, the report can more accurately describe the lesion characteristics when describing the lesion, which helps to improve the quality of the report.

[0129] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A chest radiology report generation method based on lesion perception and domain knowledge, characterized in that: The method specifically includes the following: Step 1: In the specific feature extraction module, extract the lesion area features and global visual features; Step 2: In the supplementary knowledge embedding module, construct a knowledge graph corresponding to the supplementary knowledge; Step 3: Introduce the knowledge-enhanced multi-head attention mechanism KG-Att to perform multimodal feature fusion, effectively combining the visual features of the radiological image in step 1 with the supplementary knowledge in step 2; Step 4: In the report generation module, the fused features obtained in step 3 are passed into the Transformer decoder to generate the final radiology diagnosis report.

2. According to claim 1, a method for generating chest radiology reports by coordinating lesion perception with domain knowledge is characterized in that: The specific feature extraction module in step 1 specifically includes the following: The input image Img is processed by the MLLA multi-disease detection module and the Vision Transformer global visual feature extraction module Vit to extract the image lesion area feature F M and the global visual feature F v , and use the feature F of the lesion area M To adjust the global visual feature F v weights to better focus on the lesion area; that is, the lesion area feature F M and the global visual feature F v Through the multi-head attention mechanism GMSA, the global visual features of the lesion area are obtained; The calculation formula is as follows: F v =Vit(Image) F M =AvgPool(MLLP(Img)) GMSA=Concat(head1,…,head h )W o Among them, AvgPool(·) represents the average pooling operation, MLLP(·) represents the linear mapping; Attention(·) represents the attention mechanism, Q M ,K v ,V v Indicates F M , F v The matrix form query, key and value obtained by linear transformation, is a scaling factor used to stabilize the Softmax operation; jead i represents the attention mechanism of the i-th head, represents the weight parameters of the query, key, and value of different heads; Concat(·) represents the concatenation operation of multiple attention heads, W o Represented as trainable weights; is the eigenvector F v Dimension, the global visual feature F of the lesion area is obtained through GMSA I .

3. According to claim 1, a method for generating chest radiology reports by coordinating lesion perception with domain knowledge, characterized in that: The supplementary knowledge embedding module in step 2 specifically includes the following: The construction and embedding of the supplementary knowledge graph is divided into two parts: (1) obtaining imaging disease labels from the MLLA multi-disease detection model; (2) aligning disease labels with entities in RadGraph, constructing a multi-hop subgraph, and mining supplementary knowledge; For the input image, the disease label distribution T is first obtained through the MLLA pre-training model. i , and its calculation formula is as follows: T i =MLLP(Img i ) in N t represents the total number of disease labels, i represents the number of disease labels in the current image, and MLLP represents the chest multi-disease detection model, which includes a positioning image with disease label annotations and bounding boxes, as well as lesion area features; Secondly, the disease label T i Align with entities in RadGraph; first, use BERT to align disease labels T i The entities in RadGraph are converted into corresponding word embedding vectors; Afterwards , by calculating the cosine similarity, evaluate the disease label encoding vector With each entity encoding vector in RadGraph The similarity between them is calculated to obtain the similarity score; the entities in RadGraph are sorted according to the similarity score, and the top-k entity formulas are selected; the calculation formula is as follows: Among them, ClinicalBERT Encode (·) is the text encoding function of the ClinicalBERT model, Cos(·) is the cosine similarity calculation function, and · represents the vector dot product; Finally, according to entity E i Construct a multi-hop subgraph to mine supplementary knowledge; for each entity e i k ∈E i , take it as the starting node, use the personalized page ranking algorithm, through m rounds of iterations, each round of iteration is called a hop, to visit its neighbor nodes and related connections; in each hop, select some neighbor nodes according to the degree of the node to control the size of the subgraph, and add these nodes to the subgraph; repeat this process and gradually build a subgraph containing multi-hop neighbors; the calculation formula is as follows: E subgraph =E subgraph ∪{(e i k ,u)|e i k ∈V subgraph , u∈SampleNeighbors(e i k ,PPR(e i k ,a,c))} G a =(V subgraph ,E subgraph ) Among them, e i k represents the starting node, γ represents the target node, which is the starting node at the beginning, α represents the random jump probability, PPR(e i k ,α,γ) represents the probability distribution of random walk from the starting node to the target node, δ represents that when the starting node is equal to the target node, δ=1, N(e i k ) represents the starting node e i k The set of neighbor nodes, d out (u) represents the out-degree of node u, E subgraph is the edge set of the subgraph, V subgraph is the node set of the subgraph, SampleNeighbors(e i k ,PPR(e i k ,α,γ)) represents the probability distribution of node e i k The neighbor nodes of G are sampled, n is the size of the entity set, m is the number of hops, a Represents the constructed supplementary knowledge graph; The supplementary knowledge graph is embedded in the graph using the graph embedding model RotatE to represent entities and relationships as vectors rotated on the complex plane to capture the information in the graph, thereby obtaining the entity embedding and relationship embedding of the supplementary knowledge. The calculation formula is as follows: {E ae ,It ar }=RotateE(G subgraph ) Where E ae and E ar Entity embedding and relation embedding representing complementary knowledge respectively.

4. According to claim 1, a method for generating chest radiology reports by coordinating lesion perception with domain knowledge, characterized in that: The knowledge-enhanced multi-head attention module in step 3 specifically includes the following: The knowledge-enhanced multi-head attention mechanism KG-Att is introduced to perform multimodal feature fusion; first, the aggregate relation embedding r is constructed from the supplementary knowledge. ij , since there may be more than one relationship between two entities, the average value is used as the aggregated relationship embedding, and its calculation formula is as follows: r ij =AvgPool(E r (i,j) ) E r (i,j) =Ralations(e i ,e j ) Where E r =E ar is the relational embedding of the acquired supplementary knowledge; AvgPool(·) refers to the average pooling function; e i ,e j Represent the source entity and target entity in the supplementary knowledge graph respectively; Ralations(·) represents all the relations between the source entity and the target entity; Secondly, the relationship deviation calculation formula is as follows: φ(r)=AvgPool(rW R ) Where W R represents the linear projection learnable parameters; φ(r) represents the aggregate relation embedding of entities in the supplementary knowledge graph; Finally, the image visual feature F I and entity embedding E in the knowledge graph ae and relation embedding E ar , input knowledge-enhanced multi-head attention, and output knowledge-enhanced visual representation; its calculation formula is as follows: KEMHA(Q,K,V,r)=[head1,…,head h ]W o head i =KG-Att(QW i Q ,KW i K ,VW i V ,φ i (r)) C a =KEMHA(F I ,IN ae ,IN ae ,φ(ar)) Where Q, K, V represent query, key and value matrices; d k represents the dimension of embedding; φ i (r) represents the relation deviation of the i-th head; W i q ,W i K ,W i V and W o represents the learnable parameter of the linear mapping; φ(ar) represents the aggregation relation bias of the supplementary knowledge graph; E ae represents the embedding of supplementary knowledge graph entities; C a Representation complements knowledge-enhanced visual representations.

5. The method for generating chest radiology reports by synergizing lesion perception with domain knowledge according to claim 1, characterized in that: The report generation module in step 4 specifically includes the following: The decoder of Transformer is used to generate radiology diagnosis reports by connecting the extracted visual features and supplementary knowledge; W t =Decoder(concat(F I ,C a ),W <t ),W0=0 Where W t represents the word generated by the current decoder, W <t Represents the word generated before time t, and W0=0 means that the initial state of the generation process is set to 0.