Radiology report generation method based on hierarchical knowledge integration framework

Abstract

The invention is suitable for the technical field of medical image analysis, and provides a radiology report generation method based on a hierarchical knowledge integration framework. Through hierarchical knowledge injection and a cross-modal fusion module, the difference between image perception and text cognition is effectively reduced, and the consistency between the image and the text is ensured. Rich medical priori knowledge is utilized, and the accuracy and correlation of the report are enhanced. The method not only provides a new scheme for radiology report generation, but also shows wide potential in the field of medical image analysis. The method can accurately align images and texts, reflect key clinical information, and enhance the practicability and reliability of reports in combination with image features and medical knowledge. Besides, complex anatomical structures and pathological features can be processed, the problem of generalization caused by detail omission in the report is reduced, the integrity, continuity and real-time performance of the report are ensured, and the method has remarkable theoretical innovation significance and practical application value.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image analysis, and in particular relates to a radiology report generation method based on a hierarchical knowledge integration framework. Background Art

[0002] In the field of medical image analysis, the generation of radiology reports has always been a research topic that has attracted much attention, especially in the field of automated report generation of radiology images. Although the existing technology has made some progress in the field of radiology report generation, there are still significant limitations, which restrict the performance and practicality of the automated radiology report generation system.

[0003] First, most existing methods rely on simple alignment of visual features and text features, which is often unsatisfactory when processing complex radiological images. Since radiological images are rich in details and contain complex anatomical structures and pathological features, existing methods often ignore these fine-grained pixel-level information, resulting in the generated reports may miss important diagnostic information or fail to accurately describe clinical significance. For example, when faced with complex lesions, the reports generated by some models may be too generalized and lack sufficient details to meet the needs of clinical diagnosis. Secondly, current models face great challenges in aligning images and texts, especially in the process of cross-modal learning, it is difficult to fully understand and integrate the implicit association between images and medical knowledge. Most models only rely on simple visual-linguistic feature matching when dealing with the relationship between images and texts, lacking deep integration of medical knowledge, resulting in insufficient medical professionalism and accuracy in the generated reports. In addition, existing methods have failed to provide effective solutions when dealing with the situation where the proportion of normal and abnormal areas in the image is uneven, resulting in data bias in the report generation process, which in turn affects the quality and consistency of the report. To this end, the present invention proposes a radiology report generation method based on a hierarchical knowledge integration framework. Summary of the invention

[0004] The purpose of the present invention is to provide a method for generating radiology reports based on a hierarchical knowledge integration framework, aiming to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions: A radiology report generation method based on a hierarchical knowledge integration framework comprises the following steps: 1. Visual-language pre-training stage, including the following tasks: Multi-task learning: Use pre-trained image encoders and text encoders to extract image features and text features respectively; perform contrastive learning tasks to establish coarse-grained associations between images and texts and achieve feature alignment; process image features and text features through a cross-modal fusion module; use the processed image features and text features to perform image-text matching tasks, and enhance fine-grained alignment between images and texts by combining organ and pathology labels; Masked autoencoders and masked language models: In masked autoencoders, parts of an image are randomly masked, and the model captures potential associations within the image by reconstructing the masked areas. In masked language model tasks, some key words are masked in the text, and the model learns the contextual relationships between texts by predicting the masked key words. 2. Radiology report generation stage, including the following steps: Image feature extraction: Use pre-trained image encoders to encode radiological images and extract image features; Knowledge feature fusion: Use the pre-trained text encoder to convert the organ knowledge and pathology knowledge related to the image into features; extract the organ knowledge features most relevant to the image through the dynamic knowledge extraction module, and combine the image features with the organ knowledge features through the cross-modal fusion module to generate image-organ knowledge features; at the same time, extract the pathology knowledge features most relevant to the image through the dynamic knowledge extraction module, and combine the image features with the pathology knowledge features through the cross-modal fusion module to generate image-pathology knowledge features; Report generation: The fused multimodal features are hierarchically input into the Transformer decoder to simulate the diagnostic process of a radiologist: starting with observing the image as a whole, analyzing the image based on organ labels, assigning pathological labels to each organ, and finally generating a detailed radiology report.

[0006] Furthermore, the loss function of the contrastive learning task is: Formula 1: ; in, Representing images With text The expected value of the sample pair; Representing images With text Similarity in a shared embedding space; Representing images With text Similarity in a shared embedding space; Representing images With text Similarity in a shared embedding space; is the number of image-text pairs in the batch; k is an index variable used for normalization, indicating that all samples (image-text pairs) in the batch are traversed to calculate the sum of similarities; and Indicated in The index number of the image or text corresponding to each image-text pair; The model aligns image features and text features by minimizing the distance between image-text pairs and maximizing the distance between dissimilar image-text pairs.

[0007] Furthermore, the loss function of the image-text matching task is: Formula 2: ; in, is a label, containing 1 or 0, where 1 indicates that the sample matches and 0 indicates that the sample does not match; is the predicted probability obtained by mapping the similarity score through the sigmoid function, where represents the Sigmoid function, represents the similarity calculation function, represents the image feature vector, Represents the text feature vector; when the image and text are matched, the goal of the model is to make Close to 1; when the image and text are not matched, the model's goal is to make Close to 0.

[0008] Furthermore, the loss function of the mask autoencoder task is: Formula 3: ; in, is the index set of the covered image blocks in the image, is the number of covered blocks; is the index of the mask block; The original image indivual The pixel value of is the corresponding reconstructed model The pixel value of .

[0009] Furthermore, the loss function of the masked language model task is: Formula 4: ; in, is the set of all masked position indices in the text; Is the model in the mask position Prediction correct probability.

[0010] Furthermore, the specific process of the image feature extraction and knowledge feature fusion steps is as follows: First, the radiological images are encoded using a pre-trained image encoder to extract image features. ; Use the pre-trained text encoder to encode the organ knowledge and pathological knowledge associated with the image into organ knowledge features respectively and pathological knowledge characteristics ; Then, through the dynamic knowledge extraction module, the organ knowledge features Selection and Image Features Matching organ knowledge features , and use the cross-modal fusion module to complete feature fusion and generate image-organ knowledge features At the same time, through the dynamic knowledge extraction module, from the pathological knowledge characteristics Selection and Image Features Matching pathological knowledge features , and use the cross-modal fusion module to complete feature fusion and generate image-pathology knowledge features ; The specific process is as follows: Formula 5: ; Formula 6: ; Formula 7: ; Formula 8: ; in, represents the dynamic knowledge extraction module, which is responsible for matching the organ knowledge features and pathological knowledge features corresponding to the image features; It is a cross-modal fusion module, which is used to integrate the features of different modalities; Fused image-organ knowledge features and image-pathology knowledge features It is input into the Transformer decoder layer by layer to simulate the diagnosis process of radiologists: Formula 9: ; in, is the Transformer decoder, are multimodal features at different levels. The field to be predicted in the current report.

[0011] Furthermore, in the dynamic knowledge extraction module, the dynamic extraction of potential knowledge information in the image is achieved through the matching mechanism between image features and organ and pathological knowledge; specifically as follows: For the input image feature vector Knowledge characteristics of organs and pathological knowledge characteristics , through the dynamic knowledge extraction module, from the organ knowledge features and pathological knowledge characteristics Selection and Image Features Matching organ knowledge features and pathological knowledge characteristics : Formula 10: ; Formula 11: ; in, is the temperature parameter, which is used to adjust the smoothness of the matching distribution; is the total number of knowledge vectors; and They are used to measure image features. Knowledge characteristics of organs Between and image features Pathological knowledge characteristics The degree of similarity between and They represent traversing all samples (image-text pairs) in the batch to calculate the similarity between the image and organ knowledge and between the image and pathology knowledge.

[0012] Furthermore, the cross-modal fusion module adopts a two-stream cross-attention mechanism to fuse the semantic features of the image and text through the following process: In the self-attention layer In the image features and text features The internal representation of each modality is enhanced through the interaction of internal self-attention mechanisms: Formula 12: ; in, It is the self-attention mechanism; is the image feature; is a text feature, which includes and ; and They are respectively the image modality features and text modality features enhanced by the self-attention mechanism; Then, in the cross-attention layer The cross-modal interaction process is realized in , integrating information from different modalities into their respective representations: Formula 13: ; in, is the cross attention mechanism, As an integration from language modality Information query, generating cross-attention visual features ; As an integrated visual modality Information query, generating cross-attention language features ; Finally, and Feed forward layer , generating multimodal representations of visual modalities and multimodal representation of language modalities : Formula 14: ; Finally, the multimodal features generated by superimposing image and text features For report generation: Formula 15: ; in, Represents the concatenation of feature vectors from different modalities.

[0013] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes a radiology report generation method based on a hierarchical knowledge integration framework. The framework effectively narrows the gap between image perception and text cognition through innovative hierarchical knowledge injection and cross-modal fusion modules, ensuring the consistency between image and text. In addition, the model uses rich medical prior knowledge to inject deep semantic information in the report generation process, enhancing the accuracy and relevance of the report. This method not only provides a new solution for radiology report generation, but also shows a wide range of potential applications in the field of medical image analysis. Through multi-level medical knowledge guidance, the hierarchical knowledge integration framework can more accurately align images and texts to ensure that the generated reports accurately reflect the key clinical information in radiological images. At the same time, the cross-modal fusion module can effectively combine image features with medical field knowledge, making the report content more in line with clinical needs and enhancing the practicality and reliability of the report. In addition, the framework can handle complex anatomical structures and pathological features, reduce the generalization problem caused by missing details in the report, generate more complete and coherent reports, and ensure the real-time nature of the application. This technology is not only innovative in theory, but also shows significant performance improvement in practical applications, and has wide practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flow chart of the present invention.

[0015] Figure 2 Reports generated for a visualization-based hierarchical knowledge integration framework. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0017] like Figure 1 As shown, a method for generating a radiology report based on a hierarchical knowledge integration framework is provided in one embodiment of the present invention, comprising the following steps: 1. Visual-language pre-training stage: In this stage, the model achieves multi-level alignment of images and texts through multi-task learning. Specifically, it includes the following tasks: Multi-task learning: Use pre-trained image encoders and text encoders to extract image features and text features respectively; perform contrastive learning tasks to establish coarse-grained associations between images and texts and achieve feature alignment; process image features and text features through a cross-modal fusion module, use the processed image features and text features to perform image-text matching tasks, and enhance fine-grained alignment between images and texts by combining organ and pathology labels; The contrastive learning task aims to capture the intrinsic connection between radiological images and text reports, and its loss function is: Formula 1: ; in, Representing images With text The expected value of the sample pair; Representing images With text Similarity in a shared embedding space; Representing images With text Similarity in a shared embedding space; Representing images With text Similarity in a shared embedding space; is the number of image-text pairs in the batch; k is an index variable used for normalization, indicating that all samples (image-text pairs) in the batch are traversed to calculate the sum of similarities; and Indicated in The index number of the image or text corresponding to each image-text pair; The model aligns image features and text features by minimizing the distance between image-text pairs and maximizing the distance between dissimilar image-text pairs.

[0018] The loss function of the image-text matching task is: Formula 2: ; in, is a label, containing 1 or 0, where 1 indicates that the sample matches and 0 indicates that the sample does not match; is the predicted probability obtained by mapping the similarity score through the sigmoid function, where represents the Sigmoid function, represents the similarity calculation function, represents the image feature vector, Represents the text feature vector; when the image and text are matched, the goal of the model is to make Close to 1; when the image and text are not matched, the model's goal is to make Close to 0.

[0019] Masked autoencoder and masked language model: In order to better extract the feature information of images and texts, the system introduces masked autoencoder and masked language model tasks. In the masked autoencoder task, some image blocks are randomly masked, and the model needs to reconstruct these masked parts to capture the internal representation of the image; in the masked language model task, the masked text parts are predicted to learn the contextual relationship of the text; The loss function for the mask autoencoder task is: Formula 3: ; in, is the index set of the covered image blocks in the image, is the number of covered blocks; is the index of the mask block; The original image indivual The pixel value of is the corresponding reconstructed model The pixel value of The loss function of the masked language model task is: Formula 4: ; in, is the set of all masked position indices in the text; Is the model in the mask position Prediction correct probability.

[0020] Second, the radiology report generation stage. In this stage, the system uses the pre-trained image encoder and text encoder to generate the radiology report. The specific steps include: Image feature extraction: Use the pre-trained image encoder to encode the radiological images and extract image features (the image dataset used for pre-training in the first stage is different from the image dataset in the current stage, and the parameters of the encoder have also changed); Knowledge feature fusion: Use pre-trained text encoders to convert organ knowledge and pathology knowledge related to the image into features. Through the dynamic knowledge extraction module, the organ knowledge most relevant to the image is extracted, and the image features are combined with the organ knowledge features through the cross-modal fusion module to generate image-organ knowledge features. At the same time, pathology knowledge is also dynamically selected and fused with the image to generate image-pathology knowledge features. Report generation: The fused multimodal features are hierarchically input into the Transformer decoder to simulate the diagnostic process of a radiologist: starting with observing the image as a whole, analyzing the image based on organ labels, assigning pathological labels to each organ, and finally generating a detailed radiology report.

[0021] As a preferred embodiment of the present invention, the specific process of the radiology report generation stage is as follows: First, the radiological images are encoded using a pre-trained image encoder to extract image features. ; Use the pre-trained text encoder to encode the organ knowledge and pathological knowledge associated with the image into organ knowledge features respectively and pathological knowledge characteristics ; Then, through the dynamic knowledge extraction module, the organ knowledge features Selection and Image Features Matching organ knowledge features , and use the cross-modal fusion module to complete feature fusion and generate image-organ knowledge features At the same time, through the dynamic knowledge extraction module, from the pathological knowledge characteristics Selection and Image Features Matching pathological knowledge features , and use the cross-modal fusion module to complete feature fusion and generate image-pathology knowledge features ; The specific process is as follows: Formula 5: ; Formula 6: ; Formula 7: ; Formula 8: ; in, represents the dynamic knowledge extraction module, which is responsible for matching the organ knowledge features and pathological knowledge features corresponding to the image features; It is a cross-modal fusion module used to integrate features of different modalities.

[0022] The fused multimodal features (i.e., image-organ knowledge features) and image-pathology knowledge features ) are layered into the Transformer decoder to simulate the diagnostic process of a radiologist: Formula 9: ; in, is the Transformer decoder, are multimodal features at different levels. The field to be predicted in the current report.

[0023] As a preferred embodiment of the present invention, the dynamic knowledge extraction module realizes the dynamic extraction of potential knowledge information in the image through the matching mechanism between image features and organ knowledge and pathological knowledge. Specifically, for the input image feature vector Knowledge characteristics of organs and pathological knowledge characteristics , through the dynamic knowledge extraction module, from the organ knowledge features and pathological knowledge characteristics Selection and Image Features Matching organ knowledge features and pathological knowledge characteristics : Formula 10: ; Formula 11: ; in, is the temperature parameter, which is used to adjust the smoothness of the matching distribution; is the total number of knowledge vectors; and They are used to measure image features. Knowledge characteristics of organs Between and image features Pathological knowledge characteristics The degree of similarity between and They represent traversing all samples (image-text pairs) in the batch to calculate the similarity between the image and organ knowledge and between the image and pathology knowledge.

[0024] As a preferred embodiment of the present invention, the cross-modal fusion module adopts a dual-stream cross-attention mechanism to fuse the semantic features of the image and text through the following process: First, in the self-attention layer In the image features and text features (include and ) enhance the internal representation of each modality through the interaction of internal self-attention mechanisms: Formula 12: ; in, is the self-attention mechanism, is the image feature, is the text feature, and They are respectively the image modality features and text modality features enhanced by the self-attention mechanism; Then, in the cross-attention layer The cross-modal interaction process is realized in , integrating information from different modalities into their respective representations: Formula 13: ; in, is the cross attention mechanism, As an integration from language modality Information query, generating cross-attention visual (image) features ,at the same time, As an integrated visual modality Information query, generating cross-attention language (text) features ; Finally, and Feed forward layer , generating multimodal representations of visual modalities and multimodal representation of language modalities : Formula 14: ; Finally, the multimodal features generated by superimposing image and text features For report generation: Formula 15: ; in, Represents the concatenation of feature vectors from different modalities.

[0025] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0026] Embodiment 1, as Figure 2As shown in the figure, the report generated based on the visual hierarchical knowledge integration framework can provide a better intuitive understanding of the method. The border colors in the figure are aligned with the colors presented in the text, and different colors represent different organs or diseases. The visual hierarchical knowledge integration framework uses radiological knowledge to guide image features to focus on organ structures and related diseases, thereby generating a structured report. Specifically, the visual hierarchical knowledge integration framework first focuses on the organ structure and then shifts attention to related abnormal areas. For example, due to the hierarchical integration of knowledge and interaction with the image, the generated report accurately describes the location of the right lung and the left thorax. It is worth noting that the generated report maintains a good balance between describing normal and abnormal areas, even when highlighting the features of normal areas.

[0027] The above are only preferred embodiments of the present invention. It should be pointed out that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These should also be regarded as the protection scope of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A radiology report generation method based on a hierarchical knowledge integration framework, characterized in that: The following steps are involved:

1. Visual-language pre-training stage, including the following tasks: Multi-task learning: Use pre-trained image encoders and text encoders to extract image features and text features respectively; perform contrastive learning tasks to establish coarse-grained associations between images and texts and achieve feature alignment; process image features and text features through a cross-modal fusion module; use the processed image features and text features to perform image-text matching tasks, and enhance fine-grained alignment between images and texts by combining organ and pathology labels; Masked autoencoders and masked language models: In masked autoencoders, parts of an image are randomly masked, and the model captures potential associations within the image by reconstructing the masked areas. In masked language model tasks, some key words are masked in the text, and the model learns the contextual relationships between texts by predicting the masked key words.

2. Radiology report generation stage, including the following steps: Image feature extraction: Use pre-trained image encoders to encode radiological images and extract image features; Knowledge feature fusion: Use the pre-trained text encoder to convert the organ knowledge and pathology knowledge related to the image into features; extract the organ knowledge features most relevant to the image through the dynamic knowledge extraction module, and combine the image features with the organ knowledge features through the cross-modal fusion module to generate image-organ knowledge features; at the same time, extract the pathology knowledge features most relevant to the image through the dynamic knowledge extraction module, and combine the image features with the pathology knowledge features through the cross-modal fusion module to generate image-pathology knowledge features; Report generation: The fused multimodal features are hierarchically input into the Transformer decoder to simulate the diagnostic process of a radiologist: starting with observing the image as a whole, analyzing the image based on organ labels, assigning pathological labels to each organ, and finally generating a detailed radiology report.

2. The radiology report generation method based on the hierarchical knowledge integration framework according to claim 1 is characterized in that: The loss function of the contrastive learning task is: Formula 1: ； in, Representing images With text The expected value of the sample pair; Representing images With text Similarity in a shared embedding space; Representing images With text Similarity in a shared embedding space; Representing images With text Similarity in a shared embedding space; is the number of image-text pairs in the batch; k is the index variable used for normalization; and Indicated in The index number of the image or text corresponding to each image-text pair; The model aligns image features and text features by minimizing the distance between image-text pairs and maximizing the distance between dissimilar image-text pairs.

3. The method for generating a radiology report based on a hierarchical knowledge integration framework according to claim 1, characterized in that: The loss function of the image-text matching task is: Formula 2: ； in, is a label, containing 1 or 0, where 1 indicates that the sample matches and 0 indicates that the sample does not match; is the predicted probability obtained by mapping the similarity score through the sigmoid function, where represents the Sigmoid function, represents the similarity calculation function, represents the image feature vector, Represents the text feature vector; when the image and text are matched, the goal of the model is to make Close to 1; when the image and text are not matched, the model's goal is to make Close to 0.

4. The method for generating a radiology report based on a hierarchical knowledge integration framework according to claim 1, characterized in that: The loss function for the mask autoencoder task is: Formula 3: ； in, is the index set of the covered image blocks in the image, is the number of covered blocks; is the index of the mask block; The original image indivual The pixel value of is the corresponding reconstructed model The pixel value of .

5. The method for generating a radiology report based on a hierarchical knowledge integration framework according to claim 1, characterized in that: The loss function of the masked language model task is: Formula 4: ； in, is the set of all masked position indices in the text; Is the model in the mask position Prediction correct probability.

6. The method for generating a radiology report based on a hierarchical knowledge integration framework according to claim 1, characterized in that: The specific process of the image feature extraction and knowledge feature fusion steps is as follows: First, the radiological images are encoded using a pre-trained image encoder to extract image features. ; Use the pre-trained text encoder to encode the organ knowledge and pathological knowledge associated with the image into organ knowledge features respectively and pathological knowledge characteristics ; Then, through the dynamic knowledge extraction module, the organ knowledge features Selection and Image Features Matching organ knowledge features , and use the cross-modal fusion module to complete feature fusion and generate image-organ knowledge features At the same time, through the dynamic knowledge extraction module, from the pathological knowledge characteristics Selection and Image Features Matching pathological knowledge features , and use the cross-modal fusion module to complete feature fusion and generate image-pathology knowledge features ; The specific process is as follows: Formula 5: ； Formula 6: ； Formula 7: ； Formula 8: ； in, represents the dynamic knowledge extraction module, which is responsible for matching the organ knowledge features and pathological knowledge features corresponding to the image features; It is a cross-modal fusion module, which is used to integrate the features of different modalities; Fused image-organ knowledge features and image-pathology knowledge features It is input into the Transformer decoder layer by layer to simulate the diagnosis process of radiologists: Formula 9: ； in, is the Transformer decoder, are multimodal features at different levels. The field to be predicted in the current report.

7. The method for generating a radiology report based on a hierarchical knowledge integration framework according to claim 6, characterized in that: In the dynamic knowledge extraction module, the dynamic extraction of potential knowledge information in the image is achieved through the matching mechanism between image features and organ and pathological knowledge; specifically as follows: For the input image feature vector Knowledge characteristics of organs and pathological knowledge characteristics , through the dynamic knowledge extraction module, from the organ knowledge features and pathological knowledge characteristics Selection and Image Features Matching organ knowledge features and pathological knowledge characteristics : Formula 10: ; Formula 11: ; in, is the temperature parameter, which is used to adjust the smoothness of the matching distribution; is the total number of knowledge vectors; and They are used to measure image features. Knowledge characteristics of organs Between and image features Pathological knowledge characteristics The degree of similarity between and They represent traversing all samples in the batch to calculate the similarity between the image and organ knowledge and between the image and pathology knowledge.

8. The method for generating a radiology report based on a hierarchical knowledge integration framework according to claim 7, characterized in that: The cross-modal fusion module adopts a two-stream cross-attention mechanism to fuse the semantic features of image and text through the following process: In the self-attention layer In the image features and text features The internal representation of each modality is enhanced through the interaction of internal self-attention mechanisms: Formula 12: ； in, It is the self-attention mechanism; is the image feature; is a text feature, which includes and ; and They are respectively the image modality features and text modality features enhanced by the self-attention mechanism; Then, in the cross-attention layer The cross-modal interaction process is realized in , integrating information from different modalities into their respective representations: Formula 13: ； in, is the cross attention mechanism, As an integration from language modality Information query, generating cross-attention visual features ; As an integrated visual modality Information query, generating cross-attention language features ; Finally, and Feed forward layer , generating multimodal representations of visual modalities and multimodal representation of language modalities : Formula 14: ； Finally, the multimodal features generated by superimposing image and text features For report generation: Formula 15: ； in, Represents the concatenation of feature vectors from different modalities.

Images