Method for generating medical image report fusing heterogeneous image under guidance of similar instances

Abstract

The invention provides a method for generating a medical image report fused with a heterogeneous image under the guidance of similar instances. The method mainly comprises two parts: an image feature memory module combined with the heterogeneous image and a similar instance feature fusion module. A report entity relation is extracted by combining an image feature memory module of a heterogeneous graph, the heterogeneous graph corresponding to the report is constructed as a bridge, the model is guided to pay attention to an image abnormal area, and abnormal visual features are enhanced; and the similar instance feature fusion module retrieves similar instances, fuses abnormal visual features of the similar instances, and obtains more comprehensive abnormal information while enhancing feature expression of an abnormal region. Experimental evaluation carried out on two medical image data sets of IU X-ray and MIMIC-CXR shows that the method provided by the invention is optimal in the aspect of BLEU series scoring indexes. Meanwhile, the performance of the method in METEOR and ROUGE-L indexes is also excellent. Experimental results show that the method is superior to an existing mainstream method in the aspects of accuracy and integrity of natural language generation indexes and generation reports, and the effectiveness of the method is proved.

Description

Technical Field

[0001] The present invention relates to a method for automatically generating a medical imaging report, and relates to the technical field of computer vision and natural language processing. Background Art

[0002] Automatic generation of medical imaging reports is a growing research hotspot, aiming to assist radiologists in improving the accuracy and efficiency of their reports. Inspired by the image captioning task, researchers are increasingly interested in medical report generation. A series of works has demonstrated promising results using encoder-decoder architectures. These models can effectively extract features from medical images and convert them into structured or unstructured text reports, achieving encouraging results in practice.

[0003] However, due to the inherent properties of medical images and the complexity of MRG tasks, there are still challenges that need to be addressed in this field. First, due to the limitations of imaging technology and the characteristics of human tissue, medical images have low contrast and abnormal areas usually account for a small proportion. Traditional convolutional neural network models have difficulty extracting abnormal information in small areas of medical images. Existing methods use visual memory modules and contrastive learning to extract discriminative information from image features to improve the accuracy of abnormality recognition, or fuse global and local visual features to promote report generation. There are also some methods that assist in medical report generation tasks by fusing internal information or introducing external information to improve the accuracy and fluency of reports. However, such methods usually require the introduction of external data, and the construction of external auxiliary information, such as knowledge graphs or disease label information, requires a lot of work.

[0004] Secondly, to address cross-modal representation differences, researchers have proposed a variety of technical approaches. PromptMRG utilizes a multi-task training mechanism to integrate disease type discrimination information into the feature space to achieve modality alignment. Some methods focus on using structured semantics such as knowledge graphs to establish cross-modal associations. Although these methods have made some progress, clinical report data often lacks standardized labels and annotations. In addition, the scarcity of labels also limits the effectiveness of supervised learning methods, making the models face challenges in accuracy and model robustness in real medical scenarios.

[0005] In addition, while capturing the features of abnormal regions, the MRG task also needs to consider the co-occurrence relationship between features of different regions to generate more accurate reports. In the process of identifying potential diseases, in addition to identifying abnormal regions, the mutual correlation between abnormal regions must also be considered. Similar instances often share similar pathological features and abnormal region distributions, which provides an effective reference for modeling co-occurrence relationships. Therefore, using similar instances to enhance the ability to capture the co-occurrence relationship of abnormal features helps to more accurately infer and describe disease characteristics and improve the accuracy of reports. Liu et al. compared the input image with normal samples to extract visual abnormality information, thereby enhancing the visual features of abnormal regions. However, this method detects abnormalities by comparing with manually selected normal samples, and does not consider using historical similar cases to assist in capturing abnormal regions of the image.

[0006] Building on these works, we propose a SIG model that combines heterogeneous graphs to capture the structural information of benchmark reports, assist in visual feature extraction, and highlight key image regions. Reports and images are mapped into the same vector space, and semantically similar instances are retrieved using image features to further enhance visual features and obtain more comprehensive anomaly characteristics. Summary of the Invention

[0007] The purpose of this invention is to address the limitations of traditional automated medical imaging report generation methods, which hinder the accurate characterization of abnormal features based solely on image information due to low image contrast and small abnormal regions, thereby improving the accuracy of reports. Furthermore, to address the challenges of complex co-occurrence relationships of abnormal features and the difficulty of capturing them with a single sample, this invention aims to enhance visual representations by introducing external knowledge and modeling co-occurrence patterns using similar examples, thereby generating more comprehensive and accurate medical imaging reports.

[0008] To achieve the above objectives, this application provides the following solutions:

[0009] S1. Visual feature extraction, extracting visual features of medical images.

[0010] S2. Dynamically construct heterogeneous graphs to assist in image feature extraction, enhance image feature expression capabilities, and improve image feature discrimination.

[0011] S3. Retrieve similar instances to obtain more comprehensive visual anomaly information.

[0012] S4. Combine the enhanced visual features to generate text and output a generation report.

[0013] The SIG model designed by this paper is based on an encoder-decoder architecture and consists of two main modules: a heterogeneous graph-assisted image feature memory module (FMHG) and a similar instance feature fusion module (SIFF). In addition to medical images (I) and medical reports (R), this paper extracts structured information from free-text reports and introduces a heterogeneous graph (G) to assist in report generation.

[0014] For image I, SIG is first input into the pre-trained image segmentation module to generate image I with segmentation information. sam . Image I and I sam Then it is sent to the pre-trained ResNet to extract visual features. The process is as follows:

[0015] I sam =SAM(I), (1)

[0016] V local =ResNet(I), (2)

[0017]

[0018] Where SAM() and ResNet() represent the corresponding module processing. local and Represent images I and I respectively sam local visual features.

[0019] The local visual feature V corresponding to the image I local Output hidden state after Transformer standard encoder processing Participate in subsequent operations, Encoder() represents the encoder operation:

[0020] h I =Encoder(V local ), (4)

[0021] In order to capture the implicit logical relationships in the report and extract the structured information in the free text report, the present invention uses radgraph to extract the entities and relationships in the report to construct a heterogeneous graph. Heterogeneous graphs can more accurately represent and analyze the complex information in medical reports and better understand the relationship between medical observations and organs. In addition to the three relationships defined in radgraph: modify, located and suggestive of, the present invention adds a self-loop relationship to connect the nodes themselves. In this way, the node can consider its own characteristics every time a message is transmitted, which helps to strengthen important features and make the node representation richer and more stable. The structure of the heterogeneous graph corresponding to each report is fixed. The "dynamic" mentioned above means that the node features will change as the training progresses. The model uses the same embedding layer as in the decoder to vectorize the words corresponding to the node. This allows the node features in the heterogeneous graph and the text vocabulary in the decoder stage to be represented in the same vector space, which helps to better capture the semantic association between the image and the text. In addition, different types of nodes have different semantic meanings and feature distributions. The present invention sets a specific linear transformation layer for each node type to make targeted feature adjustments based on its characteristics. The formula is as follows:

[0022] X i =Embedding(word i ), (5)

[0023] N i =X i +linear(X i ), (6)

[0024] word i Indicates that the corresponding word of the i-th entity is extracted from the report, Embedding() represents the text embedding layer, linear() is the linear layer, different linear layers are selected for different entity types, and finally the node features are generated.

[0025] After constructing the heterogeneous graph, the present invention will sam Corresponding local visual features On the one hand, input to FMHG, and the sample corresponding benchmark report R g The constructed heterogeneous graph G participates in the operation together to obtain the memory-enhanced visual feature V memory :

[0026]

[0027] on the other hand, As the input of SIFF module, combined with the retrieved similar instance report R si The corresponding heterogeneous graph G si, get the visual features V enhanced by similar instances SIFF , f SIFF () indicates the corresponding module processing:

[0028]

[0029] (7), (8) where f FMHG (), f SIFF () respectively indicate the corresponding module processing.

[0030] Next, we will explain the two module processing in more detail:

[0031] This paper proposes an image feature enhancement module (FMHG) that combines a heterogeneous graph with an image feature enhancement module (FMHG). This module extracts entity relationships from the benchmark report corresponding to the current sample and constructs a heterogeneous graph to assist in image feature extraction. During testing, no corresponding report is available for input image I, making it impossible to use text to highlight key areas of image features. To address this, the present invention constructs a memory module to memorize the image feature enhancements made by the FEM. Without the aid of a report, different key areas are adaptively highlighted based solely on the input image. The FMHG consists of two components: a heterogeneous graph-based image feature enhancement module (FEM) and a memory module.

[0032] In the image feature enhancement module (FEM) combined with the heterogeneous graph, in order to capture the complex relationships between nodes in the heterogeneous graph, the graph is input into the graph neural network for processing before entering the FEM module. Since there are only more than ten nodes in the graph on average, the 512-dimensional node features may appear redundant and it is difficult to fully utilize the high-dimensional feature space. Therefore, before graph convolution, the present invention reduces the dimensionality of the node features to reduce redundancy and help the model better capture the complex relationships in the graph. After the convolution operation of formula (10) is completed, the feature dimension is restored. The specific process can be given by the following formula:

[0033] F de =DimensionDecrease(N), (9)

[0034] F Conv =GraphConv(F de ), (10)

[0035] N Conv =DimensionIncrease(F Conv ), (11)

[0036] is the node feature of the graph, and i represents the number of nodes in the graph. DimensionDecrease() and DimensionIncrease() respectively represent the mapping of node features to reduce and increase dimensions, and are composed of multiple linear layers. The present invention sets the node feature dimension of graph convolution to 20. After the convolution is completed, the enhanced node representation is obtained by dimension upgrading. Then add it to the original node feature to get the final node feature representation FG:

[0037] GF=N+N Conv , (12)

[0038] The attention mechanism assigns different weights to elements at different positions in the input, highlighting important information and suppressing irrelevant content. It can also flexibly process information of different modalities. This is consistent with the idea of ​​using heterogeneous graphs to assist feature extraction in this paper. Therefore, the FEM module is constructed based on the attention mechanism:

[0039] X n =FCN(MultiHeadAtt(X n-1 ,GF,GF)), (13)

[0040] FCN() represents the fully connected layer, MultiHeadAtt() represents the multi-head attention mechanism, GF acts as K, V; n represents the number of operation cycles. The present invention sets n to 2, and the initial input X0 is the visual feature with segmentation information To ensure consistency between the training and testing phases, the two outputs X1 and X2 of the FEM obtained by formula (13) do not directly participate in the report generation phase, but are used to calculate the loss with the output of the memory module and participate in back propagation.

[0041] The memory module adaptively prompts abnormal areas according to the input image and enhances image features under the premise of ensuring the consistency of input in the training and testing stages. As input, the final output is V memory Input to the Transformer encoder to get h memory To participate in decoder report generation:

[0042]

[0043] V memory =Encoderlayer2(M1), (15)

[0044] M1 represents the output of the first encoder layer, which is used to participate in the loss calculation. memory Represents the memory-enhanced visual features and participates in subsequent report generation. The reason for stacking two encoder layers is to adapt to the number of iterations n of the FEM module to calculate the memory loss L meam :

[0045]

[0046] In formula (16), B represents the batch size; N is the sequence length; K is the feature dimension; meam In the calculation of, the two encoder layers output M1, V memory , and the two outputs X1 and X2 of the FEM module are used for loss calculation, and the memory of different granularity features is balanced during the training process, so that the final output V memory Have a more comprehensive memory of key areas.

[0047] Another component of the present invention is the similar instance feature fusion module SIFF, which is processed according to formula (8). As the input of SIFF module, using image features To retrieve similar instances, and then construct the heterogeneous graph G corresponding to the similar instance report si To assist in image feature extraction.

[0048] This paper defines cases with similar report semantics as “historical similar instances” rather than similarity measurement based on image features. DB , using SentenceBert to convert reports from a portion of the training dataset into 384-dimensional tensors. SentenceBert was chosen because it uses a twin network to generate semantic sentence embedding vectors suitable for clustering and similarity comparison. This method retrieves similar samples from a similar sample database using image features. This ensures consistency between training and testing, despite the modality gap between images and text.

[0049] The retrieval first maps the image to the same vector space as the text and uses cosine similarity to measure image features. and historical sample library reports S DB Semantic gap between features, take the top N historical cases with the highest cosine similarity N Participate in subsequent operations:

[0050] Case N ←max(cosine(S I ,S DB )), (17)

[0051] To reduce the impact of modality gaps, this paper clusters text vectors after dimensionality reduction. The clustering information can be considered as the potential knowledge topics of the report. The clustering results of the report are then converted into a one-hot encoding (TR), and the 384-dimensional features of the image are mapped to the same dimension as the one-hot encoding to obtain the image's knowledge topics (TI). Losses are calculated between TI and TR. The losses involved in the retrieval process of similar samples include the following:

[0052]

[0053] In formula (18), B represents the batch size, cosine() represents the cosine similarity operation, and L semantic Let the image feature vector S I Report feature S to the benchmark R Close, improve the cosine similarity between the image and the corresponding report feature. In formula (19), L represents the number of reported clustering results, L topic Make the constructed image theme tend to the theme of the corresponding report, reducing the modality gap between the image and the text; in formula (20), N represents the number of similar instances retrieved, and K represents the feature dimension. selected Calculate the top N samples S with the highest similarity retrieved instance The benchmark reports a tensor S with the image R The gap between them is reduced, so that the report tensors of the first N similar instances are as semantically close to the benchmark report as possible. The similar instances selected during the training process do not necessarily match the semantics of the benchmark report, and there may be large semantic differences. Therefore, the present invention introduces the cosine similarity calculated during the retrieval process as the coefficient of the semantic gap between the similar instance and the benchmark report. The higher the cosine similarity value, the larger the coefficient, the greater the penalty for the similar instance, making its semantics close to the benchmark report; the lower the cosine similarity, the smaller the coefficient, the weaker the penalty, and avoids over-closing instances with large semantic differences. selected Through this dynamic coefficient, the retrieved similar instances are semantically closer to the benchmark report, avoiding forcibly pulling in samples with large semantic gaps.

[0054] After retrieving similar instances, we use the FEM submodule in FMHG to enhance image features using similar instances:

[0055]

[0056] In formula (21) represents the corresponding heterogeneous graph constructed from the retrieved N similar samples; f FEM () represents the enhancement operation of FMHG; as shown in formula (22), the image features after the N similar samples are enhanced are obtained Fusion and L in the memory module selected The loss calculation is similar, using cosine similarity as a coefficient. Samples with higher similarity have a greater impact on the subsequent generation process.

[0057] So far, three types of visual features are obtained: V local , V memory and V SIFFThe above visual features are input into the encoder to obtain the corresponding three hidden states: h I , h memory and h SIFF The input h of the Transformer decoder is combined Taking the three hidden states into consideration, we get:

[0058]

[0059] As shown in formula (23), four trainable parameters a, b, c, and d are defined to control the weights of different parts. During training, h combined

[0060] Input to decoder for report generation.

[0061] During training, by reducing the prediction report The cross entropy loss between the true report Y is used to optimize the model. The loss function L RG As follows, N g Represents the length of the forecast report, y i and Represents the word at position i of the real report and the predicted report respectively:

[0062]

[0063] In addition, in order to make the prediction report semantically closer to the benchmark report, SentenceBert is used to convert the prediction report into a 384-dimensional tensor, and the loss is calculated with the tensor corresponding to the benchmark report:

[0064]

[0065] SB() represents the text-to-data processing, and B represents the batch size. The present invention expresses the total loss function as:

[0066] L osstotal =L RG +L semantic +L topic +L selected +L mem +L meaning , (26)

[0067] Finally, the training method of the invented method for generating medical imaging reports by fusing heterogeneous images under the guidance of similar examples is as follows:

[0068] The present invention uses ResNet-101 pre-trained on ImageNet as a visual feature extractor. According to the work of Chen et al., for the IU X-Ray dataset, the present invention uses front and side views as input, and uses a single image as input on the mimic dataset. In the model of the present invention, the dimension of the multi-head attention is set to 512, the number of heads is set to 8, and the number of Transformer encoder layers is set to 3. In the SIFF module, the number of similar instances is set to 3. Under the action of the ADAM optimizer, the model is trained for 100 times. The present invention sets the initial learning rate to 1e-4, and the learning rate decays to 0.8 of the original value for each iteration. The beam search size in the report generation is set to 3. The initial values ​​of a, b, c, and d in formula (23) are all set to 1. The model is implemented on Python 3.7 and trained on an NVIDIA Tesla P100 GPU.

[0069] Compared with the existing technology, the beneficial effects of the present invention are:

[0070] 1) A heterogeneous graph-assisted image feature memory module was constructed. During training, heterogeneous graphs were dynamically constructed to assist in image feature extraction, enhancing the expressive power of image features and improving the discrimination of image features.

[0071] 2) Design a similar instance-assisted medical report generation strategy. Utilize historical similar instances to assist report generation, obtain more comprehensive abnormal features, and capture the co-occurrence relationship of abnormal features. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 For chest CT images and corresponding reports, we demonstrate the inherent properties of medical images and the complexity of MRG tasks;

[0073] Figure 2 This is the model structure diagram of the medical imaging report generation method (SIG) that fuses heterogeneous images under the guidance of similar instances;

[0074] Figure 3 This is the structure diagram of the image feature memory module (FMHG) combined with heterogeneous graphs;

[0075] Figure 4 It is the process of constructing heterogeneous graphs;

[0076] Figure 5 This is the structure diagram of the similar sample feature fusion module (SIFF);

[0077] Figure 6 Comparison of natural language generation results of different models on IU-X-Ray and MIMIC-CXR.

[0078] Figure 7Generate boxplots of natural language metrics for the model on two datasets;

[0079] Figure 8 This is the image segmentation processing effect diagram;

[0080] Figure 9 The ablation experiment results of the model on the IU-X-Ray dataset;

[0081] Figure 10 The experimental results of parameter analysis of the method of the present invention on the IU X-Ray dataset are shown in Figure 2.

[0082] Figure 11 The loss ablation experimental results of the method of the present invention on IU X-Ray;

[0083] Figure 12 Comparison of real reports with those generated by the baseline model and the proposed model (SIG); DETAILED DESCRIPTION

[0084] The accompanying drawings are only for illustrative purposes and should not be construed as limiting this patent.

[0085] The present invention is further described below with reference to the accompanying drawings and embodiments.

[0086] Figure 1 The inherent properties of medical images and the complexity of MRG tasks show that there are some challenges that need to be addressed in this field. Medical images are highly similar, and abnormal regions usually occupy a smaller part of the image. This data bias causes the model to over-learn the features of normal regions during training and ignore the subtle differences in abnormal regions. Figure 1 As shown on the right, an "oblique scarring" is highlighted in a small abnormal area on the left chest CT image. This area is small in size and not significantly different from the normal area. This characteristic makes it difficult for traditional image processing methods to extract the salient features of this area. To address this difficulty, FMHG combines the structured information of the image with the heterogeneous graph to better guide the model's focus on potential key areas, enhance the visual characteristics of the abnormality, and thus improve the quality of the generated medical report.

[0087] Figure 2 This is the SIG overall framework, built on an encoder-decoder architecture, and consists of four main components: (a) Visual Feature Extraction, which extracts local and global medical image features. (b) An Image Feature Memory Module that combines heterogeneous graphs to construct and combine heterogeneous graphs to enhance medical visual features. (c) A Similar Instance Feature Fusion Module that retrieves similar instances and obtains more comprehensive visual anomaly information. (d) Text Generation.

[0088] Figure 3This is the FMHG structure diagram. This module is designed to simulate the process of doctors focusing their attention on the lesion area based on experience during clinical diagnosis. Figure 3 As shown in Figure 1, FMHG consists of two components: an image feature enhancement module (FEM) that combines heterogeneous maps with image correspondence reports, and a memory module. The FEM uses the heterogeneous map constructed from image correspondence reports to highlight key image regions and enhance image features. The memory module adaptively highlights abnormal regions based on the input image and enhances image features, while ensuring consistent input during training and testing.

[0089] Figure 4 This paper demonstrates the construction process of a heterogeneous graph. This paper uses RadGraph to extract entities and relationships from reports and constructs a heterogeneous graph. Heterogeneous graphs can more accurately represent and analyze complex information in medical reports, providing a better understanding of the relationships between medical observations and organs.

[0090] Figure 5 Figure 1 shows the SIFF structure. In the process of generating medical image reports, the doctor's experience is crucial. Experience can essentially be understood as similar examples encountered in one's career. The design of the similar example feature fusion module is based on this understanding.

[0091] Figure 6 Comparing the SIG model with existing research on two datasets, the model outperformed other methods in almost all metrics, demonstrating its ability to generate more fluent and accurate reports. Compared to the next highest values, the average increase from BLEU-1 to BLEU-4 was 4.4% on the IU X-Ray dataset and 3.1% on MIMIC-CXR. On the IU X-Ray dataset, METEOR improved by 6.6%, demonstrating that the generated text excels in accuracy, fluency, and diversity.

[0092] Figure 7 The box plots of the evaluation indicators of the training phase on the two datasets are shown. From the figure, it can be found that the overall trend of the distribution of different experimental indicators on the two datasets is consistent, reflecting the robustness of the model.

[0093] Figure 8 To demonstrate the effect of image segmentation processing, in order to avoid the influence of different color blocks on image features, unlike the original model, the present invention deepens the color of the segmented areas uniformly to emphasize the segmented areas.

[0094] Figure 9The results of the ablation experiment of the present invention on the IU-X-Ray dataset are presented. After the introduction of FMHG on the baseline model, all indicators have been significantly improved, indicating that the fluency and accuracy of report generation have been significantly enhanced, proving that the FMHG module can use heterogeneous graphs to assist in image feature extraction, and that the memory module in FMHG can effectively memorize the enhancement effect of heterogeneous graphs on image features. After the introduction of SIFF on the baseline model, all indicators have been significantly improved, indicating that SIFF can effectively improve the quality of generated reports by retrieving semantically similar instances and fusing instance features. In addition to the two main modules mentioned above, the present invention also conducted an ablation experiment on SAM. The experiment found that after the introduction of SAM, all indicators have been improved, indicating that its image segmentation processing can focus visual features on the key areas of the image, reduce the noise impact of irrelevant areas, and thus improve the model effect.

[0095] Figure 10 The results of parameter analysis experiments using the IU X-Ray dataset are presented. The model achieves optimal performance when the number of reference similarity instances is 3. Too few similarity instances prevents the model from capturing sufficient comprehensive information and enhancing image features. Excessive similarity may introduce noise and irrelevant information, leading to performance degradation. Therefore, selecting an appropriate number of similarity instances, balancing information richness with interference, is a key factor influencing model performance.

[0096] Figure 11 Explore the effects of different components in the loss function on model training. In the figure, semantic, topic, selected, mem, and meaning represent the losses of the corresponding subscripts. The above losses are divided into three categories for analysis. The first category of loss is the FMHG module related loss L mem The second type of loss is SIFF module related loss, including L semantic 、L topic and L selected The third type of loss is semantic loss L meaning ; Remove the first type of loss L mem After that, the output of the memory module that actually participates in report generation in FMHG cannot remember the image feature enhancement effect contained in the FEM output. The experimental results are lower than those of the complete model in Table 2, and are close to the performance after removing the FMHG module. This shows that the role of the FMHG module depends on L mem The existence of the second type of loss; experimental analysis was conducted on the second type of loss, and the three losses included were removed respectively. The experimental results showed that the evaluation indicators all decreased to varying degrees. semantic It is related to the retrieval process of similar instances. After removal, the accuracy of retrieving similar instances cannot be guaranteed. The experimental results are basically close to the base, indicating that the loss of L semainticAfter that, the similar instances introduced not only fail to promote the generation of accurate reports, but also introduce a lot of noise, which has a negative impact on the MRG task. topic The post-evaluation index dropped significantly, proving that topic The penalty can effectively reduce the modality gap between images and texts, thereby improving the retrieval accuracy of similar instances and thus improving model performance; the third type of loss L meaning The evaluation indicators also decreased after removal. Experimental results show that by reducing the semantic gap between the prediction report and the benchmark report, the performance of the model can be effectively improved.

[0097] Figure 12 Details of real-world reports, reports generated by a baseline model, and reports generated by our proposed model (SIG) are presented. In the first example, the real-world report mentioned "no focal comorbidities" and "no signs of pneumothorax or pleural effusion," which the baseline model did not. The SIG-predicted report was largely consistent with the real-world report. In the second example, the SIG-generated report was more accurate than the baseline model, hitting most of the keywords in the real-world report. Furthermore, the report was longer, demonstrating that SIG maintained accurate lesion prediction while generating complex sentences.

[0098] This paper proposes a similar instance-guided medical image report generation method (SIG), which combines heterogeneous graphs to capture the structured information of the benchmark report, assists in visual feature extraction and prompts key areas of the image. The report and image are mapped to the same vector space, and semantically similar instances are retrieved using image features to further enhance the visual features and obtain more comprehensive abnormality features. Experimental results on the IU X-Ray and MIMIC-CXR datasets show that the SIG method is superior to existing methods. The research focus of this paper is to explore the impact of visual information on the medical report generation process and to enhance the model's understanding and utilization of medical image features. However, the cross-modal semantic gap poses a challenge to the performance of the model. Not limited to the optimization of visual information, future work will explore the deep fusion of text and visual features to achieve more accurate and efficient medical report generation.

[0099] Finally, the details of the above embodiments of the present invention are merely examples for explaining the present invention. For those skilled in the art, any modifications, improvements and replacements of the above embodiments should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for generating a medical imaging report by fusing heterogeneous images under the guidance of similar instances, the method comprising the following steps: S1. Visual feature extraction, extracting visual features of medical images. S2. Dynamically construct heterogeneous graphs to assist in image feature extraction, enhance image feature expression capabilities, and improve image feature discrimination. S3. Retrieve similar instances to obtain more comprehensive visual anomaly information. S4. Combine the enhanced visual features to generate text and output a generation report.

2. The method for generating a medical imaging report by fusing heterogeneous images under similarity instance guidance according to claim 1, wherein the specific process of S1 is as follows: For image I, SIG is first input into the pre-trained image segmentation module to generate image I with segmentation information. sam . Image I and I sam Then it is sent to the pre-trained ResNet to extract visual features. The process is as follows: AND sam =Alone(I), (1) V local =ResNet(I), (2) Where SAM() and ResNet() represent the corresponding module processing. local and Represent images I and I respectively sam local visual features. The local visual feature V corresponding to the image I local Output hidden state after Transformer standard encoder processing Participate in subsequent operations, Encoder() represents the encoder operation: h I =Encoder(V local ), (4) 3. The method for generating a medical imaging report by fusing heterogeneous graphs under similarity instance guidance according to claim 1, wherein the specific process of S2 is as follows: To capture implicit logical relationships within reports and extract structured information from free-text reports, the present invention utilizes Radgraph to extract entities and relationships within reports and construct a heterogeneous graph. Heterogeneous graphs can more accurately represent and analyze complex information within medical reports, providing a better understanding of the associations between medical observations and organs. In addition to the three relationships defined in Radgraph: modify, located, and suggestive of, the present invention adds self-loop relationships, connecting nodes to themselves. This allows nodes to consider their own characteristics during each message transmission, helping to strengthen important features and enriching and stabilizing node representation. The structure of the heterogeneous graph corresponding to each report is fixed; the "dynamic" mentioned above refers to the fact that node features change as training progresses. The model uses the same embedding layer as the decoder to vectorize the words corresponding to nodes. This allows node features in the heterogeneous graph to be represented in the same vector space as the text vocabulary at the decoder stage, helping to better capture the semantic connections between image and text. Furthermore, different node types have different semantic meanings and feature distributions. The present invention employs a specific linear transformation layer for each node type to tailor its features to its specific characteristics. The formula for this is as follows: X i =Embedding(word i ), (5) N i =X i +linear(X i ), (6) word i Indicates that the corresponding word of the i-th entity is extracted from the report, Embedding() represents the text embedding layer, linear() is the linear layer, different linear layers are selected for different entity types, and finally the node features are generated. After constructing the heterogeneous graph, the present invention will sam Corresponding local visual features On the one hand, input to FMHG, and the sample corresponding benchmark report R g The constructed heterogeneous graph G participates in the operation together to obtain the memory-enhanced visual feature V memory : f FMHG () represents the processing of the image feature enhancement module FMHG combined with the heterogeneous graph. Specifically, FMHG consists of two parts: the image feature enhancement module (FEM) combined with the heterogeneous graph and the memory module. In the image feature enhancement module (FEM) combined with the heterogeneous graph, in order to capture the complex relationship between the nodes in the heterogeneous graph, the heterogeneous graph is input into the graph neural network for processing before entering the FEM module. Since there are only more than ten nodes in the graph on average, the 512-dimensional node features may appear redundant and it is difficult to fully utilize the high-dimensional feature space. Therefore, before graph convolution, the present invention reduces the dimensionality of the node features to reduce redundancy and help the model better capture the complex relationships in the graph. After the convolution operation of formula (9) is completed, the feature dimension is restored. The specific process can be given by the following formula: F de =DimensionDecrease(N), (8) F Conv =GraphConv(F de ), (9) N Conv =DimensionIncrease(F Conv ), (10) is the node feature of the graph, and i represents the number of nodes in the graph. DimensionDecrease() and DimensionIncrease() respectively represent the mapping of node features to reduce and increase dimensions, and are composed of multiple linear layers. The present invention sets the node feature dimension of graph convolution to 20. After the convolution is completed, the enhanced node representation is obtained by dimension upgrading. Then add it to the original node feature to get the final node feature representation GF: GF=N+N Conv , (11) After obtaining the node feature representation GF, it is input into the FEM module for processing: X n =FCN(MultiHeadAtt(X n-1 ,GF,GF)), (12) FCN() represents the fully connected layer, MultiHeadAtt() represents the multi-head attention mechanism, GF acts as K, V; n represents the number of operation cycles. The present invention sets n to 2, and the initial input X0 is the visual feature with segmentation information To ensure consistency between the training and testing phases, the two outputs X1 and X2 of the FEM obtained by formula (12) do not directly participate in the report generation phase, but are used to calculate the loss with the output of the memory module and participate in back propagation. The present invention stacks two encoder layers as a memory module to receive As input, the final output is V memory Input to the Transformer encoder to get h memory To participate in decoder report generation: V memory =Encoderlayer2(M1), (14) M1 represents the output of the first encoder layer, which is used to participate in the loss calculation. memory Represents the memory-enhanced visual features and participates in subsequent report generation. The reason for stacking two encoder layers is to adapt to the number of iterations n of the FEM module to calculate the memory loss L mem : In formula (15), B represents the batch size; N is the sequence length; K is the feature dimension; mem In the calculation of, the two encoder layers output M1, V memory , and the two outputs X1 and X2 of the FEM module are used for loss calculation, and the memory of different granularity features is balanced during the training process, so that the final output V memory Have a more comprehensive memory of key areas.

4. The method for generating a medical imaging report by fusing heterogeneous graphs under similarity instance guidance according to claim 1, wherein the specific process of S3 is as follows: Another component of the present invention is the similar instance feature fusion module SIFF. SIFF receives As the input of SIFF module, using image features To retrieve similar instances, and then construct the heterogeneous graph G corresponding to the similar instance report si To assist in image feature extraction. This paper defines cases with similar report semantics as “historical similar instances” instead of similarity measurement based on image features. DB , using SentenceBert to convert reports from a portion of the training dataset into 384-dimensional tensors. SentenceBert was chosen because it uses a twin network to generate semantic sentence embedding vectors suitable for clustering and similarity comparison. This method retrieves similar samples from a similar sample database using image features. This ensures consistency between training and testing, despite the modality gap between images and text. The retrieval first maps the image to the same vector space as the text and uses cosine similarity to measure image features. and historical sample library reports S DB Semantic gap between features, take the top N historical cases with the highest cosine similarity N Participate in subsequent operations: Case N ←max(cosine(S I ,S DB )), (16) To reduce the impact of modality gaps, this paper clusters text vectors after dimensionality reduction. The clustering information can be considered as the potential knowledge topics of the report. The clustering results of the report are then converted into a one-hot encoding (TR), and the 384-dimensional features of the image are mapped to the same dimension as the one-hot encoding to obtain the image's knowledge topics (TI). Losses are calculated between TI and TR. The losses involved in the retrieval process of similar samples include the following: In formula (17), B represents the batch size, cosine() represents the cosine similarity operation, and L semantic Let the image feature vector S I Report feature S to the benchmark R Close, improve the cosine similarity between the image and the corresponding report feature. In formula (18), L represents the number of reported clustering results, L topic Make the constructed image theme tend to the theme of the corresponding report, reducing the modality gap between the image and the text; in formula (19), N represents the number of similar instances retrieved, and K represents the feature dimension. selected Calculate the top N samples S with the highest similarity retrieved instance The benchmark reports a tensor S with the image R The gap between them is reduced, so that the report tensors of the first N similar instances are as semantically close to the benchmark report as possible. The similar instances selected during the training process do not necessarily match the semantics of the benchmark report, and there may be large semantic differences. Therefore, the present invention introduces the cosine similarity calculated during the retrieval process as the coefficient of the semantic gap between the similar instance and the benchmark report. The higher the cosine similarity value, the larger the coefficient, the greater the penalty for the similar instance, making its semantics close to the benchmark report; the lower the cosine similarity, the smaller the coefficient, the weaker the penalty, and avoids over-closing instances with large semantic differences. selected Through this dynamic coefficient, the retrieved similar instances are semantically closer to the benchmark report, avoiding forcibly pulling in samples with large semantic gaps. After retrieving similar instances, we use the FEM submodule in FMHG to enhance image features using similar instances: In formula (20) represents the corresponding heterogeneous graph constructed from the retrieved N similar samples; f FEM () represents the enhancement operation of FMHG; as shown in formula (21), the image features after the N similar samples are enhanced are obtained Fusion and L in the memory module selected The loss calculation is similar, using cosine similarity as a coefficient. Samples with higher similarity have a greater impact on the subsequent generation process.

5. The method for generating a medical imaging report by fusing heterogeneous graphs under similarity instance guidance according to claim 1, wherein the specific process of S4 is as follows: The three types of visual features obtained by S1, S2, and S3: V loal , V memory and V SIFF Input to the encoder to obtain the corresponding three hidden states: h I , h memory and h SIFF The input h of the Transformer decoder is combined Taking the three hidden states into consideration, we get: As shown in formula (22), four trainable parameters a, b, c, and d are defined to control the weights of different parts. During training, h combined Input to decoder for report generation. During training, by reducing the prediction report The cross entropy loss between the true report Y is used to optimize the model. The loss function L RG As follows, N g Represents the length of the forecast report, y i and Represents the word at position i of the real report and the predicted report respectively: In addition, in order to make the prediction report semantically closer to the benchmark report, SentenceBert is used to convert the prediction report into a 384-dimensional tensor, and the loss is calculated with the tensor corresponding to the benchmark report: SB() represents the text-to-data processing, and B represents the batch size. The present invention expresses the total loss function as: Loss total =L RG +L semantic +L topic +L selected +L mem +L meaning , (25)。

Images