Radiology report generation method and system based on feature fusion and medical guidance

Abstract

The invention belongs to the technical field of multi-modal natural language generation, and provides a radiology report generation method and system based on feature fusion and medical guidance, and the method comprises the steps: carrying out the feature extraction of medical image data and text report data, obtaining the text features with enhanced medical knowledge, and carrying out the feature extraction of the text features; obtaining image space-time characteristics reflecting the evolution information of the focus; injecting the medical knowledge enhanced text features into image space-time features by using a cross attention mechanism to generate multi-modal space-time fusion features; and according to the multi-modal space-time fusion features and a preset decoder, generating a radiology department medical text report, and by integrating historical and current multi-temporal and multi-view image data of a patient, combining anatomical structure constraints and lesion evolution rule modeling, on the premise of guaranteeing term normalization and logic coherence, establishing a medical text report of the radiology department. Longitudinal lesion evolution information and multi-view complementary information can be fully utilized, and the professionality and accuracy of reports are improved.

Description

Technical Field

[0001] This invention belongs to the field of multimodal natural language generation technology, and in particular relates to a method and system for generating radiology reports based on feature fusion and medical guidance. Background Technology

[0002] With the rapid development of medical imaging equipment and hospital informatization, clinical practice generates massive amounts of images and reports. Radiologists are under constant pressure to review images and write reports, making them susceptible to subjective experience and fatigue, leading to problems such as missed diagnoses, misdiagnoses, and inconsistent report quality. How to utilize artificial intelligence to automatically or semi-automatically generate high-quality radiology reports, thereby reducing the burden on doctors and improving diagnostic efficiency and consistency, has become an important research direction in medical auxiliary diagnosis.

[0003] While existing image-to-report generation methods have made some progress, they mostly rely on images from a single time point, neglecting the information on lesion evolution contained in the patient's longitudinal images; they do not make sufficient use of prior knowledge on the relationship between anatomical structures, lesion types, etc., and the generated text still falls short in terms of professionalism, accuracy, and interpretability; at the same time, their ability to model the spatial location of lesions and the fusion of multi-view and multi-view images is limited, making it difficult to fully reflect real clinical description habits. Summary of the Invention

[0004] To address the aforementioned issues, this invention proposes a method and system for generating radiological reports based on feature fusion and medical guidance. By integrating patient history and current multi-temporal and multi-view imaging data, and combining anatomical constraints with lesion evolution modeling, this invention significantly improves the clinical usability and physician trust of the generated reports while ensuring the standardization of terminology and logical coherence.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solution: In a first aspect, the present invention provides a method for generating radiology reports based on feature fusion and medical guidance, comprising: Acquire the patient's current medical imaging data, historical medical imaging data, and corresponding text report data; Feature extraction was performed on medical image data and text report data respectively to obtain medical knowledge-enhanced text features and image spatiotemporal features reflecting lesion evolution information; By utilizing a cross-attention mechanism, the textual features enhanced with the medical knowledge are injected into the spatiotemporal features of the image, generating multimodal spatiotemporal fusion features; Based on multimodal spatiotemporal fusion features and a pre-defined decoder, a radiology medical text report is generated.

[0006] Furthermore, the text report data feature extraction includes: A graph was constructed based on text report data and introduced external medical knowledge. and its adjacency matrix The adjacency matrix elements in Indicates the first The word and the first Dependencies between words; Use natural language processing tools to build graphs for the report. and adjacency matrix The node value is Integrate the medical knowledge subtrees into a complete syntactic-medical knowledge graph. : ; The value of each node The sum of corresponding elements of two adjacent matrices: = + +1; For the input text, the Bio-ClinicalBERT medical feature extraction model is used to extract basic semantic representations of the text. The encoded output of the text is: ; Encoded output As node weights in the constructed knowledge graph, features from neighboring nodes are then aggregated and node embeddings are updated; j For nodes i Attention coefficient Defined as: ; in, It is a learnable matrix. This indicates a concatenation operation. It is a feedforward neural network whose parameters are determined by the weight vector. Definition; final node weight Represented as: ; in, , Indicates the activation function; Representative node The adjacent nodes are identified; finally, the fused syntactic-medical knowledge graph is semantically encoded to obtain text features.

[0007] Furthermore, the feature extraction of the current image data and the historical image data includes: medical image data arranged in chronological order, extracting features of the current image and features of historical images of the same patient respectively; if there are no historical images, a trainable context vector is used to replace the historical features, and concatenated with the feature vector processed by the knowledge-guided spatiotemporal alignment module and input into the encoder; subsequently, the knowledge-guided spatiotemporal alignment module is used to perform lesion region matching and difference enhancement on the current features and historical features.

[0008] Furthermore, the processing steps of the knowledge-guided spatiotemporal alignment module include: dividing the feature map after encoding the main-view image into multiple quadrants to enhance the prior understanding of the spatial distribution of lesions; and performing cross-attention calculation on the image encoding content of one quadrant of the current image and the text features to obtain the fused features of spatial and text features. Identify key semantic regions: ; in, Indicates the first Each quadrant; yes dimensionality Q= ，K= ，V= , , , A learnable weight vector; , , Each is the current number i Query vectors, key vectors, and value vectors for each image quadrant; The first result obtained after medical feature encoding of the text report i A contextual semantic representation vector of each text token; The calculated results are concatenated with the corresponding quadrant features from historical images, and then input into a convolutional block for dimensionality reshaping and layer normalization. Channel attention is used to highlight the response intensity of key feature channels, and a gating mechanism is used for dynamic fusion. Subsequently, the quadrant representations are concatenated again, and channel attention calculation, gating fusion, and a feedforward neural network are performed again to finally output the spatiotemporally aligned image representation. If there are K images from different perspectives, the other images are also input in parallel to guide the spatiotemporal alignment module to obtain multi-view image representations. ,in, .

[0009] Furthermore, the multi-view feature fusion module is used to model the relationships between images from different body positions: from front view features get and Other perspective features get Cross-attention calculation is performed to enhance semantic alignment across multiple perspectives, and the final attention score is: ; in Q= , K= , V= , , , This is a learnable weight vector.

[0010] Furthermore, deep global context modeling is performed on multi-view images. Specifically, the front view features processed by the knowledge-guided spatiotemporal alignment module and the local features of other viewpoints are linearly projected and mapped into a set of embedded vector sequences, which are then input into an encoder with a 12-layer Transformer structure. Three additional learnable redundant sanitized embedding labels are introduced at the end of the first layer input sequence. After the 12-layer encoder alternately performs multi-head self-attention and multilayer perceptron operations, the image features of each viewpoint are deeply evolved.

[0011] Furthermore, the text generator is subjected to supervised learning by minimizing the cross-entropy loss function between radiology reports generated by the text generator and those written by real radiologists. This loss function can be expressed as: : ; in, Indicates the length of the generated report; The model attempts to predict the first... One word; This indicates the previously entered text. This represents the learnable parameters of the model.

[0012] Secondly, the present invention also provides a radiology report generation system based on feature fusion and medical guidance, comprising: The data acquisition module is configured to acquire the patient's current medical image data, historical medical image data, and corresponding text report data. The feature extraction module is configured to extract features from medical image data and text report data respectively, to obtain medical knowledge-enhanced text features, and to obtain spatiotemporal image features that reflect lesion evolution information; The feature fusion module is configured to: inject the medical knowledge-enhanced text features into the image spatiotemporal features using a cross-attention mechanism to generate multimodal spatiotemporal fusion features; The report generation module is configured to generate radiology medical text reports based on multimodal spatiotemporal fusion features and a preset decoder.

[0013] Thirdly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the feature fusion and medical-guided radiology report generation method described in the first aspect.

[0014] Fourthly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the program to implement the steps of the feature fusion and medical-guided radiology report generation method described in the first aspect.

[0015] Fifthly, the present invention also provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of the feature fusion and medical-guided radiology report generation method described in the first aspect.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention extracts features from medical imaging data and text report data to obtain medical knowledge-enhanced text features and image spatiotemporal features reflecting lesion evolution information. It then uses a cross-attention mechanism to inject the medical knowledge-enhanced text features into the image spatiotemporal features, generating multimodal spatiotemporal fusion features. Based on the multimodal spatiotemporal fusion features and a preset decoder, it generates a radiology medical text report. By integrating patient history and current multi-temporal and multi-view image data, and combining anatomical constraints with lesion evolution modeling, it can more fully utilize longitudinal lesion evolution information and multi-view complementary information while ensuring terminology standardization and logical coherence, thus improving the professionalism and accuracy of the report.

[0017] 2. This invention divides image features into multiple spatial regions and aligns and fuses corresponding regions in current and historical images, thereby more accurately depicting the spatial changes of lesions, improving the accuracy of lesion evolution description, and finally outputting spatiotemporally aligned image representations.

[0018] 3. This invention uses the frontal view features as global anchor points and increases feature interaction between multi-position images by calculating the correlation scores between other view features and the frontal view features. Attached Figure Description

[0019] The accompanying drawings, which form part of this embodiment, are used to provide a further understanding of this embodiment. The illustrative embodiments and their descriptions are used to explain this embodiment and do not constitute an improper limitation of this embodiment.

[0020] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention. Detailed Implementation

[0021] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0023] Example 1: This embodiment proposes a radiology report generation method based on feature fusion and medical guidance. First, initial data is acquired and preprocessed to obtain a processed dataset. A text generation model is constructed using a dual-branch multimodal coding approach, extracting multimodal spatiotemporal features from images combined with historical diagnoses and medically enhanced text. Then, the spatiotemporal features of the images are input into a multi-view feature fusion enhancement module, and a cross-attention mechanism is used to inject medical knowledge features into the image representation, generating multimodal spatiotemporal fusion features. Finally, the fusion features are input into a decoder with a memory retention mechanism, generating medical descriptive text that conforms to clinical narrative logic. This method can more fully utilize longitudinal lesion evolution information and multi-view complementary information, improving the professionalism and accuracy of the report. Figure 1 As shown, the method specifically includes: S1. Obtain the patient's current medical image data, historical medical image data, and corresponding text report data.

[0024] S2. Construct a report generation model, which includes a text feature extraction branch, an image feature extraction branch, and a decoder.

[0025] S2.1 The text feature extraction branch combines medical knowledge graphs and syntactic analysis to extract text features including medical entities and syntactic logic. The specific content of the text feature extraction branch includes: Input a segment of length text RadGraph is introduced as an external medical knowledge supplement to construct a medical knowledge graph for the input text. and its adjacency matrix , among which the figure Representing the relationships between words, nodes are... Words containing medical knowledge It is a set of nodes. It is the set of edges, adjacency matrix elements in Indicates the first The word and the first Dependencies between words: ; in, For the image Two words in the text; This indicates a grammatical dependency between the two words. (The formula has been modified in yellow, and the explanation follows the modified formula.) Use natural language processing tools to construct syntactic dependency graphs for the report. and adjacency matrix , among which the figure Representing the relationships between words, nodes are... All the words that make up, It is a set of nodes. It is the set of edges, adjacency matrix elements in Indicates the first The word and the first Dependencies between words: ; in, For the image Two words in the text; This indicates that there is a grammatical dependency between the two words.

[0026] The medical knowledge subtrees are integrated into a complete syntactic-medical knowledge graph. : ; The value of each node This is the sum of the absolute values ​​of corresponding elements in two adjacent matrices: = |+ +1; For the input text, the Bio-ClinicalBERT medical feature extraction model is used to extract basic semantic representations of the text. The final output after encoding is: ; Will As node weights in the constructed knowledge graph, a Graph Attention Network (GAT) is then used to aggregate features from neighboring nodes and update node embeddings. j For nodes iAttention coefficient Defined as: ; in, It is a learnable matrix. This indicates a concatenation operation. It is a feedforward neural network whose parameters are determined by the weight vector. Definition. Final node weight. Represented as: ; in , Indicates the activation function; Representative node The adjacent nodes are then used. Finally, a two-layer graph attention network is used to perform deep semantic encoding on the fused syntactic-medical knowledge graph to obtain text features.

[0027] S2.1.2 The image feature extraction branch extracts current image features and historical image features respectively, performs feature fusion through the knowledge-guided spatiotemporal alignment module, and performs global context modeling through the multi-view feature fusion module.

[0028] Optionally, the medical image data is arranged chronologically, and the image feature extraction branch adopts a two-stage feature extraction process combining ResNet-101 and ViT. First, ResNet-101 is used to extract the features P of the current image and the features C of recent historical images of the same patient; if no historical images are available, a trainable context vector is used. The replaced historical features are concatenated with the feature vectors processed by the knowledge-guided spatiotemporal alignment module and then input into the transformer encoder. Subsequently, the knowledge-guided spatiotemporal alignment module performs lesion region matching and difference enhancement on the current and historical features.

[0029] Optionally, the processing steps of the knowledge-guided spatiotemporal alignment module include: dividing the feature map after inputting the main view image into ResNet-101 encoding into four quadrants to enhance the prior understanding of the spatial distribution of lesions; and performing cross-attention calculation on the image encoding content of a certain quadrant of the current image and the text features to obtain the fused features of spatial and text features. Identify key semantic regions: ； in, Indicates the first Each quadrant; yes dimensionality Q= ，K= ，V= , , , This is a learnable weight vector. , , Each is the current number i Query vectors, key vectors, and value vectors for each image quadrant; The first result obtained after medical feature encoding of the text report i A contextual semantic representation vector for each text token.

[0030] Compare the calculation results with the corresponding quadrant features in historical images. The input convolutional blocks are stitched together, undergoing dimensionality reshaping and layer normalization. Channel attention is used to highlight the response intensity of key feature channels, and a gating mechanism is employed for dynamic fusion. Subsequently, the quadrant representations are stitched together again, followed by channel attention calculation, gating fusion, and a feedforward neural network. The final output is a spatiotemporally aligned image representation. If there are K images from different perspectives, then the other images are input in parallel to guide the spatiotemporal alignment module to obtain multi-view image representations. ,in, .

[0031] If no historical imagery is available, a trainable context vector is used. The replaced historical features are concatenated with the feature vectors from the knowledge-guided spatiotemporal alignment module and then input into the downstream module.

[0032] In some embodiments, ViT is used to perform deep global context modeling on multi-view images. Specifically, the frontal view features processed by the knowledge-guided spatiotemporal alignment module and the local features of other views are linearly projected and mapped into a set of embedding vector sequences, which are then input into an encoder with a 12-layer Transformer structure. To alleviate feature redundancy generated by ViT when processing long sequences of multi-view images, this embodiment introduces three additional learnable redundancy-cleaning embedding tokens (Register Tokens) at the end of the first-layer input sequence, denoted as... The initial input representation of the first layer of ViT can then be expressed as: ,in, Category tags used to extract global information, Position encoding is performed. After alternating multi-head self-attention and multi-layer perceptron operations by a 12-layer ViT encoder, the image features from each viewpoint undergo deep evolution.

[0033] In some embodiments, the multi-view feature fusion module is used to model the relationships between images from different body positions: from front view features get and Other perspective features get Cross-attention calculation is performed to enhance semantic alignment across multiple perspectives. The final attention score is: ; in Q= , K= , V= , , , A learnable weight vector; S2.1.3 Input the fused multimodal spatiotemporal features into the decoder to generate a radiology medical text report.

[0034] After obtaining the global image representation enhanced by multi-view collaboration and the text features enhanced with medical knowledge, this embodiment utilizes a cross-attention mechanism to achieve precise guidance and injection of medical expertise into the visual features. The final attention score is: ; in Q= , K= , V= , , , The weight vectors are learnable. Through the attention weight map generated by the Softmax function, the model can dynamically adjust its attention to different spatial locations and viewpoints in the image based on the semantic logic extracted from the external medical knowledge graph. Finally, the output of the attention mechanism is normalized by layers and connected to residuals to obtain a multimodal spatiotemporal feature fusion representation guided by external knowledge. This fusion representation not only includes longitudinal evolution information of current and historical images and spatial complementary information from multiple viewpoints, but also deeply integrates explicit medical priors provided by the RadGraph knowledge graph.

[0035] The decoder takes the externally knowledge-guided multimodal spatiotemporal feature fusion representation u as input and adopts a structure based on cross-modal memory enhancement and state preservation mechanisms, employing a dual mechanism of cross-modal memory enhancement and memory preservation. First, a learnable global memory matrix is ​​constructed, designed to simulate the clinical experience of radiologists, storing the association representations between common image patterns and standard diagnostic descriptions. At each time step of report generation, the decoder uses the current multimodal spatiotemporal fusion representation as the query matrix Q, performing similarity matching retrieval with the memory matrix to extract the most relevant prior terminology. Subsequently, a gated recurrent unit (GRU) is used to dynamically filter and retain historical context information during the decoding process. Furthermore, this invention introduces a knowledge-guided attention bias mechanism, injecting hierarchical semantic priors from the medical ontology into the Softmax layer. By deeply coupling "current image differences," "historical evolution information," and "memory matrix priors," the model effectively overcomes the problem of traditional models generating reports with monotonous content, significantly improving the consistency, professionalism, and accuracy of terminology descriptions of complex lesions.

[0036] S3. Update the report generation model weights using the loss function to obtain the trained report generation model.

[0037] The text generator is subjected to supervised learning by minimizing the cross-entropy loss function between radiology reports generated by the text generator and those written by real radiologists. The loss function can be expressed as follows: The cross-entropy loss function is used to maximize the conditional log-likelihood. The generator's loss function is as follows: ; in, Indicates the length of the generated report; The model attempts to predict the first... One word; This indicates the previously entered text. This represents the learnable parameters of the model.

[0038] Table 1 shows a performance comparison of the current state-of-the-art methods on the MIMIC-CXR-JPG dataset, and Table 2 shows a performance comparison of the current state-of-the-art methods on the IU X-Ray dataset. Table 1. Performance comparison with current state-of-the-art methods on the MIMIC-CXR-JPG dataset. Comparison Methods BLEU-1 BLEU-2 BLEU-3 BLEU-4 METEOR ROUGE-L CheXbert RadGraph F1 R2Gen 0.353 0.218 0.145 0.103 0.142 0.277 0.331 0.176 CMN 0.353 0.218 0.148 0.106 0.142 0.278 0.342 0.174 CMM 0.381 0.232 0.155 0.109 0.151 0.287 - - GSK 0.363 0.228 0.156 0.115 - 0.284 - - Clinical-BERT 0.383 0.23 0.151 0.106 0.144 0.275 - - DCL - - - 0.109 0.15 0.284 - - METransformer 0.386 0.25 0.169 0.124 0.152 0.291 - - R2GenGPT 0.409 0.261 0.179 0.128 0.16 0.287 0.381 0.212 RECAP 0.429 0.267 0.177 0.125 0.168 0.288 - - HERGen 0.395 0.248 0.169 0.122 0.156 0.285 - - Mei et al. 0.436 0.275 0.184 0.129 0.177 0.305 - - RAMT 0.362 0.229 0.157 0.113 0.153 0.284 - - FMVP 0.389 0.236 0.156 0.108 0.15 0.284 - - PromptMRG 0.398 - - 0.112 0.157 0.268 0.439 0.177 MA 0.396 0.244 0.162 0.115 0.151 0.274 - - STREAM 0.437 0.278 0.192 0.139 0.172 0.297 0.405 0.223 This invention 0.447 0.283 0.199 0.145 0.179 0.313 0.422 0.254 Table 2 Performance comparison with current state-of-the-art methods on the IU X-Ray dataset Comparison Methods BLEU-1 BLEU-2 BLEU-3 BLEU-4 METEOR ROUGE-L CheXbert RadGraph F1 R2Gen 0.47 0.304 0.219 0.165 0.187 0.371 0.53 0.318 CMN 0.475 0.309 0.222 0.17 0.191 0.375 0.552 0.295 CMM 0.494 0.321 0.235 0.181 0.201 0.384 - - GSK 0.496 0.327 0.238 0.178 - 0.381 - - Clinical-BERT 0.495 0.330 0.231 0.17 0.209 0.376 - - DCL - - - 0.163 0.193 0.383 - - METransformer 0.483 0.332 0.228 0.172 0.192 0.380 - - R2GenGPT 0.491 0.323 0.234 0.18 0.211 0.376 0.601 0.321 RAMT 0.482 0.31 0.221 0.165 0.195 0.377 - - FMVP 0.485 0.315 0.225 0.169 0.201 0.398 - - PromptMRG 0.401 - - 0.098 0.160 0.281 0.556 0.249 MA 0.501 0.328 0.23 0.17 0.213 0.386 - - STREAM 0.506 0.338 0.248 0.188 0.215 0.387 0.615 0.357 This invention 0.512 0.344 0.255 0.194 0.222 0.395 0.632 0.373 The radiology report generation method based on spatiotemporal feature fusion and medical knowledge proposed in this invention is compared with 16 state-of-the-art methods, including R2Gen, CMN, CMM, GSK, Clinical-BERT, DCL, METransformer, RAMT, FMVP, MA, R2GenGPT, recap, HERGen, Mei et al., PromptMRG, and STREAM. All of these methods employ an encoder-decoder architecture. For the image encoder, R2Gen, CMN, CMM, GSK, and PromptMRG use ResNet-101, while Clinical-BERT, RAMT, FMVP, Mei et al., and MA use DenseNet-121. DCL, METransformer, R2GenGPT, recap, HERGen, STREAM, and this invention all use VisionTransformer or its variants (such as SwingTransformer) as the visual encoder. Regarding the decoder, most contrastive methods use Transformer or its variants, while the streaming models of R2GenGPT and STREAM employ Large Language Models (LLMs) for decoding. For multi-view image analysis, FMVP, Mei et al., and STREAM use multi-view CXR images as input. Regarding historical information, recap, HERGen, Mei et al., and STREAM integrate temporal studies during the generation process.

[0039] Tables 1 and 2 present the performance comparison on the MIMIC-CXR-JPG and IU X-Ray datasets, respectively, with metrics including NLG, CheXbert vector similarity, and Radgraph F1. The best results are highlighted in bold. It can be seen that the present invention demonstrates superior performance on most metrics across both datasets. Experimental results on the IU X-Ray dataset show that the present invention achieves significant improvements across all metrics (shown in bold), far exceeding existing methods. Experimental results on the MIMIC-CXR-JPG dataset show that the present invention achieves the highest scores except for the CheXbert metric (shown in bold). This is because PromptMRG achieves optimal CheXbert vector similarity by adding a classification branch specifically designed to improve CE accuracy. The results on both datasets demonstrate that the method of the present invention achieves optimal performance on almost all major metrics, proving its significant advantages in improving report generation quality, accuracy, and medical consistency.

[0040] Example 2: This embodiment provides a radiology report generation system based on feature fusion and medical guidance, including: The data acquisition module is configured to acquire the patient's current medical image data, historical medical image data, and corresponding text report data. The feature extraction module is configured to extract features from medical image data and text report data respectively, to obtain medical knowledge-enhanced text features, and to obtain spatiotemporal image features that reflect lesion evolution information; The feature fusion module is configured to: inject the medical knowledge-enhanced text features into the image spatiotemporal features using a cross-attention mechanism to generate multimodal spatiotemporal fusion features; The report generation module is configured to generate radiology medical text reports based on multimodal spatiotemporal fusion features and a preset decoder.

[0041] The working method of the system is the same as that of the feature fusion and medical-guided radiology report generation method in Example 1, and will not be repeated here.

[0042] Example 3: This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the feature fusion and medical-guided radiology report generation method described in Embodiment 1.

[0043] Example 4: This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the program, it implements the steps of the feature fusion and medical-guided radiology report generation method described in Embodiment 1.

[0044] Example 5: This embodiment provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the radiology report generation method based on feature fusion and medical guidance described in Embodiment 1.

[0045] The above description is merely a preferred embodiment of this practice and is not intended to limit the scope of this practice. Various modifications and variations can be made to this practice by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this practice should be included within the protection scope of this practice.

Claims

1. A method for generating radiology reports based on feature fusion and medical guidance, characterized in that, include: Acquire the patient's current medical imaging data, historical medical imaging data, and corresponding text report data; Feature extraction was performed on medical image data and text report data respectively to obtain medical knowledge-enhanced text features and image spatiotemporal features reflecting lesion evolution information; By utilizing a cross-attention mechanism, the textual features enhanced with the medical knowledge are injected into the spatiotemporal features of the image, generating multimodal spatiotemporal fusion features; Based on multimodal spatiotemporal fusion features and a pre-defined decoder, a radiology medical text report is generated.

2. The method for generating radiology reports based on feature fusion and medical guidance as described in claim 1, characterized in that, The feature extraction of the text report data includes: A graph was constructed based on text report data and introduced external medical knowledge. and its adjacency matrix The adjacency matrix elements in Indicates the first The word and the first Dependencies between words; Use natural language processing tools to build graphs for the report. and adjacency matrix The node value is Integrate the medical knowledge subtrees into a complete syntactic-medical knowledge graph. : ； The value of each node The sum of corresponding elements of two adjacent matrices: = + +1； For the input text, the Bio-ClinicalBERT medical feature extraction model is used to extract basic semantic representations of the text. The encoded output of the text is: ； Encoded output As node weights in the constructed knowledge graph, features from neighboring nodes are then aggregated and node embeddings are updated; j For nodes i Attention coefficient Defined as: ； in, It is a learnable matrix. This indicates a concatenation operation. It is a feedforward neural network whose parameters are determined by the weight vector. Definition; final node weight Represented as: ； in, , Indicates the activation function; Representative node The adjacent nodes are identified; finally, the fused syntactic-medical knowledge graph is semantically encoded to obtain text features.

3. The radiology report generation method based on feature fusion and medical guidance as described in claim 1, characterized in that, The feature extraction of the current image data and the historical image data includes: medical image data are arranged in chronological order, and features of the current image and features of the same patient's historical images are extracted respectively. If there are no historical images, a trainable context vector is used to replace the historical features, and the vector is concatenated with the feature vector of the knowledge-guided spatiotemporal alignment module and input into the encoder; then the knowledge-guided spatiotemporal alignment module is used to perform lesion region matching and difference enhancement on the current features and historical features.

4. The radiology report generation method based on feature fusion and medical guidance as described in claim 3, characterized in that, The processing steps of the knowledge-guided spatiotemporal alignment module include: dividing the feature map after encoding the main-view image into multiple quadrants to enhance the prior understanding of the spatial distribution of lesions; and performing cross-attention calculation on the image encoding content of one quadrant of the current image and the text features to obtain the fused features of spatial and text features. Identify key semantic regions: ； in, Indicates the first Each quadrant; yes dimensionality Q= ，K= ，V= , , , A learnable weight vector; , , Each is the current number i Query vectors, key vectors, and value vectors for each image quadrant; The first result obtained after medical feature encoding of the text report i A contextual semantic representation vector of each text; The calculated results are concatenated with the corresponding quadrant features from historical images, and then input into a convolutional block for dimensionality reshaping and layer normalization. Channel attention is used to highlight the response intensity of key feature channels, and a gating mechanism is used for dynamic fusion. Subsequently, the quadrant representations are concatenated again, and channel attention calculation, gating fusion, and a feedforward neural network are performed again to finally output the spatiotemporally aligned image representation. If there are K images from different perspectives, the other images are input in parallel to guide the spatiotemporal alignment module to obtain multi-view image representations. ,in, .

5. The radiology report generation method based on feature fusion and medical guidance as described in claim 4, characterized in that, The multi-view feature fusion module is used for relationship modeling of images from different body positions: from front view features get and Other perspective features get Cross-attention calculation is performed to enhance semantic alignment across multiple perspectives, and the final attention score is: ； in Q= , K= , V= , , , This is a learnable weight vector.

6. The radiology report generation method based on feature fusion and medical guidance as described in claim 3, characterized in that, Deep global context modeling is performed on multi-view images. Specifically, the front view features processed by the knowledge-guided spatiotemporal alignment module and the local features of other views are linearly projected and mapped into a set of embedding vector sequences, which are then input into an encoder with a 12-layer Transformer structure. Three additional learnable redundant sanitized embedding labels are introduced at the end of the first layer input sequence. After the 12-layer encoder alternately performs multi-head self-attention and multilayer perceptron operations, the image features of each view are deeply evolved.

7. The radiology report generation method based on feature fusion and medical guidance as described in claim 4, characterized in that, The text generator is subjected to supervised learning by minimizing the cross-entropy loss function between radiology reports generated by the text generator and those written by real radiologists. The loss function can be expressed as follows: : ； in, Indicates the length of the generated report; The model attempts to predict the first... One word; This indicates the previously entered text. This represents the learnable parameters of the model.

8. A radiology report generation system based on feature fusion and medical guidance, characterized in that, include: The data acquisition module is configured to acquire the patient's current medical image data, historical medical image data, and corresponding text report data. The feature extraction module is configured to extract features from medical image data and text report data respectively, to obtain medical knowledge-enhanced text features, and to obtain spatiotemporal image features that reflect lesion evolution information; The feature fusion module is configured to: inject the medical knowledge-enhanced text features into the image spatiotemporal features using a cross-attention mechanism to generate multimodal spatiotemporal fusion features; The report generation module is configured to generate radiology medical text reports based on multimodal spatiotemporal fusion features and a preset decoder.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the program, it implements the steps of the radiology report generation method based on feature fusion and medical guidance as described in any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the feature fusion and medical-guided radiology report generation method as described in any one of claims 1-7.

Images