A Method and System for Generating Medical Reports Based on Cross-Modal Interactive Memory Networks

Abstract

This invention proposes a medical report generation method and system based on a cross-modal interactive memory network, relating to image processing technology and natural language processing. It addresses the problems of traditional models, such as the high time and effort required to incorporate medical knowledge, the inability to effectively extract anomalous fine-grained textual knowledge, and the lack of accurate loss functions, leading to incomplete and inaccurate reports. The method includes: acquiring a training set consisting of medical images and reports of known healthy and diseased areas; inputting the training set into a cross-modal interactive memory network model for training; and acquiring the medical image for which the report is to be generated and inputting it into the trained cross-modal interactive memory network model to obtain the report generation result. This invention solves the problems of ineffective extraction of anomalous fine-grained textual knowledge, improving the accuracy of the model's report generation.

Description

Technical Field

[0001] This invention belongs to the fields of image processing technology and natural language processing, and particularly relates to a method and system for generating medical reports based on cross-modal interactive memory networks. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Radiological examinations, based on modalities such as X-rays, ultrasound (US), computed tomography (CT), and magnetic resonance imaging (MRI), provide a detailed understanding of a patient's body. As a type of medical examination, radiological examinations can help screen for and diagnose diseases, as well as monitor responses to treatment. Therefore, radiological examinations are the most common type of imaging examination.

[0004] However, the enormous daily diagnostic demands and the shortage of radiologists further exacerbate problems such as diagnostic errors and discrepancies in interpretation between radiologists and physicians. Imaging reports are the most important means of communication between radiologists and referring medical professionals, both committed to providing high-quality patient care. All radiologists need observational skills to identify abnormalities and analytical skills to correlate observed abnormalities with underlying pathology. The lack of these skills and the increasing number of radiological examinations underscore the need for automation in this field, hence the proposal of Radiology Report Generation (RRG). RRG refers to the use of deep learning technology by computers to automatically generate specialized medical terminology from radiological images. On the one hand, it can alleviate the workload of radiologists; on the other hand, it can alert physicians to abnormalities, reducing the occurrence of misdiagnosis and missed diagnosis.

[0005] The inventors discovered that while some recent studies have improved the performance of models generating radiology reports by incorporating medical or prior knowledge, this requires manually pre-built knowledge graphs or knowledge template libraries, which is time-consuming, labor-intensive, and difficult to transfer to other datasets. Furthermore, existing models cannot effectively handle data bias in textual data within datasets (abnormal reports constitute only a small portion of the dataset, and abnormal descriptions also constitute only a small portion of these reports), leading models to more readily learn normal descriptions while ignoring abnormal ones. This makes existing models challenging for clinical decision support. Moreover, previous methods primarily rely on traditional loss mechanisms to optimize the model, lacking robust supervision, resulting in incomplete (lacking descriptions of certain abnormalities from real reports) and inconsistent (contradictory to descriptions in real reports) reports. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, this invention provides a medical report generation method and system based on a cross-modal interactive memory network. By constructing a medical report generation model based on a cross-modal interactive memory network, the automated injection and utilization of medical knowledge is achieved, while simultaneously enhancing the extraction of anomalous fine-grained knowledge features and reducing the differences between features from different modalities.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0008] The first aspect of this invention discloses a method for generating medical reports based on a cross-modal interactive memory network, comprising:

[0009] Obtain a training set, which includes medical images of known healthy and diseased regions, and medical reports of known healthy and diseased regions;

[0010] The training set is input into the cross-modal interactive memory network model to train the model and obtain the trained cross-modal interactive memory network model.

[0011] The medical image to be generated is obtained, and the medical image is input into the trained cross-modal interactive memory network model to obtain the report generation result of the medical image.

[0012] The cross-modal interactive memory network model includes an abnormal medical knowledge enhancement module and a cross-modal interactive memory module.

[0013] During training, medical images and medical reports are input into a cross-modal interactive memory network model for processing to obtain global image features and initial medical text knowledge features;

[0014] The enhanced medical text knowledge features are extracted from the initial medical text knowledge features using the abnormal medical knowledge enhancement module.

[0015] The extracted enhanced medical text knowledge features are stored in a matrix through a cross-modal interactive memory module, and medical knowledge related to medical images in the matrix is ​​extracted based on global image features.

[0016] As a further technical solution, medical images and medical reports are input into a cross-modal interactive memory network model for processing. This model further includes a relation-enhancing feature encoder and a text feature encoder, specifically:

[0017] The medical image is input into the relation enhancement feature encoder to obtain relation-enhanced image features and global image features;

[0018] The medical report is input into the text feature encoder to obtain the initial medical text knowledge features.

[0019] As a further technical solution, the medical image is input into a relation enhancement feature encoder to obtain relation-enhanced image features and global image features. The relation enhancement feature encoder includes a convolutional neural network and an intrinsic relation encoder. The specific process is as follows:

[0020] Medical images are input into a convolutional neural network to extract visual region features and convert them into visual sequence labels;

[0021] Visual sequence labels are converted into visual sequence features through linear transformation;

[0022] Visual sequence features are combined with global CLS tags to form new visual sequence features, and the new visual sequence features are input into the intrinsic relation encoder;

[0023] The key-value pairs of multi-head self-attention in the encoding of intrinsic relations are obtained through the learnable memory matrix;

[0024] Based on multi-head attention key-value pairs, the features of the new visual sequence are encoded to obtain relation-enhanced image features and global image features.

[0025] As a further technical solution, the enhanced medical text knowledge features are extracted from the initial medical text knowledge features through the abnormal medical knowledge enhancement module. The specific process is as follows:

[0026] Obtain the disease topic vector and initialize the disease state vector;

[0027] By interacting disease topic vectors with initial medical text knowledge features, fine-grained features related to disease topics are obtained.

[0028] The disease state features are obtained by interacting with the initial disease state vector and the real disease state vector.

[0029] By fusing fine-grained features related to the disease topic with disease state features, a fused feature is obtained.

[0030] The fused features are combined with the initial medical text knowledge features to obtain the enhanced medical text knowledge features.

[0031] As a further technical solution, the extracted enhanced medical text knowledge features are stored in a matrix through a cross-modal interactive memory module, and medical knowledge related to the medical image is extracted from the matrix based on global image features. The specific process is as follows:

[0032] Initialize a dynamic medical matrix to store medical knowledge;

[0033] Interact with the matrix and enhanced medical text knowledge features to obtain new knowledge from the enhanced medical text knowledge features;

[0034] The newly acquired knowledge is processed using layer normalization and multi-layer perceptron mechanisms to obtain an updated knowledge matrix;

[0035] The updated knowledge matrix is ​​then subjected to a cross-attention operation with global image features to obtain medical knowledge related to the medical image.

[0036] As a further technical solution, after inputting the medical report into the text feature encoder, global report features are also obtained. The text feature encoder includes a SciBert encoder and a fully connected layer. The specific process is as follows:

[0037] The medical report is encoded using a SciBert encoder to obtain an aggregated representation of the medical report.

[0038] The aggregated representation of the medical report is input into the fully connected layer to obtain the global report features.

[0039] As a further technical solution, the model is optimized during the training process by constructing a joint loss function. The specific process of constructing the joint loss function is as follows:

[0040] The predicted values ​​are obtained by linear transformation of global image features, and multi-label binary cross-entropy is performed with the ground truth labels to construct a multimodal disease classification loss.

[0041] Construct a cross-modal global semantic loss based on global report features and global image features;

[0042] The joint loss is obtained by adding the cross-modal global semantic loss, the multimodal disease classification loss, and the report generation loss.

[0043] The second aspect discloses a medical report generation system based on a cross-modal interactive memory network, including:

[0044] A training data acquisition module is used to obtain medical images and medical reports of known healthy and diseased areas, including the training set;

[0045] A cross-modal interactive memory network model construction and training module is used to input the training set into the cross-modal interactive memory network model, train the model, and obtain the trained cross-modal interactive memory network model; wherein, the cross-modal interactive memory network model includes an abnormal medical knowledge enhancement module and a cross-modal interactive memory module;

[0046] During training, medical images and medical reports are input into a cross-modal interactive memory network model for processing to obtain global image features and initial medical text knowledge features;

[0047] The enhanced medical text knowledge features are extracted from the initial medical text knowledge features using the abnormal medical knowledge enhancement module.

[0048] The extracted enhanced medical text knowledge features are stored in a matrix through a cross-modal interactive memory module, and medical knowledge related to medical images is extracted from the matrix based on global image features.

[0049] The medical report generation module is used to acquire the medical image to be generated into the report, input the medical image into the trained cross-modal interactive memory network model, and obtain the report generation result of the medical image.

[0050] A third aspect of the present invention is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0051] A fourth aspect of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above-described method.

[0052] The above one or more technical solutions have the following beneficial effects:

[0053] In this embodiment, a medical report generation model is constructed, which includes a feature encoder, an abnormal medical knowledge enhancement module, a cross-modal interactive memory module, a report generation decoder, and a cross-modal joint semantic loss unit. This model solves the problems of traditional models, such as the high time and effort required to introduce medical knowledge, the inability to extract abnormal fine-grained text knowledge, and the lack of an accurate loss function, which easily leads to the generation of incomplete and inaccurate reports.

[0054] In this embodiment, the abnormal medical knowledge enhancement module alleviates the text data bias problem, namely, the problem that abnormal reports and abnormal descriptions only account for a small part of the dataset, which can help the model strengthen its attention to and learning of abnormal descriptions (fine-grained knowledge features); the cross-modal interactive memory module realizes the automatic introduction and utilization of medical text knowledge, improving the accuracy of the model's report generation; the designed cross-modal joint semantic loss unit improves the model's visual representation ability while promoting semantic consistency between image and text semantic features.

[0055] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0056] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0057] Figure 1 This is a flowchart of the medical report generation model based on a cross-modal interactive memory network in this embodiment 1;

[0058] Figure 2 This is a schematic diagram of the abnormal medical knowledge enhancement module in this embodiment. Detailed Implementation

[0059] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0060] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0061] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0062] The overall approach proposed in this invention is as follows: medical images and paired medical reports obtain image features and text features through a feature encoder; an abnormal medical knowledge enhancement module extracts fine-grained textual knowledge from the text features; a cross-modal interactive memory module stores the extracted coarse-grained and fine-grained medical textual knowledge features in a matrix and extracts relevant knowledge from the matrix; a report generation decoder receives the merged features of the input image features and knowledge features, and continues to predict the next word based on the input sequence. This predicted word is added to the input sequence to continue the above process until a complete report is generated; a cross-modal joint semantic loss unit, while retaining the text generation loss, incorporates multi-label disease classification loss and cross-modal global semantic loss during the training phase to optimize and supervise the model.

[0063] Example 1

[0064] This embodiment discloses a medical report generation method based on a cross-modal interactive memory network.

[0065] To more clearly illustrate this embodiment, the implementation process of medical report generation based on cross-modal interactive memory networks can be specifically described as follows:

[0066] A method for generating medical reports based on a cross-modal interactive memory network is provided, including:

[0067] S1. Obtain a training set, which includes medical images of known healthy and diseased areas, and medical reports of known healthy and diseased areas;

[0068] S2. Input the training set into the cross-modal interactive memory network model, train the model, and obtain the trained cross-modal interactive memory network model.

[0069] S3. Obtain the medical image for which the report is to be generated, input the medical image for which the report is to be generated into the trained cross-modal interactive memory network model, and obtain the report generation result of the medical image.

[0070] like Figure 1 As shown, in step S1, a training set is obtained, which includes medical images of known healthy areas and diseased areas, and medical reports of known healthy areas and diseased areas.

[0071] The datasets were selected from MIMIC-CXR and IU-Xray, and were divided into three subsets: training, testing, and validation sets. These subsets were used to train, test, and validate the cross-modal interactive memory network model. The training set included medical images and corresponding medical reports, while the testing and validation sets contained only medical images. The training set was obtained by providing medical images and reports of known healthy and diseased regions.

[0072] like Figure 1 As shown, in step S2, the training set is input into the cross-modal interactive memory network model to train the model and obtain the trained cross-modal interactive memory network model.

[0073] S201. Construct a cross-modal interactive memory network model.

[0074] The cross-modal interactive memory network model includes a relation-enhanced feature encoder, a text feature encoder, an abnormal medical knowledge enhancement module, a cross-modal interactive memory module, a report generation decoder, and a cross-modal joint semantic loss unit.

[0075] S202. Train the cross-modal interactive memory network model and extract its features.

[0076] S2021. Input the medical image into the relation enhancement feature encoder to obtain relation-enhanced image features and global image features.

[0077] Relation enhancement feature encoders include convolutional neural networks and intrinsic relation encoders.

[0078] The specific process for obtaining relation-enhanced image features and global image features is as follows:

[0079] (1) Input medical images into a convolutional neural network, extract visual region features and convert them into visual sequence labels.

[0080] like Figure 1As shown, in this embodiment, the radiographic images I∈R in the dataset are... H×W×3 The input is fed into a pre-trained ResNet-101 network. Visual region features are extracted in the last convolutional layer of the ResNet-101 network and converted into visual sequence labels. Where N I This represents 49 visual region features, and the dimension of C is 2048.

[0081] (2) Visual sequence labels are converted into visual sequence features through linear transformation;

[0082] Then, a linear transformation is used to map the visual sequence labels to visual sequence features X. P The formula is:

[0083]

[0084] Among them, f img (·) represents ResNet, W g A learnable parameter matrix, W g ∈R C×E E represents the visual feature dimension, which is 512.

[0085] An intrinsic relation encoder with a 3-layer standard Transformer encoder structure was constructed to obtain relation-enhanced visual features. Each encoder layer uses a multi-head self-attention mechanism, as shown in the following formula:

[0086]

[0087] Q i =XW i q K i =YW i k V i =YW i v (3)

[0088] MHA(X,Y)=Concat(Head1(X,Y),…Head i (X,Y),…Head H (X,Y))W h (4)

[0089] Where X is the query matrix, Y is the key / value matrix, and W... i q W i k W i v W hFor learnable parameters; Concat(·) represents a join operation; e n =E / H; H is the number of attention heads, Q i K represents the query matrix of the i-th attention head in multi-head attention. i and V i Let represent the key matrix and value matrix of the i-th attention head in a multi-head self-attention system, respectively.

[0090] (3) Combine the visual sequence features with the global CLS label to form a new visual sequence feature, and input the new visual sequence feature into the intrinsic relation encoder. Through the learnable memory matrix, the key-value pairs of multi-head self-attention in the intrinsic relation encoding are obtained.

[0091] like Figure 1 As shown, in this embodiment, a global CLS marker and a visual sequence marker X are introduced. P Together they form a new sequence As input to the intrinsic relationship encoder, the key-value pairs in the multi-head self-attention module of the intrinsic relationship encoder are expanded by adding a learnable memory matrix, which helps to obtain the intrinsic relationships between image regions. The formula is as follows:

[0092]

[0093] in and V i ^ It is the expanded key-value pair of the i-th header, and [;] is the join operation Mk. i and Mv i It is a learning matrix containing M rows.

[0094] (3) Encode the new visual sequence features based on the multi-head attention key-value pairs to obtain relation-enhanced image features and global image features.

[0095] An intrinsic relational encoder layer f e The process of (·) is as follows:

[0096] Attz = AddNorm(MHA(z,z))(6)

[0097] f e (z)=AddNorm(FFN(Attz))(7)

[0098] Where z represents the input of each encoder layer, AddNorm consists of layer normalization and residual connections, and FFN is a feedforward network layer.

[0099] CLS markers and visual sequence features X p Image features enhanced with relational enhancement are obtained after passing through the intrinsic relational encoder. and global image features Gimg ∈R E .

[0100] Through the above steps, the image features of the medical images were accurately extracted, providing data support for the subsequent generation of medical reports and improving the accuracy of the model.

[0101] (4) The global image features are transformed linearly to obtain the predicted values, and multi-label binary cross-entropy is performed with the ground real labels to construct a multimodal disease classification loss.

[0102] In this embodiment, to improve the model's visual representation capability, we perform multi-label disease classification prediction. We use the CheXpert model to extract the ground truth labels from the corresponding reports. We then use the global image features G... img The predicted value is obtained through linear transformation. With ground real labels The multi-label binary cross-entropy loss is calculated using the following formula:

[0103] Y ^ =G img W V +b V (8)

[0104]

[0105] Among them W V b is a learnable parameter matrix V For bias; σ(·) is the sigmoid function; N d For the number of disease topics, y i ∈{0,1} is the true label of the i-th disease. It is the predicted value for the i-th disease.

[0106] Through the above steps, multi-label disease classification prediction is achieved, improving the model's visual representation capabilities.

[0107] S2022. Input the medical report into the text feature encoder to obtain the initial medical text knowledge features.

[0108] The text feature encoder consists of a SciBert encoder and a fully connected layer.

[0109] like Figure 1 As shown, in this embodiment, the SciBert architecture is used to process the real report. Medical knowledge feature extraction is performed, where N R The length of a report is indicated as follows:

[0110] H = TextEncoder(T)(10)

[0111] Where T represents the ground truth report of the corresponding input image, and TextEncoder(·) represents the text encoder. This represents the text features obtained after the text encoder, i.e., the initial medical text knowledge features.

[0112] In addition, the aggregated representation T of the medical report is obtained through the SciBert architecture. CLS Then, it is passed through a fully connected layer to obtain the global reporting feature G. text The formula is:

[0113] G text = T CLS W G +b G (11)

[0114] Among them, G text ∈R E W G b is a learnable parameter matrix G For bias.

[0115] Through the above steps, medical knowledge related to the medical report was accurately extracted, providing data support for subsequent report generation and improving the model's accuracy.

[0116] To better align global semantic information across modalities, a multimodal contrastive loss is employed to learn the latent semantic features between images and text.

[0117] Specifically, we will use the target global image features G img The corresponding global reporting feature G text The features that are paired are called positive sample pairs, and the remaining unpaired features are called negative sample pairs. Through training, the distance between positive sample pairs is reduced, and the distance between negative sample pairs is increased. To maximize the alignment of semantic information between paired features, we perform contrastive learning using the image and its paired report as target points, obtaining a cross-modal global semantic loss L. S The formula is:

[0118]

[0119] L S =L img-text +L text-img (14)

[0120] Where N represents the training batch size, S<,> represents the cosine similarity function, τ represents the learnable temperature coefficient, and G img G represents global image features. text This indicates the characteristics of a global report.

[0121] By following the steps described above, we can better align global semantic information across modalities.

[0122] S2023. Extract enhanced medical text knowledge features from the initial medical text knowledge features through the abnormal medical knowledge enhancement module.

[0123] Significant data bias (abnormal case reports constitute only a small fraction, and descriptions of the abnormal regions are minimal) causes the model to easily overlook fine-grained features of these regions. Since crucial clinical information primarily derives from the abnormal parts, we propose an abnormal medical knowledge enhancement module.

[0124] (1) Obtain the disease topic vector and initialize the disease state vector.

[0125] like Figure 2 As shown, in this embodiment, 14 disease topics and their corresponding disease topic states are introduced, such as pulmonary edema, pneumonia, and other disease-related topics and their various disease topic states: positive, negative, uncertain, and not mentioned, etc., and a disease topic vector is introduced. Where N d Let E represent the number of disease topics, and E represent the feature dimension of each disease topic.

[0126] Initialize disease state vector Where N k The number of disease states.

[0127] (2) Interact with the disease topic vectors and the initial medical text knowledge features to obtain fine-grained features related to the disease topic.

[0128] like Figure 2 As shown, in this embodiment, T topic Initial medical text knowledge features obtained from medical reports through the SciBert architecture Interact to obtain fine-grained features related to the disease topic.

[0129] (3) Interact with the initial disease state vector and the real disease state vector to obtain disease state features.

[0130] like Figure 2 As shown, in this embodiment, the disease state vector T will be initialized. states Compared with the true disease state vector Interact with the system to obtain rich disease state features.

[0131] In this embodiment, the abnormal medical knowledge enhancement module is only used during the training phase. Realistic disease states help our model describe diseases and related symptoms based on accurate information (teacher coercion mechanism).

[0132] (4) The fine-grained features related to the disease theme are fused with the disease state features to obtain the fused features.

[0133] In this embodiment, fine-grained features related to the disease topic are... (Integrating disease-themed features) with rich disease state features The fusion feature is obtained by adding (whether or not one suffers from a certain disease).

[0134] (5) The fusion features are combined with the initial medical text knowledge features to obtain the enhanced medical text knowledge features.

[0135] In this embodiment, the fused features are combined with the initial medical text knowledge features H (i.e., the disease details contained in the actual report) to obtain the enhanced medical text knowledge features.

[0136] The above steps can alleviate the text data bias problem and help the model strengthen its attention to and learning of anomaly descriptions (fine-grained knowledge features).

[0137] S2024. The extracted enhanced medical text knowledge features are stored in a matrix through the cross-modal interactive memory module, and medical knowledge related to medical images in the matrix is ​​extracted based on global image features.

[0138] Previous studies have primarily introduced medical or prior knowledge by pre-constructing knowledge graphs or knowledge template libraries. These methods are time-consuming and labor-intensive, and difficult to transfer to other datasets. Since constructing knowledge graphs and template libraries is time-consuming and labor-intensive, we aim to achieve the automatic storage and utilization of medical text knowledge features. Therefore, we propose a method for dynamically storing medical knowledge.

[0139] The specific process for extracting medical knowledge related to medical images from the matrix is ​​as follows:

[0140] (1) Initialize the dynamic medical matrix to store medical knowledge.

[0141] like Figure 1 As shown in this embodiment, during the training phase, we randomly initialize a dynamic medical knowledge matrix. To store medical knowledge, among which K b E represents the number of rows in the matrix, i.e., how much knowledge it can store, and E is the feature dimension of each row.

[0142] (2) The matrix is ​​interacted with the enhanced medical text knowledge features to obtain new knowledge from the enhanced medical text knowledge features. The new knowledge is processed by layer normalization and multi-layer perception mechanism to obtain the updated knowledge matrix.

[0143] like Figure 1 As shown, in this embodiment, matrix B interacts with enhanced medical text knowledge features (features containing fine-grained medical knowledge) H^ to update the medical knowledge in the matrix.

[0144] Specifically, when the knowledge matrix at step l is B l New knowledge is obtained from the enhanced medical text knowledge features of the current input report, using the following formula:

[0145] B′ l =MHA(B l ,H^)(15)

[0146] B l+1 =B l +MLP(LN(B′ l ))(16)

[0147] Among them, B′ l It is new knowledge obtained from the enhanced medical knowledge text features of the current input report. MLP stands for Multilayer Perceptron, LN stands for Layer Normalization, and B... l+1 This is the updated knowledge matrix.

[0148] (3) Perform cross-attention operation between the updated knowledge matrix and global image features to obtain medical knowledge related to medical images.

[0149] The updated knowledge matrix B l+1 With global image features G img Perform cross-attention operations to obtain medical knowledge related to the current medical image. The formula is:

[0150]

[0151] in, This represents medical knowledge related to the current medical image. CA(,) represents the cross-attention operation.

[0152] S2025. Combine the medical knowledge and relationship-enhanced image features related to the medical image to obtain the image-knowledge merging feature.

[0153] Medical knowledge related to medical images Image features X with enhanced relationship V Parallel operations are performed to obtain image-knowledge merging features V. merge The formula is:

[0154]

[0155] in,

[0156] During the testing phase, the matrix remains unchanged and image features are used to generate reports using the knowledge stored in the knowledge matrix.

[0157] Through the above steps, the introduction and utilization of automated medical text knowledge were realized, improving the accuracy of model-generated reports.

[0158] S2026. The report generation decoder receives the incoming image-knowledge merging features and predicts the next word based on the input sequence. The predicted word is added to the input sequence, and the process continues to predict the next word until a complete medical report is generated.

[0159] The report generation process is an autoregressive process, where each word is predicted based on a previously generated sequence of words. Specifically, the decoder uses the input sequence and image-knowledge fusion features V. merge The system then predicts the next word, adds this predicted word to the input sequence, and continues predicting the next word. Our model follows this process until a complete report is generated. The formula for predicting probability is:

[0160]

[0161] in, V is the word generated at time t. merge Represents image-knowledge merging features.

[0162]

[0163] During the training phase, we will provide a full ground-based real-world report. The word sequence is input into the decoder. Then, an upper triangular masking operation is performed on the input sequence and the upper triangular matrix; that is, each word is predicted based on the word sequence to its left. The model is optimized by minimizing the cross-entropy loss.

[0164]

[0165] Where, N R Y indicates the length of the report. t This represents the t-th word generated.

[0166] In the cross-modal joint semantic loss unit, a joint loss is constructed, which is the sum of all loss components. Multimodal disease classification loss and cross-modal global semantic loss are added to improve the model's visual representation ability and enhance the global semantic alignment between modalities, respectively.

[0167] Specifically, in order to balance the relationship between losses, the loss L obtained through equations (9) and (14) will be... V and L S Combined with the balance coefficients β1 and β2 respectively, and then with the reported generation loss L G Adding them together, we get the joint loss L. loss The formula is:

[0168] L loss =L G +β1L V +β2L S (twenty one)

[0169] The above steps improve the model's visual representation capabilities while promoting semantic consistency between image and text semantic features.

[0170] Following the steps outlined above, the dataset is divided into three subsets: a training set, a test set, and a validation set. The cross-modal interactive memory network model is then trained, tested, and validated. The training set includes medical images and corresponding medical reports, while the test and validation sets contain only medical images. Furthermore, the model's effectiveness is verified by comparing it with a series of baseline models.

[0171] The model presented in this embodiment was compared with other baseline models on the MIMIC-CXR and IU-Xray datasets, and the experimental results showed a significant improvement. It addresses the shortcomings of traditional models, such as the significant time and effort required to incorporate medical knowledge, the inability to effectively extract anomalous fine-grained textual knowledge, and the lack of a precise loss function, which often results in incomplete and inaccurate reports.

[0172] In this embodiment, in step S3, the medical image to be generated is obtained, and the medical image to be generated is input into the trained cross-modal interactive memory network model to obtain the report generation result of the medical image.

[0173] In practical applications, after obtaining medical images of a patient's lungs through X-ray scanning, these images can be input into a cross-modal interactive memory network model. The feature encoder automatically identifies potential lesions such as pulmonary nodules, pneumonia, and pneumothorax in the medical images and extracts image features associated with the lesion areas and global image features. Based on the image features, the model finds medical knowledge related to the current lung medical image from the knowledge matrix. The extracted image features and valuable medical knowledge are then input into the model's report generation decoder to generate a complete medical report of the current medical image for doctors' reference. This reduces the burden on doctors and helps them discover details that are difficult to detect with the naked eye, improving the accuracy and efficiency of diagnosis.

[0174] Example 2

[0175] The purpose of this embodiment is to provide a medical report generation system based on a cross-modal interactive memory network, including:

[0176] A training data acquisition module is used to obtain medical images and medical reports of known healthy and diseased areas, including the training set;

[0177] A cross-modal interactive memory network model construction and training module is used to input the training set into the cross-modal interactive memory network model, train the model, and obtain the trained cross-modal interactive memory network model; wherein, the cross-modal interactive memory network model includes an abnormal medical knowledge enhancement module and a cross-modal interactive memory module;

[0178] During training, medical images and medical reports are input into a cross-modal interactive memory network model for processing to obtain global image features and initial medical text knowledge features;

[0179] The enhanced medical text knowledge features are extracted from the initial medical text knowledge features using the abnormal medical knowledge enhancement module.

[0180] The extracted enhanced medical text knowledge features are stored in a matrix through a cross-modal interactive memory module, and medical knowledge related to medical images is extracted from the matrix based on global image features.

[0181] The medical report generation module is used to acquire the medical image to be generated into the report, input the medical image into the trained cross-modal interactive memory network model, and obtain the report generation result of the medical image.

[0182] This invention provides a medical report generation system based on a cross-modal interactive memory network, implementing the method steps in Embodiment 1.

[0183] Example 3

[0184] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0185] Example 4

[0186] The purpose of this embodiment is to provide a computer-readable storage medium.

[0187] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above method.

[0188] The steps and methods involved in the apparatus of the above embodiments correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0189] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0190] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for generating medical reports based on cross-modal interactive memory networks, characterized in that, include: Obtain a training set, which includes medical images of known healthy and diseased regions, and medical reports of known healthy and diseased regions; The training set is input into the cross-modal interactive memory network model to train the model and obtain the trained cross-modal interactive memory network model. The medical image to be generated is obtained, and the medical image is input into the trained cross-modal interactive memory network model to obtain the report generation result of the medical image. The cross-modal interactive memory network model includes a relation enhancement feature encoder, a text feature encoder, an abnormal medical knowledge enhancement module, a cross-modal interactive memory module, a report generation decoder, and a cross-modal joint semantic loss unit. The specific process for training the cross-modal interactive memory network model is as follows: The medical image is input into the relation enhancement feature encoder to obtain relation-enhanced image features and global image features; The medical report is input into the text feature encoder to obtain the initial medical text knowledge features; The enhanced medical text knowledge features are extracted from the initial medical text knowledge features using the abnormal medical knowledge enhancement module. The extracted enhanced medical text knowledge features are stored in a matrix through a cross-modal interactive memory module, and medical knowledge related to medical images is extracted from the matrix based on global image features. Image-knowledge merging features are obtained by concatenating medical knowledge and relationship-enhanced image features associated with medical images. The report generation decoder receives the incoming image-knowledge merging features and predicts the next word based on the input sequence. The predicted word is then added to the input sequence, and the process continues to predict the next word until a complete medical report is generated.

2. The medical report generation method based on a cross-modal interactive memory network as described in claim 1, characterized in that, The medical image is input into the relation enhancement feature encoder to obtain relation-enhanced image features and global image features. The relation enhancement feature encoder includes a convolutional neural network and an intrinsic relation encoder. The specific process is as follows: Medical images are input into a convolutional neural network to extract visual region features and convert them into visual sequence labels; Visual sequence labels are converted into visual sequence features through linear transformation; Visual sequence features are combined with global CLS tags to form new visual sequence features, and the new visual sequence features are input into the intrinsic relation encoder; The key-value pairs of multi-head self-attention in the intrinsic relation encoder are obtained through the learnable memory matrix; Based on the key-value pairs of multi-head self-attention, the features of the new visual sequence are encoded to obtain relation-enhanced image features and global image features.

3. The medical report generation method based on a cross-modal interactive memory network as described in claim 1, characterized in that, The enhanced medical text knowledge features are extracted from the initial medical text knowledge features using the abnormal medical knowledge enhancement module. The specific process is as follows: Obtain the disease topic vector and initialize the disease state vector; By interacting disease topic vectors with initial medical text knowledge features, fine-grained features related to disease topics are obtained. The disease state features are obtained by interacting with the initial disease state vector and the real disease state vector. By fusing fine-grained features related to the disease topic with disease state features, a fused feature is obtained. The fused features are combined with the initial medical text knowledge features to obtain the enhanced medical text knowledge features.

4. The medical report generation method based on a cross-modal interactive memory network as described in claim 1, characterized in that, The extracted enhanced medical text knowledge features are stored in a matrix through a cross-modal interactive memory module, and medical knowledge related to the medical images is extracted from the matrix based on global image features. The specific process is as follows: Initialize a dynamic medical matrix to store medical knowledge; Interacting the matrix with enhanced medical text knowledge features to obtain new knowledge from the enhanced medical text knowledge features; The newly acquired knowledge is processed using layer normalization and multi-layer perceptron mechanisms to obtain an updated knowledge matrix; The updated knowledge matrix is ​​then subjected to a cross-attention operation with global image features to obtain medical knowledge related to the medical image.

5. The medical report generation method based on a cross-modal interactive memory network as described in claim 1, characterized in that, After inputting the medical report into the text feature encoder, global report features are obtained. The text feature encoder includes a SciBert encoder and a fully connected layer. The specific process is as follows: The medical report is encoded using a SciBert encoder to obtain an aggregated representation of the medical report. The aggregated representation of the medical report is input into the fully connected layer to obtain the global report features.

6. The medical report generation method based on a cross-modal interactive memory network as described in claim 5, characterized in that, During training, a joint loss function is constructed using a cross-modal joint semantic loss unit to optimize the model. The specific process for constructing the joint loss function is as follows: Construct a cross-modal global semantic loss based on global report features and global image features; The predicted values ​​are obtained by linear transformation of global image features, and multi-label binary cross-entropy is performed with the ground truth labels to construct a multimodal disease classification loss. The joint loss is obtained by adding the cross-modal global semantic loss, the multimodal disease classification loss, and the report generation loss.

7. A medical report generation system based on a cross-modal interactive memory network, characterized in that, include: The training data acquisition module is used to acquire a training set, which includes medical images of known healthy areas and diseased areas, and medical reports of known healthy areas and diseased areas. A cross-modal interaction memory network model construction and training module is used to input the training set into the cross-modal interaction memory network model for training, resulting in the trained cross-modal interaction memory network model. The cross-modal interaction memory network model includes a relation enhancement feature encoder, a text feature encoder, an abnormal medical knowledge enhancement module, a cross-modal interaction memory module, a report generation decoder, and a cross-modal joint semantic loss unit. The specific process for training the cross-modal interaction memory network model is as follows: The medical image is input into the relation enhancement feature encoder to obtain relation-enhanced image features and global image features; The medical report is input into the text feature encoder to obtain the initial medical text knowledge features; The enhanced medical text knowledge features are extracted from the initial medical text knowledge features using the abnormal medical knowledge enhancement module. The extracted enhanced medical text knowledge features are stored in a matrix through a cross-modal interactive memory module, and medical knowledge related to medical images is extracted from the matrix based on global image features. Image-knowledge merging features are obtained by concatenating medical knowledge and relationship-enhanced image features associated with medical images. The report generation decoder receives the incoming image-knowledge merging features and predicts the next word based on the input sequence. The predicted word is added to the input sequence, and the process continues to predict the next word until a complete medical report is generated. The medical report generation module is used to acquire the medical image to be generated into the report, input the medical image into the trained cross-modal interactive memory network model, and obtain the report generation result of the medical image.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it performs the steps of the method described in any one of claims 1-6 above.

Images