Medical report generation method and system based on cross-modal interactive memory network

Abstract

The invention provides a medical report generation method and system based on a cross-modal interactive memory network, and relates to the field of image processing technologies and natural language processing. In order to solve the problems that when medical knowledge is introduced into a traditional model, a large amount of time and energy are consumed, abnormal fine-grained text knowledge cannot be better extracted, an accurate loss function is lacked, and an incomplete and inaccurate report is easily generated, the method comprises the steps of obtaining a training set, the training set is medical images and medical reports of a known healthy area and a lesion area; inputting the training set into a cross-modal interactive memory network model, and training the model to obtain a trained cross-modal interactive memory network model; and acquiring a medical image of a report to be generated, and inputting the medical image into the trained cross-modal interactive memory network model to obtain a report generation result of the medical image. According to the method, the problem that abnormal fine-grained text knowledge cannot be better extracted is solved, and the precision of a model generation report is improved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology and natural language processing, and in particular to a medical report generation method and system based on a cross-modal interactive memory network. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Radiology examinations based on modalities such as X-ray, ultrasound (US), computed tomography (CT), and magnetic resonance imaging (MRI) provide a detailed understanding of the patient's body. As a type of medical examination, radiology examinations can help screen and diagnose diseases, as well as monitor the response to treatment. Therefore, radiology examinations are the most common type of imaging examinations.

[0004] However, the huge daily diagnostic demand and the shortage of radiologists have further exacerbated the problems of diagnostic errors and differences in interpretation between radiologists and physicians. Imaging reports are the most important means of communication between radiologists and referring medical professionals, and both parties are committed to providing high-quality patient care. All radiologists need to have some observation skills to identify abnormalities and analytical skills to link observed abnormalities to underlying pathology. The lack of these skills and the increasing number of radiological examinations emphasize the need for automation in this field, so the Radiology Report Generation task (RRG) was proposed. The Radiology Report Generation task refers to the automatic generation of professional medical terms based on radiological images by computers using deep learning technology. On the one hand, it can reduce the burden on radiologists, and on the other hand, it can alert doctors to abnormalities and reduce the occurrence of misdiagnosis and missed diagnosis.

[0005] The inventors have found that in recent years, some existing studies have improved the performance of the model in generating radiology reports by introducing medical knowledge or prior knowledge, but medical knowledge needs to be introduced through artificially pre-built knowledge graphs or knowledge template libraries, which consumes a lot of time and effort and is difficult to migrate to other data sets. In addition, the existing models cannot handle the data bias problem in the text data in the data set well (abnormal reports only account for a small part of the data set, and abnormal descriptions only account for a small part of the abnormal reports), which makes it easier for the model to learn normal descriptions and ignore abnormal descriptions, which makes the existing models challenging as clinical decision support. In addition, previous methods mainly rely on traditional losses to optimize the model, lack of strong supervision methods, resulting in incomplete (lack of descriptions of certain abnormal conditions in real reports) and inconsistency problems in the generated reports (contradictory to the conditions described in the real reports). Summary of the invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a medical report generation method and system based on a cross-modal interactive memory network. By constructing a medical report generation model based on a cross-modal interactive memory network, the injection and utilization of automated medical knowledge is realized, while the extraction of abnormal fine-grained knowledge features is strengthened and the differences between different modal features are reduced.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0008] The first aspect of the present invention discloses a method for generating a medical report based on a cross-modal interactive memory network, comprising:

[0009] Acquire a training set, wherein the training set includes medical images of known healthy areas and lesion areas, and medical reports of known healthy areas and lesion areas;

[0010] Inputting the training set into the cross-modal interactive memory network model, training the model, and obtaining a trained cross-modal interactive memory network model;

[0011] Acquire a medical image for which a report is to be generated, input the medical image for which a report is to be generated into a trained cross-modal interactive memory network model, and obtain a report generation result of the medical image;

[0012] Wherein, the cross-modal interactive memory network model includes an abnormal medical knowledge enhancement module and a cross-modal interactive memory module;

[0013] During training, medical images and medical reports are input into the cross-modal interactive memory network model for processing to obtain global image features and initial medical text knowledge features;

[0014] Extracting enhanced medical text knowledge features from initial medical text knowledge features through an abnormal medical knowledge enhancement module;

[0015] The extracted enhanced medical text knowledge features are stored in a matrix through a cross-modal interactive memory module, and medical knowledge related to medical images in the matrix is ​​extracted based on global image features.

[0016] As a further technical solution, the medical images and medical reports are input into a cross-modal interactive memory network model for processing, wherein the cross-modal interactive memory network model also includes a relation enhancement feature encoder and a text feature encoder, specifically:

[0017] Inputting the medical image into a relation-enhanced feature encoder to obtain relation-enhanced image features and global image features;

[0018] The medical report is input into the text feature encoder to obtain the initial medical text knowledge features.

[0019] As a further technical solution, the medical image is input into a relation-enhanced feature encoder to obtain relation-enhanced image features and global image features, wherein the relation-enhanced feature encoder includes a convolutional neural network and an intrinsic relation encoder, and the specific process is as follows:

[0020] Input the medical image into the convolutional neural network, extract the visual area features and convert them into visual sequence tags;

[0021] Convert visual sequence tags into visual sequence features through linear transformation;

[0022] The visual sequence features and the global CLS tag are combined into a new visual sequence feature, and the new visual sequence feature is input into the intrinsic relation encoder;

[0023] Through the learnable memory matrix, the key-value pairs of multi-head self-attention in the intrinsic relationship encoding are obtained;

[0024] According to the multi-head attention key-value pairs, the new visual sequence features are encoded to obtain relation-enhanced image features and global image features.

[0025] As a further technical solution, the enhanced medical text knowledge features in the initial medical text knowledge features are extracted through the abnormal medical knowledge enhancement module. The specific process is as follows:

[0026] Get the disease topic vector and initialize the disease state vector;

[0027] Interact the disease topic vector with the initial medical text knowledge features to obtain fine-grained features related to the disease topic;

[0028] Interact the initialized disease state vector with the true disease state vector to obtain the disease state feature;

[0029] Fusing fine-grained features related to the disease topic with disease state features to obtain fused features;

[0030] The fused features are combined with the initial medical text knowledge features to obtain enhanced medical text knowledge features.

[0031] As a further technical solution, the extracted enhanced medical text knowledge features are stored in a matrix through a cross-modal interactive memory module, and the medical knowledge related to the medical image in the matrix is ​​extracted according to the global image features. The specific process is as follows:

[0032] Initialize the dynamic medical matrix to store medical knowledge;

[0033] Interact the matrix with the enhanced medical text knowledge features to obtain new knowledge from the enhanced medical text knowledge features;

[0034] Layer normalization and multi-layer perception mechanism are used to process the new knowledge and obtain the updated knowledge matrix;

[0035] The updated knowledge matrix is ​​cross-attentionally operated with the global image features to obtain the medical knowledge related to the medical image.

[0036] As a further technical solution, after the medical report is input into the text feature encoder, the global report feature is also obtained, wherein the text feature encoder includes the SciBert encoder and the fully connected layer. The specific process is as follows:

[0037] Encode the medical report through the SciBert encoder to obtain the medical report aggregation representation;

[0038] The aggregated representation of medical reports is input into the fully connected layer to obtain the global report features.

[0039] As a further technical solution, during the training process, the model is optimized by constructing a joint loss function, wherein the specific process of constructing the joint loss function is:

[0040] The global image features are linearly transformed to obtain the predicted values, and multi-label binary cross entropy is performed with the ground truth labels to construct the multimodal disease classification loss;

[0041] Construct a cross-modal global semantic loss based on global report features and global image features;

[0042] The cross-modal global semantic loss, multimodal disease classification loss and report generation loss are added to obtain the joint loss.

[0043] The second aspect discloses a medical report generation system based on a cross-modal interactive memory network, comprising:

[0044] Acquiring a training data module, wherein the training set includes medical images of known healthy areas and lesion areas, and medical reports of known healthy areas and lesion areas;

[0045] A cross-modal interactive memory network model construction training module is used to input the training set into the cross-modal interactive memory network model, train the model, and obtain a trained cross-modal interactive memory network model; wherein the cross-modal interactive memory network model includes an abnormal medical knowledge enhancement module and a cross-modal interactive memory module;

[0046] During training, medical images and medical reports are input into the cross-modal interactive memory network model for processing to obtain global image features and initial medical text knowledge features;

[0047] Extracting enhanced medical text knowledge features from initial medical text knowledge features through an abnormal medical knowledge enhancement module;

[0048] The extracted enhanced medical text knowledge features are stored in a matrix through a cross-modal interactive memory module, and medical knowledge related to medical images in the matrix is ​​extracted based on global image features;

[0049] The medical report generation module is used to obtain the medical image for which the report is to be generated, input the medical image for which the report is to be generated into the trained cross-modal interactive memory network model, and obtain the report generation result of the medical image.

[0050] The third aspect of the present invention is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0051] A fourth aspect of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which executes the steps of the above method when executed by a processor.

[0052] One or more of the above technical solutions have the following beneficial effects:

[0053] In this embodiment, a medical report generation model including a feature encoder, an abnormal medical knowledge enhancement module, a cross-modal interactive memory module, a report generation decoder and a cross-modal joint semantic loss unit is constructed, which solves the problems in traditional models that the introduction of medical knowledge consumes a lot of time and energy, cannot better extract abnormal fine-grained text knowledge, and lacks precise loss functions, which easily generates incomplete and inaccurate reports.

[0054] In this embodiment, the abnormal medical knowledge enhancement module alleviates the problem of text data bias, that is, the problem that abnormal reports and abnormal descriptions only account for a small part of the data set, and can help the model strengthen its attention and learning of abnormal descriptions (fine-grained knowledge features); the cross-modal interactive memory module realizes the introduction and utilization of automated medical text knowledge, and improves the accuracy of model-generated reports; the designed cross-modal joint semantic loss unit promotes the semantic consistency between image and text semantic features while improving the model's visual representation capabilities.

[0055] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0057] Figure 1 This is a flow chart of a medical report generation model based on a cross-modal interactive memory network according to the first embodiment of the present invention;

[0058] Figure 2 This is a schematic diagram of the abnormal medical knowledge enhancement module of the first embodiment. DETAILED DESCRIPTION

[0059] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0060] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0061] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0062] The overall idea proposed by the present invention is as follows: medical images and paired medical reports obtain image features and text features through a feature encoder; an abnormal medical knowledge enhancement module extracts fine-grained text knowledge from text features; a cross-modal interactive memory module stores the extracted coarse-grained and fine-grained medical text knowledge features in a matrix, and extracts relevant knowledge from the matrix; a report generation decoder receives the merged features of the incoming image features and knowledge features, and continues to predict the next word according to the input sequence. The predicted word is added to the input sequence to continue the above process until a complete report is generated; a cross-modal joint semantic loss unit adds multi-label disease classification loss and cross-modal global semantic loss to optimize and supervise the model during the training phase on the basis of retaining the text generation loss.

[0063] Embodiment 1

[0064] This embodiment discloses a medical report generation method based on a cross-modal interactive memory network.

[0065] In order to more clearly illustrate this embodiment, the process of implementing medical report generation based on the cross-modal interactive memory network can be specifically described as follows:

[0066] A method for generating a medical report based on a cross-modal interactive memory network is provided, comprising:

[0067] S1. Obtain a training set, wherein the training set includes medical images of known healthy areas and lesion areas, and medical reports of known healthy areas and lesion areas;

[0068] S2. Input the training set into the cross-modal interactive memory network model, train the model, and obtain a trained cross-modal interactive memory network model;

[0069] S3. Obtain a medical image for which a report is to be generated, and input the medical image for which a report is to be generated into a trained cross-modal interactive memory network model to obtain a report generation result of the medical image.

[0070] like Figure 1 As shown, in step S1, a training set is obtained, wherein the training set includes medical images of known healthy areas and lesion areas, and medical reports of known healthy areas and lesion areas.

[0071] The dataset is selected from MIMIC-CXR and IU-Xray, and the dataset is divided into three subsets: training set, test set and validation set. The cross-modal interactive memory network model is trained, tested and validated. The training set includes medical images and corresponding medical reports, and the test set and validation set only contain medical images. The training set is obtained, and the training set includes medical images of known healthy areas and lesion areas, and medical reports of known healthy areas and lesion areas.

[0072] like Figure 1 As shown, in step S2, the training set is input into the cross-modal interactive memory network model, the model is trained, and a trained cross-modal interactive memory network model is obtained.

[0073] S201. Construct a cross-modal interactive memory network model.

[0074] The cross-modal interactive memory network model includes a relationship enhancement feature encoder, a text feature encoder, an abnormal medical knowledge enhancement module, a cross-modal interactive memory module, a report generation decoder and a cross-modal joint semantic loss unit.

[0075] S202. Train the cross-modal interactive memory network model and perform feature extraction.

[0076] S2021. Input the medical image into a relationship-enhanced feature encoder to obtain relationship-enhanced image features and global image features.

[0077] The relation-enhanced feature encoder consists of a convolutional neural network and an intrinsic relation encoder.

[0078] The specific process of obtaining relationship-enhanced image features and global image features is as follows:

[0079] (1) The medical image is input into a convolutional neural network to extract visual area features and convert them into visual sequence labels.

[0080] like Figure 1As shown, in this embodiment, the radiological image I∈R in the data set E×W×3 Input to the pre-trained ResNet-101 network, extract the visual area features in the last convolutional layer of the ResNet-101 network, and convert them into visual sequence tags Where N i Representing 49 visual area features, the dimension of C is 2048.

[0081] (2) Convert visual sequence tags into visual sequence features through linear transformation;

[0082] Then the visual sequence labels are mapped to visual sequence features X through linear transformation P , the formula is:

[0083]

[0084] in, ResNet, W S A learnable parameter matrix, W S ∈R C×E , E is the visual feature dimension 512.

[0085] An intrinsic relation encoder with a three-layer standard Transformer encoder structure is constructed to obtain relation-enhanced visual features. Each encoder layer uses a multi-head self-attention mechanism with the formula:

[0086]

[0087] MHA(X,Y)=Concat(Head 1 (X,Y),…Head l (X, Y), …Head H (X, Y))W k (4)

[0088] Where X is the query matrix, Y is the key / value matrix, W h is a learnable parameter; Concat(·) represents a concatenation operation; e n =E / H; H is the number of attention heads, Q i represents the query matrix of the i-th attention head in multi-head attention, K i and V i denote the key matrix and value matrix of the i-th attention head in the multi-head self-attention.

[0089] (3) The visual sequence features and the global CLS tag are combined into a new visual sequence feature, and the new visual sequence feature is input into the intrinsic relationship encoder. Through the learnable memory matrix, the key-value pairs of the multi-head self-attention in the intrinsic relationship encoding are obtained.

[0090] like Figure 1 As shown, in this embodiment, a global CLS marker and a visual sequence marker X are introduced. P Together they form a new sequence As the input of the intrinsic relation encoder, the key-value pairs in the multi-head self-attention in the intrinsic relation encoder are expanded by adding a learnable memory matrix to help obtain the intrinsic relationship between image regions. The formula is:

[0091]

[0092] in and is the extended key-value pair of the i-th header, Is the connection operation MK i and Mv i is a learning matrix containing M rows.

[0093] (3) According to the multi-head attention key-value pairs, the new visual sequence features are encoded to obtain relation-enhanced image features and global image features.

[0094] An intrinsic relation encoder layer f e The process of (·) is as follows:

[0095] Attz=AddNorm(MHA(z, z)) (6)

[0096] f e (z) = AddNorm (FFN (Attz)) (7)

[0097] Among them, z represents the input of each encoder layer, AddNorm consists of layer normalization and residual connection, and FFN is the feed-forward network layer.

[0098] CLS Marking and Visual Sequence Feature X P After the intrinsic relation encoder, the relation-enhanced image features are obtained respectively. and global image features

[0099] After the above steps, the image features of the medical image are accurately extracted, providing data support for the subsequent generation of medical reports and improving the accuracy of the model.

[0100] (4) The global image features are linearly transformed to obtain the predicted values, and multi-label binary cross entropy is performed with the ground truth labels to construct the multimodal disease classification loss.

[0101] In this example, in order to improve the model's visual representation capabilities, we perform multi-label disease classification prediction. We use the CheXpert model to extract the ground truth labels in the corresponding reports. We use the global image features The predicted value is obtained by linear transformation With ground truth labels Perform multi-label binary cross entropy loss, the formula is:

[0102]

[0103]

[0104] Where W V is the learnable parameter matrix, b V is the bias; σ(·) is the sigmoid function; N d is the number of disease topics, y i ∈{0, 1} is the true label of the i-th disease, is the predicted value of the ith disease.

[0105] After the above steps, multi-label disease classification prediction is achieved and the visual representation ability of the model is improved.

[0106] S2022. Input the medical report into a text feature encoder to obtain initial medical text knowledge features.

[0107] The text feature encoder consists of a SciBert encoder and a fully connected layer.

[0108] like Figure 1 As shown, in this embodiment, the SciBert framework is used to analyze the real report Perform medical knowledge feature extraction, where N S Indicates the length of a report, as follows:

[0109] H = TextEncoder(T) (10)

[0110] Where T represents the ground truth report corresponding to the input image, TextEncoder(·) represents the text encoder, Represents the text features obtained through the text encoder, that is, the initial medical text knowledge features.

[0111] In addition, the aggregated representation of medical reports is obtained through the SciBert architecture. CLS , and then pass it through the fully connected layer to obtain the global report feature G bext , the formula is:

[0112] Gbext =T CLS W G +b G (11)

[0113] Among them, G bext ∈R E , W G is the learnable parameter matrix, b G For bias.

[0114] After the above steps, the medical knowledge related to the medical report is accurately extracted, providing data support for the subsequent report generation and improving the accuracy of the model.

[0115] In order to better align cross-modal global semantic information, multimodal contrastive loss is adopted to learn the latent semantic features between images and texts.

[0116] Specifically, we define the target global image feature G img The paired global report feature G text The distance between the positive pairs is smaller and the distance between the negative pairs is larger through training. In order to maximize the alignment of the semantic information between the paired features, we use the image and the report paired with it as the target points for comparative learning to obtain the cross-modal global semantic loss L S , the formula is:

[0117]

[0118] Where N represents a training batch size, S<,> represents the cosine similarity function, represents the learnable temperature coefficient, G img Represents the global image feature G bext Indicates the global reporting characteristic.

[0119] After the above steps, the cross-modal global semantic information can be better aligned.

[0120] S2023. Extract enhanced medical text knowledge features from the initial medical text knowledge features through the abnormal medical knowledge enhancement module.

[0121] Serious data bias (abnormal case reports only account for a small part and there are few descriptions of abnormal areas) makes the model easily ignore the fine-grained features of abnormal areas. The key clinical information mainly comes from the abnormal part, so we propose an abnormal medical knowledge enhancement module.

[0122] (1) Obtain the disease topic vector and initialize the disease state vector.

[0123] like Figure 2As shown, in this embodiment, 14 disease topics and their corresponding disease topic states are introduced, such as pulmonary edema, pneumonia, and multiple disease-related topics and their multiple disease topic states: positive, negative, uncertain, and not mentioned, etc., and the disease topic vector is introduced Where N d is the number of disease topics, and E is the characteristic dimension of each disease topic.

[0124] Initialize the disease state vector Where N k is the number of disease states.

[0125] (2) The disease topic vector is interacted with the initial medical text knowledge features to obtain fine-grained features related to the disease topic.

[0126] like Figure 2 As shown, in this embodiment, The initial medical text knowledge features obtained from the medical report through the SciBert architecture Interact to obtain fine-grained features related to disease topics

[0127] (3) The initialized disease state vector is interacted with the real disease state vector to obtain the disease state characteristics.

[0128] like Figure 2 As shown, in this embodiment, the disease state vector is initialized The true disease state vector Interact to obtain rich disease status characteristics

[0129] In this embodiment, the abnormal medical knowledge enhancement module is only used in the training phase, and the real disease state helps our model to describe the disease and related symptoms based on accurate information (teacher forcing mechanism).

[0130] (4) The fine-grained features related to the disease topic are fused with the disease state features to obtain fused features.

[0131] In this embodiment, fine-grained features related to disease topics are (Combines disease theme features) with rich disease state features (whether suffering from a certain disease) are added to obtain the fusion feature.

[0132] (5) The fused features are combined with the initial medical text knowledge features to obtain enhanced medical text knowledge features.

[0133] In this embodiment, the fusion feature is combined with the initial medical text knowledge feature H (i.e., the disease details contained in the real report) to obtain the enhanced medical text knowledge feature

[0134] After the above steps, the text data bias problem can be alleviated, helping the model to pay more attention to and learn about abnormal descriptions (fine-grained knowledge features).

[0135] S2024. The extracted enhanced medical text knowledge features are stored in a matrix through a cross-modal interactive memory module, and medical knowledge related to the medical image in the matrix is ​​extracted based on global image features.

[0136] Previous studies mainly introduced medical or prior knowledge by pre-building knowledge graphs or knowledge template libraries. These methods are time-consuming and labor-intensive and difficult to migrate to other datasets. Since building knowledge graphs and template libraries is time-consuming and labor-intensive, we hope to achieve automatic storage and utilization of medical text knowledge features. Therefore, we propose a method for dynamically storing medical knowledge.

[0137] Extract medical knowledge related to medical images in the matrix. The specific process is:

[0138] (1) Initialize the dynamic medical matrix to store medical knowledge.

[0139] like Figure 1 As shown, in this embodiment, during the training phase, we randomly initialize a dynamic medical knowledge matrix To store medical knowledge, K b It indicates how many rows the matrix contains, that is, how much knowledge can be stored, and E is the characteristic dimension of each row.

[0140] (2) The matrix is ​​interacted with the enhanced medical text knowledge features to obtain new knowledge from the enhanced medical text knowledge features. The new knowledge is processed using layer normalization and multi-layer perception mechanisms to obtain an updated knowledge matrix.

[0141] like Figure 1 As shown, in this embodiment, the matrix B interacts with the enhanced medical text knowledge features (features containing fine-grained medical knowledge) H^ to update the medical knowledge in the matrix.

[0142] Specifically, when the knowledge matrix of the training to step l is B l , obtain new knowledge from the enhanced medical text knowledge features of the current input report, the formula is:

[0143] B′ l =MHA(B l ,H^) (15)

[0144] B l+1 =B l +MLP(LN(B′ l )) (16)

[0145] Among them, B′ l is the new knowledge obtained from the enhanced medical knowledge text features of the current input report, MLP is a multi-layer perceptron, LN is layer normalization, B l+1 is the updated knowledge matrix.

[0146] (3) Perform a cross-attention operation on the updated knowledge matrix and the global image features to obtain medical knowledge related to the medical image.

[0147] The updated knowledge matrix B l+1 With the global image feature G img Perform cross-attention operation to obtain medical knowledge related to the current medical image The formula is:

[0148]

[0149] in, Represents the medical knowledge related to the current medical image, CA(,) denotes the cross attention operation.

[0150] S2025. Combine the medical knowledge related to the medical image with the relationship-enhanced image features to obtain an image-knowledge combined feature.

[0151] Medical knowledge related to medical images Image features X enhanced with relations V Perform parallel operations to obtain image-knowledge merging features The formula is:

[0152]

[0153] in,

[0154] During the testing phase, the matrix is ​​kept unchanged and the image features are reported using the knowledge stored in the knowledge matrix.

[0155] After the above steps, the introduction and utilization of automated medical text knowledge is realized, and the accuracy of model-generated reports is improved.

[0156] S2026. The report generation decoder receives the incoming image-knowledge merging features, and predicts the next word based on the input sequence, adds the predicted word to the input sequence, and continues to predict the next word until a complete medical report is generated.

[0157] The report generation process is an autoregressive process, where each word is predicted based on the previously generated word sequence. Specifically, the decoder uses the input sequence and the image-knowledge merge feature V merge To predict the next word, the predicted word is then added to the input sequence to continue predicting the next word. Our model follows the above process until a complete report is generated. The prediction probability formula is:

[0158]

[0159] in, is the word generated at time t, V merge represents the image-knowledge merging feature,

[0160] During the training phase, we report the entire ground truth The word sequence is input into the decoder. Then the upper triangular mask operation is performed on the input sequence and the upper triangular matrix, that is, each word is predicted based on the word sequence on its left, and the model is optimized by minimizing the cross entropy loss:

[0161]

[0162] Among them, N R Indicates the length of the report, Y t Represents the tth word generated.

[0163] In the cross-modal joint semantic loss unit, a joint loss is constructed, which is the sum of all loss components, and multimodal disease classification loss and cross-modal global semantic loss are added to improve the visual representation ability of the model and strengthen the global semantic alignment between modalities, respectively.

[0164] Specifically, in order to balance the relationship between the losses, the loss L obtained by equations (9) and (14) is V and L S and the balance coefficient β 1 and β 2 Combined and then reported to generate the loss L G Add together to get the joint loss L loss , the formula is:

[0165] L loss =L G +β1 L V +β 2 L S (twenty one)

[0166] After the above steps, the visual representation ability of the model is improved while promoting the semantic consistency between image and text semantic features.

[0167] According to the above steps, the dataset is divided into three subsets: training set, test set and validation set. The cross-modal interactive memory network model is trained, tested and validated. The training set includes medical images and corresponding medical reports, while the test set and validation set only contain medical images. The model is compared with a series of baseline models to verify its effectiveness.

[0168] The model in this embodiment is compared with other baseline models on the datasets MIMIC-CXR and IU-Xray, and the experimental results show significant improvement. This solves the problems in traditional models that it takes a lot of time and effort to introduce medical knowledge, cannot better extract abnormal fine-grained text knowledge, lacks precise loss functions, and easily generates incomplete and inaccurate reports.

[0169] In this embodiment, in step S3, a medical image for which a report is to be generated is obtained, and the medical image for which a report is to be generated is input into a trained cross-modal interactive memory network model to obtain a report generation result of the medical image.

[0170] In practical applications, after using X-rays to scan the patient's lungs to obtain a lung medical image, the medical image can be input into a cross-modal interactive memory network model. The feature encoder can automatically identify potential lesions such as lung nodules, pneumonia, and pneumothorax in the medical image, and extract image features and global image features associated with the lesion areas. According to the image features, medical knowledge related to the current lung medical image can be found from the knowledge matrix in the model. The extracted image features and valuable medical knowledge are input into the report generation decoder of the model to generate a complete medical report on the current medical image for the doctor's reference. On the one hand, it can reduce the burden on doctors, and on the other hand, it can assist doctors in discovering some details that are difficult to detect with the naked eye, thereby improving the accuracy and efficiency of diagnosis.

[0171] Embodiment 2

[0172] The purpose of this embodiment is to provide a medical report generation system based on a cross-modal interactive memory network, including:

[0173] Acquiring a training data module, wherein the training set includes medical images of known healthy areas and lesion areas, and medical reports of known healthy areas and lesion areas;

[0174] A cross-modal interactive memory network model construction training module is used to input the training set into the cross-modal interactive memory network model, train the model, and obtain a trained cross-modal interactive memory network model; wherein the cross-modal interactive memory network model includes an abnormal medical knowledge enhancement module and a cross-modal interactive memory module;

[0175] During training, medical images and medical reports are input into the cross-modal interactive memory network model for processing to obtain global image features and initial medical text knowledge features;

[0176] Extracting enhanced medical text knowledge features from initial medical text knowledge features through an abnormal medical knowledge enhancement module;

[0177] The extracted enhanced medical text knowledge features are stored in a matrix through a cross-modal interactive memory module, and medical knowledge related to medical images in the matrix is ​​extracted based on global image features;

[0178] The medical report generation module is used to obtain the medical image for which the report is to be generated, input the medical image for which the report is to be generated into the trained cross-modal interactive memory network model, and obtain the report generation result of the medical image.

[0179] Based on providing a medical report generation system based on a cross-modal interactive memory network, the method steps in embodiment 1 are implemented.

[0180] Embodiment 3

[0181] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0182] Embodiment 4

[0183] The purpose of this embodiment is to provide a computer-readable storage medium.

[0184] A computer-readable storage medium stores a computer program, which executes the steps of the above method when executed by a processor.

[0185] The steps involved in the apparatus of the above embodiment correspond to the method embodiment 1, and the specific implementation method can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0186] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0187] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A medical report generation method based on a cross-modal interactive memory network, characterized in that: include: Acquire a training set, wherein the training set includes medical images of known healthy areas and lesion areas, and medical reports of known healthy areas and lesion areas; Inputting the training set into the cross-modal interactive memory network model, training the model, and obtaining a trained cross-modal interactive memory network model; Acquire a medical image for which a report is to be generated, input the medical image for which a report is to be generated into a trained cross-modal interactive memory network model, and obtain a report generation result of the medical image; Wherein, the cross-modal interactive memory network model includes an abnormal medical knowledge enhancement module and a cross-modal interactive memory module; During training, medical images and medical reports are input into the cross-modal interactive memory network model for processing to obtain global image features and initial medical text knowledge features; Extracting enhanced medical text knowledge features from initial medical text knowledge features through an abnormal medical knowledge enhancement module; The extracted enhanced medical text knowledge features are stored in a matrix through a cross-modal interactive memory module, and medical knowledge related to medical images in the matrix is ​​extracted based on global image features.

2. A method for generating a medical report based on a cross-modal interactive memory network as claimed in claim 1, characterized in that: The medical images and medical reports are input into the cross-modal interactive memory network model for processing, wherein the cross-modal interactive memory network model also includes a relation enhancement feature encoder and a text feature encoder, specifically: Inputting the medical image into a relation-enhanced feature encoder to obtain relation-enhanced image features and global image features; The medical report is input into the text feature encoder to obtain the initial medical text knowledge features.

3. A medical report generation method based on a cross-modal interactive memory network as described in claim 2, characterized in that: The medical image is input into the relation-enhanced feature encoder to obtain relation-enhanced image features and global image features, wherein the relation-enhanced feature encoder includes a convolutional neural network and an intrinsic relation encoder. The specific process is as follows: Input the medical image into the convolutional neural network, extract the visual area features and convert them into visual sequence tags; Convert visual sequence tags into visual sequence features through linear transformation; The visual sequence features and the global CLS tag are combined into a new visual sequence feature, and the new visual sequence feature is input into the intrinsic relation encoder; Through the learnable memory matrix, the key-value pairs of multi-head self-attention in the intrinsic relationship encoding are obtained; According to the multi-head attention key-value pairs, the new visual sequence features are encoded to obtain relation-enhanced image features and global image features.

4. The method for generating a medical report based on a cross-modal interactive memory network according to claim 1, characterized in that: The enhanced medical text knowledge features in the initial medical text knowledge features are extracted through the abnormal medical knowledge enhancement module. The specific process is as follows: Get the disease topic vector and initialize the disease state vector; Interact the disease topic vector with the initial medical text knowledge features to obtain fine-grained features related to the disease topic; Interact the initialized disease state vector with the true disease state vector to obtain the disease state feature; Fusing fine-grained features related to the disease topic with disease state features to obtain fused features; The fused features are combined with the initial medical text knowledge features to obtain enhanced medical text knowledge features.

5. The method for generating a medical report based on a cross-modal interactive memory network according to claim 1, characterized in that: The extracted enhanced medical text knowledge features are stored in the matrix through the cross-modal interactive memory module, and the medical knowledge related to the medical image in the matrix is ​​extracted according to the global image features. The specific process is as follows: Initialize the dynamic medical matrix to store medical knowledge; Interact the matrix with the enhanced medical text knowledge features to obtain new knowledge from the enhanced medical text knowledge features; Layer normalization and multi-layer perception mechanism are used to process the new knowledge and obtain the updated knowledge matrix; The updated knowledge matrix is ​​cross-attentionally operated with the global image features to obtain the medical knowledge related to the medical image.

6. A medical report generation method based on a cross-modal interactive memory network as claimed in claim 1, characterized in that: After the medical report is input into the text feature encoder, the global report features are also obtained. The text feature encoder includes the SciBert encoder and the fully connected layer. The specific process is as follows: Encode the medical report through the SciBert encoder to obtain the medical report aggregation representation; The aggregated representation of medical reports is input into the fully connected layer to obtain the global report features.

7. A medical report generation method based on a cross-modal interactive memory network as claimed in claim 1, characterized in that: During the training process, the model is optimized by constructing a joint loss function, where the specific process of constructing the joint loss function is: Construct a cross-modal global semantic loss based on global report features and global image features; The global image features are linearly transformed to obtain the predicted values, and multi-label binary cross entropy is performed with the ground truth labels to construct the multimodal disease classification loss; The cross-modal global semantic loss, multimodal disease classification loss and report generation loss are added to obtain the joint loss.

8. A medical report generation system based on a cross-modal interactive memory network, characterized in that: include: Acquiring a training data module, wherein the training set includes medical images of known healthy areas and lesion areas, and medical reports of known healthy areas and lesion areas; A cross-modal interactive memory network model construction training module is used to input the training set into the cross-modal interactive memory network model, train the model, and obtain a trained cross-modal interactive memory network model; wherein the cross-modal interactive memory network model includes an abnormal medical knowledge enhancement module and a cross-modal interactive memory module; During training, medical images and medical reports are input into the cross-modal interactive memory network model for processing to obtain global image features and initial medical text knowledge features; Extracting enhanced medical text knowledge features from initial medical text knowledge features through an abnormal medical knowledge enhancement module; The extracted enhanced medical text knowledge features are stored in a matrix through a cross-modal interactive memory module, and medical knowledge related to medical images in the matrix is ​​extracted based on global image features; The medical report generation module is used to obtain the medical image for which the report is to be generated, input the medical image for which the report is to be generated into the trained cross-modal interactive memory network model, and obtain the report generation result of the medical image.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are performed.

Images