A method and device for generating medical image report

Abstract

The present invention discloses a method and device for generating a medical image report, including: obtaining a target medical image; processing the target medical image based on a target medical image semantic association model to obtain multimodal topic distribution information corresponding to the target medical image; generating multi-level text description information of the target medical image based on the multimodal topic distribution information; optimizing the multi-level text description information to obtain a medical image report corresponding to the target medical image. The present invention realizes modeling of potential semantic associations of medical images by related topics with complementary modalities based on the target medical image semantic association model, can interpret the semantics represented by the medical image, and realizes the generation of objective and accurate text information based on the multi-level text information description, thereby improving the accuracy of automatic generation of medical image reports.

Description

Technical Field

[0001] The present invention relates to the technical field of medical information processing, and in particular to a method and device for generating a medical image report. Background Art

[0002] With the massive growth of digital medical imaging data, researchers at home and abroad have used computer vision, natural language understanding, deep learning and other technologies to conduct a series of research on medical image classification and recognition, positioning and monitoring, tissue organ and lesion segmentation, etc. for medical imaging data with different imaging principles such as MRI, CT, X-ray, ultrasound, PRT, etc. However, using computers to identify abnormal lesion features in images is only the first step in clinical image-assisted diagnosis. In clinical practice, radiologists analyze the normal or abnormal manifestations in the images by reading medical images and combining their own experience and knowledge to form imaging examination reports. The relevant text descriptions of medical images in the reports are an important basis for clinical diagnosis.

[0003] Since the language descriptions of normal manifestations in imaging reports are very similar, radiologists often spend a lot of time repeatedly writing reports with similar content. In order to assist doctors in writing reports efficiently, there are studies on automatic report generation through deep learning of medical images. However, the existing methods have problems with reliability and poor interpretability, and cannot improve the accuracy of automatic report generation. Summary of the invention

[0004] In view of the above problems, the present invention provides a method and device for generating a medical imaging report, which achieves the purpose of improving the reliability, interpretability and accuracy of automatically generated medical imaging reports.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A method for generating a medical imaging report, comprising:

[0007] Acquire target medical images;

[0008] The target medical image is processed based on a target medical image semantic association model to obtain multimodal topic distribution information corresponding to the target medical image, wherein the target medical image semantic association model is obtained by multimodal topic modeling based on a target training sample, wherein each image-text pair in the target training sample is an information pair of a medical image and text information, and the multimodal topic includes at least a modality-shared topic and a modality-related topic, wherein the modality-shared topic represents a topic in which the medical image and text information in the image-text pair correspond to the same topic, and the modality-related topic represents a topic in which the medical image and text information in the image-text pair correspond to different topics;

[0009] Based on the multimodal topic distribution information, generating multi-level text description information of the target medical image;

[0010] The multi-level text description information is optimized to obtain a medical image report corresponding to the target medical image.

[0011] Optionally, it also includes:

[0012] Get initial training samples;

[0013] The initial training sample is optimized to obtain a target training sample:

[0014] Determine the image-text pairs with the target neighborhood structure in the target training samples as multimodal samples;

[0015] Calculating multimodal topic similarities for the multimodal samples respectively to obtain modal shared topics and modal related topics;

[0016] determining the modality-shared topics and the modality-related topics as high-level semantic feature descriptors;

[0017] Performing semantic association analysis based on the processing logic in the medical image report and the high-level semantic feature descriptor to obtain medical image semantic association information;

[0018] Machine learning is performed on the medical image semantic association information to obtain a target medical image semantic association model.

[0019] Optionally, the optimizing the initial training sample to obtain a target training sample includes:

[0020] Performing training based on the initial training samples to obtain an initial model, wherein the initial model is a model capable of determining a matching relationship between medical images and text information, and the initial model includes a unimodal encoder, an image-based text encoder, and an image-based text decoder;

[0021] Encoding the medical image and text information corresponding to each image-text pair in the initial training sample by the single-modal encoder to obtain image encoding information and text encoding information;

[0022] Adjusting the image-based text decoder based on the manually annotated image-text pairs to obtain a descriptor;

[0023] Using the descriptor to generate text description information of the first medical image without manually annotated text information in the initial training sample, to obtain a first type of image-text pairs, wherein the first type of image-text pairs are image-text pairs corresponding to the medical image and the text description information generated by the descriptor;

[0024] Adjusting the image-based text encoder based on the manually annotated image-text pairs to obtain a filter;

[0025] Filtering the first type of image-text pairs and the second type of image-text pairs using the filter to obtain a third type of image-text pairs, wherein the second type of image-text pairs are image-text pairs consisting of medical images without manually annotated text information and corresponding automatically extracted text information, and the third type of image-text pairs include the filtered first type of image-text pairs and the filtered second type of image-text pairs;

[0026] Target medical image-text sample data is generated based on the manually annotated image-text pairs and the third type of image-text pairs.

[0027] Optionally, generating multi-level text description information of the target medical image based on the multimodal topic distribution information includes:

[0028] Determining, based on the multimodal topic distribution information, an association probability between each topic and the target medical image;

[0029] Based on the association probability, determining a target topic from each topic corresponding to the multimodal topic distribution;

[0030] Determine the target subject to generate supervision information for describing the text, and generate a text sentence corresponding to the target medical image based on a hierarchical long short-term memory network and a visual attention mechanism;

[0031] The text sentences are processed to obtain multi-level text description information.

[0032] Optionally, the optimizing the multi-level text description information to obtain a medical image report corresponding to the target medical image includes:

[0033] Based on the display visualization information and the image report description mode, the multi-level text description information is optimized to obtain a medical image report corresponding to the target medical image.

[0034] A medical imaging report generating device, comprising:

[0035] An acquisition unit, used for acquiring a target medical image;

[0036] A processing unit, configured to process the target medical image based on a target medical image semantic association model to obtain multimodal topic distribution information corresponding to the target medical image, wherein the target medical image semantic association model is obtained by performing multimodal topic modeling based on a target training sample, wherein each image-text pair in the target training sample is an information pair of a medical image and text information, and wherein the multimodal topic includes at least a modality-shared topic and a modality-related topic, wherein the modality-shared topic represents a topic in which the medical image and text information in the image-text pair correspond to the same topic, and the modality-related topic represents a topic in which the medical image and text information in the image-text pair correspond to different topics;

[0037] A generating unit, configured to generate multi-level text description information of the target medical image based on the multimodal topic distribution information;

[0038] The optimization unit is used to optimize the multi-level text description information to obtain a medical image report corresponding to the target medical image.

[0039] Optionally, it also includes:

[0040] A sample acquisition unit, used to acquire initial training samples;

[0041] The sample optimization unit is used to optimize the initial training sample to obtain a target training sample:

[0042] A first determination unit is used to determine the image-text pairs with the target neighborhood structure in the target training sample as multimodal samples;

[0043] A calculation unit, used to calculate multimodal topic similarity for the multimodal samples respectively, to obtain modal shared topics and modal related topics;

[0044] A second determining unit, configured to determine the modality shared topics and the modality related topics as high-level semantic feature descriptors;

[0045] An analysis unit, configured to perform semantic association analysis based on the processing logic in the medical image report and the high-level semantic feature descriptor to obtain semantic association information of the medical image;

[0046] The machine learning unit is used to perform machine learning on the medical image semantic association information to obtain a target medical image semantic association model.

[0047] Optionally, the sample optimization unit is specifically used for:

[0048] Performing training based on the initial training samples to obtain an initial model, wherein the initial model is a model capable of determining a matching relationship between medical images and text information, and the initial model includes a unimodal encoder, an image-based text encoder, and an image-based text decoder;

[0049] Encoding the medical image and text information corresponding to each image-text pair in the initial training sample by the single-modal encoder to obtain image encoding information and text encoding information;

[0050] Adjusting the image-based text decoder based on the manually annotated image-text pairs to obtain a descriptor;

[0051] Using the descriptor to generate text description information of the first medical image without manually annotated text information in the initial training sample, to obtain a first type of image-text pairs, wherein the first type of image-text pairs are image-text pairs corresponding to the medical image and the text description information generated by the descriptor;

[0052] Adjusting the image-based text encoder based on the manually annotated image-text pairs to obtain a filter;

[0053] Filtering the first type of image-text pairs and the second type of image-text pairs using the filter to obtain a third type of image-text pairs, wherein the second type of image-text pairs are image-text pairs consisting of medical images without manually annotated text information and corresponding automatically extracted text information, and the third type of image-text pairs include the filtered first type of image-text pairs and the filtered second type of image-text pairs;

[0054] Target medical image-text sample data is generated based on the manually annotated image-text pairs and the third type of image-text pairs.

[0055] Optionally, the generating unit includes:

[0056] A first determination subunit, configured to determine an association probability between each topic and the target medical image based on the multimodal topic distribution information;

[0057] A second determination subunit, configured to determine a target topic from among the topics corresponding to the multimodal topic distribution based on the association probability;

[0058] A generating subunit, configured to determine the target subject to generate supervisory information for describing the text, and generate a text sentence corresponding to the target medical image based on a hierarchical long short-term memory network and a visual attention mechanism;

[0059] The processing subunit is used to process the text sentence to obtain multi-level text description information.

[0060] Optionally, the optimization unit is specifically used for:

[0061] Based on the display visualization information and the image report description mode, the multi-level text description information is optimized to obtain a medical image report corresponding to the target medical image.

[0062] Compared with the prior art, the present invention provides a method and device for generating a medical image report, including: obtaining a target medical image; processing the target medical image based on a target medical image semantic association model to obtain multimodal topic distribution information corresponding to the target medical image; generating multi-level text description information of the target medical image based on the multimodal topic distribution information; optimizing the multi-level text description information to obtain a medical image report corresponding to the target medical image. The present invention implements modeling of potential semantic associations of medical images with related topics of modal complementarity based on the target medical image semantic association model, can interpret the semantics represented by the medical image, and realizes the generation of objective and accurate text information based on the multi-level text information description, thereby improving the accuracy of automatic generation of medical image reports. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0064] Figure 1 A schematic diagram of a flow chart of a method for generating a medical imaging report provided by an embodiment of the present invention;

[0065] Figure 2 A partial schematic diagram of a Chinese chest imaging report provided by an embodiment of the present invention;

[0066] Figure 3 A schematic diagram of the structure of a medical imaging report generating device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] The terms "first" and "second" and the like in the specification and claims of the present invention and the above drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0069] In an embodiment of the present invention, a method for generating a medical image report is provided, and the method can be applied to an application scenario of automatically generating a medical image report that matches the medical image. Specifically, the present invention aims at the problem of insufficient high-level semantic modeling capabilities of medical images, and constructs a medical image semantic association model based on multimodal semantic topic modeling based on a knowledge representation model that integrates large-scale medical images and text modalities. According to the multimodal semantic topics predicted by the semantic association of medical images, an interpretable description of the content of medical images is achieved using multi-topic hierarchical natural language text. It can be applied to a medical image report generation system based on a medical image description method for intelligent auxiliary diagnosis scenarios.

[0070] See also Figure 1 , is a flow chart of a method for generating a medical imaging report provided by an embodiment of the present invention, the method may include the following steps:

[0071] S101. Acquire target medical images.

[0072] Among them, the target medical image is the image for which a medical imaging report needs to be generated, and can be an image acquired by any medical device, such as an ultrasound device, a computer tomography device, a magnetic resonance imaging device, etc. Correspondingly, the target medical image can be a medical image or a group of related medical images, such as an image composed of local medical images of a certain part.

[0073] S102: Process the target medical image based on the target medical image semantic association model to obtain multimodal topic distribution information corresponding to the target medical image.

[0074] Among them, the target medical image semantic association model is obtained by multimodal topic modeling based on the target training samples, each image-text pair in the target training samples is an information pair of medical image and text information, and the multimodal topic includes at least modality-shared topics and modality-related topics. The modality-shared topics represent the same topics corresponding to the medical image and text information in the image-text pairs, and the modality-related topics represent different topics corresponding to the medical image and text information in the image-text pairs.

[0075] The target training samples are obtained by optimizing the initial training samples, wherein the initial training samples are obtained by preprocessing and extracting standardized information from large-scale public medical imaging data, Chinese and English clinical imaging reports, case reports, and imaging-related scientific discussions, and combining them with the medical knowledge base to carry out medical "image-text" joint representation learning. After obtaining the initial training samples, a multi-stage multimodal pre-training method based on the idea of ​​data bootstrapping can be used to obtain the target training samples. The specific sample processing process will be described in detail in the subsequent embodiments of the present invention and will not be described in detail here.

[0076] Medical images are vectorized based on the joint representation of target training samples, the semantic association between medical images and their descriptive texts is summarized as a measure of cross-modal latent semantic topic relevance and fuzzy reasoning, and the latent semantic topics of medical images are modeled using topic models. In one embodiment, it also includes: obtaining initial training samples; optimizing the initial training samples to obtain target training samples: determining the image-text pairs with target neighborhood structures in the target training samples as multimodal samples; performing multimodal topic similarity calculations on the multimodal samples to obtain modal shared topics and modal related topics; determining the modal shared topics and modal related topics as high-level semantic feature descriptors; performing semantic association analysis based on the processing logic in the medical image report and the high-level semantic feature descriptors to obtain medical image semantic association information; performing machine learning on the medical image semantic association information to obtain a target medical image semantic association model.

[0077] If there are medical images with highly similar visual features (abnormal manifestations), the high-level semantic topic distributions (such as lesion location, morphology, edges, surrounding conditions, etc.) of the descriptor lesion abnormalities are also similar. Based on the joint representation of "medical image-text information" in the target training sample, the present invention reduces the potential semantic associations between medical images to a measure of cross-modal semantic topic relevance, and can use the latent Dirichlet topic model (Latent Dirichlet Allocation, LDA) theory to perform topic modeling on the multimodal shared space and unimodal related space of similar medical images. Among them, the topic modeling here is to predict the shared topic distribution of images and texts based on the assumed semantic space. At the same time, it is also assumed that there is redundant information in the text itself, so there is also a topic distribution for the description in the text that is not related to the image, that is, a unimodal related topic. That is to say, this step first uses the topic model to predict the topic distribution of the sample.

[0078] Specifically, if there is a single-modality medical image sample S i (V i , *), where V iRepresents the image information in the image-text pair. If it is a target medical image, it only has medical images and no text information. Use the joint representation model to vectorize it, and then determine the image-text pair with the target neighborhood structure in the image-text pair based on the vectorized representation. The target neighborhood structure refers to the image-text pair with similar neighborhood structure under the same modality, and determine it as a multimodal sample. i Image information V i , its vectorized representation, calculates other annotated (text) samples with high vector similarity, that is, finds images with similar visual features, and then analyzes their topic distribution based on the text information of similar images. For example, the multimodal sample is represented as S sim (V, D), where V represents medical images and D represents text information related to medical images. S And the modal correlation space L M Perform topic modeling and analyze the topic distribution (Z M , Z S ) to calculate semantic similarity. For multimodal sample S j (V j , D j ), and also vectorize it, find samples with similar neighborhood structures, and calculate multimodal topic similarity. Among them, the modal shared space refers to a semantic space shared by images and texts, such as local features such as specific anatomical parts and lesions in images, and the corresponding texts of different granularity describing these features (mainly semantic topics composed of words and phrases), which are significantly associated; the modal related space refers to the semantic spaces of images and texts, and there is no direct association between the two.

[0079] Distribute the multimodal shared topics Z S And the modality-related topic distribution Z M They are respectively used as high-level semantic feature descriptors, and then semantic association analysis is performed according to the processing logic in the medical imaging report to obtain medical imaging semantic association information; machine learning is performed on the medical imaging semantic association information to obtain the target medical imaging semantic association model. Among them, the processing logic in the medical imaging report is mainly combined with the imaging diagnosis logic formed around the disease site, signs, nature, etiology, etc. in the medical imaging diagnosis guide. And based on the parallel fuzzy rules of multimodal topic similarity, the clinical significance represented by the abnormal imaging performance is summarized and associated, that is, medical imaging semantic association.

[0080] S103: Generate multi-level text description information of the target medical image based on the multimodal topic distribution information.

[0081] The obtained multimodal topic distribution information can be used to obtain the semantically relevant topics of the target medical image. Combined with the visual attention mechanism, hierarchical medical image description information containing multi-granular texts such as keyword, sentence, paragraph, etc. is generated according to specific topics.

[0082] Specifically, in one embodiment, the multi-level text description information of the target medical image is generated based on the multimodal topic distribution information, including: determining the association probability between each topic and the target medical image based on the multimodal topic distribution information; determining a target topic from each topic corresponding to the multimodal topic distribution based on the association probability; determining the target topic to generate supervision information for describing the text, and generating a text sentence corresponding to the target medical image based on a hierarchical long short-term memory network and a visual attention mechanism; and processing the text sentence to obtain multi-level text description information.

[0083] After obtaining the multimodal topic distribution information, which contains multiple topics, each topic has a different association probability with the current target medical image. Usually, the topic with the largest association probability is selected as the supervisory information of the description text. new (V new ,*), the visual modality topic distribution Modeling and deriving other modes Semantically related topics Predict the multimodal topic distribution of the sample Among them, V new Target medical imaging.

[0084] Generate hierarchical medical image descriptions based on specific topics: If each sentence generated focuses on a specific topic, using multimodal topics As supervisory information, a hierarchical long short-term memory network (Hierarchical Long Short-Term Memory) with good effect on generating long text sequences is added, combined with the visual attention mechanism to automatically generate natural language sentences associated with the content of specific areas of the image. All generated text sequences are spliced ​​and summarized to obtain hierarchical imaging descriptions and diagnostic opinions. Hierarchical refers to the granularity of the text. According to the previously predicted semantic themes, the generated text includes words, concepts, phrases, and complete sentences.

[0085] S104: Optimize the multi-level text description information to obtain a medical image report corresponding to the target medical image.

[0086] This process is mainly based on displaying visualization information and image report description mode, optimizing multi-level text description information to obtain a medical image report corresponding to the target medical image. Among them, displaying visualization information mainly refers to the display mode or visualization format information of the report, and the image report description mode refers to the description logic of the usual image report. That is, the neural network feature visualization method can be used to display the generation process of medical image description from a visual and logical perspective, which is convenient for manual evaluation and analysis.

[0087] See also Figure 2 , which is a partial schematic diagram of a Chinese chest imaging report provided in an embodiment of the present invention. In this application scenario, chest imaging examination (X-ray) with relatively wide clinical application is selected as the research object, and a Chinese chest imaging report generation system is constructed by combining medical image description, transfer learning and machine translation technologies. Based on the medical imaging report generation method provided in an embodiment of the present invention, a Chinese chest imaging report is generated through machine-assisted generation and human-computer interactive feedback.

[0088] The present invention provides a method for generating a medical image report, comprising: obtaining a target medical image; processing the target medical image based on a target medical image semantic association model to obtain multimodal topic distribution information corresponding to the target medical image; generating multi-level text description information of the target medical image based on the multimodal topic distribution information; optimizing the multi-level text description information to obtain a medical image report corresponding to the target medical image. The present invention implements modeling of potential semantic associations of medical images with related topics of modal complementarity based on the target medical image semantic association model, can interpret the semantics represented by the medical image, and realizes the generation of objective and accurate text information based on the multi-level text information description, thereby improving the accuracy of automatic generation of medical image reports.

[0089] The following is a detailed description of the process of optimizing the initial training samples in the embodiment of the present invention. This process can learn a generalizable medical "image-text" joint representation, which is an important basis for obtaining high-quality multimodal semantic features. Based on large-scale multimodal medical imaging data and text data such as imaging diagnosis reports, case reports, and scientific discussions describing imaging findings, the present invention adopts a multi-stage multimodal pre-training method based on the idea of ​​data bootstrapping to establish a medical "image-text" joint representation model. The data involved in the learning of the medical "image-text" joint representation model include other data such as public medical imaging data sets, clinical imaging reports, medical case reports, and imaging-related scientific discussions. First, pre-processing and normalized information extraction are performed on medical data of different sources and types, including extracting medical "image-text" parallel pairs from imaging reports, case reports, etc.; extracting medical images of different modalities and their semantic label information from open medical imaging data. Secondly, a visual language model pre-training method based on data bootstrapping is combined with a medical knowledge base to construct a learning framework to form a medical knowledge representation with complementary modal information fusion.

[0090] In one implementation, the optimizing the initial training sample to obtain a target training sample includes:

[0091] Performing training based on the initial training samples to obtain an initial model, wherein the initial model is a model capable of determining a matching relationship between medical images and text information, and the initial model includes a unimodal encoder, an image-based text encoder, and an image-based text decoder;

[0092] Encoding the medical image and text information corresponding to each image-text pair in the initial training sample by the single-modal encoder to obtain image encoding information and text encoding information;

[0093] Adjusting the image-based text decoder based on the manually annotated image-text pairs to obtain a descriptor;

[0094] Using the descriptor to generate text description information of the first medical image without manually annotated text information in the initial training sample, to obtain a first type of image-text pairs, wherein the first type of image-text pairs are image-text pairs corresponding to the medical image and the text description information generated by the descriptor;

[0095] Adjusting the image-based text encoder based on the manually annotated image-text pairs to obtain a filter;

[0096] Filtering the first type of image-text pairs and the second type of image-text pairs using the filter to obtain a third type of image-text pairs, wherein the second type of image-text pairs are image-text pairs consisting of medical images without manually annotated text information and corresponding automatically extracted text information, and the third type of image-text pairs include the filtered first type of image-text pairs and the filtered second type of image-text pairs;

[0097] Target medical image-text sample data is generated based on the manually annotated image-text pairs and the third type of image-text pairs.

[0098] Among them, the initial model is a multi-stage multimodal training mode, which can actually be trained in stages from smaller semantic label units to larger sentence units of medical image-text data, so as to learn image-text correspondence information of different granularities at different stages. Specifically, the unimodal encoder encodes the input data of text and image respectively, which is equivalent to a feature extractor, and provides input vectors of text modality and image modality for subsequent image-based text encoder and image-based text decoder. The process of unimodal encoder encoding medical images and related texts by unimodal encoder mainly includes: encoding medical images using ViT (Vision Transformer) model, dividing the input image into a group of image patches, and then inputting the transformer unit to obtain the output image, and the output image embedding sequence plus a [CLS] tag to constitute the global image feature, wherein the [CLS] tag is a special tag placed before the image or text input content, representing the entire image or sentence information. Correspondingly, the encoding method of the text modality is consistent with the BERT model (Bidirectional Encoder Representations from Transformer, a pre-trained language representation model), and a [CLS] tag is added to represent the global text information. The image-based text encoder needs to obtain the image encoding information obtained by the unimodal encoder, that is, the unimodal image input vector, which can be used alone for image-text matching filters, or combined with the image-based text decoder to form a codec. Based on the BERT model, the image-based text encoder adds a cross attention layer (Cross Attention, CA) between the feedforward network layer (FFN) and the self-attention layer (SA) to inject image information into the network. An [Encode] tag is appended to the text, and the output embedding representation is used as the multimodal representation information of the image-text pair. The image-based text decoder needs to decode the image / text encoding information of the unimodal encoder to generate text description information of the image. That is, the bidirectional self-attention layer in the image-based text encoder is replaced with a causal self-attention layer for decoding operations. At the same time, a [Decode] marker is added to the text to indicate the beginning of the sequence, and the [EOS] marker indicates the end of the sequence.

[0099] It should be noted that the unimodal encoder, the image-based text encoder and the image-based text decoder can all be applied to the learning process of the image-text pair during the training process of the initial pre-training model. In addition, in the embodiment of the present invention, the above structure can also be used to optimize the initial medical image-text sample data.

[0100] In the embodiment of the present invention, the image-based text encoder can be used as a "filter", and the image-based text decoder can be used as a "descriptor", so as to realize the data bootstrapping process to solve the problem of data noise in the initial medical image and text sample data. In this embodiment, when the image-based text encoder can be used as a "filter" and the image-based text decoder can be used as a "descriptor", they are initialized from the same pre-trained initial model and then fine-tuned on high-quality manually annotated image and text pairs. The descriptor generates corresponding synthetic descriptive text information based on a given medical image, and the filter can be used to determine whether the medical image and text pairs obtained by automatically extracting text, or the description information generated for the medical image by the descriptor matches, and if not, the corresponding image and text pairs are filtered out. Therefore, this embodiment is convenient for generating high-quality training samples, and can solve the problem of limited sample quantity, thereby improving the accuracy of subsequent applications.

[0101] See also Figure 3 In another embodiment of the present invention, a medical image report generating device is provided, which is characterized by comprising:

[0102] An acquisition unit 301 is used to acquire a target medical image;

[0103] A processing unit 302 is used to process the target medical image based on a target medical image semantic association model to obtain multimodal topic distribution information corresponding to the target medical image, wherein the target medical image semantic association model is obtained by multimodal topic modeling based on a target training sample, wherein each image-text pair in the target training sample is an information pair of a medical image and text information, and the multimodal topic includes at least a modality-shared topic and a modality-related topic, wherein the modality-shared topic represents a topic in which the medical image and text information in the image-text pair correspond to the same topic, and the modality-related topic represents a topic in which the medical image and text information in the image-text pair correspond to different topics;

[0104] A generating unit 303, configured to generate multi-level text description information of the target medical image based on the multimodal topic distribution information;

[0105] The optimization unit 304 is used to optimize the multi-level text description information to obtain a medical image report corresponding to the target medical image.

[0106] In one embodiment, it further includes:

[0107] A sample acquisition unit, used to acquire initial training samples;

[0108] The sample optimization unit is used to optimize the initial training sample to obtain a target training sample:

[0109] A first determination unit is used to determine the image-text pairs with the target neighborhood structure in the target training sample as multimodal samples;

[0110] A calculation unit, used to calculate multimodal topic similarity for the multimodal samples respectively, to obtain modal shared topics and modal related topics;

[0111] A second determining unit, configured to determine the modality shared topics and the modality related topics as high-level semantic feature descriptors;

[0112] An analysis unit, configured to perform semantic association analysis based on the processing logic in the medical image report and the high-level semantic feature descriptor to obtain semantic association information of the medical image;

[0113] The machine learning unit is used to perform machine learning on the medical image semantic association information to obtain a target medical image semantic association model.

[0114] Furthermore, the sample optimization unit is specifically used for:

[0115] Performing training based on the initial training samples to obtain an initial model, wherein the initial model is a model capable of determining a matching relationship between medical images and text information, and the initial model includes a unimodal encoder, an image-based text encoder, and an image-based text decoder;

[0116] Encoding the medical image and text information corresponding to each image-text pair in the initial training sample by the single-modal encoder to obtain image encoding information and text encoding information;

[0117] Adjusting the image-based text decoder based on the manually annotated image-text pairs to obtain a descriptor;

[0118] Using the descriptor to generate text description information of the first medical image without manually annotated text information in the initial training sample, to obtain a first type of image-text pairs, wherein the first type of image-text pairs are image-text pairs corresponding to the medical image and the text description information generated by the descriptor;

[0119] Adjusting the image-based text encoder based on the manually annotated image-text pairs to obtain a filter;

[0120] Filtering the first type of image-text pairs and the second type of image-text pairs using the filter to obtain a third type of image-text pairs, wherein the second type of image-text pairs are image-text pairs consisting of medical images without manually annotated text information and corresponding automatically extracted text information, and the third type of image-text pairs include the filtered first type of image-text pairs and the filtered second type of image-text pairs;

[0121] Target medical image-text sample data is generated based on the manually annotated image-text pairs and the third type of image-text pairs.

[0122] In one embodiment, the generating unit comprises:

[0123] A first determination subunit, configured to determine an association probability between each topic and the target medical image based on the multimodal topic distribution information;

[0124] A second determination subunit, configured to determine a target topic from among the topics corresponding to the multimodal topic distribution based on the association probability;

[0125] A generating subunit, configured to determine the target subject to generate supervisory information for describing the text, and generate a text sentence corresponding to the target medical image based on a hierarchical long short-term memory network and a visual attention mechanism;

[0126] The processing subunit is used to process the text sentence to obtain multi-level text description information.

[0127] Optionally, the optimization unit is specifically used for:

[0128] Based on the display visualization information and the image report description mode, the multi-level text description information is optimized to obtain a medical image report corresponding to the target medical image.

[0129] The embodiment of the present invention provides a medical image report generation device, comprising: an acquisition unit acquires a target medical image; a processing unit processes the target medical image based on a target medical image semantic association model to obtain multimodal topic distribution information corresponding to the target medical image; a generation unit generates multi-level text description information of the target medical image based on the multimodal topic distribution information; an optimization unit optimizes the multi-level text description information to obtain a medical image report corresponding to the target medical image. The present invention implements modeling of potential semantic associations of medical images with related topics of modal complementarity based on the target medical image semantic association model, can interpret the semantics represented by the medical image, and realizes the generation of objective and accurate text information based on the multi-level text information description, thereby improving the accuracy of automatic generation of medical image reports.

[0130] Based on the aforementioned embodiments, an embodiment of the present invention provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any of the above medical imaging report generation methods.

[0131] An embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of a method for generating a medical imaging report are implemented when the processor executes the program.

[0132] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0133] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0134] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a medical imaging report, characterized in that: include: Acquire target medical images; The target medical image is processed based on a target medical image semantic association model to obtain multimodal topic distribution information corresponding to the target medical image, wherein the target medical image semantic association model is obtained by multimodal topic modeling based on a target training sample, and the target training sample is obtained by optimizing an initial training sample; each image-text pair in the target training sample is an information pair of a medical image and text information, and the multimodal topic includes at least a modality-shared topic and a modality-related topic, wherein the modality-shared topic represents the topic corresponding to the same medical image and text information in the image-text pair, and the modality-related topic represents the topic corresponding to the medical image and text information in the image-text pair The information corresponds to different topics; wherein the process of optimizing the initial training sample to obtain the target training sample includes: training based on the initial training sample to obtain an initial model, wherein the initial model is a model for determining the matching relationship between medical images and text information, and the initial model includes a unimodal encoder, an image-based text encoder, and an image-based text decoder; encoding the medical image and text information corresponding to each image-text pair in the initial training sample by the unimodal encoder to obtain image encoding information and text encoding information; adjusting the image-based text decoder based on the manually annotated image-text pairs to obtain a descriptor; The descriptor is used to generate text description information of the first medical image without manually annotated text information in the initial training sample to obtain a first type of image-text pairs, wherein the first type of image-text pairs are image-text pairs corresponding to the medical image and the text description information generated by the descriptor; the image-based text encoder is adjusted based on the manually annotated image-text pairs to obtain a filter; the first type of image-text pairs and the second type of image-text pairs are filtered by the filter to obtain a third type of image-text pairs, wherein the second type of image-text pairs are image-text pairs consisting of medical images without manually annotated text information and the corresponding automatically extracted text information, and the third type of image-text pairs include the filtered first type of image-text pairs and the filtered second type of image-text pairs; target training samples are generated according to the manually annotated image-text pairs and the third type of image-text pairs; Based on the multimodal topic distribution information, generating multi-level text description information of the target medical image; The multi-level text description information is optimized to obtain a medical image report corresponding to the target medical image.

2. The method according to claim 1, characterized in that Also includes: Get initial training samples; The initial training sample is optimized to obtain a target training sample: Determining the image-text pairs with the target neighborhood structure in the target training samples as multimodal samples; Performing multimodal topic similarity calculations on the multimodal samples respectively to obtain modality-shared topics and modality-related topics; determining the modality-shared topics and the modality-related topics as high-level semantic feature descriptors; Performing semantic association analysis based on the processing logic in the medical image report and the high-level semantic feature descriptor to obtain medical image semantic association information; Machine learning is performed on the medical image semantic association information to obtain a target medical image semantic association model.

3. The method according to claim 1, characterized in that The step of generating multi-level text description information of the target medical image based on the multimodal topic distribution information includes: Determining, based on the multimodal topic distribution information, an association probability between each topic and the target medical image; Based on the association probability, determining a target topic from each topic corresponding to the multimodal topic distribution; Determine the target subject to generate supervision information for describing the text, and generate a text sentence corresponding to the target medical image based on a hierarchical long short-term memory network and a visual attention mechanism; The text sentences are processed to obtain multi-level text description information.

4. The method according to claim 1, characterized in that: The step of optimizing the multi-level text description information to obtain a medical image report corresponding to the target medical image includes: Based on the display visualization information and the image report description mode, the multi-level text description information is optimized to obtain a medical image report corresponding to the target medical image.

5. A medical imaging report generating device, characterized in that: include: An acquisition unit, used for acquiring a target medical image; A processing unit, used to process the target medical image based on a target medical image semantic association model to obtain multimodal topic distribution information corresponding to the target medical image, wherein the target medical image semantic association model is obtained by multimodal topic modeling based on a target training sample, and the target training sample is obtained by optimizing an initial training sample; each image-text pair in the target training sample is an information pair of a medical image and text information, and the multimodal topic includes at least a modality-shared topic and a modality-related topic, wherein the modality-shared topic represents the topic corresponding to the same medical image and text information in the image-text pair, and the modality-related topic represents the medical image in the image-text pair Different topics correspond to the text information; wherein the process of optimizing the initial training sample to obtain the target training sample includes: training based on the initial training sample to obtain an initial model, the initial model is a model for determining the matching relationship between the medical image and the text information, and the initial model includes a unimodal encoder, an image-based text encoder and an image-based text decoder; encoding the medical image and text information corresponding to each image-text pair in the initial training sample by the unimodal encoder to obtain image encoding information and text encoding information; adjusting the image-based text decoder based on the manually annotated image-text pairs to obtain a descriptor; The descriptor is used to generate text description information of the first medical image without manually annotated text information in the initial training sample to obtain a first type of image-text pairs, wherein the first type of image-text pairs are image-text pairs corresponding to the medical image and the text description information generated by the descriptor; the image-based text encoder is adjusted based on the manually annotated image-text pairs to obtain a filter; the first type of image-text pairs and the second type of image-text pairs are filtered by the filter to obtain a third type of image-text pairs, wherein the second type of image-text pairs are image-text pairs consisting of medical images without manually annotated text information and the corresponding automatically extracted text information, and the third type of image-text pairs include the filtered first type of image-text pairs and the filtered second type of image-text pairs; target training samples are generated according to the manually annotated image-text pairs and the third type of image-text pairs; A generating unit, configured to generate multi-level text description information of the target medical image based on the multimodal topic distribution information; The optimization unit is used to optimize the multi-level text description information to obtain a medical image report corresponding to the target medical image.

6. The device according to claim 5, characterized in that Also includes: A sample acquisition unit, used to acquire initial training samples; The sample optimization unit is used to optimize the initial training sample to obtain a target training sample: A first determining unit, configured to determine the image-text pairs having a target neighborhood structure in the target training sample as multimodal samples; A calculation unit, used for performing multimodal topic similarity calculation on the multimodal samples respectively to obtain modality shared topics and modality related topics; A second determining unit, configured to determine the modality shared topics and the modality related topics as high-level semantic feature descriptors; An analysis unit, configured to perform semantic association analysis based on the processing logic in the medical image report and the high-level semantic feature descriptor to obtain semantic association information of the medical image; The machine learning unit is used to perform machine learning on the medical image semantic association information to obtain a target medical image semantic association model.

7. The device according to claim 5, characterized in that The generating unit comprises: A first determination subunit, configured to determine an association probability between each topic and the target medical image based on the multimodal topic distribution information; A second determination subunit, configured to determine a target topic from among the topics corresponding to the multimodal topic distribution based on the association probability; A generating subunit, configured to determine the target subject to generate supervisory information for describing the text, and generate a text sentence corresponding to the target medical image based on a hierarchical long short-term memory network and a visual attention mechanism; The processing subunit is used to process the text sentence to obtain multi-level text description information.

8. The device according to claim 5, characterized in that The optimization unit is specifically used for: Based on the display visualization information and the image report description mode, the multi-level text description information is optimized to obtain a medical image report corresponding to the target medical image.

Images