Medical report generating model based on relation model, and generating method thereof

Abstract

The invention discloses a medical report generating model based on a relation model, and a generating method thereof. The model comprises the components of a deep convolutional network which is used for classifying abnormal terminologies and extracting a visual characteristic, inputs the visual characteristic into a two-stage bidirectional long-and-short-period memory network layer for performingattention operation, and is connected with a full connecting layer after each stage of bidirectional long-and-short-period memory network layer for selecting a template and a word; a hierarchical recursive neural network which comprises two bidirectional long-and-short-period memory network layers, wherein the top bidirectional long-and-short-period memory network layer is operated through attention and is connected with the full connecting layer for selecting an accurate template, and the lower bidirectional long-and-short-period memory network layer receives the information of the top layerand is connected with the full connecting layer for selecting an accurate word; and a report generating module which comprises a template decoder for predicting an abnormal sentence and a word decoderfor generating a normal sentence, thereby selecting generating using template searching or sentence generation on the current sentence according to decision of the hierarchical recursive neural network, and finally connects all sentences which are obtained through searching or automatic generation for forming the medical report.

Description

technical field [0001] The invention relates to the technical field of machine learning, in particular to a relational model-based medical report generation model and a generation method thereof. Background technique [0002] Automatic generation of medical image reports has recently attracted increasing research interest, which has significant potential for simplifying diagnostic procedures and reducing the burden on physicians. Unlike the traditional visual image understanding task of generating a single sentence, generating long, thematically coherent reports describing patient conditions and symptoms poses a more challenging task at the intersection of computer vision and natural language processing. In addition to the difficulties shared with image understanding and visual question answering (VQA) (e.g., fine-grained visual processing and reasoning, bridging visual and linguistic modalities), medical report generation is a long-form narrative consisting of multiple sent...

AI Technical Summary

Problems solved by technology

However, in the absence of other prior knowledge about visual content, such computational visual attention may focus on irrelevant regions.

While some approaches have been proposed to alleviate this problem by resorting to ensemble-based retrieval, hierarchical architectures, and multi-task learning, few consider key entities, topic relations, and paragraph coherence

Images