A medical image report generation method based on fine-grained attention

Abstract

A medical image report generation method based on fine-grained attention, comprising: first, extracting teeth and symptom areas in the original X-ray panoramic picture through a pre-trained yolov5 target detection model; extracting features of the original X-ray panoramic picture through a denseNet-121 convolutional neural network, and fusing the features with candidate disease label texts through cross-modal attention to obtain coarse-grained global cross-modal representation. Subsequently, a geometric connection graph is constructed based on the geometric position distribution of the teeth and the symptom areas, so that the teeth and the corresponding lesion areas are aligned in space. Then, the region image features are fused according to the belonging relationship represented by the geometric connection graph through graph attention to obtain fine-grained tooth symptom representation. Finally, the coarse-grained global cross-modal features and the fine-grained tooth symptom representation are input into a transformer bridge tower text generator with two layers of cross-modal attention and controlled by a sentence state forgetting gate to generate a final report text.

Description

TECHNICAL FIELD

[0001] The present application relates to a medical image report generation method based on fine-grained attention, in particular to a medical image report generation method in the field of dentistry. The present application first detects the tooth region and disease region existing in the image by using the yolov5 target detection method, aligns the tooth and the lesion region thereon by establishing the geometric connection graph between the regions through the relative position, and then uses the self-attention and graph attention mechanism to model the fine-grained object-level image representation. Then, the object-level image representation and the global-level image representation fused with the global-level text modal features extracted by the convolutional neural network are input into the transformer bridge text generator with two layers of cross-modal attention and controlled by a sentence state forgetting gate to generate the final report text. The present application relates to the fields of deep learning, oral medicine, medical image, etc., and specifically relates to the multi-modal field based on deep learning. BACKGROUND

[0002] Medical image data contains rich health information of patients, and is a key basis for early screening, differential diagnosis and treatment of various diseases in clinic. However, with the rapid development of medical image technology, medical image data increases exponentially, which brings great diagnostic pressure and report writing load to clinicians and radiologists. Radiologists need to carefully examine each image and write a full-text report to describe the results. Diagnosing medical images requires radiologists to provide an appropriate amount of experience to develop more confident and accurate reports. In addition, a more compelling problem is the time required for radiologists to write full-text reports. According to the experience level of radiologists, it takes an average of 10 minutes or more. Therefore, many researchers have begun to devote themselves to the research of using machine learning technology to automatically generate results and descriptions from medical images to reduce the burden of doctors.

[0003] In recent years, with the rapid development of artificial intelligence technology, the application of deep learning in the medical field has greatly increased, driving a large number of research on deep learning-based image report automatic generation. However, there are still several problems in using deep learning technology to realize the image report generation in the field of oral medicine.

[0004] (1) At present, many methods match the corresponding template sentences by classifying the image features through a retrieval method, and then combine the template sentences to form a complete report. However, this retrieval method cannot produce language diversified descriptions and depict rare but prominent medical findings, and the quality of the generated text is highly dependent on the manually constructed template sentence library, which is difficult to cope with the changing pathological conditions in the actual environment.

[0005] (2) Compared with general images, oral medical images have more similar objects in the picture and higher similarity, and the objects are arranged densely, which has high local specificity. It is necessary to detect the fine-grained diseases in the picture by taking each tooth as a unit. This leads to the fact that the current mainstream medical image report generation method cannot accurately generate fine-grained reports.

[0006] (3) Since the diseases in the oral medical image are anchored by teeth, there is inevitably a problem of label scope in image coding and text generation: if the disease is bound to the wrong tooth, even if the disease itself is detected correctly, the diagnosis is chaotic. How to generate a correct and orderly report is a challenge.

[0007] In the field of medical image report generation, in order to convert the disease-related features captured from the image into report text generation, the current mainstream method is to use CNN-RNN structure. First, use convolutional neural networks such as ResNet to extract overall features from the image. Then, input these image features as intermediate layers to extract additional auxiliary features, such as using a classifier to preliminarily classify diseases, introducing a knowledge graph as prior knowledge, etc. Finally, the extracted image features and these auxiliary features are fused into multi-modal features, which are input into a generator such as LSTM to generate text across modalities.

[0008] The above method has made some effective progress in the field of medical image report generation, but it has ignored the fine-grained features of medical images and text data in the dental field. Fine-grained features refer to the fact that disease labels in the dental field are bound to specific teeth where the disease is located, and the image features of the disease are highly related to the local image features of the tooth where the lesion is located, but are irrelevant to the surrounding teeth. This makes it difficult for traditional medical image report generation models to extract fine-grained local image features and diagnose diseases. In addition, the errors caused by not controlling the scope of the extracted fine-grained features are devastating, which will lead to chaotic diagnosis results and loss of practical significance. Therefore, the present application constructs a new medical image report generation method based on fine-grained hierarchical attention to extract fine-grained features of the image and generate accurate and orderly text according to the fine-grained features.

[0009] Therefore, how to solve the problem of fine-grained medical image data and label scope in the dental field is a difficult problem that needs to be solved urgently in the field of medical image report generation in today's big data era. SUMMARY

[0010] In order to solve the problem that the prior art cannot extract fine-grained features of medical images and control the label scope during feature fusion and text generation, the present application proposes a medical image report generation method based on fine-grained attention.

[0011] On the basis of extracting global image features by DenseNet-121 and cross-modal fusion of global text label features, the application can use attention mechanism and target detection results to extract fine-grained local image features and use relative position relationship to establish the geometric relationship graph between targets to control the label scope. Finally, the global level image features and fine-grained object level image features are input into a transformer bridge text generator with two layers of cross-modal attention and controlled by a sentence state forgetting gate to generate the final report text. Compared with the existing methods, the method can achieve more advanced performance.

[0012] To solve the above problems, the technical scheme provided by the application is:

[0013] A medical image report generation method based on fine-grained attention, comprising the following steps:

[0014] Step 1: Use DenseNet-121 convolutional neural network to extract features of the whole image, and use yolov5 target detection tool to detect tooth and lesion target frames in the image and extract local image features of each target.

[0015] Step 2: Use Bert to obtain word embedding of candidate disease labels, input global image features and label word embedding into cross-modal mutual attention encoder to encode cross-modal global level image representation.

[0016] Step 3: According to the relative position of teeth and lesions, a geometric relationship graph is constructed to bind related lesion areas and tooth areas and constrain the coding scope of lesion areas.

[0017] Step 4: Add the fine-grained image features of the lesion area and the corresponding label word embedding to obtain the cross-modal fine-grained object level disease image representation.

[0018] Step 5: According to the position coordinates of tooth targets and lesion target frames in the image, encode the absolute position and relative position embedding of the target, embed the respective position information into the fine-grained object level disease image representation and fine-grained object level tooth image representation, and then use geometric self-attention mechanism to encode the mutual relationship between tooth objects and the mutual relationship between lesion objects.

[0019] Step 6: According to the geometric relationship graph, use graph mutual attention mechanism to encode the related fine-grained object level disease image representation and fine-grained object level tooth image representation obtained in the last step to model the detailed relationship between related disease areas and tooth areas, and then obtain fine-grained object level image features through feedforward neural network.

[0020] Step 7: input the global-level image features and the fine-grained object-level image features into a transformer bridge text generator with two layers of cross-modal attention and controlled by a sentence state forgetting gate to generate text. The generator adds two levels of cross-modal attention modules on the basis of a general transformer: a sentence-level cross-modal attention module first applies cross-modal attention to the global-level image features as a whole to assign weights to local regions on the global image; then a word-level cross-modal attention module further applies cross-modal attention to the fine-grained object-level image features to determine which tooth object information needs to be specifically expressed. In addition, the sentence state forgetting gate control unit assigns a forgetting weight to the currently input fine-grained object-level image features according to the sentence state in which the generator is currently located and the region weight focused on by the sentence-level cross-modal attention module, controls the tooth object specifically focused on in the current sentence, and restricts the decoding scope of the disease label.

[0021] Step 8: the generator iteratively generates the final text.

[0022] Further, the convolutional neural network in step 1 includes ResNet, AlexNet, etc.; and the target detection model includes Faster-RCNN, YOLO, etc.

[0023] Further, the word embedding acquisition model in step 2 includes Word2vec, Glove, etc.

[0024] The present application introduces a position information encoded attention mechanism to fuse the fine-grained relationship information between the tooth region and the disease region, and uses it together with the global cross-modal image features to enhance the fine-grained representation of the image. Furthermore, the present application constructs a geometric connection graph based on the geometric position relationship between regions, and uses a graph mutual attention mechanism to model the detailed relationship between related disease regions and tooth regions, and combines a text generator with a hierarchical cross-modal attention structure and a sentence state forgetting gate control unit to restrict the encoding and decoding scope of the disease label. Compared with previous methods, the present application has certain improvement in accuracy and efficiency.

[0025] The advantages of the present application are:

[0026] 1. The present application introduces a position information encoded attention mechanism to fuse the fine-grained relationship information between the tooth region and the disease region, and uses it together with the global cross-modal image features to enhance the fine-grained representation of the image, solving the defect that the traditional medical image report generation method cannot capture fine-grained image features and object relationships.

[0027] 2. The application is based on the geometric position relationship between regions to construct a geometric connection graph, and uses a graph mutual attention mechanism to model the detailed relationship between related disease regions and tooth regions. Combined with a text generator with a hierarchical cross-modal attention structure and a sentence state forgetting gate control unit, the encoding and decoding scope of the disease label can be effectively constrained, and finally accurate and orderly medical report text can be generated. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is the overall flowchart of the application.

[0029] Figure 2 is the overall architecture diagram of the application. DETAILED DESCRIPTION

[0030] The technical solutions of the application will be further described below with reference to the drawings.

[0031] A medical image report generation method based on fine-grained attention, comprising the following steps:

[0032] Step 1: compress the oral X-ray panoramic picture to 512*512 size, then use DenseNet-121 convolutional neural network to extract the whole picture features, and use yolov5 target detection tool to detect the tooth target frame in the picture and the lesion target frame For each tooth and lesion target frame, the subgraph image features are also extracted using the DenseNet-121 convolutional neural network. Finally, the whole image features V, tooth subgraph feature set lesion subgraph feature set tooth subgraph position set and lesion subgraph position set and lesion corresponding disease keyword embedding Where i and j represent the number of tooth objects and the number of lesion objects in the image. Wherein

[0033] Step 2: use Bert to obtain the word embedding of the candidate disease label Input the global image features and label word embedding into the cross-modal mutual attention encoder to encode the cross-modal global image representation. The cross-modal mutual attention encoder consists of two multi-head mutual attention modules. The first multi-head mutual attention module takes V as Query, W des as Key and Value, which is used to assign weights to possible disease labels according to global image features, to obtain the encoded label weight The second multi-head mutual attention module takes As Query, V as Key and Value, used to assign weights to each region of the global image according to the encoded label weights, to get the focus region in the image Finally, normalize and add them to get the cross-modal global-level image representation V' pic The whole process can be represented by the following formula:

[0034]

[0035]

[0036]

[0037] Where MHA represents the multi-head attention mechanism:

[0038]

[0039] MHA(X, Y) = [Att1(X, Y);...; Att n (X, Y)]W O (5)

[0040] Where W Q , W K , W V , are learnable parameter matrices.

[0041] Step 3: According to the relative position of teeth and lesions, a geometric relationship graph G is constructed to bind the relevant lesion regions and tooth regions and constrain the encoding scope of the lesion regions. Specifically, an undirected graph is constructed with the elements in the tooth object set E tooth and the lesion object set E des as nodes. For each lesion object , find the nearest tooth object according to its spatial coordinates and connect a edge between them.

[0042] Step 4: Add the fine-grained image features of the lesion region and the corresponding label word embedding to get the cross-modal fine-grained object-level disease image representation

[0043]

[0044] Step 5: Encode the absolute position and relative position of the targets according to the position coordinates of the tooth target and lesion target boxes in the figure, embed the respective position information into the fine-grained object-level disease image representation and fine-grained object-level tooth image representation, and then use geometric self-attention mechanism to encode the mutual relationship between tooth objects and the mutual relationship between lesion objects respectively. Get tooth object self-attention representation and lesion object self-attention representation

[0045] Define the position coordinates of each target box as B i = (x, y, w, h), where x, y, w, h represent the center horizontal coordinate, the center vertical coordinate, the width and the height of the target box respectively. Thus we obtain the absolute position encoding (RPE) of each target box:

[0046] RPE(i) = B i W emb (7)

[0047] where is an absolute position embedding parameter matrix.

[0048] For each two target boxes B i and B j , their relative geometric position relationship can be represented by a four-tuple as:

[0049]

[0050] Thus we obtain the relative position encoding of each target box <B i , B j >:

[0051] RltPE(i, j) = Ω(i, j)W emb′ (9)

[0052] where is a relative position embedding parameter matrix.

[0053] Define the geometric self-attention mechanism (MHCSA) as:

[0054] MHCSA(X, Y) = [CSAtt1(X, Y);...; CSAtt n (X, Y)]W O (10)

[0055]

[0056] where, W Q , W K , W V , are learnable parameter matrices.

[0057] Step 6: According to the geometric relationship graph G, use the graph mutual attention mechanism to encode the relevant fine-grained tooth object self-attention representation obtained in the last step and the lesion object self-attention representation Model the detailed relationship between the relevant disease area and the tooth area, and then obtain the fine-grained object-level image feature V' through the feedforward neural network obj .

[0058] The graph mutual attention mechanism (MHGCA) aims to enhance the representation of the source node by embedding the information of the target node into the source node. Specifically, for each tooth node in the geometric relationship graph G , apply the attention mechanism to encode the information of all disease nodes connected to it. For disease nodes not connected to it, set their attention weights to 0. Define the graph mutual attention mechanism as follows:

[0059] MHGCA(X, Y) = [GCAtt1(X, Y);...; GCAtt n (X, Y)]W O (12)

[0060]

[0061] where W Q , W K , W V , are learnable parameter matrices.

[0062] Step 7: Combine the global-level image feature V' pic and the fine-grained object-level image feature V' objThe input is a Transformer Bridge Tower text generator (Region Control Generator) with two layers of cross-modal attention and controlled by a sentence-state forget gate. The generator adds two levels of cross-modal attention modules to a typical Transformer: the sentence-level cross-modal attention module first applies cross-modal attention to the global image features, assigning weights to local regions on the global image; then, the word-level cross-modal attention module further applies cross-modal attention to fine-grained object-level image features, determining which specific tooth object's information needs to be described. The sentence-state forget gate control unit assigns a forgetting weight to the current input fine-grained object-level image features based on the generator's current sentence state and the weights of the regions currently focused on by the sentence-level cross-modal attention module, controlling the specific tooth object of interest in this sentence and constraining the decoding scope of the disease label. Finally, the hidden layer output is fed into a linear layer and SOFIMAX to predict the next word.

[0063] Specifically, for each decoding time step t, RCG will use the current input word x t =w t +e t As input (w t : Word embedding, e t (Location embedding):

[0064] h t =MHA(x t x t-1 (14)

[0065] Then, h t and global image features V′ pic Fine-grained object-level image features V′ obj and the current sentence state. t Input hierarchy scope control for cross-modal attention module:

[0066]

[0067] Wherein, the current sentence state is State. t This refers to separating the output text using periods, with words within the same sentence assigned the same sentence state value. From this, we obtain the Sentence State Encoding (SSE) for each word at each time step:

[0068] SSE(x t ) = State t W sen (16)

[0069] in It is a sentence state embedding parameter matrix.

[0070] The hierarchical scope control cross-modal attention module contains two cross-modal attention modules: sentence-level cross-modal mutual attention module and scope control word-level cross-modal mutual attention module (rcWMHCA). Specifically, h t and V′ pic First, the input sentence-level cross-modal mutual attention module is executed to perform cross-modal attention:

[0071] h′ t = MHA(h t , V′ pic ) (17)

[0072] Subsequently, by adding the sentence state encoding to the sentence-level mutual attention layer output h t ′, the sentence state enhanced current hidden layer representation is obtained, and then a sigmoid layer is output to obtain the scope control threshold:

[0073] gate t = σ((h′ t +SSE(x t ))W a ) (18)

[0074] Subsequently, h′ t , gate t and V′ obj are input into the scope control word-level cross-modal mutual attention module to encode the fine-grained mutual attention representation:

[0075] h ″ t = rcWMHCA(h′ t , V′ obj , gate t ) (19)

[0076] The scope control word-level cross-modal mutual attention is defined as follows:

[0077] rcWMHCA(X, Y, τ) = [rcWCAtt1(X, Y, τ);...; rcWCAtt n (X, Y, τ)]W O (20)

[0078]

[0079] Finally, h is passed to the FFN and linear layer to predict the next word:

[0080]

[0081] Step 8: The generator iteratively generates the final text.

[0082] The present application mainly solves the problems of fine-grained feature extraction of medical images and label scope in feature fusion and text generation which cannot be solved by the prior art. A medical image report generation method based on fine-grained attention is proposed. On the basis of extracting global image features by DenseNet-121 and cross-modal fusion of global text label features, the attention mechanism and target detection results are used to extract fine-grained local image features, and the geometric relationship graph between targets is established by using the relative position relationship to control the label scope. Finally, the global level image features and fine-grained object level image features are input into the transformer bridge tower text generator with two layers of cross-modal attention and controlled by a sentence state forgetting gate to generate the final report text.

[0083] The content described in the embodiments of the present specification is only a list of implementation forms of the inventive concept, and the protection scope of the present application should not be regarded as being limited to the specific forms stated in the embodiments, and the protection scope of the present application also extends to equivalent technical means that can be thought of by those skilled in the art according to the inventive concept.

Claims

1. A medical image report generation method based on fine-grained attention, comprising the following steps: Step 1: compress the oral X-ray panoramic picture to 512*512 size, then use DenseNet-121 convolutional neural network to extract the whole picture features, and use yolov5 target detection tool to detect the tooth target box in the picture and the lesion target box For each tooth and lesion target box, the image features of the subgraph are also extracted using the DenseNet-121 convolutional neural network; finally, the whole image features V, the tooth subgraph feature set the lesion subgraph feature set the tooth subgraph position set and the lesion subgraph position set and the lesion corresponding disease keyword embedding Where i, j represent the number of tooth objects and the number of lesion objects in the image; wherein Step 2: Obtain word embeddings of candidate disease labels using BERT Input the global image features and label word embeddings into the cross-modal mutual attention encoder to encode the cross-modal global-level image representation; The cross-modal mutual attention encoder consists of two multi-head mutual attention modules, the first multi-head mutual attention module takes V as Query, W des as Key and Value, to assign weights to possible disease labels according to global image features, to get the encoded label weights The second multi-head mutual attention module takes as Query, V as Key and Value, to assign weights to each region of the global image according to the encoded label weights, to get the key regions in the image Finally, normalize and add them to get the cross-modal global-level image representation V' pic The whole process can be represented by the following formula: Wherein, MHA represents a multi-head attention mechanism: MHA(X,Y) = [Att1(X,Y);... ; Att n (X,Y)]W O (5) wherein, is a learnable parameter matrix; Step 3: According to the relative position of teeth and lesions, a geometric relationship graph G is constructed, and the relevant lesion area and tooth area are bound to constrain the coding scope of the lesion area; Step 4: Add the fine-grained image features of the lesion area and the corresponding label word embedding to obtain a cross-modal fine-grained object-level disease image representation Step 5: encode the absolute position and relative position of the target according to the position coordinates of the tooth target and the lesion target frame in the figure, embed the respective position information, add the fine-grained object-level disease image representation and the fine-grained object-level tooth image representation, and then use geometric self-attention mechanism to encode the mutual relationship between tooth objects and the mutual relationship between lesion objects respectively; obtain tooth object self-attention representation and lesion object self-attention representation Let B denote the position coordinates of each bounding box i = (x, y, w, h), where x, y, w, h represent the center horizontal coordinate, the center vertical coordinate, the width and the height of the bounding box, respectively; thus we obtain the absolute position encoding RPE of each bounding box: RPE(i) = B i W emb (7) wherein is an absolute position embedding parameter matrix; For each two target boxes B i and B j , their relative geometric position relationship can be represented by a quadruple as: Thus we obtain the relative position encoding of each target box pair <B i ,B j > RltPE(i,j) = Ω(i,j) W emb′ (9) wherein is a relative position embedding parameter matrix; Define the geometric self-attention mechanism MHCSA as: MHCSA(X,Y) = [CSAtt1(X,Y);... ; CSAtt n (X,Y)]W O (10) wherein, is a learnable parameter matrix; Step 6: encode the relevant fine-grained tooth object self-attention representation obtained in the last step using the graph mutual attention mechanism according to the geometric relationship graph G and the lesion object self-attention representation Model the detailed relationship between the relevant disease area and the tooth area, and then obtain the fine-grained object-level image feature V' through the feedforward neural network obj The graph mutual attention mechanism MHGCA aims to enhance the representation of the source node by embedding the information of the target node into the source node; Step 7: global-level image features V' pic and fine-grained object-level image features V' obj The transformer-based text generator Region ControlGenerator with two layers of cross-modal attention and controlled by a sentence state forget gate is inputted to generate text; the generator is based on the general transformer and adds two levels of cross-modal attention modules: the sentence-level cross-modal attention module first applies cross-modal attention to the global-level image features as a whole, assigning weights to local regions on the global image; then the word-level cross-modal attention module further applies cross-modal attention to the fine-grained object-level image features, determining which tooth object information needs to be specifically expressed; the sentence state forget gate control unit assigns a forget weight to the currently input fine-grained object-level image features according to the sentence state in which the current time step of the generator is located and the region weight to which the sentence-level cross-modal attention module is currently paying attention, controls the tooth object to which the current sentence is specifically paying attention, and restricts the decoding scope of the disease label; finally, the hidden layer output is sent to a linear layer and a softmax to predict the next word; Step 8: The generator iteratively generates the final text.

2. The method of claim 1, wherein the method is a fine-grained attention-based medical image reporting generation method. Step 3 specifically comprises: constructing an undirected graph with the elements of the set of tooth objects E tooth and the set of lesion objects E des as nodes; for each lesion object finding the closest tooth object based on their spatial coordinates and connecting an edge between them.

3. The method of claim 1, wherein the method is based on fine-grained attention. The graph mutual attention mechanism MHGCA described in step 6 specifically includes: for each tooth node in the geometric relationship graph G all disease nodes connected thereto applying an attention mechanism to encode their information; for disease nodes not connected thereto, setting their attention weight to 0; the graph mutual attention mechanism is defined as follows: MHGCA(X,Y) = [GCAtt1(X,Y);... ; GCAtt n (X,Y)]W O (12) wherein, is a learnable parameter matrix.

4. The method of claim 1, wherein the method is based on fine-grained attention. Step 7 specifically includes: for each decoding time step t, the RCG takes the current input word x t = w t + e t as input, w t : a word embedding, e t : a position embedding; h t = MHA(x t ,x t-1 ) (14) Then, h t and global-level image features V′ pic , fine-grained object-level image features V′ obj and the current sentence state State t Input hierarchical role scope control cross-modal attention module HrcWMHCA: where the current sentence state State t is the sentence state of the current word. We obtain the sentence state encoding SSE of each time step word by separating the output text by periods and assigning the same sentence state value to the words within the same sentence. SSE(x t ) = State t W sen (16) wherein is a sentence state embedding parameter matrix; The hierarchical scope control cross-modal attention module includes two cross-modal attention modules: a sentence-level cross-modal mutual attention module and a scope control word-level cross-modal mutual attention module rcWMHCA; specifically, h t and V' pic First, the sentence-level cross-modal mutual attention module is input to perform cross-modal attention: h' t = MHA(h t , V pic ) (17) Subsequently, by adding the sentence state code to the sentence-level mutual attention layer output ht', the sentence state enhanced current hidden layer representation is obtained, and then a sigmoid layer is output to output the scope control threshold: gate t =σ((h′ t +SSE(xt))WG) (18) Subsequently, h' is computed as t , gate t , and V' obj input the scope control word-level cross-modal self-attention module to encode the fine-grained cross-modal self-attention representation: h ″ t = rcWMHCA(h', h) (18) t ,V′ obj ,gate t ) (19) Define the scope control word-level cross-modal mutual attention as follows: rcWMHCA(X,Y,τ)=[rcWCAtt1(X,Y,τ);…;rcWCAttn(X,Y,τ)]WO(20) Finally, the passed to the FFN and linear layer to predict the next word:

5. The method of claim 1, wherein: The convolutional neural network of step 1 includes ResNet, AlexNet; the target detection model includes Faster-RCNN, YOLO.

6. The method of claim 1, wherein: The word embedding acquisition model of step 2 includes Word2vec, Glove.

Images