A medical image report generation method combining language prior knowledge

Abstract

The application provides a medical image report generation method combining language prior knowledge, and belongs to the technical fields of computer vision and text generation. The application comprises a phrase template library construction stage and a report generation stage based on retrieval. The method represents language prior knowledge by a template library constructed by phrases, and generates a medical image report by using a report generation method based on retrieval. The method designs a phrase score combining document frequency and semantic information to select important phrases; in the report generation stage, a visual-language retrieval model is used to query templates matching the input image, and a retrieval-fusion strategy is used to fuse the feature representation of the templates, so as to learn the interaction between different templates; finally, a phrase decoder generates a more accurate report under the guidance of the template features. The method provided by the application solves the problems of semantic information confusion and information degree redundancy caused by sentence templates.

Description

Technical Field

[0001] This invention relates to a method for generating medical image reports that incorporates prior linguistic knowledge, belonging to the fields of computer vision and text generation technology. Background Technology

[0002] Medical text reports typically have fixed vocabulary and grammatical structures, meaning they follow a fixed pattern and represent prior linguistic knowledge within the report. This prior linguistic knowledge provides the decoder with semantic and structural information, thereby guiding the model to generate more professional and accurate text reports.

[0003] Existing retrieval-based report generation methods first select important patterns to build a template library, then retrieve templates that match the images, and finally combine or interpret the retrieved templates to generate the report. These methods use sentences or the entire report as patterns to build the template library. However, this template library can lead to confusion during the retrieval process, reducing the overall performance of the model. This is because medical image reports contain many sentences with different expressions but similar meanings. When both sentences are used as templates, the retrieval model's choice between them is redundant, negatively impacting overall performance. Furthermore, there are cases where a single sentence contains multiple semantic meanings; when one organ is in a normal state while another is in an abnormal state, the sentence will negatively affect the retrieval module.

[0004] Compared to sentences, phrases contain more singular and finer-grained semantic information. Using phrases to build a template library helps mitigate the negative impacts of the two aforementioned issues. Therefore, this invention employs phrases with fine-grained information as templates and designs phrase scores to select key phrases as templates. Summary of the Invention

[0005] The purpose of this invention is to address the semantic confusion and information redundancy issues caused by sentence templates, and to propose a medical image report generation method that incorporates prior linguistic knowledge. This invention includes a phrase template library construction stage and a retrieval-based report generation stage.

[0006] During the template library construction phase, key phrases are selected as templates from the collected phrase set based on phrase scores, and the features of the templates are explicitly stored in an external template feature library.

[0007] The retrieval-based report generation stage comprises a visual feature extractor and three sub-modules: a template feature library dynamic update module, a visual-language retrieval module, and a phrase decoding module. The template feature library dynamic update module progressively updates the template features in the feature library during training to learn better template feature representations and increase the stability of the retrieval model. The visual-language retrieval module searches for relevant template features from the template feature library based on the input medical image and fuses multiple template features to obtain a more suitable template feature representation. The phrase decoding module then expands the matched phrase template features based on visual features and the fused template features to generate the final text report.

[0008] A method for generating medical image reports that incorporates prior linguistic knowledge includes the following steps:

[0009] Step 1: Collect a set of candidate phrases from the training corpus and extract the corresponding N-gram phrase substrings from all reported sentences;

[0010] Where N is the length of each phrase, and the set of phrases of length N is represented as follows:

[0011] Step 2: Score each phrase based on its phrase score.

[0012] The phrase score is shown in the following formula:

[0013]

[0014] The first term represents the phrase frequency term. #,i This represents the number of times the i-th phrase substring appears in the k-th sentence, where K is the total number of occurrences in all sentences of the training corpus.

[0015] The second term is the phrase semantic term, which measures whether the candidate phrase maintains semantic consistency with its corresponding parent sentence. Here, τ represents the balance coefficient, and d(·) is the cosine similarity, used to measure the similarity between phrase features. and p(s,r) represents the encoding features of the i-th phrase r and its corresponding k-th sentence s. λ This is a length penalty term to prevent the construction of a template library that is indistinguishable from a sentence-level template library based on phrase scores. Here, λ represents a hyperparameter that balances the length of candidate phrases.

[0016] Step 3: Perform duplicate removal on candidate phrases. For all candidate phrase substrings within the same parent sentence, nonmaximum suppression is used to retain the candidate phrase with the highest phrase score as the candidate template. For candidate phrases from different parent sentences, the cosine similarity between features is used to determine whether the phrases are duplicated, and then the candidate phrase with the higher phrase score is retained.

[0017] Step 4: For the number T of phrase templates in the template library, determine the number of phrases of different lengths in the template library according to the following formula.

[0018] in, Let N represent the number of phrases of length N in the corpus. In the experiment, N ∈ {3, 4, ..., 12}. Furthermore, if a candidate phrase is a substring of another phrase and has a low phrase score, it is removed, resulting in the final phrase template library.

[0019] Step 5: Template retrieval, specifically: using visual features and template feature library as input, the visual-language retrieval module retrieves features that match the image;

[0020] The visual-language retrieval module is implemented based on an attention mechanism.

[0021]

[0022] Where g represents the attention score map corresponding to the template, and k / and These represent the key vectors corresponding to visual features and template features, respectively, and L1(·) represents the L1 normalization function.

[0023] Step 6: Learn the relationship between the retrieved template and visual features based on the attention mechanism, and fuse them to obtain fused template features.

[0024] Step 7: Update template features. Specifically, learn which template features need to be updated in the previous time step through the forget gate, and learn which parts of the new template features should be added to the template library through the input gate.

[0025] The new template features are obtained by weighted summation of word embeddings in the template using gated linear units;

[0026]

[0027] in, It is the template feature input at the current moment, e i,j Let σ(·) represent the word embedding of the j-th word in the i-th template, σ(·) represent the Sigmoid function, ⊙ represent the Hadamard product, W and V represent the learnable weights, and b and c represent the learnable biases.

[0028] In this process, the new template features are fused with the template features from the previous time step to obtain the final updated template features.

[0029] Step 8: Using visual features and template features as input, generate the final text report through the phrase decoding module.

[0030] Beneficial effects

[0031] This invention proposes a retrieval-based report generation model by representing language prior knowledge through a template library constructed from phrases. Compared with existing technologies, it has the following advantages:

[0032] This invention designs a phrase score that combines document frequency and semantic information to select important phrases, mitigating the semantic confusion and information redundancy problems caused by sentence templates. In the report generation stage, a visual-language retrieval model queries templates that match the input image, and a retrieval-fusion strategy is used to fuse the feature representations of the templates, thereby learning the interactions between different templates. Finally, the phrase decoder generates a more accurate report under the guidance of template features. Attached Figure Description

[0033] Figure 1 This is a schematic diagram illustrating a specific implementation of the medical image report generation method that incorporates prior language knowledge according to the present invention.

[0034] Figure 2 The structural diagram of the template feature library dynamic update module constructed for this invention;

[0035] Figure 3 This is a structural diagram of the visual-language retrieval module constructed in this invention. Detailed Implementation

[0036] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0037] Example

[0038] like Figure 1 As shown, for a given training corpus, important phrases are first selected as templates. Where r i This represents the i-th phrase template, where T is the number of phrase templates in the template library (corresponding to steps 1-4 in the invention description); the template feature library is represented as... At time t during training, the template feature library dynamic update module updates the representation of the feature library at the current time based on the feature library from the previous time step and the word embeddings: Where e t Represents the word embedding at time t, such as Figure 2 As shown (corresponding to step 7 in the invention description). For a given medical image I, its visual features are represented as f / The visual-language retrieval module then uses f / Retrieve the template set that matches the image. The corresponding template features are represented as like Figure 3 As shown (corresponding to steps 5-6 in the invention description). Finally, the phrase decoding module decodes the template features. and visual features f / Generate text report (Corresponding to step 8 in the invention description)

[0039] (1) Construct a phrase template library

[0040] This invention first constructs a phrase template library containing the number of phrase templates T in the template library under the guidance of phrase scores, following steps 1 to 4, and initializes the template features using the BioSentVec model.

[0041] (2) Constructing a retrieval-based report generation network

[0042] The retrieval-based report generation stage of this invention includes a visual feature extractor and three sub-modules: a template feature library dynamic update module, a visual-language retrieval module, and a phrase decoding module.

[0043] Visual feature extractors are composed of convolutional neural networks, and typical convolutional neural networks such as ResNet and DenseNet can be selected.

[0044] The template feature library dynamic update module consists of a forget gate and an input gate. The forget gate and input gate have the same structure, consisting of multiple fully connected layers and non-linear activation functions, as expressed in the formula:

[0045]

[0046]

[0047] in, This represents the template features from the previous time step. W represents the template features input at the current time. Fo and W In V Fo and V In All of these are learnable weights in a fully connected layer, and tanh is a non-linear activation function.

[0048] To ensure the stability of the training process, this invention... and Introduce residual connections to update template features: Z i This represents the residual term, which is the new template feature learned through the forget gate and the input gate. Represents the template features at the current moment after fusion;

[0049] Zi =W(σ(G) Fo )⊙G Fo +σ(G In )⊙G In )+b

[0050] Where W is the learnable weight of the fully connected layer, b is the bias, σ is the Softmax function, and ⊙ is the Hadamard product.

[0051] The visual-language retrieval module adopts an attention structure, consisting of multiple fully connected layers and activation functions, to learn the matching relationship between template features and visual features, and generate an attention map corresponding to the template.

[0052]

[0053] in, `g` represents the matrix transpose operation, and `L1(·)` represents the L1 normalization function. The score in `g` represents the correlation between the template and the input image; the higher the score, the stronger the correlation. Then, the template features are fused based on `g`. in This represents matrix multiplication. The final fused template features are represented as f. RA =FC([f R ;v V ]), where [·; ·] represents a cascading operation.

[0054] The phrase decoding module consists of a BERT encoder, a generation branch, and a diagnostic branch. The generation and diagnostic branches are composed of multiple cascaded fully connected layers. The generation branch predicts word probabilities based on encoded features, while the diagnostic branch learns multi-label classification tasks using visual and template features as input.

[0055] (3) Train the report generation network

[0056] The method described in this invention is trained in an end-to-end manner during the report generation phase, and the overall loss function is: Where λ gen , λ diF and λ ret The coefficient represents the balance between different loss terms.

[0057] Branch generation uses cross-entropy as the loss function.

[0058]

[0059] Among them, y i This represents the index of the i-th word. Let p represent the index of the i-th word obtained from the prediction, p represent the prediction probability, and l represent the report length.

[0060] Calculate the binary cross-entropy loss as the retrieval loss function:

[0061]

[0062] Where T represents the number of templates in the template library. This indicates the actual template annotation obtained based on the phrase template library. It is the calculated template prediction score.

[0063] The diagnostic branch is composed of a binary cross-entropy loss function. optimization

[0064]

[0065] Where C represents the number of disease categories, This indicates whether the j-th disease is positive, with 1 indicating positive and 0 indicating negative. This represents the disease prediction score.

[0066] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the invention. Any modifications, equivalent substitutions, or partial improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating a medical image report incorporating linguistic prior knowledge, characterized by, The method comprises a phrase template library construction stage and a search-based report generation stage; in the template library construction stage, key phrases are selected as templates from the collected phrase set according to the phrase scores, and the features of the templates are stored in an external template feature library; the search-based report generation stage comprises a visual feature extractor and three sub-modules: a template feature library dynamic updating module, a visual-linguistic search module and a phrase decoding module; The method specifically comprises the following steps: Step 1, collecting a candidate phrase set from a training corpus, and extracting corresponding N-gram phrase subsequences from all report sentences; where N is the length of each phrase, and the set of phrases of length N is denoted as Step 2, scoring each phrase according to a phrase score; The phrase score is as shown in the following formula: where the first term represents the phrase frequency term, n #,i represents the number of occurrences of the ith phrase substring in the kth sentence, and K is the number of occurrences of all sentences in the training corpus. The second is the phrase semantic item, which measures whether the candidate phrase is semantically consistent with the corresponding parent sentence, where τ represents the balance coefficient, d(·) is the cosine similarity used to measure the similarity between phrase features, and are the encoded features of the i-th phrase r and the k-th sentence s where the phrase r is located, respectively, p(s, r) λ is the length penalty item, which prevents the construction of a template library that is no different from the sentence-level template library according to the phrase score, where λ represents the hyperparameter for balancing the length of the candidate phrase; Step 3, performing a de-duplication operation on the candidate phrases, for all candidate phrase subsequences of the same parent sentence, the non-maximum suppression is adopted to retain the candidate phrase with the highest phrase score as a candidate template, and for the candidate phrases of different parent sentences, whether the candidate phrases are duplicated is determined according to whether the cosine similarity between the features exceeds a certain threshold, and then the candidate phrase with a higher phrase score is retained; Step 4. For the number of phrase templates T in the template library, determine the number of phrases of different lengths in the template library according to the following formula wherein, denotes the number of all phrases of length N in the corpus, N e {3,4,..., 12}, and, in addition, if a candidate phrase is a substring of other phrases and the phrase score is low, it is removed, resulting in a final phrase template library; Step 5, searching for a template, specifically: taking the visual features and the template feature library as inputs, searching for features matching the image through the visual-linguistic search module; wherein the visual-linguistic search module is realized based on an attention mechanism; Step 6, learning the relationship between the retrieved template and the visual features based on the attention mechanism, and fusing to obtain fused template features; Step 7, updating the template features, specifically: learning which template features need to be updated through a forget gate at the previous moment, and learning which parts of the new template features should be added to the template library through an input gate; The new template features are obtained by weighted summation of the word embeddings in the template through a gated linear unit; wherein, is the template feature input at the current time, e i,j denotes the word embedding of the j-th word in the i-th template, σ(·) represents the Sigmoid function, ⊙ denotes the Hadamard product, W and V represent learnable weights, and b and c represent learnable biases; wherein the new template feature is fused with the template feature of the previous time to obtain the final updated template feature Step 8, taking the visual features and the template features as inputs, generating a final text report through the phrase decoding module.

2. The method of claim 1, wherein the language prior knowledge comprises at least one of a medical terminology, a medical report style, and a medical report structure. The visual feature extractor is composed of a convolutional neural network, and typical convolutional neural networks such as ResNet and DenseNet can be selected.

3. The method of claim 2, wherein the language prior knowledge comprises a medical report template. The template feature library dynamic updating module is composed of a forget gate and an input gate, and the forget gate and the input gate have the same structure and are composed of multiple fully connected layers and nonlinear activation functions, and the formula is represented as: wherein, denotes the template feature at the previous time, denotes the template feature input at the current time, W Fo and W In , V Fo and V In are all learnable weights in the fully connected layer, and tanh is a nonlinear activation function. In order to ensure the stability of the training process, the present application introduces a residual connection update template feature between and : Wherein Z i represents a residual term, that is, a new template feature learned through a forgetting gate and an input gate, represents a template feature at the current moment after fusion. Z i = W(σ(G Fo ) ⊙ G Fo + σ(G In ) ⊙ G In ) + b Wherein, W is the learnable weight of the fully connected layer, b is the bias, σ is the Softmax function, and is the Hadamard product.

4. The method of claim 3, wherein the language prior knowledge comprises at least one of a medical terminology, a medical report style, and a medical report structure. The visual-linguistic search module adopts an attention structure and is composed of multiple fully connected layers and activation functions, learns the matching relationship between the template features and the visual features, and generates an attention map corresponding to the template: wherein, represents a matrix transpose operation, k V and respectively represent the key vector corresponding to the visual feature and the template feature, L1(·) represents an L1 normalization function, g represents an attention score map corresponding to the template, the score in g represents the correlation between the template and the input image, the higher the score, the higher the correlation, then, the template feature is fused according to g wherein represents matrix multiplication, and the finally fused template feature is represented as f 0A = FC([f 0 ; v V ]), wherein [·; ·] represents a concatenation operation.

5. The method of claim 4, wherein the language prior knowledge comprises a medical report template. The phrase decoding module is composed of a BERT encoder, a generation branch and a diagnosis branch, wherein the generation branch and the diagnosis branch are composed of multiple fully connected layers in cascade, the generation branch predicts the word probability according to the encoded features, and the diagnosis branch learns a multi-label classification task with the visual features and the template features as inputs.

6. The method of generating a medical image report in conjunction with linguistic prior knowledge according to any one of claims 1 to 5, characterized in that, The method is trained in an end-to-end manner in a report generation stage, and the overall loss function is where λ gen , λ dia and λ re / represent coefficients for balancing different loss terms; Generating branches with cross-entropy as loss function where y i denotes the sequence number of the i-th word, denotes the sequence number of the i-th word predicted, p denotes the prediction probability, and l denotes the report length; Computing binary cross-entropy loss as a retrieval loss function Wherein, T represents the number of templates in the template library, represents the real template label obtained according to the phrase template library label, is the calculated template prediction score; The diagnostic branch is by binary cross-entropy loss function Optimization: wherein C denotes the number of disease categories, denotes whether the jth disease is positive, 1 denotes positive, and 0 denotes negative, denotes the disease prediction score.

Images