Hierarchical image report generation method and device combined with sentence-level contrast learning
By employing sentence-level image-text contrast learning and a hierarchical report generation module, the problem of low accuracy in image report generation in existing methods is solved, thereby improving the accuracy of reports and the feature extraction capability of image encoders.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TSINGHUA UNIVERSITY
- Filing Date
- 2023-03-29
- Publication Date
- 2026-05-01
AI Technical Summary
Existing methods treat the entire image report as a single text unit for generation, which leads to reduced accuracy.
We employ sentence-level image-text contrastive learning and a hierarchical report generation module. Sentence-level image-text contrastive learning forces the image encoder to learn the corresponding image features for each topic. The hierarchical report generation module first generates the topic of each sentence, and then generates the actual sentence based on the topic and the image.
It improves the accuracy of image report generation, enhances the feature extraction capability of the image encoder, and makes the generated reports more in line with doctors' report writing habits.
Smart Images
Figure CN116525052B_ABST
Abstract
Description
A hierarchical image report generation method and apparatus combining sentence-level contrastive learning Technical Field
[0001] This application relates to the fields of computer vision, natural language processing and computer-aided diagnostics, and in particular to a hierarchical image report generation method and apparatus that combines sentence-level contrastive learning. Background Technology
[0002] An imaging report is a textual description of a medical image, containing detailed information about physiological structures, abnormalities, and diseases within the image. Its main content can be abstracted into a paragraph composed of multiple sentences. Different sentences focus on different content and topics, and are relatively independent of each other. For example, in a chest X-ray report, some sentences focus on heart size, some on the presence of pneumonia, and some on bones, etc. Manually writing imaging reports is time-consuming and labor-intensive, but using deep learning methods, neural network models can be trained using large amounts of image and report data to automatically generate reports.
[0003] Image-text contrastive learning is a weakly supervised learning method that can be used for model pre-training or combined with other tasks to achieve better results. In the context of image report generation, image-text contrastive learning generally treats the entire report as a single text. However, since reports contain multiple sentences, each with different topics and content, treating the entire report as a whole is unreasonable. Therefore, how to accurately and automatically generate image reports for medical images is an urgent problem to be solved. Summary of the Invention
[0004] This application aims to at least partially address one of the technical problems in the related art.
[0005] Therefore, the first objective of this application is to propose a hierarchical image report generation method that combines sentence-level contrastive learning. This method addresses the technical problem of existing methods that treat the entire image report as a single text unit, which reduces accuracy. By using sentence-level image-text contrastive learning as an auxiliary task for report generation, the image encoder is forced to learn corresponding image features for each topic, thereby enhancing the feature extraction capability of the image encoder. At the same time, by utilizing a hierarchical report generation module, the topic of each sentence is generated first, and then the actual sentence is generated based on the topic and the image, thus improving the accuracy of the generated image report.
[0006] The second objective of this application is to propose a hierarchical image report generation device that incorporates sentence-level contrastive learning.
[0007] The third objective of this application is to provide a non-transitory computer-readable storage medium.
[0008] To achieve the above objectives, the first aspect of this application proposes a hierarchical image report generation method combining sentence-level contrastive learning, comprising: acquiring multiple medical image-image report pairs and constructing an image report generation model, wherein the image report generation model includes a sentence-level image-text contrastive learning module and a hierarchical report generation module; training the sentence-level image-text contrastive learning module with multiple medical image-image report pairs based on a first joint loss function, outputting feature vectors of medical images, topic vectors and content vectors of each sentence in the image report; training the hierarchical report generation module with feature vectors of medical images, topic vectors of each sentence in the image report and word sequences of each sentence in the image report based on a second joint loss function, obtaining a trained image report generation model; acquiring a medical image to be identified, inputting the medical image to be identified into the trained image report generation model, and generating an image report of the medical image to be identified.
[0009] Optionally, in one embodiment of this application, an image report generation model is constructed, including:
[0010] A sentence-level image-text contrast learning module is constructed based on the image encoder and the text encoder;
[0011] A hierarchical report generation module is constructed based on the vector quantization module, sentence decoder, and word decoder.
[0012] Based on the sentence-level image-text contrast learning module and the hierarchical report generation module, an image report generation model is obtained.
[0013] Optionally, in one embodiment of this application, multiple medical image-image report pairs are trained using a sentence-level image-text contrast learning module based on a first joint loss function, outputting feature vectors of the medical images, topic vectors and content vectors of each sentence in the image reports, including:
[0014] The medical image is input into the convolutional neural network of the image encoder to obtain a feature map. The feature map is then unfolded and used as a sequence input into the Transformer encoder of the image encoder to obtain the feature vector of the medical image.
[0015] The image report is input into the text encoder, each sentence in the image report is processed to generate a sequence, and the sequence is input into the Transformer Encoder of the text encoder to generate the topic vector and content vector of each sentence in the image report;
[0016] Based on the feature vector of the medical image, the topic vector of each sentence in the image report, and the content vector of each sentence in the image report, a first joint loss function is constructed, and the image encoder and text encoder are trained based on the first joint loss function.
[0017] Optionally, in one embodiment of this application, a first joint loss function is constructed based on the feature vector of the medical image, the topic vector of each sentence in the image report, and the content vector of each sentence in the image report, including:
[0018] Based on the topic vector of each sentence in the image report and the feature vector of the medical image, calculate the topic-guided image features of each sentence;
[0019] Calculate the similarity between the topic-guided image features and content vectors for each sentence, and construct a contrastive loss based on the similarity.
[0020] Based on the contrast loss, a first joint loss function is constructed by introducing topic loss and topic inconsistency loss;
[0021] The formula for calculating similarity is expressed as:
[0022]
[0023] Where S represents a sentence in the imaging report, I represents a medical image, C represents the content vector of sentence S in the imaging report, and T represents the content vector of sentence S in the imaging report. i Let q be the topic vector of sentence S in the image report. i Let be the i-th feature vector of medical image I, and let cos represent the cosine of the angle between the two vectors.
[0024] The contrast loss is expressed as:
[0025]
[0026] Among them, L silc For comparison of loss, b is the training batch size, and R0 is the training loss. i S represents the i-th image report in the training batch, and R represents the i-th image report. i One of the sentences, I i For the i-th medical image in the training batch, I k For the k-th medical image in the training batch, sim(x,y) represents the similarity between sentence x and medical image y, and t is a trainable temperature parameter;
[0027] The topic loss is represented as:
[0028]
[0029] Among them, L topic Let λ be the topic loss, λ be a hyperparameter, n be the number of sentences in the image report, and H() be the vector entropy function. N is the length of vector v, vi G is the i-th element of v. i Let i be the topic vector of the i-th sentence in the image report;
[0030] The topic inconsistency loss is represented as:
[0031]
[0032] Among them, L diff Let represent the topic inconsistency loss, m represent the number of sentences in the image report, cos represent the cosine of the angle between the two vectors, and T represent the topic inconsistency loss. i Let T represent the topic vector of the i-th sentence in the image report. k This represents the topic vector of the k-th sentence in the image report.
[0033] Optionally, in one embodiment of this application, the feature vector of the medical image, the topic vector of each sentence in the image report, and the word sequence of each sentence in the image report are input into a hierarchical report generation module for training based on a second joint loss function, including:
[0034] The topic vector of each sentence in the image report is input into the vector quantization module, and the target vector that is closest to the topic vector of each sentence is obtained through the discrete space of the vector quantization module.
[0035] Based on the topic vector and the obtained target vector for each sentence, a loss function is constructed to train the vector quantization module;
[0036] The loss function of the vector quantization module is expressed as:
[0037]
[0038] Among them, L vq Here, sg() is the loss function for the vector quantization module, v is the input vector, and e is the loss function for the vector quantization module. v Let e be the vector in the discrete space of the vector quantization module that is closest to the input vector v, and let e be an unused vector in the discrete space of the vector quantization module. e The vector that is closest to vector e in the input vector.
[0039] Optionally, in one embodiment of this application, the feature vector of the medical image, the topic vector of each sentence in the image report, and the word sequence of each sentence in the image report are input into a hierarchical report generation module for training based on a second joint loss function, including:
[0040] The feature vector of the medical image is input into the sentence decoder, and the sentence decoder predicts the sentence topic vector of the medical image. Based on the topic vector of each sentence in the image report and the predicted sentence topic vector, a loss function is constructed to train the sentence decoder.
[0041] The loss function of the sentence decoder is expressed as:
[0042]
[0043] Among them, L sg Let P be the loss function of the sentence decoder, and t be the probability function. i Let t be the topic vector of the i-th sentence in the image report. 1:-1 Let Q be the topic vector sequence of the first to the (i-1)th sentences in the image report, Q be the feature vector of the medical image, and θ be the parameters of the sentence decoder.
[0044] Optionally, in one embodiment of this application, the feature vector of the medical image, the topic vector of each sentence in the image report, and the word sequence of each sentence in the image report are input into a hierarchical report generation module for training based on a second joint loss function, including:
[0045] The feature vectors of medical images and the topic vectors of each sentence in the image report are input into the word decoder, and the word sequence of each sentence is predicted by the word decoder.
[0046] Based on the word sequence of each sentence in the image report and the predicted word sequence of the sentence, a loss function is constructed to train the word decoder;
[0047] The loss function of the word decoder is expressed as:
[0048]
[0049] Among them, L wg Let P be the loss function of the word decoder, T be the probability function, and w be the topic vector of the sentence. i w is the i-th word in the sentence. 1:-1 Let Q be the sequence of words from the first to the (i-1)th word in the sentence, Q be the feature vector of the medical image, and θ be the parameters of the word decoder.
[0050] Optionally, in one embodiment of this application, the medical image to be identified is input into a trained image report generation model to generate an image report of the medical image to be identified, including:
[0051] The medical image to be identified is input into the image encoder and encoded to obtain the feature vector of the medical image to be identified.
[0052] The feature vector of the medical image to be identified is input into the sentence decoder to predict the subscripts of the topic vectors of multiple sentences. The topic vectors of multiple sentences are then obtained from the vector quantization module based on the subscripts of the topic vectors of multiple sentences.
[0053] The topic vectors of multiple sentences are input into the word decoder. The word sequence of each sentence is generated sequentially based on the feature vector of the medical image to be identified and the topic vector of each sentence. Multiple sentences are then concatenated to obtain the image report of the medical image to be identified.
[0054] To achieve the above objectives, a second aspect of this application proposes a hierarchical image report generation apparatus that incorporates sentence-level contrastive learning, comprising:
[0055] The acquisition module is used to acquire multiple medical image-image report pairs and construct an image report generation model. The image report generation model includes a sentence-level image-text comparison learning module and a hierarchical report generation module.
[0056] The first training module is used to train multiple medical image-image report pairs into a sentence-level image-text comparison learning module based on the first joint loss function, and outputs the feature vector of the medical image, the topic vector and the content vector of each sentence in the image report;
[0057] The second training module is used to train the hierarchical report generation module by inputting the feature vector of the medical image, the topic vector of each sentence in the image report, and the word sequence of each sentence in the image report into the hierarchical report generation module based on the second joint loss function, so as to obtain the trained image report generation model.
[0058] The generation module is used to acquire the medical image to be identified, input the medical image to be identified into the trained image report generation model, and generate an image report of the medical image to be identified.
[0059] To achieve the above objectives, a third aspect of this application proposes a non-transitory computer-readable storage medium that, when the instructions in the storage medium are executed by a processor, can perform a hierarchical image report generation method incorporating sentence-level contrastive learning.
[0060] The hierarchical image report generation method, apparatus, and non-transitory computer-readable storage medium combining sentence-level contrastive learning in this application solve the technical problem that existing methods treat the entire image report as a single text unit, reducing accuracy. By using sentence-level image-text contrastive learning as an auxiliary task for report generation, the image encoder is forced to learn corresponding image features for each topic, enhancing the feature extraction capability of the image encoder. At the same time, by utilizing a hierarchical report generation module, the topic of each sentence is generated first, and then the actual sentence is generated based on the topic and the image, improving the accuracy of the generated image report.
[0061] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description
[0062] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:
[0063] Figure 1 is a flowchart illustrating a hierarchical image report generation method combining sentence-level contrastive learning provided in Embodiment 1 of this application.
[0064] Figure 2 is a schematic diagram of the training process of the image report generation model of the hierarchical image report generation method combining sentence-level contrastive learning according to an embodiment of this application.
[0065] Figure 3 is a schematic diagram of the actual usage process of the image report generation model of the hierarchical image report generation method combining sentence-level contrastive learning according to an embodiment of this application.
[0066] Figure 4 is a schematic diagram of a hierarchical image report generation device that combines sentence-level contrastive learning, provided in Embodiment 2 of this application. Detailed Implementation
[0067] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.
[0068] Deep learning is an artificial intelligence method that achieves specific functions by building neural network models and training them on large amounts of data. Image report generation methods based on deep learning generally use an encoder-decoder structure. The input image is first encoded into feature vectors or feature maps by the encoder, and then the decoder generates a text sequence of the report. The decoder typically employs two methods: direct generation and hierarchical generation. Direct generation treats the entire image report as a sequence and generates each character or word in that sequence sequentially; hierarchical generation first generates the features of each sentence in the report sequentially, and then generates characters or words for each sentence sequentially. In terms of model architecture, the encoder is generally based on a convolutional neural network (CNN) or a Transformer, and the decoder is generally based on a recurrent neural network (RNN) or a Transformer. However, in hierarchical generation models, the decoder is generally based on a recurrent neural network.
[0069] Image-text contrastive learning is a weakly supervised learning method that can be used for model pre-training or combined with other tasks to achieve better results. In the context of image report generation, image-text contrastive learning generally treats the entire report as a single text. However, since a report contains multiple sentences, each with different topics and content, treating the entire report as a whole is unreasonable.
[0070] This application proposes a deep learning-based method for automatic image report generation, which treats the image report as multiple independent sentences and performs comparative learning between the sentences and the images. It mainly includes two modules: sentence-level comparative learning and hierarchical image report generation. In this application, the hierarchical decoding uses Transformer as the decoder.
[0071] The following description, with reference to the accompanying drawings, describes a hierarchical image report generation method and apparatus that incorporates sentence-level contrastive learning according to embodiments of this application.
[0072] Figure 1 is a flowchart illustrating a hierarchical image report generation method combining sentence-level contrastive learning provided in Embodiment 1 of this application.
[0073] As shown in Figure 1, the hierarchical image report generation method combining sentence-level contrastive learning includes the following steps:
[0074] Step 101: Obtain multiple medical image-image report pairs and construct an image report generation model, wherein the image report generation model includes a sentence-level image-text comparison learning module and a hierarchical report generation module;
[0075] Step 102: Based on the first joint loss function, train the sentence-level image-text comparison learning module of multiple medical images-image reports to output the feature vector of the medical images, the topic vector and content vector of each sentence in the image report;
[0076] Step 103: Based on the second joint loss function, the feature vector of the medical image, the topic vector of each sentence in the image report, and the word sequence of each sentence in the image report are input into the hierarchical report generation module for training to obtain the trained image report generation model.
[0077] Step 104: Obtain the medical image to be identified, input the medical image to be identified into the trained image report generation model, and generate an image report of the medical image to be identified.
[0078] The hierarchical image report generation method combining sentence-level contrastive learning in this application embodiment acquires multiple medical image-image report pairs and constructs an image report generation model. The image report generation model includes a sentence-level image-text contrastive learning module and a hierarchical report generation module. Based on a first joint loss function, the multiple medical image-image report pairs are input into the sentence-level image-text contrastive learning module for training, outputting feature vectors of the medical images, topic vectors of each sentence in the image reports, and content vectors. Based on a second joint loss function, the feature vectors of the medical images, the topic vectors of each sentence in the image reports, and the word sequences of each sentence in the image reports are input into the hierarchical report generation module for training, resulting in a trained image report generation model. Finally, a medical image to be identified is acquired and input into the trained image report generation model to generate an image report for the identified medical image. Therefore, this method can solve the technical problem that existing methods treat the entire image report as a single text and generate it as a whole, which reduces accuracy. By using sentence-level image-text contrast learning as an auxiliary task for report generation, the image encoder is forced to learn the corresponding image features for each topic, which enhances the feature extraction capability of the image encoder. At the same time, by using a hierarchical report generation module, the topic of each sentence is generated first, and then the actual sentence is generated based on the topic and the image, which improves the accuracy of the generated image report.
[0079] Furthermore, in this embodiment of the application, the image report generation model is constructed, including:
[0080] A sentence-level image-text contrast learning module is constructed based on the image encoder and the text encoder;
[0081] A hierarchical report generation module is constructed based on the vector quantization module, sentence decoder, and word decoder.
[0082] Based on the sentence-level image-text contrast learning module and the hierarchical report generation module, an image report generation model is obtained.
[0083] The image report generation model in this application mainly includes two modules: sentence-level image-text comparison learning and hierarchical report generation. The sentence-level image-text comparison learning module mainly includes two neural networks: an image encoder and a text encoder. The hierarchical report generation module is used to generate image reports. During training, the hierarchical report generation module and the sentence-level image-text comparison learning module share the image encoder and the text encoder. The hierarchical report generation module also includes three neural networks: a vector quantization module, a sentence decoder, and a word decoder.
[0084] In the sentence-level image-text contrastive learning module, for a given image-report pair, the image encoder encodes the image (i.e., the video) into N D-dimensional feature vectors, each vector encoding content related to a specific topic in the image (e.g., the first vector encodes information about the heart in the image, the second vector encodes information about the skeleton, etc.). The text encoder encodes each sentence in the report into an N-dimensional topic vector T and a D-dimensional content vector C. All elements of T are non-negative and sum to 1, serving as a soft representation of the sentence's topic (e.g., if T = [0.9, 0.1, 0, 0...], it means the sentence is 90% related to the first topic and 10% to the second topic). The image encoder, text encoder, and their encoded vectors are all trained from the training data through contrastive learning.
[0085] In the hierarchical report generation module, the vector quantization module maps the topic vectors T of different sentences in the dataset to a relatively small discrete space E. The sentence decoder predicts (generates) the topic vectors of several sentences that should appear in the image report based on the feature vectors of the input image (these topics are selected from the discrete space E). The word decoder predicts (generates) the words of a sentence based on the feature vectors of the input image and the topic vectors of that sentence. In practice, the multiple topic vectors predicted by the sentence decoder are each processed by the word decoder to generate sentences, and then concatenated to obtain the final report.
[0086] Furthermore, in this embodiment, multiple medical image-image report pairs are trained using a sentence-level image-text comparison learning module based on a first joint loss function, outputting feature vectors of the medical images, topic vectors and content vectors of each sentence in the image reports, including:
[0087] The medical image is input into the convolutional neural network of the image encoder to obtain a feature map. The feature map is then unfolded and used as a sequence input into the Transformer encoder of the image encoder to obtain the feature vector of the medical image.
[0088] The image report is input into the text encoder, each sentence in the image report is processed to generate a sequence, and the sequence is input into the Transformer Encoder of the text encoder to generate the topic vector and content vector of each sentence in the image report;
[0089] Based on the feature vector of the medical image, the topic vector of each sentence in the image report, and the content vector of each sentence in the image report, a first joint loss function is constructed, and the image encoder and text encoder are trained based on the first joint loss function.
[0090] In this embodiment, the image encoder consists of a convolutional neural network and a Transformer Encoder neural network. The input medical image is first processed by the convolutional neural network, which outputs a feature map (the specific size is determined by the convolutional neural network; for example, if a ResNet18 network is used, the output is a 7*7*512 feature map). This feature map, after being flattened in spatial dimensions (for ResNet18, it is flattened into 49 512-dimensional feature vectors), is input as a sequence into a Transformer Encoder, which outputs N D-dimensional feature vectors (for ResNet18, N=49, D=512).
[0091] In this embodiment, the text encoder consists of a Transformer Encoder and an output layer. A sentence is first processed by operations such as token embedding and position embedding, and then fed into the Transformer Encoder as a sequence. The output vector at the beginning of the sentence is passed through a fully connected layer and a Softmax layer to obtain an N-dimensional topic vector T; after passing through a fully connected layer, a D-dimensional content vector C is obtained.
[0092] Further, in this embodiment, a first joint loss function is constructed based on the feature vector of the medical image, the topic vector of each sentence in the image report, and the content vector of each sentence in the image report, including:
[0093] Based on the topic vector of each sentence in the image report and the feature vector of the medical image, calculate the topic-guided image features of each sentence;
[0094] Calculate the similarity between the topic-guided image features and content vectors for each sentence, and construct a contrastive loss based on the similarity.
[0095] Based on the contrast loss, a first joint loss function is constructed by introducing topic loss and topic inconsistency loss;
[0096] The formula for calculating similarity is expressed as:
[0097]
[0098] Where S represents a sentence in the imaging report, I represents a medical image, C represents the content vector of sentence S in the imaging report, and T represents the content vector of sentence S in the imaging report. i Let q be the topic vector of sentence S in the image report. i Let be the i-th feature vector of medical image I, and let cos represent the cosine of the angle between the two vectors.
[0099] The contrast loss is expressed as:
[0100]
[0101] Among them, L silc For comparison of loss, b is the training batch size, and R0 is the training loss. i S represents the i-th image report in the training batch, and R represents the i-th image report. i One of the sentences, I i For the i-th medical image in the training batch, I k For the k-th medical image in the training batch, sim(x,y) represents the similarity between sentence x and medical image y, and t is a trainable temperature parameter;
[0102] The topic loss is represented as:
[0103]
[0104] Among them, L topic Let λ be the topic loss, λ be a hyperparameter, n be the number of sentences in the image report, and H() be the vector entropy function. N is the length of vector v, v i G is the i-th element of v. i Let i be the topic vector of the i-th sentence in the image report;
[0105] The topic inconsistency loss is represented as:
[0106]
[0107] Among them, L diff Let represent the topic inconsistency loss, m represent the number of sentences in the image report, cos represent the cosine of the angle between the two vectors, and T represent the topic inconsistency loss. i Let T represent the topic vector of the i-th sentence in the image report. k This represents the topic vector of the k-th sentence in the image report.
[0108] The loss function for sentence-level image-text contrastive learning consists of three main parts.
[0109] First, there's the contrastive loss. For a sentence, its N-dimensional topic vector T is used as weights to weight and sum the N D-dimensional feature vectors of an image, resulting in a D-dimensional vector called Topic-guided Image Content (TGIC). This vector represents the content of the image corresponding to the topic. When the sentence and image match (i.e., they are a pair), the TGIC should be similar to the sentence content C; conversely, when the sentence and image do not match (i.e., they are not a pair), the TGIC should be dissimilar to C. During training, for a sentence in the i-th sample of a batch, its topic T is used to extract the TGIC from all images in the same batch, and the cosine similarity between these TGICs and the sentence content C is calculated. The formula for calculating the similarity is:
[0110]
[0111] Where S represents a sentence in the imaging report, I represents a medical image, C represents the content vector of sentence S in the imaging report, and T represents the content vector of sentence S in the imaging report. i Let q be the i-th element of the topic vector of sentence S in the image report. i Let be the i-th feature vector of medical image I, and let cos represent the cosine of the angle between the two vectors.
[0112] The contrast loss is expressed as:
[0113]
[0114] Among them, L silc For comparison of loss, b is the training batch size, and R0 is the training loss. i S represents the i-th image report in the training batch, and R represents the i-th image report. i One of the sentences, I i For the i-th medical image in the training batch, I k For the k-th medical image in the training batch, sim(x,y) represents the similarity between sentence x and medical image y, and t is a trainable temperature parameter;
[0115] Secondly, the topic vector T must satisfy both local sparsity and global balance. That is, for a given vector T, only a few elements should be large, because a sentence won't involve too many topics; simultaneously, for many vectors T, globally, every element at each position must be used to ensure fine-grained topic segmentation. Therefore, another loss function is introduced. For n sentences in a report, let their topic vector be G∈R. n×N Then add the topic loss:
[0116]
[0117] Among them, L topic Let λ be the topic loss, λ be a hyperparameter, n be the number of sentences in the image report, and H() be the vector entropy function. N is the length of vector v, v i G is the i-th element of v. i Let i be the topic vector of the i-th sentence in the image report;
[0118] Meanwhile, the topic inconsistency loss is expressed as:
[0119]
[0120] Among them, L diff Let represent the topic inconsistency loss, m represent the number of sentences in the image report, cos represent the cosine of the angle between the two vectors, and T represent the topic inconsistency loss. i Let T represent the topic vector of the i-th sentence in the image report. k This represents the topic vector of the k-th sentence in the image report.
[0121] Furthermore, in this embodiment, the feature vector of the medical image, the topic vector of each sentence in the image report, and the word sequence of each sentence in the image report are input into the hierarchical report generation module for training based on the second joint loss function, including:
[0122] The topic vector of each sentence in the image report is input into the vector quantization module, and the target vector that is closest to the topic vector of each sentence is obtained through the discrete space of the vector quantization module.
[0123] Based on the topic vector and the obtained target vector for each sentence, a loss function is constructed to train the vector quantization module;
[0124] The loss function of the vector quantization module is expressed as:
[0125]
[0126] Among them, L vq Here, sg() is the loss function for the vector quantization module, v is the input vector, and e is the loss function for the vector quantization module. v Let e be the vector in the discrete space of the vector quantization module that is closest to the input vector v, and let e be an unused vector in the discrete space of the vector quantization module. e The vector that is closest to vector e in the input vector.
[0127] The Vector quantization (VQ) module maps the input vector v to a discrete space E. Specifically, the VQ module contains K N-dimensional vectors, i.e., E∈RK×N. For the input N-dimensional vector v, the VQ module finds the closest vector ev in E and outputs ev along with its corresponding index. The loss function during training of the vector quantization module consists of two parts: one part moves the vector in E closest to the input vector towards the input vector, and the other part moves the least recently used vector in E towards its closest counterpart.
[0128] Furthermore, in this embodiment, the feature vector of the medical image, the topic vector of each sentence in the image report, and the word sequence of each sentence in the image report are input into the hierarchical report generation module for training based on the second joint loss function, including:
[0129] The feature vector of the medical image is input into the sentence decoder, and the sentence decoder predicts the sentence topic vector of the medical image. Based on the topic vector of each sentence in the image report and the predicted sentence topic vector, a loss function is constructed to train the sentence decoder.
[0130] The loss function of the sentence decoder is expressed as:
[0131]
[0132] Among them, L sg Let P be the loss function of the sentence decoder, and t be the probability function. i Let t be the topic vector of the i-th sentence in the image report. 1:-1 Let Q be the topic vector sequence of the first to the (i-1)th sentences in the image report, Q be the feature vector of the medical image, and θ be the parameters of the sentence decoder.
[0133] The sentence decoder consists of a Transformer Decoder that predicts the topic of each sentence in the report based on the image feature vectors. The N feature vectors obtained from the input image through the image encoder serve as the input to the sentence decoder's cross-attention function. The sentence decoder autoregressively predicts (generates) sentence topic vectors. However, it does not directly predict vectors; instead, it predicts a sequence of indices, from which the corresponding topic vector can be retrieved from the VQ module.
[0134] For a report, if the true topic vectors of multiple sentences (i.e., the topic vectors obtained by the text encoder) correspond to T = t1, t2, ... t in the VQ module, respectively... n Then the loss function of the sentence decoder is:
[0135]
[0136] Where P is the probability function, Q is the sequence of feature vectors of the image corresponding to this report, and θ is the parameter of the sentence decoder.
[0137] Furthermore, in this embodiment, the feature vector of the medical image, the topic vector of each sentence in the image report, and the word sequence of each sentence in the image report are input into the hierarchical report generation module for training based on the second joint loss function, including:
[0138] The feature vectors of medical images and the topic vectors of each sentence in the image report are input into the word decoder, and the word sequence of each sentence is predicted by the word decoder.
[0139] Based on the word sequence of each sentence in the image report and the predicted word sequence of the sentence, a loss function is constructed to train the word decoder;
[0140] The loss function of the word decoder is expressed as:
[0141]
[0142] Among them, L wg Let P be the loss function of the word decoder, T be the probability function, and w be the topic vector of the sentence. i w is the i-th word in the sentence. 1:-1 Let Q be the sequence of words from the first to the (i-1)th word in the sentence, Q be the feature vector of the medical image, and θ be the parameters of the word decoder.
[0143] The word decoder consists of a Transformer Decoder that predicts the sequence of words in a sentence based on the image feature vector and the sentence's topic vector. The sentence's topic vector is obtained by the text encoder during training and predicted by the sentence decoder during actual use. The image feature vector, like that of the sentence decoder, serves as the input to the decoder's cross-attention. The sentence topic vector is used as the input to the first position of the sequence, and the word decoder uses this as a basis to autoregressively generate the sequence of words in the sentence.
[0144] For a sentence, if its word sequence is represented as W = w1, w2, ... w m Then the loss function of the word decoder is:
[0145]
[0146] Where P is the probability function, T is the topic vector of this sentence, Q is the image feature vector sequence, and θ is the parameter of the word decoder.
[0147] As shown in Figure 2, the sentence-level image-text generation module of the image report generation model is trained, including: acquiring multiple image-report pairs; inputting the images into the convolutional neural network and Transformer encoder of the image encoder to obtain the feature vector of the medical image; inputting the image report into the text encoder to generate the topic vector and content vector of each sentence in the image report; and constructing a contrastive loss L based on the feature vector of the medical image, the topic vector of each sentence in the image report, and the content vector of each sentence in the image report. silc Topic loss L topic Loss L due to topic inconsistency diff The image encoder and text encoder are trained.
[0148] The hierarchical report generation module of the image report generation model is trained, including: inputting the topic vector of each sentence in the image report into the vector quantization module VQ; obtaining the target vector that is closest to the topic vector of each sentence through the discrete space of the vector quantization module; and constructing the loss function L. vq The vector quantization module is trained; the feature vectors of the medical images are input into the sentence decoder, and the sentence decoder predicts the sentence topic vectors of the medical images, thus constructing a loss function L. sg The sentence decoder is trained; the feature vectors of the medical images and the topic vectors of each sentence in the image report are input into the word decoder, and the word sequence of each sentence is predicted by the word decoder; based on the word sequence of each sentence in the image report and the predicted word sequence of that sentence, a loss function L is constructed. wg Train the word decoder.
[0149] Furthermore, in this embodiment of the application, the medical image to be identified is input into a trained image report generation model to generate an image report of the medical image to be identified, including:
[0150] The medical image to be identified is input into the image encoder and encoded to obtain the feature vector of the medical image to be identified.
[0151] The feature vector of the medical image to be identified is input into the sentence decoder to predict the subscripts of the topic vectors of multiple sentences. The topic vectors of multiple sentences are then obtained from the vector quantization module based on the subscripts of the topic vectors of multiple sentences.
[0152] The topic vectors of multiple sentences are input into the word decoder. The word sequence of each sentence is generated sequentially based on the feature vector of the medical image to be identified and the topic vector of each sentence. Multiple sentences are then concatenated to obtain the image report of the medical image to be identified.
[0153] As shown in Figure 3, the medical image to be identified is input to the image encoder for encoding, resulting in a feature vector of the medical image. This feature vector is then input to the sentence decoder and word decoder. Based on the feature vector of the medical image, the sentence decoder predicts the subscript T of the topic vectors for multiple sentences. 47 ···T 213 T 189 The topic vectors of multiple sentences are input into the vector quantization module, and the topic vectors of multiple sentences are obtained from the vector quantization module according to the index. The word decoder generates the word sequence of each sentence according to the feature vector of the medical image to be identified and the topic vector of each sentence, resulting in multiple sentences. The multiple sentences are then connected to obtain the image report of the medical image to be identified.
[0154] This application uses sentence-level image-text contrast learning as an auxiliary task for report generation, forcing the image encoder to learn corresponding image features for each topic, thereby enhancing the feature extraction capability of the image encoder. At the same time, through a hierarchical report generation module, the topic of each sentence is generated first, and then the actual sentence is generated based on the topic and the image. This is consistent with the nature of imaging reports and the habits of doctors in writing imaging reports, and can generate more accurate reports.
[0155] Figure 4 is a schematic diagram of a hierarchical image report generation device that combines sentence-level contrastive learning, provided in Embodiment 2 of this application.
[0156] As shown in Figure 4, this hierarchical image report generation device combining sentence-level contrastive learning includes:
[0157] The acquisition module 10 is used to acquire multiple medical image-image report pairs and construct an image report generation model. The image report generation model includes a sentence-level image-text comparison learning module and a hierarchical report generation module.
[0158] The first training module 20 is used to train multiple medical image-image report pairs into a sentence-level image-text comparison learning module based on a first joint loss function, and outputs the feature vector of the medical image, the topic vector and the content vector of each sentence in the image report;
[0159] The second training module 30 is used to train the hierarchical report generation module by inputting the feature vector of the medical image, the topic vector of each sentence in the image report and the word sequence of each sentence in the image report into the hierarchical report generation module based on the second joint loss function, so as to obtain the trained image report generation model.
[0160] The generation module 40 is used to acquire the medical image to be identified, input the medical image to be identified into the trained image report generation model, and generate an image report of the medical image to be identified.
[0161] The hierarchical image report generation device combining sentence-level contrastive learning according to this application embodiment includes: an acquisition module, used to acquire multiple medical image-image report pairs and construct an image report generation model, wherein the image report generation model includes a sentence-level image-text contrastive learning module and a hierarchical report generation module; a first training module, used to train the sentence-level image-text contrastive learning module by inputting multiple medical image-image report pairs into the sentence-level image-text contrastive learning module based on a first joint loss function, and output the feature vector of the medical image, the topic vector of each sentence in the image report, and the content vector; a second training module, used to train the hierarchical report generation module by inputting the feature vector of the medical image, the topic vector of each sentence in the image report, and the word sequence of each sentence in the image report based on a second joint loss function, and obtain a trained image report generation model; and a generation module, used to acquire the medical image to be identified, input the medical image to be identified into the trained image report generation model, and generate an image report of the medical image to be identified. Therefore, this method can solve the technical problem that existing methods treat the entire image report as a single text and generate it as a whole, which reduces accuracy. By using sentence-level image-text contrast learning as an auxiliary task for report generation, the image encoder is forced to learn the corresponding image features for each topic, which enhances the feature extraction capability of the image encoder. At the same time, by using a hierarchical report generation module, the topic of each sentence is generated first, and then the actual sentence is generated based on the topic and the image, which improves the accuracy of the generated image report.
[0162] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the hierarchical image report generation method combining sentence-level contrastive learning as described in the above embodiments.
[0163] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0164] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0165] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.
[0166] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.
[0167] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0168] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0169] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0170] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.
Claims
1. A hierarchical image report generation method combining sentence-level contrastive learning, characterized in that, The process includes the following steps: acquiring multiple medical image-image report pairs and constructing an image report generation model, wherein the image report generation model includes a sentence-level image-text comparison learning module and a hierarchical report generation module; training the sentence-level image-text comparison learning module based on a first joint loss function, outputting feature vectors of the medical images, topic vectors and content vectors of each sentence in the image report; training the hierarchical report generation module based on a second joint loss function, obtaining a trained image report generation model; acquiring a medical image to be identified, inputting the medical image to be identified into the trained image report generation model, and generating an image report of the medical image to be identified; wherein, constructing the image report generation model includes: based on image encoding... A sentence-level image-text comparison learning module is constructed using a coder and a text encoder; a hierarchical report generation module is constructed based on a vector quantization module, a sentence decoder, and a word decoder; an image report generation model is obtained based on the sentence-level image-text comparison learning module and the hierarchical report generation module; the step of training the sentence-level image-text comparison learning module with multiple medical images and image reports based on a first joint loss function, and outputting feature vectors of medical images, topic vectors and content vectors of each sentence in the image report, includes: inputting the medical images into the convolutional neural network of the image encoder to obtain feature maps, unfolding the feature maps as sequences and inputting them as sequences into the Transformer encoder of the image encoder to obtain feature vectors of the medical images; inputting the image reports into the text encoder, processing each sentence of the image reports to generate sequences, and inputting the sequences into the Transformer encoder of the text encoder. The encoder generates topic vectors and content vectors for each sentence in the image report; it constructs a first joint loss function based on the feature vectors of the medical image, the topic vectors of each sentence in the image report, and the content vectors of each sentence in the image report, and trains the image encoder and text encoder based on the first joint loss function; the construction of the first joint loss function based on the feature vectors of the medical image, the topic vectors of each sentence in the image report, and the content vectors of each sentence in the image report includes: calculating the topic-guided image features of each sentence based on the topic vectors of each sentence in the image report and the feature vectors of the medical image, wherein, for a sentence, its N-dimensional topic vector T is used as the weight to perform a weighted summation of the N D-dimensional feature vectors of the image to obtain a D-dimensional vector, called the topic-guided image feature; calculating the similarity between the topic-guided image features and the content vector of each sentence, and constructing a contrastive loss based on the similarity;Based on the contrastive loss, a first joint loss function is constructed by introducing topic loss and topic inconsistency loss.
2. The method as described in claim 1, characterized in that, The formula for calculating the similarity is as follows: Where S represents a sentence in the image report, I represents a medical image, and C represents the content vector of sentence S in the image report. Let q be the topic vector of sentence S in the image report. i Let be the i-th feature vector of medical image I, and let cos represent the cosine of the angle between the two vectors; the contrast loss is expressed as: in, For comparison of losses, b is the training batch size. S represents the i-th image report in the training batch. One of the sentences, For the i-th medical image in the training batch, For the k-th medical image in the training batch, sim(x,y) represents the similarity between sentence x and medical image y, and t is a trainable temperature parameter; the topic loss is expressed as: ;in, Loss of topic Here, n is a hyperparameter, and n is the number of sentences in the image report. Let entropy be the vector function. N is the length of vector v, v i Let i be the i-th element of v. Let be the topic vector of the i-th sentence in the image report; the topic inconsistency loss is represented as: in, Let represent the topic inconsistency loss, m represent the number of sentences in the image report, and cos represent the cosine of the angle between the two vectors. Let i represent the topic vector of the i-th sentence in the image report. This represents the topic vector of the k-th sentence in the image report.
3. The method as described in claim 1, characterized in that, The step of training the hierarchical report generation module by inputting the feature vector of the medical image, the topic vector of each sentence in the image report, and the word sequence of each sentence in the image report into the module based on the second joint loss function includes: inputting the topic vector of each sentence in the image report into the vector quantization module, obtaining the target vector that is closest to the topic vector of each sentence through the discrete space of the vector quantization module; and constructing a loss function to train the vector quantization module based on the topic vector of each sentence and the obtained target vector. The loss function of the vector quantization module is expressed as follows: in, The loss function for the vector quantization module. This is a gradient cutoff operation, where v is the input vector. Let be the vector closest to the input vector v in the discrete space of the vector quantization module, and let e be an unused vector in the discrete space of the vector quantization module. The vector that is closest to vector e in the input vector.
4. The method as described in claim 1, characterized in that, The step of training the hierarchical report generation module by inputting the feature vector of the medical image, the topic vector of each sentence in the image report, and the word sequence of each sentence in the image report into the module based on the second joint loss function includes: inputting the feature vector of the medical image into the sentence decoder; predicting the sentence topic vector of the medical image through the sentence decoder; and constructing a loss function to train the sentence decoder based on the topic vector of each sentence in the image report and the predicted sentence topic vector. The loss function of the sentence decoder is expressed as follows: in, Let this be the loss function of the sentence decoder. Let be a probability function. Let i be the topic vector of the i-th sentence in the image report. Let i be the topic vector sequence from the first to the (i-1)th sentence in the image report. For medical images, These are the parameters for the sentence decoder.
5. The method as described in claim 1, characterized in that, The step of training the hierarchical report generation module by inputting the feature vector of the medical image, the topic vector of each sentence in the image report, and the word sequence of each sentence in the image report into the second joint loss function includes: inputting the feature vector of the medical image and the topic vector of each sentence in the image report into the word decoder, and predicting the word sequence of each sentence through the word decoder; constructing a loss function based on the word sequence of each sentence in the image report and the predicted word sequence of that sentence to train the word decoder; the loss function of the word decoder is expressed as: in, The loss function for the word decoder, Let be a probability function. This represents the topic vector of the sentence. For the i-th word of the sentence, Let i be the sequence of words from the first to the (i-1)th word in the sentence. For medical images, These are the parameters for the word decoder.
6. The method as described in claim 1, characterized in that, The step of inputting the medical image to be identified into the trained image report generation model to generate an image report of the medical image to be identified includes: inputting the medical image to be identified into an image encoder for encoding to obtain a feature vector of the medical image to be identified; inputting the feature vector of the medical image to be identified into a sentence decoder to predict the subscripts of topic vectors of multiple sentences; obtaining the topic vectors of multiple sentences from a vector quantization module based on the subscripts of the topic vectors of the multiple sentences; inputting the topic vectors of the multiple sentences into a word decoder; generating a word sequence for each sentence sequentially based on the feature vector of the medical image to be identified and the topic vector of each sentence to obtain multiple sentences; and concatenating the multiple sentences to obtain an image report of the medical image to be identified.
7. A hierarchical image report generation device combining sentence-level contrastive learning, characterized in that, include: An acquisition module is used to acquire multiple medical image-image report pairs and construct an image report generation model, wherein the image report generation model includes a sentence-level image-text comparison learning module and a hierarchical report generation module; a first training module is used to train the sentence-level image-text comparison learning module based on a first joint loss function, outputting feature vectors of the medical images, topic vectors and content vectors of each sentence in the image reports; a second training module is used to train the hierarchical report generation module based on a second joint loss function, inputting the feature vectors of the medical images, the topic vectors of each sentence in the image reports, and the word sequences of each sentence in the image reports, to obtain a trained image report generation model; a generation module is used to acquire a medical image to be identified, input the medical image to be identified into the trained image report generation model, and generate an image report of the medical image to be identified; wherein, the construction of the image report... The generative model includes: constructing a sentence-level image-text contrast learning module based on an image encoder and a text encoder; constructing a hierarchical report generation module based on a vector quantization module, a sentence decoder, and a word decoder; obtaining an image report generation model based on the sentence-level image-text contrast learning module and the hierarchical report generation module; the step of training the sentence-level image-text contrast learning module with multiple medical images and image reports based on a first joint loss function, and outputting feature vectors of medical images, topic vectors and content vectors of each sentence in the image report, includes: inputting the medical images into the convolutional neural network of the image encoder to obtain feature maps, unfolding the feature maps as sequences and inputting them as sequences into the Transformer encoder of the image encoder to obtain feature vectors of the medical images; inputting the image reports into the text encoder, processing each sentence of the image reports to generate sequences, and inputting the sequences into the Transformer encoder of the text encoder. The encoder generates topic vectors and content vectors for each sentence in the image report; based on the feature vectors of the medical image, the topic vectors of each sentence in the image report, and the content vectors of each sentence in the image report, a first joint loss function is constructed, and the image encoder and text encoder are trained based on the first joint loss function; the construction of the first joint loss function based on the feature vectors of the medical image, the topic vectors of each sentence in the image report, and the content vectors of each sentence in the image report includes: calculating the topic-guided image features of each sentence based on the topic vectors of each sentence in the image report and the feature vectors of the medical image, wherein for a sentence, its N-dimensional topic vector T is used as the weight to perform a weighted summation of the N D-dimensional feature vectors of the image to obtain a D-dimensional vector, called the topic-guided image feature; calculating the similarity between the topic-guided image features and the content vector of each sentence, and constructing a contrastive loss based on the similarity;Based on the contrastive loss, a first joint loss function is constructed by introducing topic loss and topic inconsistency loss.
8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-5.