A multi-modal pre-training method for generating CT image representation and image report
By employing a multimodal pre-training method, combining CT images and image reports, and utilizing stochastic data augmentation and the Transformer model to optimize model parameters, the limitations of labeled data in the medical field for deep learning models have been addressed, enabling efficient CT image feature extraction and report generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TSINGHUA UNIVERSITY
- Filing Date
- 2023-02-16
- Publication Date
- 2026-04-14
AI Technical Summary
The application of deep learning models in the medical field is limited by the high cost of labeled data, and existing methods are difficult to effectively extract fine-grained features from CT images.
A multimodal pre-training method is adopted. By acquiring combined data of CT images and image reports, random data augmentation and Transformer model are used to extract image features and text features, and the model parameters are optimized by a hybrid loss function.
This improved the model's ability to represent CT images, reduced reliance on labeled data, increased data utilization efficiency, and enabled efficient CT image feature extraction and report generation.
Smart Images
Figure CN116433934B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of natural language processing, and in particular to a multimodal pre-training method for generating CT image representations and image reports. Background Technology
[0002] In recent years, deep learning technology has developed rapidly and has been widely used in various fields. The success of deep learning technology largely depends on massive amounts of labeled data. For specialized fields such as medicine, chemistry, and biology, obtaining accurate annotations requires annotators with certain professional knowledge, and acquiring massive amounts of labeled data is extremely costly, significantly limiting the development and application of deep learning technology. Pre-training a basic model using large-scale data and applying the basic model to downstream tasks with smaller datasets using transfer learning methods helps improve the model's ability to handle downstream tasks. Compared to predefined labels or image self-supervision, natural language has richer semantic information. Using natural language and images for multimodal pre-training can provide more accurate supervision information for fine-grained features in images, helping the model to extract fine-grained image representations more accurately.
[0003] With the development of deep learning technology, Transformer-based natural language processing methods can effectively analyze large-scale text data, are suitable for parsing natural language text data, extracting key information, and encoding text, and can also efficiently generate target text based on the encoded information; Convolutional neural network-based image processing methods can efficiently extract features from images and obtain representations of key image information; Image self-supervised learning methods can help models mine more general image features. Summary of the Invention
[0004] To address the aforementioned issues, a multimodal pre-training method for generating CT image representations and image reports is proposed, which helps improve the model's ability to handle CT image-related tasks.
[0005] The first aspect of this application proposes a multimodal pre-training method for generating CT image representations and image reports, including:
[0006] Acquire multimodal data combinations, wherein the multimodal data combinations include CT images and image reports;
[0007] Random data enhancement is performed on the CT images, and the enhanced data is input into an image encoder for encoding to determine image features, wherein the image features include first image features and second image features;
[0008] The image report is input into a text encoder for encoding to determine text features, and the image features are input into a text decoder to determine the image description;
[0009] The hybrid loss function is determined based on the similarity in feature space between the first image feature and the second image feature, the text feature and the image feature, and the accuracy between the image description and the image report, respectively.
[0010] Based on the hybrid loss function, the gradient descent algorithm is used to train the model and update the parameters of the image encoder, the text encoder, and the text decoder.
[0011] Optionally, the step of performing random data augmentation on the CT images and inputting the augmented data into an image encoder for encoding to determine image features includes:
[0012] The image encoder is constructed based on a convolutional neural network, and the parameters of the image encoder are initialized according to the model parameters pre-trained on the ImageNet dataset.
[0013] The CT image is randomly rotated, flipped, cropped and scaled twice to obtain two different perspectives of the CT image;
[0014] The images from two different perspectives are input into the image encoder for encoding to obtain the image features of the CT images from the two different perspectives in the feature space.
[0015] Optionally, the step of inputting the image report into a text encoder for encoding to determine text features includes:
[0016] Construct a Transformer-based text encoder and initialize the text encoder and text embedding matrix according to the parameters of the BERT model;
[0017] The text embedding of the image report is calculated based on the text embedding matrix, and the position encoding is added to the text embedding before it is input into the text encoder.
[0018] The text features of the image report are obtained by the text encoder and mapped to the feature space.
[0019] Optionally, inputting the image features into a text decoder to determine the image description includes:
[0020] Construct a Transformer-based text decoder and input the image features into the text decoder;
[0021] The image description, based on natural language, is generated by decoding and corresponds to the image features.
[0022] Optionally, determining the hybrid loss function includes:
[0023] The single-modal contrast loss function is determined based on the similarity between the first image features and the second image features in the feature space.
[0024] The multimodal contrast loss function is determined based on the similarity between the text features and the image features in the feature space;
[0025] The cross-entropy loss function is determined based on the accuracy of the image description and the image report.
[0026] Optionally, determining the single-modal contrast loss function based on the similarity between the first image features and the second image features in the feature space includes:
[0027] Obtain a binary feature pair consisting of the first image feature and the second image feature;
[0028] In the feature space, the similarity between the first image feature and the second image feature within the binary feature pair is calculated;
[0029] The single-modal contrast loss function is calculated by maximizing the similarity between paired first image features and second image features in the feature space, and minimizing the similarity between unpaired first image features and second image features in the feature space.
[0030] Optionally, determining the multimodal contrast loss function based on the similarity between the text features and the image features in the feature space includes:
[0031] Obtain a ternary feature pair consisting of the text feature, the first image feature, and the second image feature;
[0032] In the feature space, the similarity between the image features and the text features in the ternary feature pairs is calculated;
[0033] The multimodal contrast loss function is calculated by maximizing the similarity between paired image features and text features in the feature space, and minimizing the similarity between unpaired image features and text features in the feature space.
[0034] Optionally, determining the cross-entropy loss function based on the accuracy of the image description and the image report includes:
[0035] Based on the image description and the image report, the cross-entropy loss function for generating the report is calculated in characters.
[0036] The second aspect of this application proposes a multimodal pre-training device for generating CT image representations and image reports, comprising:
[0037] A data collection module is used to acquire multimodal data combinations, wherein the multimodal data combinations include CT images and image reports;
[0038] The first processing module is used to perform random data enhancement on the CT image and input the enhanced data into an image encoder for encoding to determine image features, wherein the image features include a first image feature and a second image feature;
[0039] The second processing module is used to input the image report into a text encoder for encoding to determine text features, and input the image features into a text decoder to determine the image description.
[0040] The loss function calculation module is used to determine the hybrid loss function based on the similarity in feature space between the first image feature and the second image feature, the text feature and the image feature, and the accuracy between the image description and the image report, respectively.
[0041] The update module is used to train the model using the gradient descent algorithm based on the hybrid loss function, and update the parameters of the image encoder, the text encoder and the text decoder.
[0042] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:
[0043] By inputting paired CT images and image reports, a random data augmentation method is used to obtain two different perspectives of the same image and input them into a convolutional neural network-based image encoder to extract their respective image features. A Transformer-based text encoder is used to extract the text features corresponding to the image report, and a Transformer-based text decoder is used to generate the corresponding report for the image. The model parameters are optimized through a multi-task learning strategy to improve the model's ability to represent CT images and improve the efficiency of data utilization.
[0044] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description
[0045] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:
[0046] Figure 1 This is a flowchart illustrating a multimodal pre-training method for generating CT image representations and image reports according to embodiments of this application;
[0047] Figure 2This is a flowchart illustrating the determination of image features according to an embodiment of this application;
[0048] Figure 3 This is a flowchart illustrating the determination of text features according to embodiments of this application;
[0049] Figure 4 This is a flowchart illustrating the determination of image description according to an embodiment of this application.
[0050] Figure 5 This is a flowchart illustrating the determination of a single-modal contrast loss function according to an embodiment of this application;
[0051] Figure 6 This is a flowchart illustrating the determination of a multimodal contrast loss function according to an embodiment of this application;
[0052] Figure 7 This is a block diagram illustrating a multimodal pre-training device for generating CT image representations and image reports according to an embodiment of this application. Detailed Implementation
[0053] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.
[0054] Figure 1 This is a flowchart illustrating a multimodal pre-training method for generating CT image representations and image reports according to embodiments of this application, including:
[0055] Step 101: Obtain multimodal data combination, wherein the multimodal data combination includes CT images and image reports.
[0056] In this embodiment of the application, a multimodal data combination with CT images and corresponding image reports of the same user as the basic data units is acquired through a PACS image management and communication system.
[0057] In order to extract potential structural and textural features from images more accurately, the images need to be normalized, incomplete samples from images and reports need to be deleted, and invalid areas such as black borders naturally present in CT images during examination need to be cropped to ensure that the model can focus on as much effective information as possible and reduce invalid interference from other areas. The normalized image-report data constitutes the modal data combination used in practice.
[0058] Step 102: Perform random data enhancement on the CT images and input the enhanced data into the image encoder for encoding to determine image features, wherein the image features include first image features and second image features.
[0059] In this embodiment of the application, step 102 further includes:
[0060] Step 201: Construct an image encoder based on a convolutional neural network, and initialize the image encoder parameters according to the model parameters pre-trained on the ImageNet dataset;
[0061] Step 202: The CT image is randomly rotated, flipped, cropped and scaled twice to obtain two different perspectives of the CT image;
[0062] Step 203: Input the images from two different perspectives into the image encoder for encoding to obtain the image features of the two different perspectives of the CT image in the feature space.
[0063] In this embodiment, the input CT image is randomly rotated, flipped, cropped, and scaled twice to obtain two different viewpoints corresponding to the same CT image. The images from the two different viewpoints are then input into an image encoder for encoding, and the images are mapped into a higher-order feature space to obtain the image features of the two different viewpoints of the same CT image in the feature space.
[0064] In addition, the ImageNet dataset is an image database organized according to the WordNet hierarchy, where each node of the hierarchy is described by hundreds or thousands of images. The dataset contains 14,197,122 images and 21,841 Synset indices. It contains a wide variety of images, and each image is associated with a label.
[0065] Step 103: Input the image report into the text encoder for encoding to determine the text features, and input the image features into the text decoder to determine the image description.
[0066] The process of determining text features is as follows:
[0067] Step 301: Construct a Transformer-based text encoder and initialize the text encoder and text embedding matrix according to the parameters of the BERT model;
[0068] Step 302: Calculate the text embedding of the image report based on the text embedding matrix, add position encoding to the text embedding, and then input it into the text encoder;
[0069] Step 303: Obtain the text features of the image report mapped to the feature space based on the text encoder.
[0070] In this embodiment, the text embedding corresponding to the input image report is first calculated using a text embedding matrix. Then, the text embedding is added with position encoding and input into a text encoder to map the image report into the same feature space as in step 203 above, thereby obtaining the corresponding text features.
[0071] The process of determining the image description is as follows:
[0072] Step 401: Construct a Transformer-based text decoder by inputting image features into the text decoder;
[0073] Step 402: Generate a natural language-based image description corresponding to the image features by decoding.
[0074] In this embodiment of the application, any one of the image features from the two perspectives obtained in step 203 above is randomly selected and input into the text decoder. The decoder generates an image description based on natural language that corresponds to the image feature.
[0075] Step 104: Determine the hybrid loss function based on the similarity between the first image features and the second image features, the text features and the image features in the feature space, and the accuracy of the image description and the image report.
[0076] In this embodiment, the hybrid loss function includes determining a single-modal contrast loss function based on the similarity between the first image features and the second image features in the feature space, determining a multimodal contrast loss function based on the similarity between text features and image features in the feature space, and determining a cross-entropy loss function based on the accuracy of image descriptions and image reports.
[0077] The process of determining the single-modal contrast loss function is as follows:
[0078] Step 501: Obtain a binary feature pair consisting of the first image feature and the second image feature;
[0079] Step 502: In the feature space, calculate the similarity between the first image feature and the second image feature within the binary feature pair;
[0080] Step 503: Calculate the single-modal contrast loss function by maximizing the similarity between paired first image features and second image features in the feature space, and minimizing the similarity between unpaired first image features and second image features in the feature space.
[0081] In this embodiment, the alignment of image features from paired viewpoints in the feature space is achieved by calculating a single-modal contrast loss function.
[0082] The process of determining the multimodal contrast loss function is as follows:
[0083] Step 601: Obtain a ternary feature pair consisting of text features, first image features, and second image features;
[0084] Step 602: In the feature space, calculate the similarity between image features and text features in the ternary feature pair;
[0085] Step 603: Calculate the multimodal contrast loss function by maximizing the similarity between paired image features and text features in the feature space, and minimizing the similarity between unpaired image features and text features in the feature space.
[0086] In this embodiment, the alignment of image features and text features corresponding to paired images and reports in the feature space is achieved by calculating a multimodal contrast loss function.
[0087] The process of determining the cross-entropy loss function is as follows:
[0088] Based on the image description and image report, the cross-entropy loss function for generating the report is calculated in characters.
[0089] In this embodiment, minimizing the cross-entropy loss function helps the image encoder learn key information in the image and output reasonable representations. Furthermore, it enables cross-modal text generation through correct decoding by the text decoder, thus achieving semantic-level supervision.
[0090] Step 105: Based on the hybrid loss function, the gradient descent algorithm is used to train the model and update the parameters of the image encoder, text encoder and text decoder.
[0091] In this embodiment, after training, the image encoder can be used as the backbone network to be applied to various downstream tasks, including CT image diagnosis and image segmentation, and can efficiently extract key features from CT images without requiring a large amount of labeled data. Furthermore, by combining the image encoder and text decoder, the task of generating CT image reports can be achieved.
[0092] This application embodiment inputs paired CT images and image reports, uses a random data augmentation method to obtain two different perspectives of the same image, and inputs them into a convolutional neural network-based image encoder to extract their respective image features. A Transformer-based text encoder is used to extract the text features corresponding to the image report, and a Transformer-based text decoder is used to generate the report corresponding to the image. A multi-task learning strategy is used for pre-training to optimize model parameters, improve the model's ability to represent CT images, and improve the efficiency of data utilization.
[0093] Figure 7This application illustrates a multimodal pre-training device for generating CT image representations and image reports, comprising a data collection module 710, a first processing module 720, a second processing module 730, a loss function calculation module 740, and an update module 750.
[0094] The data collection module 710 is used to acquire multimodal data combinations, which include CT images and image reports;
[0095] The first processing module 720 is used to perform random data enhancement on CT images and input the enhanced data into an image encoder for encoding to determine image features, wherein the image features include first image features and second image features.
[0096] The second processing module 730 is used to input the image report into the text encoder for encoding to determine the text features, and input the image features into the text decoder to determine the image description.
[0097] The loss function calculation module 740 is used to determine the hybrid loss function based on the similarity between the first image features and the second image features, the text features and the image features in the feature space, and the accuracy of the image description and the image report, respectively.
[0098] Update module 750 is used to update the parameters of the image encoder, text encoder and text decoder by training the model based on the hybrid loss function and using the gradient descent algorithm.
[0099] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.
[0100] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0101] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A multi-modal pre-training method for generating CT image representation and image report, characterized in that, include: Acquire multimodal data combinations, wherein the multimodal data combinations include CT images and image reports; Random data enhancement is performed on the CT images, and the enhanced data is input into an image encoder for encoding to determine image features, wherein the image features include first image features and second image features; The image report is input into a text encoder for encoding to determine text features, and the image features are input into a text decoder to determine the image description; The hybrid loss function is determined based on the similarity in feature space between the first image feature and the second image feature, the text feature and the image feature, and the accuracy between the image description and the image report, respectively. Based on the hybrid loss function, the gradient descent algorithm is used to train the model and update the parameters of the image encoder, the text encoder, and the text decoder. The step of inputting the image report into a text encoder for encoding and determining text features includes: Construct a Transformer-based text encoder and initialize the text encoder and text embedding matrix according to the parameters of the BERT model; The text embedding of the image report is calculated based on the text embedding matrix, and the position encoding is added to the text embedding before it is input into the text encoder. The text features of the image report are obtained by the text encoder and mapped to the feature space. The step of inputting the image features into a text decoder to determine the image description includes: Construct a Transformer-based text decoder and input the image features into the text decoder; The image description, based on natural language, is generated by decoding and corresponds to the image features.
2. The method of claim 1, wherein, The step of performing random data augmentation on the CT images and inputting the augmented data into an image encoder for encoding to determine image features includes: The image encoder is constructed based on a convolutional neural network, and the parameters of the image encoder are initialized according to the model parameters pre-trained on the ImageNet dataset. The CT image is randomly rotated, flipped, cropped and scaled twice to obtain two different perspectives of the CT image; The images from two different perspectives are input into the image encoder for encoding to obtain the image features of the CT images from the two different perspectives in the feature space.
3. The method of claim 2, wherein, Determining the hybrid loss function includes: The single-modal contrast loss function is determined based on the similarity between the first image features and the second image features in the feature space. The multimodal contrast loss function is determined based on the similarity between the text features and the image features in the feature space; The cross-entropy loss function is determined based on the accuracy of the image description and the image report.
4. The method according to claim 3, characterized in that, The step of determining the single-modal contrast loss function based on the similarity between the first image features and the second image features in the feature space includes: Obtain a binary feature pair consisting of the first image feature and the second image feature; In the feature space, the similarity between the first image feature and the second image feature within the binary feature pair is calculated; The single-modal contrast loss function is calculated by maximizing the similarity between paired first image features and second image features in the feature space, and minimizing the similarity between unpaired first image features and second image features in the feature space.
5. The method according to claim 4, characterized in that, The step of determining the multimodal contrast loss function based on the similarity between the text features and the image features in the feature space includes: Obtain a ternary feature pair consisting of the text feature, the first image feature, and the second image feature; In the feature space, the similarity between the image features and the text features in the ternary feature pairs is calculated; The multimodal contrast loss function is calculated by maximizing the similarity between paired image features and text features in the feature space, and minimizing the similarity between unpaired image features and text features in the feature space.
6. The method according to claim 5, characterized in that, The step of determining the cross-entropy loss function based on the accuracy of the image description and the image report includes: Based on the image description and the image report, the cross-entropy loss function for generating the report is calculated in characters.
7. A multimodal pre-training device for generating CT image representations and image reports, characterized in that, include: A data collection module is used to acquire multimodal data combinations, wherein the multimodal data combinations include CT images and image reports; The first processing module is used to perform random data enhancement on the CT image and input the enhanced data into an image encoder for encoding to determine image features, wherein the image features include a first image feature and a second image feature; The second processing module is used to input the image report into a text encoder for encoding to determine text features, and input the image features into a text decoder to determine the image description. The loss function calculation module is used to determine the hybrid loss function based on the similarity in feature space between the first image feature and the second image feature, the text feature and the image feature, and the accuracy between the image description and the image report, respectively. The update module is used to train the model based on the hybrid loss function using the gradient descent algorithm, and update the parameters of the image encoder, the text encoder and the text decoder. The step of inputting the image report into a text encoder for encoding and determining text features includes: Construct a Transformer-based text encoder and initialize the text encoder and text embedding matrix according to the parameters of the BERT model; The text embedding of the image report is calculated based on the text embedding matrix, and the position encoding is added to the text embedding before it is input into the text encoder. The text features of the image report are obtained by the text encoder and mapped to the feature space. The step of inputting the image features into a text decoder to determine the image description includes: Construct a Transformer-based text decoder and input the image features into the text decoder; The image description, based on natural language, is generated by decoding and corresponds to the image features.
Citation Information
Patent Citations
Multi-mode three-dimensional medical image fusion method and system and electronic equipment
CN110580695A
Pre-training method and device for medical multi-modal model
CN114972929A