Training method of medical image analysis model based on comparative learning

Abstract

The invention relates to model training, in particular to a method for training a medical image analysis model based on comparative learning, which comprises the following steps of: constructing a report library containing disease or important discovery and anatomy areas; extracting disease or important discovery and anatomical region information in each report, describing fine-grained information corresponding to the identified disease, and constructing a structured report library; respectively vectorizing the disease description and the anatomical region in each structured report to obtain a corresponding disease query vector and an anatomical region query vector; converting the medical images corresponding to the structured reports into visual feature vectors to capture key visual features of the medical images; respectively carrying out feature interaction learning on the disease query vector and the visual feature vector as well as the anatomical region query vector and the visual feature vector; according to the technical scheme provided by the invention, the defect that the disease recognition and focus positioning performance of the model is difficult to be enhanced by fully utilizing limited annotation data in the prior art can be effectively overcome.

Description

Technical Field

[0001] The present invention relates to model training, and in particular to a training method for a medical image analysis model based on contrastive learning. Background Art

[0002] In the field of medical image analysis, deep learning technology, especially convolutional neural networks (CNNs), has made great progress. However, many high-performance deep learning models rely on a large amount of labeled data for supervised learning. In practical scenarios, high-quality medical image labeled data is often difficult to obtain. Not only is the acquisition cost high, but it also requires professional medical knowledge. Therefore, how to make full use of limited labeled data, and how to extract and utilize complementary information from different medical imaging modalities, has become the key to improving the performance of medical image analysis models.

[0003] In recent years, contrastive learning, as an effective unsupervised or semi-supervised learning strategy, has shown great potential in the field of natural image processing by constructing positive and negative sample pairs to learn data representation. Compared with the traditional training methods of medical image analysis models, multimodal contrastive learning pre-training can better capture and utilize complex features in medical images in the absence of labeled data by comparing medical images of different modalities, thereby improving the generalization ability and performance of the model.

[0004] In the invention patent application with application publication number CN116525052A, a hierarchical image report generation method and device combined with sentence-level contrastive learning is disclosed, which adopts a contrastive learning strategy to enhance the image encoder's ability to capture features by comparing sentences with images. The fundamental idea is similar to the way of learning in natural scenes, that is, by emphasizing the overall correlation between sentences and images, the model's ability to understand overall features is cultivated. However, this method has certain limitations, especially for medical images that require a high degree of attention to local details. The local lesion areas in medical images are usually extremely small and have considerable complexity. The model is prone to neglecting the capture of these key local details during the learning process, resulting in the loss of many subtle and key image feature information. Summary of the invention

[0005] 1. Technical issues to be solved

[0006] In view of the above-mentioned shortcomings of the prior art, the present invention provides a training method for a medical image analysis model based on contrastive learning, which can effectively overcome the defect of the prior art that it is difficult to fully utilize limited annotated data to enhance the model's performance in disease identification and lesion localization.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A training method for a medical image analysis model based on contrastive learning comprises the following steps:

[0010] S1. Build a library of reports containing diseases or important findings and anatomical regions;

[0011] S2, extract the diseases or important findings and anatomical region information in each report, and construct a structured report library with fine-grained information corresponding to the identified disease descriptions;

[0012] S3, vectorizing the disease description and anatomical region in each structured report to obtain corresponding disease query vectors and anatomical region query vectors;

[0013] S4, converting the medical images corresponding to each structured report into a visual feature vector to capture the key visual features of the medical images;

[0014] S5, performing feature interactive learning on the disease query vector and the visual feature vector, as well as the anatomical region query vector and the visual feature vector, respectively, to obtain corresponding visually weighted disease representations and visually weighted anatomical region representations;

[0015] S6. Perform comparative learning on visually weighted disease representations and disease labels, as well as visually weighted anatomical region representations and anatomical region labels, to enhance the performance of medical image analysis models in disease recognition and lesion localization.

[0016] Preferably, a report library containing diseases or important findings and anatomical regions is constructed in S1, including:

[0017] Based on the expertise of clinicians, a report library containing 100 diseases or important findings and 60 anatomical regions was constructed;

[0018] Each disease or significant finding was arbitrarily combined with each anatomical region.

[0019] Preferably, S2 extracts the diseases or important findings and anatomical region information in each report, and describes the fine-grained information corresponding to the identified diseases to construct a structured report library, including:

[0020] S21. Use the multimodal large language model GPT-4 to extract information about diseases or important findings and anatomical regions in each report;

[0021] S22. Build a structured report library with fine-grained information including manifestation, shape and texture corresponding to the identified disease description.

[0022] Preferably, in S3, the disease descriptions and anatomical regions in each structured report are vectorized to obtain corresponding disease query vectors and anatomical region query vectors, including:

[0023] S31. Use the MedBert-base-Chinese model to vectorize the disease descriptions in each structured report to obtain the corresponding disease query vector:

[0024] Q m =f text (Description(S m ));

[0025] Among them, Description(S m ) is disease S m Corresponding disease description, m is the disease number, m∈[1,100], f text represents the MedBert-base-Chinese model, Q m For disease m The corresponding disease query vector has a size of (B, 100, D), where B is the batch size of the input network and D is the feature dimension;

[0026] S32. Use the MedBert-base-Chinese model to vectorize the anatomical regions in each structured report to obtain the corresponding anatomical region query vector:

[0027] Q n =f text (S n );

[0028] Among them, S n is the anatomical region, n is the anatomical region number, n∈[1,60], Q n For anatomical region S n The corresponding anatomical region query vector is of size (B, 60, D).

[0029] Preferably, in S4, the medical images corresponding to each structured report are converted into visual feature vectors to capture key visual features of the medical images, including:

[0030] The ResNet image encoder is used to convert the medical images corresponding to each structured report into visual feature vectors:

[0031] V k =f vision (I k );

[0032] Among them, I k is the input medical image, k is the medical image serial number, fvision ResNet image encoder, V k For Medical Imaging I k The corresponding visual feature vector has a size of (B, N, D), where N is the number of tokens after feature extraction by the ResNet image encoder.

[0033] Preferably, in S5, feature interactive learning is performed on the disease query vector and the visual feature vector, and the anatomical region query vector and the visual feature vector, respectively, to obtain corresponding visually weighted disease representations and visually weighted anatomical region representations, including:

[0034] S51. Take the disease query vector as the query Query, and the visual feature vector as the key Key and value Value. Use the Diseasequerytransformer model to perform feature interactive learning on the disease query vector and the visual feature vector to obtain the corresponding visual weighted disease representation:

[0035] A i =CrossAttention(Q m ,V k );

[0036] Among them, A i For disease m The corresponding disease query vector Q m and Medical Imaging I k The corresponding visual feature vector V k The visually weighted disease representation obtained after deep interaction is of size (B, 100, D). CrossAttention represents the cross attention mechanism.

[0037] S52, taking the anatomical region query vector as the query Query, taking the visual feature vector as the key Key and the value Value, and using the AnatomyQueryTransformer model to perform feature interactive learning on the anatomical region query vector and the visual feature vector, to obtain the corresponding visually weighted anatomical region representation:

[0038] A j =CrossAttention(Q n ,V k );

[0039] Among them, A j For anatomical region S n The corresponding anatomical region query vector Q n and Medical Imaging I k The corresponding visual feature vector V kVisually weighted anatomical region representation after deep interaction, size (B, 60, D).

[0040] Preferably, in S6, comparative learning is performed on the visually weighted disease representation and the disease label, and the visually weighted anatomical region representation and the anatomical region label, respectively, to enhance the performance of the medical image analysis model in disease identification and lesion localization, including:

[0041] S61, extracting diseases or important findings and anatomical region information from each report using the multimodal large language model GPT-4 as disease labels and anatomical region labels, respectively;

[0042] S62, respectively performing average pooling on the visually weighted disease representation and the visually weighted anatomical region representation to obtain corresponding pooled disease representation and pooled anatomical region representation;

[0043] S63. Compare the pooled disease representation with the disease label and use the loss function Loss disease for:

[0044]

[0045] Among them, Meanpooling (A i ) is the disease representation after pooling, L disease Meanpooling for disease labels T (A i )、L T disease They are the pooled disease representation and the transposition of the disease label, respectively. SCE represents the Softmax Cross-Entropy loss function.

[0046] S64. Compare the pooled anatomical region representation with the anatomical region label, and use the loss function Loss anatomy for:

[0047]

[0048] Among them, Meanpooling (A j ) is the anatomical region representation after pooling, L anatomy For anatomical region labels, Meanpooling T (A j )、L T anatomy They are the anatomical region representation after pooling and the transposition of the anatomical region label, respectively.

[0049] (III) Beneficial effects

[0050] Compared with the prior art, the present invention provides a training method for a medical image analysis model based on contrastive learning, which uses a multimodal large language model to structure the report, thereby clearly defining 100 diseases or important findings and 60 anatomical regions in medical images, and constructs a new model training system based on location perception based on these precisely labeled disease labels and anatomical region labels. In this system, disease query vectors and anatomical region query vectors are used as query queries, and feature interaction learning is performed with visual feature vectors respectively, and then the obtained deep interaction results are contrasted with disease labels and anatomical region labels respectively. The model obtained by this model training system can not only accurately understand the global structure of medical images, but also has the ability to keenly identify local lesion areas, so that it can make full use of limited labeled data to enhance the model's performance in disease identification and lesion localization, providing a more accurate and reliable medical image analysis model for clinical diagnosis, which can effectively improve the accuracy and efficiency of medical image analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 It is a schematic diagram of the process of the present invention;

[0053] Figure 2 A schematic diagram of the disease or important discovery or anatomical region in the present invention;

[0054] Figure 3 The figure is a schematic diagram of constructing a structured report library in the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] A training method for a medical image analysis model based on contrastive learning, such as Figure 1 As shown in S1, build a report library containing diseases or important findings and anatomical regions, including:

[0057] Based on the expertise of clinicians, a comprehensive set of 100 diseases or important findings and 60 anatomical regions (such as Figure 2 Report library shown in the figure);

[0058] Each disease or significant finding was arbitrarily combined with each anatomical region.

[0059] S2, extract the diseases or important findings and anatomical region information in each report, and describe the corresponding fine-grained information of the identified diseases (such as Figure 3 As shown), build a structured report library, including:

[0060] S21. Use the multimodal large language model GPT-4 to extract information about diseases or important findings and anatomical regions in each report;

[0061] S22. Build a structured report library with fine-grained information including manifestation, shape and texture corresponding to the identified disease description.

[0062] like Figure 3 As shown in the figure, the report is "Linear and flake-like increased density shadows are seen in the left upper lung field, and the left costophrenic angle is blurred." After the multimodal large language model GPT-4 structures the report, the diseases or important findings are obtained: pneumonia, pleural effusion; anatomical area: pneumonia-->left upper lung, pleural effusion-->left pleura; manifestation, shape and texture: pneumonia-->manifested as an opaque area on imaging..., pleural effusion: manifested as excessive fluid accumulation in the pleural cavity...

[0063] S3. Vectorize the disease descriptions and anatomical regions in each structured report to obtain corresponding disease query vectors and anatomical region query vectors, including:

[0064] S31. Use the MedBert-base-Chinese model to vectorize the disease descriptions in each structured report to obtain the corresponding disease query vector:

[0065] Q m =f text (Description(S m ));

[0066] Among them, Description(S m ) is disease S m Corresponding disease description, m is the disease number, m∈[1,100], f text represents the MedBert-base-Chinese model, Q m For disease mThe corresponding disease query vector has a size of (B, 100, D), where B is the batch size of the input network and D is the feature dimension;

[0067] S32. Use the MedBert-base-Chinese model to vectorize the anatomical regions in each structured report to obtain the corresponding anatomical region query vector:

[0068] Q n =f text (S n );

[0069] Among them, S n is the anatomical region, n is the anatomical region number, n∈[1,60], Q n For anatomical region S n The corresponding anatomical region query vector is of size (B, 60, D).

[0070] S4. Convert the medical images (including X-ray, CT, MRI, etc.) corresponding to each structured report into visual feature vectors to capture the key visual features of the medical images, including:

[0071] The ResNet image encoder is used to convert the medical images corresponding to each structured report into visual feature vectors:

[0072] V k =f vision (I k );

[0073] Among them, I k is the input medical image, k is the medical image serial number, f vision ResNet image encoder, V k For Medical Imaging I k The corresponding visual feature vector has a size of (B, N, D), where N is the number of tokens after feature extraction by the ResNet image encoder.

[0074] S5. Perform feature interactive learning on the disease query vector and the visual feature vector, and the anatomical region query vector and the visual feature vector, respectively, to obtain corresponding visually weighted disease representations and visually weighted anatomical region representations, specifically including:

[0075] S51. Take the disease query vector as the query Query, take the visual feature vector as the key Key and value Value, and use the Disease query transformer model to perform feature interactive learning on the disease query vector and the visual feature vector to obtain the corresponding visual weighted disease representation:

[0076] Ai =CrossAttention(Q m ,V k );

[0077] Among them, A i For disease m The corresponding disease query vector Q m and Medical Imaging I k The corresponding visual feature vector V k The visually weighted disease representation obtained after deep interaction is of size (B, 100, D). CrossAttention represents the cross attention mechanism.

[0078] S52, taking the anatomical region query vector as the query Query, taking the visual feature vector as the key Key and the value Value, and using the Anatomy query transformer model to perform feature interactive learning on the anatomical region query vector and the visual feature vector, to obtain the corresponding visually weighted anatomical region representation:

[0079] A j =CrossAttention(Q n ,V k );

[0080] Among them, A j For anatomical region S n The corresponding anatomical region query vector Q n and Medical Imaging I k The corresponding visual feature vector V k Visually weighted anatomical region representation after deep interaction, size (B, 60, D).

[0081] S6. Comparative learning is performed on visually weighted disease representations and disease labels, as well as visually weighted anatomical region representations and anatomical region labels, to enhance the performance of medical image analysis models in disease identification and lesion localization, including:

[0082] S61, extracting diseases or important findings and anatomical region information from each report using the multimodal large language model GPT-4 as disease labels and anatomical region labels, respectively;

[0083] S62, respectively performing average pooling on the visually weighted disease representation and the visually weighted anatomical region representation to obtain corresponding pooled disease representation and pooled anatomical region representation;

[0084] S63. Compare the pooled disease representation with the disease label and use the loss function Loss disease for:

[0085]

[0086] Among them, Meanpooling (A i ) is the disease representation after pooling, L disease Meanpooling for disease labels T (A i )、L T disease They are the pooled disease representation and the transposition of the disease label, respectively. SCE represents the Softmax Cross-Entropy loss function.

[0087] S64. Compare the pooled anatomical region representation with the anatomical region label, and use the loss function Loss anatomy for:

[0088]

[0089] Among them, Meanpooling (A j ) is the anatomical region representation after pooling, L anatomy For anatomical region labels, Meanpooling T (A j )、L T anatomy They are the anatomical region representation after pooling and the transposition of the anatomical region label, respectively.

[0090] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a medical image analysis model based on contrastive learning, characterized in that: The following steps are involved: S1. Build a library of reports containing diseases or important findings and anatomical regions; S2, extract the diseases or important findings and anatomical region information in each report, and construct a structured report library with fine-grained information corresponding to the identified disease descriptions; S3, vectorizing the disease description and anatomical region in each structured report to obtain corresponding disease query vectors and anatomical region query vectors; S4, converting the medical images corresponding to each structured report into a visual feature vector to capture the key visual features of the medical images; S5, performing feature interactive learning on the disease query vector and the visual feature vector, as well as the anatomical region query vector and the visual feature vector, respectively, to obtain corresponding visually weighted disease representations and visually weighted anatomical region representations; S6. Perform comparative learning on visually weighted disease representations and disease labels, as well as visually weighted anatomical region representations and anatomical region labels, to enhance the performance of medical image analysis models in disease recognition and lesion localization.

2. The training method of the medical image analysis model based on contrastive learning according to claim 1, characterized in that: S1 builds a library of reports containing diseases or important findings and anatomical regions, including: Based on the expertise of clinicians, a report library containing 100 diseases or important findings and 60 anatomical regions was constructed; Each disease or significant finding was arbitrarily combined with each anatomical region.

3. The training method of the medical image analysis model based on contrastive learning according to claim 2, characterized in that: S2 extracts diseases or important findings and anatomical region information from each report, and describes the corresponding fine-grained information of the identified diseases to build a structured report library, including: S21. Use the multimodal large language model GPT-4 to extract information about diseases or important findings and anatomical regions in each report; S22. Build a structured report library with fine-grained information including manifestation, shape and texture corresponding to the identified disease description.

4. The training method of the medical image analysis model based on contrastive learning according to claim 3, characterized in that: In S3, the disease descriptions and anatomical regions in each structured report are vectorized to obtain the corresponding disease query vectors and anatomical region query vectors, including: S31. Use the MedBert-base-Chinese model to vectorize the disease descriptions in each structured report to obtain the corresponding disease query vector: Q m =f text (Description(S m )); Among them, Description(S m ) is disease S m Corresponding disease description, m is the disease number, m∈[1,100], f text represents the MedBert-base-Chinese model, Q m For disease m The corresponding disease query vector has a size of (B, 100, D), where B is the batch size of the input network and D is the feature dimension; S32. Use the MedBert-base-Chinese model to vectorize the anatomical regions in each structured report to obtain the corresponding anatomical region query vector: Q n =f text (S n ); Among them, S n is the anatomical region, n is the anatomical region number, n∈[1,60], Q n For anatomical region S n The corresponding anatomical region query vector is of size (B, 60, D).

5. The training method of the medical image analysis model based on contrastive learning according to claim 4, characterized in that: In S4, the medical images corresponding to each structured report are converted into visual feature vectors to capture the key visual features of the medical images, including: The ResNet image encoder is used to convert the medical images corresponding to each structured report into visual feature vectors: V k =f vision (I k ); Among them, I k is the input medical image, k is the medical image serial number, f vision ResNet image encoder, V k For Medical Imaging I k The corresponding visual feature vector has a size of (B, N, D), where N is the number of tokens after feature extraction by the ResNet image encoder.

6. The training method of the medical image analysis model based on contrastive learning according to claim 5, characterized in that: In S5, feature interactive learning is performed on the disease query vector and the visual feature vector, as well as the anatomical region query vector and the visual feature vector, to obtain the corresponding visually weighted disease representation and visually weighted anatomical region representation, including: S51. Take the disease query vector as the query Query, and the visual feature vector as the key Key and value Value. Use the Diseasequerytransformer model to perform feature interactive learning on the disease query vector and the visual feature vector to obtain the corresponding visual weighted disease representation: A i =CrossAttention(Q m ,V k ); Among them, A i For disease m The corresponding disease query vector Q m and Medical Imaging I k The corresponding visual feature vector V k The visually weighted disease representation obtained after deep interaction is of size (B, 100, D). CrossAttention represents the cross attention mechanism. S52, taking the anatomical region query vector as the query Query, taking the visual feature vector as the key Key and the value Value, and using the AnatomyQueryTransformer model to perform feature interactive learning on the anatomical region query vector and the visual feature vector, to obtain the corresponding visually weighted anatomical region representation: A j =CrossAttention(Q n ,V k ); Among them, A j For anatomical region S n The corresponding anatomical region query vector Q n and Medical Imaging I k The corresponding visual feature vector V k Visually weighted anatomical region representation after deep interaction, size (B, 60, D).

7. The training method of the medical image analysis model based on contrastive learning according to claim 6, characterized in that: S6 conducts comparative learning on visually weighted disease representations and disease labels, as well as visually weighted anatomical region representations and anatomical region labels, to enhance the performance of medical image analysis models in disease identification and lesion localization, including: S61, extracting diseases or important findings and anatomical region information from each report using the multimodal large language model GPT-4 as disease labels and anatomical region labels, respectively; S62, respectively performing average pooling on the visually weighted disease representation and the visually weighted anatomical region representation to obtain corresponding pooled disease representation and pooled anatomical region representation; S63. Compare the pooled disease representation with the disease label and learn the loss function Loss disease for: Among them, Meanpooling (A i ) is the disease representation after pooling, L disease Meanpooling for disease labels T (A i ), They are the pooled disease representation and the transposition of the disease label, respectively. SCE represents the Softmax Cross-Entropy loss function. S64. Compare the pooled anatomical region representation with the anatomical region label, and use the loss function Loss anatomy for: Among them, Meanpooling (A j ) is the anatomical region representation after pooling, L anatomy For anatomical region labels, Meanpooling T (A j ), They are the anatomical region representation after pooling and the transposition of the anatomical region label, respectively.

Images