Training method of medical image visual encoder based on cross-sequence local contrast
By adopting a cross-sequence local contrast training method in 3D medical imaging vision encoder, combined with self-distillation framework and contrast learning loss, the problem of insufficient representation ability of visual encoder for micro-local features in the prior art is solved, and the perception ability of micro- and fuzzy lesions is significantly improved.
Patent Information
- Application Number
- CN202411974153.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-05-06
AI Technical Summary
When existing 3D medical imaging vision encoders process highly redundant 3D medical imaging data, it is difficult to effectively characterize micro-local features, resulting in weak perception of micro- and fuzzy lesions.
Using a training method based on cross-sequence local comparison, multiple global and local views are generated by randomly cropping 3D medical image data, and using a self-distillation framework combined with contrast learning losses, the student model and teacher model are guided to update under the supervision of overall loss, enhancing the ability to characterize local features.
It effectively improves the visual encoder's ability to characterize tiny local features in high redundant 3D medical image data, thereby improving the perception of tiny and fuzzy lesions, and promoting the full use of multi-modal large models of medical image.
Smart Images
Figure CN119940464A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to model training, and in particular to a training method for a medical image visual encoder based on cross-sequence local comparison. Background Art
[0002] Reliable medical image visual encoders are the basis for the multimodal large model of medical images to achieve many tasks such as disease identification, reasoning, and positioning. Among them, CT, MRI, etc. are the most commonly used 3D medical images in clinical diagnosis. They can clearly display information at all levels of tissues and organs, provide a complete three-dimensional structural view, and help doctors more clearly understand the size, shape, location and other details of the lesion.
[0003] There are two main training methods for existing 3D medical image visual encoders: one is image + text contrastive learning (CLIP, Contrastive Language-Image Pre-training), which is to collect a large number of 3D medical images and corresponding reports, image descriptions and other texts, and train the visual encoder through contrastive learning; the other is pure image self-supervised learning (SSL, Self-Supervised Learning for Images), that is, using image data alone without image annotation, and training the visual encoder through self-supervised agent tasks such as contrastive learning, image reconstruction, and spatiotemporal sequence prediction. Currently, the best method for this training method is the DINO series.
[0004] Since the training method of image + text contrastive learning requires images and corresponding text descriptions to form samples, and text descriptions are often scarce, the amount of training data is relatively limited. However, the text descriptions corresponding to the images can help the visual encoder learn more fine-grained local features of each organ or a specific location.
[0005] For the training method of pure image self-supervised learning, since only image data is needed for training, more data can usually be used for training. However, the lack of guidance from annotations and text descriptions makes it difficult for the visual encoder to learn local features. For example, the classic SSL method DINO constrains an image to maintain a consistent distribution between class tokens generated by different views to supervise the training of the visual encoder. However, the class token is essentially a global description of the image, so such a contrast constraint cannot directly make the visual encoder focus on local features, resulting in the visual encoder's weak ability to represent tiny local features, which is necessary for medical image analysis. Summary of the invention
[0006] 1. Technical issues to be resolved
[0007] In view of the above-mentioned shortcomings of the prior art, the present invention provides a training method for a medical image visual encoder based on cross-sequence local comparison, which can effectively overcome the defect of the prior art that the visual encoder has a weak ability to represent tiny local features in highly redundant 3D medical image data.
[0008] (II) Technical solution
[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions:
[0010] The training method of the medical image visual encoder based on cross-sequence local comparison includes the following steps:
[0011] S1. Randomly cropping a plurality of first global views and first local views from a sequence of 3D medical image data, inputting the first global views into the student model and the teacher model respectively, and inputting the first local views into the student model;
[0012] S2, randomly cropping a plurality of second global views and second local views from another sequence of 3D medical image data, and inputting the second global views and the second local views into the teacher model;
[0013] S3. According to the output result of the student model, select the first local view that best matches the first global view, and use the first local view as the anchor local view;
[0014] S4. Construct positive and negative sample pairs for contrastive learning based on the anchor point local view and calculate the contrastive learning loss;
[0015] S5. Calculate the cross entropy loss of the self-distillation framework itself and combine it with the contrastive learning loss to determine the overall loss.
[0016] S6. Through the supervision of the overall loss, guide the student model parameters and the teacher model parameters to update.
[0017] Preferably, in S1, a plurality of first global views and first local views are randomly cropped from a sequence of 3D medical image data, the first global views are respectively input into the student model and the teacher model, and the first local views are input into the student model, including:
[0018] Randomly crop two first global views g1 and g2 and eight first local views l1, l2, ..., l8 from a sequence of 3D medical image data, input the two first global views g1 and g2 into the student model and the teacher model respectively, and input the eight first local views l1, l2, ..., l8 into the student model;
[0019] Among them, the size of 3D medical image data is 448*448*64, the two first global views g1 and g2 represent the global description of the image under different perturbations, and the size is consistent with the original image. The eight first local views l1, l2, …, l8 represent the local description of the image at different positions, and the size is 112*112*16.
[0020] Preferably, in S2, a plurality of second global views and second local views are randomly cropped from another sequence of 3D medical image data, and the second global views and the second local views are input into the teacher model, including:
[0021] Randomly crop two second global views g3 and g4 and eight second local views from another sequence of 3D medical image data, and input the two second global views and the eight second local views into the teacher model;
[0022] Among them, the positions of 3D medical imaging data of different sequences remain fixed when randomly cropped.
[0023] Preferably, in S3, according to the output result of the student model, the first local view that best matches the first global view is selected, and the first local view is used as the anchor local view, including:
[0024] S31, perform dot product operations on the class token vectors of the first local views l1, l2, ..., l8 output by the student model and the class token vector of the first global view g1 output by the student model, and obtain the corresponding similarity scores:
[0025]
[0026] in, The first local view l i The class token vector, is the classtoken vector of the first global view g1 The transpose of Indicates the calculation of similarity score, The first local view l i Similarity score with the first global view g1, i∈[1,8];
[0027] S32, taking the first local view with the highest similarity score as the anchor local view l that best matches the first global view g1 * :
[0028]
[0029] Preferably, in S4, constructing positive and negative sample pairs for contrastive learning based on the anchor point local view and calculating the contrastive learning loss includes:
[0030] S41, anchor point local view l * A positive sample pair is formed between it and the first global view g1;
[0031] S42, local view 1 except anchor point * The first partial view and all second partial views other than the anchor partial view l * A negative sample pair is formed between them;
[0032] S43. Calculate the contrastive learning loss L based on the positive sample pairs and negative sample pairs contr :
[0033]
[0034] in, is the anchor point local view l * classtoken vector, sim(·) represents the calculation of cosine similarity, τ is a hyperparameter for controlling contrastive learning loss, is the class token vector of the local view in the negative sample pair, and m is the number of local views in the negative sample pair.
[0035] Preferably, in S5, the cross entropy loss of the self-distillation framework itself is calculated, and the overall loss is determined in combination with the contrastive learning loss, including:
[0036] S51, local view l of anchor point output by student model * , the class token vector of the first global view g1, and the class token vectors of the first global view g2, the second global view g3, and the second global view g4 output by the teacher model, and calculate the cross entropy loss L of the self-distillation framework itself dino :
[0037]
[0038] in, are the classtoken vectors of the first global view g2, the second global view g3, and the second global view g4, respectively, H(a,b)=-alogb;
[0039] S52. According to the contrastive learning loss L contr and the cross entropy loss L dino Determine the overall loss.
[0040] Preferably, in S6, the student model parameters and the teacher model parameters are guided to be updated through the supervision of the overall loss, including:
[0041] Through the supervision of the overall loss, the student model parameters are guided to be updated through gradient back propagation, and the teacher model parameters are updated through EMA exponential sliding average;
[0042] The student model and the teacher model use the same network parameters and use the 3Dvit-base model as the backbone network. The input batch size is 16*16*16 and the output feature dimension is 768.
[0043] (III) Beneficial effects
[0044] Compared with the prior art, the training method of the medical image visual encoder based on cross-sequence local contrast provided by the present invention follows the DINO self-distillation framework in the overall structure of the visual encoder, innovatively introduces the contrast learning loss of the local area as a regularization term, guides the model to explore the differences between different local areas, and enhances the visual encoder's ability to represent tiny local features in highly redundant 3D medical image data, thereby improving the visual encoder's perception of tiny and blurred lesions. At the same time, the present invention further combines multi-sequence 3D medical image data to construct positive and negative sample pairs between cross-sequence medical image data, helping the visual encoder to align input images of different sequences and map them to the joint feature space, which is conducive to the medical image multimodal large model to make more full use of multi-sequence medical image information. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0046] Figure 1 It is a schematic diagram of the process of the present invention;
[0047] Figure 2 It is a schematic diagram of the process of screening anchor point local views in the present invention. DETAILED DESCRIPTION
[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0049] Training methods for medical image visual encoders based on local comparison across sequences, such as Figure 1 As shown, S1, randomly cropping multiple first global views and first local views from a sequence of 3D medical image data, inputting the first global views into the student model and the teacher model respectively, and inputting the first local views into the student model, specifically including:
[0050] Randomly crop two first global views g1 and g2 and eight first local views l1, l2, ..., l8 from a sequence of 3D medical image data, input the two first global views g1 and g2 into the student model and the teacher model respectively, and input the eight first local views l1, l2, ..., l8 into the student model;
[0051] Among them, the size of 3D medical image data is 448*448*64, the two first global views g1 and g2 represent the global description of the image under different perturbations, and the size is consistent with the original image. The eight first local views l1, l2, …, l8 represent the local description of the image at different positions, and the size is 112*112*16.
[0052] In the technical solution of the present application, the cross-sequence 3D medical imaging data may be lung window, mediastinum window, bone window sequences in CT, and T1, T2, and Flair sequences in MRI.
[0053] S2, randomly cropping a plurality of second global views and second local views from another sequence of 3D medical image data, and inputting the second global views and the second local views into the teacher model, specifically comprising:
[0054] Randomly crop two second global views g3 and g4 and eight second local views from another sequence of 3D medical image data, and input the two second global views and the eight second local views into the teacher model;
[0055] Among them, the positions of 3D medical imaging data of different sequences remain fixed when randomly cropped.
[0056] DINO constrains the output distribution of all local views to be consistent with the output distribution of the global view. Although this method helps to align global and local feature representations, it also hinders the visual encoder from exploring the differences between different local regions. Therefore, the technical solution of this application is improved here to construct positive and negative sample pairs between cross-sequence medical imaging data, replacing the original local view and global view output distribution consistency constraints.
[0057] Specifically, it is necessary to construct positive and negative sample pairs for contrastive learning. Considering that directly removing the contrast between all local views and the global view may damage the consistency between local features and global features, forcing all local views to have consistent output distributions with the global view will inhibit the ability of the visual encoder to explore local regional differences. In order to strike a balance between the two, the technical solution of this application selects the local view that best matches the global view as the anchor local view, and constructs positive and negative sample pairs between cross-sequence medical imaging data.
[0058] S3. According to the output result of the student model, a first local view that best matches the first global view is selected, and the first local view is used as the anchor local view, specifically including:
[0059] S31, perform dot product operations on the class token vectors of the first local views l1, l2, ..., l8 output by the student model and the class token vector of the first global view g1 output by the student model, and obtain the corresponding similarity scores:
[0060]
[0061] in, The first local view l i The class token vector, is the classtoken vector of the first global view g1 The transpose of Indicates the calculation of similarity score, The first local view l i Similarity score with the first global view g1, i∈[1,8];
[0062] S32, taking the first local view with the highest similarity score as the anchor local view l that best matches the first global view g1 * :
[0063]
[0064] S4. Construct positive and negative sample pairs for contrastive learning based on the anchor point local view and calculate the contrastive learning loss, including:
[0065] S41, anchor point local view l * A positive sample pair is formed between it and the first global view g1;
[0066] S42, local view 1 except anchor point * The first partial view and all second partial views other than the anchor partial view l * A negative sample pair is formed between them;
[0067] S43. Calculate the contrastive learning loss L based on the positive sample pairs and negative sample pairs contr :
[0068]
[0069] in, is the anchor point local view l * classtoken vector, sim(·) represents the calculation of cosine similarity, τ is a hyperparameter for controlling contrastive learning loss, is the class token vector of the local view in the negative sample pair, and m is the number of local views in the negative sample pair.
[0070] S5. Calculate the cross entropy loss of the self-distillation framework itself and combine it with the contrastive learning loss to determine the overall loss, including:
[0071] S51, local view l of anchor point output by student model * , the class token vector of the first global view g1, and the class token vectors of the first global view g2, the second global view g3, and the second global view g4 output by the teacher model, and calculate the cross entropy loss L of the self-distillation framework itself dino :
[0072]
[0073] in, are the classtoken vectors of the first global view g2, the second global view g3, and the second global view g4, respectively, H(a,b)=-alogb;
[0074] S52. According to the contrastive learning loss L contr and the cross entropy loss L dino Determine the overall loss.
[0075] S6. Guide the update of student model parameters and teacher model parameters through the supervision of overall loss, including:
[0076] Through the supervision of the overall loss, the student model parameters are guided to be updated through gradient back propagation, and the teacher model parameters are updated through EMA exponential sliding average;
[0077] The student model and the teacher model use the same network parameters and use the 3Dvit-base model as the backbone network. The input batch size is 16*16*16 and the output feature dimension is 768.
[0078] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A training method for medical image visual encoder based on cross-sequence local comparison, characterized by: The following steps are involved: S1. Randomly cropping a plurality of first global views and first local views from a sequence of 3D medical image data, inputting the first global views into the student model and the teacher model respectively, and inputting the first local views into the student model; S2, randomly cropping a plurality of second global views and second local views from another sequence of 3D medical image data, and inputting the second global views and the second local views into the teacher model; S3. According to the output result of the student model, select the first local view that best matches the first global view, and use the first local view as the anchor local view; S4. Construct positive and negative sample pairs for contrastive learning based on the anchor point local view and calculate the contrastive learning loss; S5. Calculate the cross entropy loss of the self-distillation framework itself and combine it with the contrastive learning loss to determine the overall loss. S6. Through the supervision of the overall loss, guide the student model parameters and the teacher model parameters to update.
2. The method for training a medical image visual encoder based on cross-sequence local comparison according to claim 1, characterized in that: In S1, a plurality of first global views and first local views are randomly cropped from a sequence of 3D medical image data, the first global views are respectively input into the student model and the teacher model, and the first local views are input into the student model, including: Randomly crop two first global views g1 and g2 and eight first local views l1, l2, ..., l8 from a sequence of 3D medical image data, input the two first global views g1 and g2 into the student model and the teacher model respectively, and input the eight first local views l1, l2, ..., l8 into the student model; Among them, the size of 3D medical image data is 448*448*64, the two first global views g1 and g2 represent the global description of the image under different perturbations, and the size is consistent with the original image. The eight first local views l1, l2, …, l8 represent the local description of the image at different positions, and the size is 112*112*16.
3. The method for training a medical image visual encoder based on cross-sequence local comparison according to claim 2, characterized in that: In S2, a plurality of second global views and second local views are randomly cropped from another sequence of 3D medical image data, and the second global views and the second local views are input into the teacher model, including: Randomly crop two second global views g3 and g4 and eight second local views from another sequence of 3D medical image data, and input the two second global views and the eight second local views into the teacher model; Among them, the positions of 3D medical imaging data of different sequences remain fixed when randomly cropped.
4. The method for training a medical image visual encoder based on cross-sequence local comparison according to claim 3, characterized in that: In S3, according to the output result of the student model, the first local view that best matches the first global view is selected, and the first local view is used as the anchor local view, including: S31, perform dot product operations on the class token vectors of the first local views l1, l2, ..., l8 output by the student model and the class token vector of the first global view g1 output by the student model, and obtain the corresponding similarity scores: in, is the first partial view l i The class token vector, is the classtoken vector of the first global view g1 The transpose of Indicates the calculation of similarity score, is the first partial view l i Similarity score with the first global view g1, i∈[1,8]; S32, taking the first local view with the highest similarity score as the anchor local view l that best matches the first global view g1 * :
5. The method for training a medical image visual encoder based on cross-sequence local comparison according to claim 4, characterized in that: In S4, positive and negative sample pairs for contrastive learning are constructed based on the local view of the anchor point, and the contrastive learning loss is calculated, including: S41, anchor point local view l * A positive sample pair is formed between it and the first global view g1; S42, local view 1 except anchor point * The first partial view and all second partial views other than the anchor partial view l * A negative sample pair is formed between them; S43. Calculate the contrastive learning loss L based on the positive sample pairs and negative sample pairs contr : in, is the anchor point local view l * classtoken vector, sim(·) represents the calculation of cosine similarity, τ is a hyperparameter for controlling contrastive learning loss, is the class token vector of the local view in the negative sample pair, and m is the number of local views in the negative sample pair.
6. The method for training a medical image visual encoder based on cross-sequence local comparison according to claim 5, characterized in that: In S5, the cross entropy loss of the self-distillation framework itself is calculated, and the overall loss is determined by combining the contrastive learning loss, including: S51, local view l of anchor point output by student model * , the class token vector of the first global view g1, and the class token vectors of the first global view g2, the second global view g3, and the second global view g4 output by the teacher model, and calculate the cross entropy loss L of the self-distillation framework itself dino : in, are the classtoken vectors of the first global view g2, the second global view g3, and the second global view g4, respectively, H(a,b)=-alogb; S52. According to the contrastive learning loss L contr and the cross entropy loss L dino Determine the overall loss.
7. The method for training a medical image visual encoder based on cross-sequence local comparison according to claim 6, characterized in that: In S6, the overall loss supervision is used to guide the update of the student model parameters and the teacher model parameters, including: Through the supervision of the overall loss, the student model parameters are guided to be updated through gradient back propagation, and the teacher model parameters are updated through EMA exponential sliding average; The student model and the teacher model use the same network parameters and use the 3Dvit-base model as the backbone network. The input batch size is 16*16*16 and the output feature dimension is 768.