A Medical Image Analysis Method Based on Prior Location Information

Abstract

This invention relates to medical image analysis, specifically to a medical image analysis method based on prior location information. The method initializes learnable queries for different regions according to the importance of different body parts; inputs a 3D medical image, extracts 3D visual features in the corresponding regions based on the prior location information, and extracts local features of specific regions from all 3D visual features; performs feature interaction learning between the learnable queries and local features of each region to obtain visual interaction features for each region; classifies the target region based on its visual interaction features using a classification head, and processes the classification results into text to obtain a textual description of the target region's classification result; and generates a textual report of the target region based on its visual interaction features, the textual description of the classification result, and user command text. The technical solution provided by this invention effectively overcomes the deficiency of accurately generating textual reports of target regions.

Description

Technical Field

[0001] This invention relates to medical image analysis, and more specifically to a medical image analysis method based on prior location information. Background Technology

[0002] The medical field contains a vast amount of multimodal information, including patient information, diagnostic reports, and medical images from various imaging modalities. The combination of diagnostic reports and medical images provides accurate and detailed descriptions, findings, and diagnoses, and is considered high-quality annotation. These diagnostic reports and medical images, along with physicians' diagnostic workflows, are stored on a large scale in databases without additional cost. The key question then becomes how to fully utilize this data for accurate analysis of medical images.

[0003] The existing technical solution mainly involves using the ViT visual encoder to encode CT images, then inputting them into a large language model (LLM) after simple resampling and dimensional transformation. The large language model (LLM) then outputs a text report, and the training method usually involves predicting the next self-supervised training mode.

[0004] The methods described above struggle to effectively learn the location of targets in 3D medical images, primarily because they lack direct and effective supervision in establishing a connection between features and text. For medical image text report generation tasks, the correlation between location descriptions and image features is crucial for the accuracy of the report; therefore, enhancing the information about the target location is an urgent issue that needs to be addressed.

[0005] Furthermore, most of these methods employ a cascaded model of "visual encoder + feature sampler + LLM," where the visual encoder undergoes pre-training in a separate stage, and then is fine-tuned in the final stage after being integrated with the feature sampler. In this training mode, for the sake of generality, the pre-training process of the visual encoder does not effectively learn and guide query information for specific regions. Although it can be easily integrated into various LLM frameworks, the extracted features lack emphasis on region-specific characteristics, resulting in poor performance for lesion detection, which heavily relies on region-specific features. Summary of the Invention

[0006] (a) Technical problems to be solved

[0007] In view of the above-mentioned shortcomings of the existing technology, the present invention provides a medical image analysis method based on prior location information, which can effectively overcome the defect of the existing technology that makes it difficult to accurately generate text reports of target areas.

[0008] (II) Technical Solution

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] A medical image analysis method based on prior location information includes the following steps:

[0011] S1. Initialize the learnable queries for each region based on the importance of different parts;

[0012] S2. Input a 3D medical image, extract 3D visual features in the corresponding area based on prior location information, and extract local features of a specific area from all 3D visual features.

[0013] S3. Perform feature interaction learning on the learnable queries of each region and the local features of each region to obtain the visual interaction features of each region.

[0014] S4. Use the segmentation head to segment the input image based on the visual interaction features of the target region to obtain the segmentation mask of the target region;

[0015] S5. Classify the target region based on the visual interaction features of the target region using the classification head, and process the classification results into text to obtain a textual description of the classification results of the target region.

[0016] S6. Input the visual interaction features of the target region, the textual description of the classification results, and the user command text into the large language model LLM to obtain the text report of the target region, and output it together with the segmentation mask of the target region.

[0017] Preferably, in S1, learnable queries for different regions are initialized according to the importance of different parts, including:

[0018] Based on the type of lesion in different locations, initialize learnable queries for different regions:

[0019] Queries={Q 左肺 Q 右肺 Q 纵膈};

[0020] Among them, Q 左肺 For the left lung queries, considering the variety of lung lesions, the number of queries was assigned to 20. 右肺 For the right lung query, considering the variety of lung lesions, the number of queries is 20. 纵膈 For mediastinal queries, considering the variety of lung lesions, the number is allocated to 10.

[0021] Preferably, in step S2, a 3D medical image is input, and 3D visual features are extracted in the corresponding region based on prior location information. Furthermore, local features of a specific region are extracted from all 3D visual features, including:

[0022] S21. Input a 3D medical image and align the input image. The positions of each part are fixed and known. Use a pre-trained SAT model to extract the region mask of the input image to obtain the prior location information proposal.

[0023] S22. Using the ViT visual encoder, 3D visual features are extracted in the corresponding region based on prior location information proposals.

[0024] S23. Use the ROIAlign method to extract local features of specific regions from all 3D visual features:

[0025] F 左肺 =ROIAlign[SAT(left lung); F];

[0026] Where SAT (left lung) represents the region mask of the left lung extracted from the input image by the SAT model, and F represents the 3D visual features of all regions in the input image. 左肺 This refers to the local features of the left lung in the input image;

[0027] S24. Based on the local features of all specific regions, obtain the local features of the input image:

[0028] F region ={F 左肺 ;F 右肺 ;F 纵膈};

[0029] Among them, F 左肺 F 右肺 F 纵膈 F represents the local features of the left lung, right lung, and mediastinum in the input image, respectively. region These are local features of the input image.

[0030] Preferably, in S3, feature interaction learning is performed on the learnable queries of each region and the local features of each region to obtain the visual interaction features of each region, including:

[0031] S31. Perform feature interaction learning on the learnable queries of each region and the local features of each region to obtain the visual interaction features of each region:

[0032] T 左肺 =CrossAttention(Q) 左肺 ;F 左肺 );

[0033] Where CrossAttention represents the cross-attention mechanism, T 左肺The visual interaction features of the left lung in the input image;

[0034] S32. Based on the visual interaction features of all regions, obtain the visual interaction features of the input image:

[0035] T region ={T 左肺 ;T 右肺 ;T 纵膈};

[0036] Among them, T 左肺 T 右肺 T 纵膈 These are the visual interaction features of the left lung, right lung, and mediastinum in the input image, respectively. region The visual interactive features of the input image.

[0037] Preferably, in step S4, the segmentation head performs region segmentation on the input image based on the visual interaction features of the target region to obtain a segmentation mask for the target region, including:

[0038] S41. Use the segmentation task to pre-train the segmentation head, directly guiding the segmentation head to learn the location and characteristics of the region, and obtain the pre-trained segmentation head;

[0039] S42. Input the visual interaction features of the target region into the pre-trained segmentation head, and have the segmentation head perform region segmentation on the input image to obtain the segmentation mask of the target region.

[0040] Preferably, in step S5, the classification head is used to classify the target region based on its visual interaction features, and the classification results are processed into text to obtain a textual description of the classification results for the target region, including:

[0041] S51. The classification head is pre-trained using a classification task. Classification labels are extracted from the text report of medical images, and then the visual interaction features of each region in the medical images are used for classification prediction. The loss function is calculated based on the classification prediction results and classification labels to guide the classification head to learn, so as to emphasize the regionality of visual interaction features and make the specified learnable queries focus more on their own regional semantics.

[0042] S52. The classification head is pre-trained using the image-text comparison task. The visual interaction features of each region in the medical image and the corresponding text report paragraphs are used as positive sample pairs, and different paragraphs in the same text report are used as negative sample pairs to form positive and negative sample pairs for image-text comparison. This guides the classification head to perform comparative learning, so as to guide the visual interaction features to expand their own semantics to the region-related descriptions. While ensuring the region semantics, more fine-grained semantic guidance is added to obtain the pre-trained classification head.

[0043] S53. Input the visual interaction features of the target region into the pre-trained classification head, and the classification head will perform classification to obtain the classification result;

[0044] S54. Perform text processing on the classification results to obtain a text description of the classification results for the target region.

[0045] Preferably, in S6, the visual interaction features of the target region, the textual description of the classification results, and the user command text are input into a large-scale language model (LLM) to obtain a textual report of the target region, which is output along with the segmentation mask of the target region, including:

[0046] S61. Use the LLM self-supervised task to pre-train the large language model LLM, so as to obtain a broader semantic guidance than the image-text comparison task through the text guidance provided by the large language model LLM, and obtain the pre-trained large language model LLM.

[0047] S62. Input the visual interaction features of the target region, the textual description of the classification results, and the user instruction text Prompt into the pre-trained large language model LLM, and the large language model LLM generates a textual report of the target region.

[0048] S63. Output the text report of the target region along with the segmentation mask of the target region;

[0049] Among them, when using the LLM self-supervised task to pre-train a large language model LLM, the obtained semantic guidance has strong noise. Therefore, the LLM self-supervised task loss function has the smallest weight in the overall loss function of pre-training.

[0050] Preferably, the overall loss function of the pre-training includes loss functions for segmentation tasks, classification tasks, image-text comparison tasks, and LLM self-supervised tasks:

[0051] Loss = αLoss DiceLoss +βLoss BCE +γLoss ContractiveLoss +δLoss LLM ;

[0052] Among them, Loss DiceLoss Loss BCE Loss ContractiveLoss Loss LLM α, β, γ, and δ are the loss functions for segmentation, classification, image-text comparison, and LLM self-supervised tasks, respectively. α, β, γ, and δ are the weights of the loss functions for segmentation, classification, image-text comparison, and LLM self-supervised tasks, respectively. Loss is the overall loss function for pre-training.

[0053] (III) Beneficial Effects

[0054] Compared with existing technologies, the medical image analysis method based on prior location information provided by this invention has the following beneficial effects:

[0055] 1) Extracting local features of a specific region based on prior location information can effectively reduce computation, improve detection accuracy, and enhance the semantic interaction between image features and target location;

[0056] 2) Learnable queries and local features of each region are learned through feature interaction learning, and pre-trained through segmentation tasks, classification tasks, image-text comparison tasks and LLM self-supervised tasks. Although these tasks are general downstream tasks, the information they guide is very consistent with the features and objectives designed in this invention. The model performance is improved through purposeful downstream task learning.

[0057] 3) Unlike existing training modes, this invention pre-trains the visual encoder together, and based on this training mode, this invention has the ability to generate text reports for a specified area. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0059] Figure 1 This is a schematic diagram of the process of the present invention;

[0060] Figure 2 This is a schematic diagram of the process for generating a segmentation mask and text report for the left lung in this invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0062] A medical image analysis method based on prior location information, such as Figure 1 As shown, S1 initializes learnable queries for different regions based on the importance of different parts, specifically including:

[0063] Based on the type of lesion in different locations, initialize learnable queries for different regions:

[0064] Queries={Q 左肺 Q 右肺 Q 纵膈};

[0065] Among them, Q 左肺 For the left lung queries, considering the variety of lung lesions, the number of queries was assigned to 20. 右肺 For the right lung query, considering the variety of lung lesions, the number of queries is 20. 纵膈 For mediastinal queries, considering the variety of lung lesions, the number is allocated to 10.

[0066] S2. Input a 3D medical image, extract 3D visual features in the corresponding region based on prior location information, and extract local features of a specific region from all 3D visual features, specifically including:

[0067] S21. Input a 3D medical image and align the input image. The positions of each part are fixed and known. Use a pre-trained SAT model to extract the region mask of the input image to obtain the prior location information proposal.

[0068] S22. Using the ViT visual encoder, 3D visual features are extracted in the corresponding region based on prior location information proposals.

[0069] S23. Use the ROIAlign method to extract local features of specific regions from all 3D visual features:

[0070] F 左肺 =ROIAlign[SAT(left lung); F];

[0071] Where SAT (left lung) represents the region mask of the left lung extracted from the input image by the SAT model, and F represents the 3D visual features of all regions in the input image. 左肺 This refers to the local features of the left lung in the input image;

[0072] S24. Based on the local features of all specific regions, obtain the local features of the input image:

[0073] F region ={F 左肺 ;F 右肺 ;F 纵膈};

[0074] Among them, F 左肺 F右肺 F 纵膈 F represents the local features of the left lung, right lung, and mediastinum in the input image, respectively. region These are local features of the input image.

[0075] In the technical solution of this application, such as Figure 1 As shown, the Target Model is suitable for large-scale multimodal models of medical images. After pre-training, the model's output features can be used as visual features to input into a large-scale language model (LLM). This includes a Visual Encoder (which can be a visual encoder with spatial semantics such as ViT or Conv) that calculates image features, and a Cross-Attention module that extracts region features, all of which are trained as a whole. After integrating the large-scale language model (LLM), only a small amount of data is needed for fine-tuning to complete the empowerment process.

[0076] S3. Perform feature interaction learning on the learnable queries of each region and the local features of each region to obtain the visual interaction features of each region, specifically including:

[0077] S31. Perform feature interaction learning on the learnable queries of each region and the local features of each region to obtain the visual interaction features of each region:

[0078] T 左肺 =CrossAttention(Q) 左肺 ;F 左肺 );

[0079] Where CrossAttention represents the cross-attention mechanism, T 左肺 The visual interaction features of the left lung in the input image;

[0080] S32. Based on the visual interaction features of all regions, obtain the visual interaction features of the input image:

[0081] T region ={T 左肺 ;T 右肺 ;T 纵膈};

[0082] Among them, T 左肺 T 右肺 T 纵膈 These are the visual interaction features of the left lung, right lung, and mediastinum in the input image, respectively. region The visual interactive features of the input image.

[0083] S4. Using a segmentation head, the input image is segmented based on the visual interaction features of the target region to obtain a segmentation mask for the target region, specifically including:

[0084] S41. Use the segmentation task to pre-train the segmentation head, directly guiding the segmentation head to learn the location and characteristics of the region, and obtain the pre-trained segmentation head;

[0085] S42. Input the visual interaction features of the target region into the pre-trained segmentation head, and have the segmentation head perform region segmentation on the input image to obtain the segmentation mask of the target region.

[0086] S5. Classify the target region based on the visual interaction features using a classification head, and then process the classification results into text to obtain a textual description of the classification results for the target region, specifically including:

[0087] S51. The classification head is pre-trained using a classification task. Classification labels are extracted from the text report of medical images, and then the visual interaction features of each region in the medical images are used for classification prediction. The loss function is calculated based on the classification prediction results and classification labels to guide the classification head to learn, so as to emphasize the regionality of visual interaction features and make the specified learnable queries focus more on their own regional semantics.

[0088] S52. The classification head is pre-trained using the image-text comparison task. The visual interaction features of each region in the medical image and the corresponding text report paragraphs are used as positive sample pairs, and different paragraphs in the same text report are used as negative sample pairs to form positive and negative sample pairs for image-text comparison. This guides the classification head to perform comparative learning, so as to guide the visual interaction features to expand their own semantics to the region-related descriptions. While ensuring the region semantics, more fine-grained semantic guidance is added to obtain the pre-trained classification head.

[0089] S53. Input the visual interaction features of the target region into the pre-trained classification head, and the classification head will perform classification to obtain the classification result;

[0090] S54. Perform text processing on the classification results to obtain a text description of the classification results for the target region.

[0091] S6. Input the visual interaction features of the target region, the textual description of the classification results, and the user command text into a large-scale language model (LLM) to obtain a textual report of the target region, which is output along with the segmentation mask of the target region. Specifically, this includes:

[0092] S61. Use the LLM self-supervised task to pre-train the large language model LLM, so as to obtain a broader semantic guidance than the image-text comparison task through the text guidance provided by the large language model LLM, and obtain the pre-trained large language model LLM.

[0093] S62. Input the visual interaction features of the target region, the textual description of the classification results, and the user instruction text Prompt into the pre-trained large language model LLM, and the large language model LLM generates a textual report of the target region.

[0094] S63. Output the text report of the target region along with the segmentation mask of the target region;

[0095] Among them, when using the LLM self-supervised task to pre-train a large language model LLM, the obtained semantic guidance has strong noise. Therefore, the LLM self-supervised task loss function has the smallest weight in the overall loss function of pre-training.

[0096] In the technical solution of this application, the overall loss function for pre-training includes loss functions for segmentation tasks, classification tasks, image-text comparison tasks, and LLM self-supervised tasks:

[0097] Loss = αLoss DiceLoss +βLoss BCE +γLoss ContractiveLoss +δLoss LLM ;

[0098] Among them, Loss DiceLoss Loss BCE Loss ContractiveLoss Loss LLM α, β, γ, and δ are the loss functions for segmentation, classification, image-text comparison, and LLM self-supervised tasks, respectively. α, β, γ, and δ are the weights of the loss functions for segmentation, classification, image-text comparison, and LLM self-supervised tasks, respectively. Loss is the overall loss function for pre-training.

[0099] In the technical solution of this application, for each case in various tasks, there are multiple lesion regions and corresponding multiple text reports paired. During pre-training, not all pairs are used; pairs are randomly discarded. This strengthens the information correlation between image and text pairs and enables the system to learn to output text reports with different region combinations based on different input queries (each region query).

[0100] To better illustrate the technical solution of this application, the following explanation uses a query for left lung disease as an example. Figure 2 As shown:

[0101] To query for diseases of the left lung, simply input the left lung query to perform feature query. Then, concatenate the corresponding visual interaction features of the left lung with the textual description of the left lung classification results and the user command text Prompt (What diseases are there in the lungs?), and input them into a large language model LLM. Finally, it generates and outputs a text report on the left lung, along with an accompanying lung segmentation mask.

[0102] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A medical image analysis method based on prior location information, characterized in that: Includes the following steps: S1. Initialize the learnable queries for each region based on the importance of different parts; S2. Input a 3D medical image, extract 3D visual features in the corresponding area based on prior location information, and extract local features of a specific area from all 3D visual features. S3. Perform feature interaction learning on the learnable queries of each region and the local features of each region to obtain the visual interaction features of each region. S4. Use the segmentation head to segment the input image based on the visual interaction features of the target region to obtain the segmentation mask of the target region; S5. Classify the target region based on the visual interaction features of the target region using the classification head, and process the classification results into text to obtain a textual description of the classification results of the target region. S6. Input the visual interaction features of the target region, the textual description of the classification results, and the user command text into the large language model LLM to obtain the text report of the target region, and output it together with the segmentation mask of the target region.

2. The medical image analysis method based on prior location information according to claim 1, characterized in that: In S1, learnable queries for different regions are initialized based on the importance of different parts, including: Based on the type of lesion in different locations, initialize learnable queries for different regions: Queries={Q 左肺 ;Q 右肺 ;Q 纵膈 }; Among them, Q 左肺 For the left lung queries, considering the variety of lung lesions, the number of queries was assigned to 20. 右肺 For the right lung query, considering the variety of lung lesions, the number of queries is 20. 纵膈 For mediastinal queries, considering the variety of lung lesions, the number is allocated to 10.

3. The medical image analysis method based on prior location information according to claim 2, characterized in that: Input a 3D medical image into S2, extract 3D visual features in the corresponding region based on prior location information, and extract local features of a specific region from all 3D visual features, including: S21. Input a 3D medical image and align the input image. The positions of each part are fixed and known. Use a pre-trained SAT model to extract the region mask of the input image to obtain the prior location information proposal. S22. Using the ViT visual encoder, 3D visual features are extracted in the corresponding region based on prior location information proposals. S23. Use the ROIAlign method to extract local features of specific regions from all 3D visual features: F 左肺 =ROIAlign[SAT(left lung); F]; Where SAT (left lung) represents the region mask of the left lung extracted from the input image by the SAT model, and F represents the 3D visual features of all regions in the input image. 左肺 This refers to the local features of the left lung in the input image; S24. Based on the local features of all specific regions, obtain the local features of the input image: F region ={F 左肺 ;F 右肺 ;F 纵膈 }; Among them, F 左肺 F 右肺 F 纵膈 F represents the local features of the left lung, right lung, and mediastinum in the input image, respectively. region These are local features of the input image.

4. The medical image analysis method based on prior location information according to claim 3, characterized in that: In S3, learnable queries for each region are used to learn the interaction features of each region with the local features of each region to obtain the visual interaction features of each region, including: S31. Perform feature interaction learning on the learnable queries of each region and the local features of each region to obtain the visual interaction features of each region: T 左肺 =CrossAttention(Q 左肺 ;F 左肺 ); Where CrossAttention represents the cross-attention mechanism, T 左肺 The visual interaction features of the left lung in the input image; S32. Based on the visual interaction features of all regions, obtain the visual interaction features of the input image: T region ={T 左肺 ;T 右肺 ;T 纵膈 }; Among them, T 左肺 T 右肺 T 纵膈 These are the visual interaction features of the left lung, right lung, and mediastinum in the input image, respectively. region The visual interactive features of the input image.

5. The medical image analysis method based on prior location information according to claim 4, characterized in that: In S4, the segmentation head performs region segmentation on the input image based on the visual interaction features of the target region, obtaining the segmentation mask of the target region, including: S41. Use the segmentation task to pre-train the segmentation head, directly guiding the segmentation head to learn the location and characteristics of the region, and obtain the pre-trained segmentation head; S42. Input the visual interaction features of the target region into the pre-trained segmentation head, and have the segmentation head perform region segmentation on the input image to obtain the segmentation mask of the target region.

6. The medical image analysis method based on prior location information according to claim 5, characterized in that: In S5, a classification head is used to classify the target region based on its visual interaction features, and the classification results are then processed into textual descriptions to obtain a textual description of the classification results for the target region, including: S51. The classification head is pre-trained using a classification task. Classification labels are extracted from the text report of medical images, and then the visual interaction features of each region in the medical images are used for classification prediction. The loss function is calculated based on the classification prediction results and classification labels to guide the classification head to learn, so as to emphasize the regionality of visual interaction features and make the specified learnable queries focus more on their own regional semantics. S52. The classification head is pre-trained using the image-text comparison task. The visual interaction features of each region in the medical image and the corresponding text report paragraphs are used as positive sample pairs, and different paragraphs in the same text report are used as negative sample pairs to form positive and negative sample pairs for image-text comparison. This guides the classification head to perform comparative learning, so as to guide the visual interaction features to expand their own semantics to the region-related descriptions. While ensuring the region semantics, more fine-grained semantic guidance is added to obtain the pre-trained classification head. S53. Input the visual interaction features of the target region into the pre-trained classification head, and the classification head will perform classification to obtain the classification result; S54. Perform text processing on the classification results to obtain a text description of the classification results for the target region.

7. The medical image analysis method based on prior location information according to claim 6, characterized in that: In S6, the visual interaction features of the target region, the textual description of the classification results, and the user command text are input into a large-scale language model (LLM) to obtain a textual report of the target region, which is output along with the segmentation mask of the target region, including: S61. Use the LLM self-supervised task to pre-train the large language model LLM, so as to obtain a broader semantic guidance than the image-text comparison task through the text guidance provided by the large language model LLM, and obtain the pre-trained large language model LLM. S62. Input the visual interaction features of the target region, the textual description of the classification results, and the user instruction text Prompt into the pre-trained large language model LLM, and the large language model LLM generates a textual report of the target region. S63. Output the text report of the target region along with the segmentation mask of the target region; Among them, when using the LLM self-supervised task to pre-train a large language model LLM, the obtained semantic guidance has strong noise. Therefore, the LLM self-supervised task loss function has the smallest weight in the overall loss function of pre-training.

8. The medical image analysis method based on prior location information according to claim 7, characterized in that: The overall loss function for pre-training includes loss functions for segmentation tasks, classification tasks, image-text comparison tasks, and LLM self-supervised tasks: Loss=αLoss DiceLoss +βLoss BCE +γLoss ContractiveLoss +δLoss LLM ; Among them, Loss DiceLoss Loss BCE Loss ContractiveLoss Loss LLM α, β, γ, and δ are the loss functions for segmentation, classification, image-text comparison, and LLM self-supervised tasks, respectively. α, β, γ, and δ are the weights of the loss functions for segmentation, classification, image-text comparison, and LLM self-supervised tasks, respectively. Loss is the overall loss function for pre-training.

Images