Lymph node nodule classification method based on visual pre-training model and VLM

Abstract

The invention discloses a lymph node and nodule classification method based on a visual pre-training model and a VLM, and the method comprises two core technology modules: the module 1 is a pre-trained visual large model, and the module is used for coding an ultrasonic image through the abundant and general representation learning capability of the pre-trained large model, and carrying out the classification of the ultrasonic image; features with generalization performance priori are extracted, and the problem of performance reduction caused by domain offset is relieved; the second module is a field-adaptive VLM model, on the basis of a Prompt Finishing design normal form, pictures are coded by using a pre-trained large model, lymph node pictures are abstracted into feature expressions, and the feature expressions have rich and robust feature semantic information and can reflect the distribution conditions of pictures with different contents in a universal feature space, so that the method is suitable for the field-adaptive VLM model; then, the initialized cue word is modified based on the unique feature expression of each image, so that the optimization update of the cue word has strong correlation with the change of the features of the image;

Description

Technical Field

[0001] This invention relates to the field of ultrasound image analysis technology, and in particular to a lymph node nodule classification method based on a visual pre-trained model and VLM. Background Technology

[0002] In recent years, the clinical detection rate of lymph node-related diseases has been rising, especially among young adults and middle-aged people, showing a significant increasing trend. Their occurrence and development are closely related to various factors such as infection, inflammation, immune abnormalities, and even malignant tumors. Lymph node nodules are one of the important clinical manifestations of these diseases, and the differentiation of their nature—especially the determination of benign or malignant—is crucial for clinical decision-making, treatment planning, and patient prognosis.

[0003] Currently, the gold standard for lymph node assessment still relies on pathological examination after puncture biopsy or surgical resection, which is an invasive procedure. This not only places a physical burden on patients but also poses operational risks and diagnostic delays. With the rapid development of artificial intelligence technology in medical imaging, AI models based on ultrasound images have shown great potential in lymph node nodule analysis. These models can automatically extract deep morphological features and texture patterns from ultrasound images, enabling intelligent auxiliary differentiation of nodule malignancy. This not only helps improve diagnostic consistency and alleviate the workload of ultrasound physicians but also reduces unnecessary invasive examinations.

[0004] However, existing methods are mostly based on supervised learning frameworks, and model performance heavily relies on high-quality labeled data. Determining the pathological results corresponding to a single lymph node requires professional clinical experience. In addition, due to differences in ultrasound equipment, imaging protocols, and operating habits used by different medical institutions, significant image domain shifts occur, often leading to a decline in the generalization ability of supervised models in new scenarios. Therefore, developing robust AI diagnostic models for lymph node diagnosis that can adapt to multi-center, cross-device data distribution has become an urgent task in current medical artificial intelligence research.

[0005] Vision Language Models (VLMs) understand the content of images by modeling the matching relationship between images and text, and inherently possess zero-shot and few-shot generalization capabilities. However, due to the significant gap between medical images (especially ultrasound images) and natural images, the generalization ability of VLMs is difficult to directly transfer to medical images. Existing technical solutions attempt to understand the content of images by designing learnable cue word modules to learn the image-text correspondence between cue words and images; however, these methods do not effectively address the challenges posed by domain shift. When the image content changes (due to the hospital, equipment, or ultrasound model being sampled), the cue words learned from existing training data struggle to adapt to the new domain data due to the lack of adjustments to the image encoder, thus limiting the performance of VLM models. Summary of the Invention

[0006] To address the above technical problems, this invention provides a lymph node nodule classification method based on a visual pre-trained model and VLM, comprising the following steps:

[0007] S1. Collect public and internal datasets. The public datasets include open-source ultrasound data covering multiple organs and tissues, including the thyroid, breast, abdomen, and heart, and contain original images and corresponding mask annotations. The internal datasets include different lymph node ultrasound images and corresponding pathological conclusions collected from multiple hospitals. Doctors manually delineate the nodule range based on the specific content of the pathological conclusions, and also delineate key structural tissues, including the thyroid gland and thyroid trachea.

[0008] S2. Doctors give the benign or malignant judgment of the nodules based on the pathological conclusions, which constitutes the lymph node dataset; the lymph node dataset is randomly divided into an in-domain dataset and an out-of-domain dataset, and the in-domain dataset is divided into an in-domain training set and an in-domain validation set according to the proportion.

[0009] S3. Construct the MAE model, load the pre-trained weights released by Facebook, adjust the MAE model to obtain a pre-trained large visual model, and reconstruct the image.

[0010] S4. Construct a loss function to supervise and backpropagate the model, and update the parameters. After each training session, calculate the reconstruction loss of the model using the in-domain validation set, and retain the model with the lowest reconstruction loss on the in-domain validation set.

[0011] S5. Construct the CLIP model, load the weights officially released by Open-AI, and adjust the CLIP model to obtain the domain-adapted VLM model; construct the encoder part of the pre-trained large model, load the pre-trained weights adjusted in the previous step; perform forward propagation, calculate the similarity score and classification loss, supervise the model and perform backpropagation, and complete the parameter update.

[0012] S6. After each training session, calculate the classification accuracy of the model using the in-domain validation set, and retain the model with the highest classification performance on the in-domain validation set.

[0013] S7. Load the encoder part of the model with the highest classification performance on the in-domain validation set and the pre-trained large vision model, and use the out-of-domain dataset to verify the generalization performance of the model and evaluate the nodule classification accuracy of the model on the out-of-domain dataset.

[0014] The technical solution further defined in this invention is:

[0015] Furthermore, in step S1, both the public dataset and the internal dataset are divided into training and validation sets in an 8:2 ratio; the domain dataset is divided into a domain training set and a domain validation set in an 8:2 ratio.

[0016] As described above, a lymph node nodule classification method based on a visual pre-trained model and VLM is used. In step S3, the image reconstruction method includes local reconstruction and specific reconstruction. Local reconstruction is completed by randomly sampling a part of the original image and predicting the sampled part by masking the image region. Specific reconstruction is completed by designing a specific mask template so that the model can predict specific content.

[0017] As described above, a lymph node nodule classification method based on a visual pre-trained model and VLM is used, where a specific mask template is used to cover key regions in the ultrasound image, including lymph node nodules, thyroid glands, thyroid nodules, and thyroid trachea.

[0018] As described above, in a lymph node nodule classification method based on a visual pre-trained model and VLM, step S3 involves customizing noise according to the image content to complete denoising and reconstruction. A single-layer linear neural network is used to generate added noise based on the image content. During the reconstruction process, the model learns to avoid noise interference and restores the denoised image, as shown in the following formula:

[0019] n i =Linear(image) i )

[0020]

[0021]

[0022] Among them, image i Linear(image) represents the original content of the i-th image. i ) represents the linear projection of the i-th image to obtain the noise to be added, i.e., n i ; This represents the content of the i-th image after noise has been added; This represents the image retained after performing a random masking operation on the i-th image; Randommask() generates a random mask of size (224, 224) for image cropping; Specificmask() generates a mask based on a specific lesion or organ; MIM represents the reconstruction model, l res This represents the reconstruction loss function.

[0023] As described above, a lymph node nodule classification method based on a visual pre-trained model and VLM is used to process the original content of the i-th image during the denoising and reconstruction process. i Image content after adding noise When using the same mask.

[0024] As described above, in a lymph node nodule classification method based on a visual pre-trained model and VLM, in step S5, the domain-adapted VLM model follows the paradigm of text encoder encoding text and image encoder encoding images, embedding domain knowledge into the token initialized by the text encoder; when performing text encoding, the pre-trained visual large model in step S3 is used to encode the visual features of each image in the general space, and these features are used to perform offset operations on the initialized text token, so that the representation of the text token is supervised by the general features of the image.

[0025] As described above, the specific operation of step S5 in the lymph node nodule classification method based on a visual pre-trained model and VLM is as follows:

[0026]

[0027]

[0028] f i =VisualEncoder(image) i )

[0029]

[0030] in, Represents the text token of the i-th category, [v1][v2]...[v n [] represents n randomly initialized learnable text tokens, and [cls] represents category information; This represents the encoder in a pre-trained large visual model, processing images. i Encode it and process it through projection P() to map its shape to the same shape as the text token of the i-th category, denoted as biasi; This represents the text token after adding the offset; TextEncoder() and VisualEncoder() represent the text encoder and image encoder in the CLIP model, respectively, which process the text token and visual token in turn to obtain the corresponding text features t. i and visual features f i ;logits iThis represents the similarity score between text features and visual features, and log_scale represents the preset scaling amount.

[0031] As described above, in a lymph node nodule classification method based on a visual pre-trained model and VLM, the cross-entropy loss function in step S5 is as follows:

[0032]

[0033] Among them, l ce This represents the cross-entropy loss function, where N represents the total number of samples, and label... i Let represent the true label of the i-th label, and CrossEntropy() represent the cross-entropy loss function.

[0034] The beneficial effects of this invention are:

[0035] (1) In this invention, a domain-adaptive VLM model is proposed. Combining the general feature learning ability of the visual pre-trained large model and the image-text matching ability of the VLM model, a method for classifying lymph node nodules is designed. The model uses the visual large model to model image features and adaptively adjusts the text token according to the features, so that the model has the ability to perceive different content and different domains of the image, which alleviates the problem of cross-domain classification performance degradation.

[0036] (2) In this invention, a pre-training fine-tuning method for a large ultrasonic vision model is proposed; customized noise is generated based on the content of the image itself to enhance the model’s understanding of the subtle details of a single image; the reconstruction task is split into local reconstruction task and specific reconstruction task, so that the encoded abstract features have both refined feature understanding ability and positional relationship understanding ability, which alleviates the phenomenon of domain offset of features extracted by traditional feature extractors.

[0037] (3) The proposed pre-trained large model combined with VLM classification model paradigm can be applied to classification tasks of multiple organs and multiple tissues. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the structure of the pre-trained large visual model in an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram of the domain-adaptive VLM model structure in an embodiment of the present invention. Detailed Implementation

[0040] This embodiment provides a lymph node nodule classification method based on a visual pre-trained model and VLM, including the following steps:

[0041] S1. Collect public and internal datasets. The public dataset includes open-source ultrasound data covering multiple organs and tissues such as the thyroid, breast, abdomen, and heart, and contains original images and corresponding mask annotations. The internal dataset includes different lymph node ultrasound images and corresponding pathological conclusions collected from multiple hospitals. Doctors manually delineate the nodule range based on the specific content of the pathological conclusions, and also delineate key structural tissues, such as the thyroid gland and thyroid trachea. Both the public and internal datasets are divided into training and validation sets in an 8:2 ratio.

[0042] S2. Doctors give the benign or malignant judgment of the nodules based on the pathological conclusions, which constitutes the lymph node dataset. The lymph node dataset is randomly divided into an in-domain dataset and an out-of-domain dataset. The in-domain dataset is divided into an in-domain training set and an in-domain validation set in an 8:2 ratio for model training and selection. The out-of-domain dataset is used as a test set to measure the overall classification performance of the model.

[0043] S3. Construct the MAE model, load the pre-trained weights released by Facebook, and adjust the MAE model to obtain the following: Figure 1 The pre-trained large visual model is shown, and the image is reconstructed.

[0044] Due to the limited availability of data, including publicly available datasets of ultrasound images and internally collected lymph node ultrasound images, this embodiment uses a fine-tuned MAE (Meta's official pure visual pre-trained large model) to obtain a pre-trained visual large model of ultrasound images. In order to alleviate domain bias, learn general feature representations, and increase the model's ability to understand image content during pre-training, this embodiment has made the following improvements compared to MAE.

[0045] Improvement 1: The reconstruction task is divided into a local reconstruction task and a specific reconstruction task. The local reconstruction task randomly samples a portion of the original image and uses a mask to occlude the image region to predict the sampled portion, thus completing the local reconstruction. The specific reconstruction task designs a specific mask template, such as occluding key regions in the ultrasound image (lymph node nodules, thyroid glands, thyroid nodules, thyroid trachea, etc.), so that the model can predict specific content and complete the specific reconstruction.

[0046] This reconstruction method enables the model to learn both the fine features of local areas and the intrinsic positional relationships of key organs in ultrasound images, giving it a stronger learning ability.

[0047] Improvement 2: Customizing noise based on image content to complete denoising and reconstruction; In addition to conventional data augmentation methods, a single-layer linear neural network is used to generate added noise based on the image content. During the reconstruction process, the model needs to learn to avoid the interference of noise and restore the denoised image. This denoising and reconstruction method enables the model to have denoising capabilities and accurately restore the fine nodules in the ultrasound image from the dryness, as shown in the following formula:

[0048] n i =Linear(image) i )

[0049]

[0050]

[0051] Among them, image i Linear(image) represents the original content of the i-th image. i ) represents the linear projection of the i-th image to obtain the noise to be added, i.e., n i ; This represents the content of the i-th image after noise has been added; This represents the image retained after performing a random mask operation on the i-th image; Randommask() generates a random mask of size (224, 224) to crop the image (although Randommask has randomness, it is still effective when processing the original content of the i-th image). i Image content after adding noise (When using the same mask); Specificmask() represents a mask generated based on a specific lesion or organ; MIM represents the reconstruction model, l res This represents the reconstruction loss function.

[0052] S4. Construct a loss function to supervise and backpropagate the model, and update the parameters. After each training session, calculate the reconstruction loss of the model using the in-domain validation set, and retain the model with the lowest reconstruction loss on the in-domain validation set.

[0053] S5. Construct the CLIP model, load the weights officially released by Open-AI, and adjust the CLIP model to obtain the domain-adapted VLM model; construct the encoder part of the pre-trained large model, load the pre-trained weights adjusted in the previous step; perform forward propagation, calculate the similarity score and classification loss, supervise the model and perform backpropagation, and complete the parameter update.

[0054] like Figure 2As shown, the domain-adaptive VLM model proposed in the patent is an improvement on the CLIP model officially released by Open-AI. It follows the paradigm of text encoders encoding text and image encoders encoding images, embedding domain knowledge into the token initialized by the text encoder.

[0055] Specifically, when performing text encoding, the pre-trained large visual model in step S3 is used to encode the visual features of each image in the general space. Such features have rich and robust information content. Due to the improvement of the pre-training method, these features have better domain generalization than the features obtained by traditional feature extractors. These features are used to perform offset operations on the initialized text token, so that the representation of the text token is supervised by the general features of the image.

[0056] The specific operation of step S5 is shown in the following formula:

[0057]

[0058] f i =VisualEncoder(image) i )

[0059]

[0060] in, Represents the text token of the i-th category, [v1][v2]...[v n [] represents n randomly initialized learnable text tokens, and [cls] represents category information; This represents the encoder in a pre-trained large visual model, processing images. i Encode the token and then project it using P() to map its shape to the same shape as the text token of the i-th category, denoted as bias. i ; This represents the text token after adding the offset; TextEncoder() and VisualEncoder() represent the text encoder and image encoder in the CLIP model, respectively, which process the text token and visual token in turn to obtain the corresponding text features t. i and visual features f i ;logits i This represents the similarity score between textual and visual features, with log_scale representing the preset scaling factor; ce This represents the cross-entropy loss function, where N represents the total number of samples, and label... iLet represent the true label of the i-th label, and CrossEntropy() represent the cross-entropy loss function.

[0061] S6. After each training session, calculate the classification accuracy of the model using the in-domain validation set, and retain the model with the highest classification performance on the in-domain validation set.

[0062] S7. Load the encoder part of the model with the highest classification performance on the in-domain validation set and the pre-trained large vision model, and use the out-of-domain dataset to verify the generalization performance of the model and evaluate the nodule classification accuracy of the model on the out-of-domain dataset.

[0063] This embodiment abstracts domain knowledge into a latent space mapping of each image content. It uses the mapped vectors and a simple adaptation network to offset and adjust the text prompts, enabling the text prompts to be adaptively modified according to different image content, thereby enhancing the domain generalization ability of the VLM model.

[0064] This embodiment includes two core technology modules. The first module is a pre-trained large visual model. The role of this module is to encode ultrasound images by utilizing the rich general representation learning capabilities of the pre-trained large model, extract features with generalization performance prior, and alleviate the performance degradation caused by domain bias. The second module is a domain-adaptive VLM model. Based on the Prompt Finetuning design paradigm, it uses the pre-trained large model to encode images, abstracting lymph node images into feature representations. These feature representations have rich and robust feature semantic information, which can reflect the distribution of images with different content in a general feature space. Then, based on the unique feature representation of each image, the initialized prompts are modified, so that the optimization and update of the prompts are strongly correlated with the changes in the features of the image itself.

[0065] In addition to the embodiments described above, the present invention may have other implementations. All technical solutions formed by equivalent substitution or equivalent transformation fall within the protection scope claimed by the present invention.

Claims

1. A lymph node nodule classification method based on a visual pre-trained model and VLM, characterized in that: Includes the following steps: S1. Collect public and internal datasets. The public datasets include open-source ultrasound data covering multiple organs and tissues, including the thyroid, breast, abdomen, and heart, and contain original images and corresponding mask annotations. The internal datasets include different lymph node ultrasound images and corresponding pathological conclusions collected from multiple hospitals. Doctors manually delineate the nodule range based on the specific content of the pathological conclusions, and also delineate key structural tissues, including the thyroid gland and thyroid trachea. S2. Doctors provide a benign or malignant assessment of the nodules based on pathological findings, which constitutes a lymph node dataset. The lymph node dataset was randomly divided into an in-domain dataset and an out-of-domain dataset. The in-domain dataset was further divided into an in-domain training set and an in-domain validation set according to a certain ratio. S3. Construct the MAE model, load the pre-trained weights released by Facebook, adjust the MAE model to obtain a pre-trained large visual model, and reconstruct the image. S4. Construct a loss function to supervise and backpropagate the model, and update the parameters. After each training session, calculate the reconstruction loss of the model using the in-domain validation set, and retain the model with the lowest reconstruction loss on the in-domain validation set. S5. Build the CLIP model, load the weights officially released by Open-AI, and adjust the CLIP model to obtain the domain-adapted VLM model; build the encoder part of the pre-trained large model, load the pre-trained weights adjusted in the previous step; perform forward propagation, calculate the similarity score and classification loss, supervise the model and perform backpropagation, and complete the parameter update. S6. After each training session, calculate the classification accuracy of the model using the in-domain validation set, and retain the model with the highest classification performance on the in-domain validation set. S7. Load the encoder part of the model with the highest classification performance on the in-domain validation set and the pre-trained large vision model, and use the out-of-domain dataset to verify the generalization performance of the model and evaluate the nodule classification accuracy of the model on the out-of-domain dataset.

2. The lymph node nodule classification method based on a visual pre-trained model and VLM as described in claim 1, characterized in that: In step S1, both the public dataset and the internal dataset are divided into training and validation sets in an 8:2 ratio; the domain dataset is divided into a domain training set and a domain validation set in an 8:2 ratio.

3. The lymph node nodule classification method based on a visual pre-trained model and VLM according to claim 1, characterized in that: In step S3, the image reconstruction method includes local reconstruction and specific reconstruction. Local reconstruction is completed by randomly sampling a part of the original image and predicting the sampled part by masking the image region. Specific reconstruction is completed by designing a specific mask template so that the model can predict specific content.

4. The lymph node nodule classification method based on a visual pre-trained model and VLM according to claim 3, characterized in that: The specific mask template includes masking key areas in the ultrasound image, including lymph node nodules, thyroid glands, thyroid nodules, and thyroid trachea.

5. The lymph node nodule classification method based on a visual pre-trained model and VLM according to claim 3, characterized in that: In step S3, noise is customized based on the image content to complete denoising and reconstruction. A single-layer linear neural network is used to generate added noise based on the image content. During the reconstruction process, the model learns to avoid noise interference and restores the denoised image, as shown in the following formula: n i =Linear(image i ) Among them, image i Liar(image) represents the original content of the i-th image. i ) represents the linear projection of the i-th image to obtain the noise to be added, i.e., n i ; This represents the content of the i-th image after noise has been added; This represents the image retained after performing a random masking operation on the i-th image; Randommask() generates a random mask of size (224, 224) for image cropping; Specificmask() generates a mask based on a specific lesion or organ; MIM represents the reconstruction model, l res This represents the reconstruction loss function.

6. The lymph node nodule classification method based on a visual pre-trained model and VLM according to claim 1, characterized in that: During the denoising and reconstruction process, the original content of the i-th image is processed. i Image content after adding noise When using the same mask.

7. The lymph node nodule classification method based on a visual pre-trained model and VLM according to claim 1, characterized in that: In step S5, the domain-adaptive VLM model follows the paradigm of text encoder encoding text and image encoder encoding images, embedding domain knowledge into the token initialized by the text encoder. When performing text encoding, the pre-trained visual large model in step S3 is used to encode the visual features of each image in the general space. These features are used to offset the initialized text token, so that the representation of the text token is supervised by the general features of the image.

8. The lymph node nodule classification method based on a visual pre-trained model and VLM according to claim 7, characterized in that: The specific operation of step S5 is shown in the following formula: f i =VisualEncoder(image i ) in, Represents the text token of the i-th category, [v1][v2]...[v n [] represents n randomly initialized learnable text tokens, and [cls] represents category information; This represents the encoder in a pre-trained large visual model, processing images. i Encode the token and then project it using P() to map its shape to the same shape as the text token of the i-th category, denoted as bias. i ; This represents the text token after adding the offset; TextEncoder() and VisualEncoder() represent the text encoder and image encoder in the CLIP model, respectively, which process the text token and visual token in turn to obtain the corresponding text features t. i and visual features f i ;logits i This represents the similarity score between text features and visual features, and log scale represents the preset scaling amount.

9. The lymph node nodule classification method based on a visual pre-trained model and VLM according to claim 8, characterized in that: In step S5, the cross-entropy loss function is shown in the following equation: Among them, l ce This represents the cross-entropy loss function, where N represents the total number of samples, and label... i Let represent the true label of the i-th label, and CrossEntropy() represent the cross-entropy loss function.

Images