A thyroid pathological image multimodal retrieval system

Abstract

The application discloses a thyroid pathological image multimodal retrieval system, comprising the following steps: based on the thyroid cell pathological grading report standard of cytological examination, the similarity definition of the thyroid pathological image is given, the definition not only emphasizes the difference between different grades, but also refers to the similarity between different grades, so that the visual encoder completes the pre-training in the grading image data set; based on the pre-trained visual encoder, the text encoder is aligned and trained through the image and text in the clinical diagnosis data and pathological books, so that the data under different modes is interacted; based on the pre-trained multimodal model, the text and image input are mapped to the same feature vector space, and the multimodal retrieval is completed by comparing the similarity. The application provides a thyroid pathological image multimodal retrieval system, which can not only complete image and text retrieval, but also encode pictures into feature vectors through a visual encoder, and further complete different classification and diagnosis tasks by connecting different classification heads.

Description

Technical Field

[0001] This invention relates to the field of thyroid pathology image retrieval technology, and in particular to a multimodal thyroid pathology image retrieval system. Background Technology

[0002] The increasing number of large-scale institutional and public pathological histological image datasets has effectively promoted applications in diagnosis, research, and education. However, diagnostic annotations are extremely scarce compared to pathological image data. How to fully utilize the limited existing annotations to construct a retrieval library for massive images has become a concern. Specifically, this involves finding regions with annotation value from a vast amount of unannotated pathological images, or finding annotated reference cases for newly acquired pathological images.

[0003] For example, thyroid cancer pathological section diagnosis involves analyzing and detecting the state of thyroid tissue from a given whole-field-of-view (WSI) thyroid pathological section image, and then classifying the pathological image, such as benign or malignant. Tumor grading is an important process in pathology, used to assess the malignancy of tumors and improve diagnostic consistency. The Bethesda System for Reporting Thyroid Cytopathology (TBSRTC), based on fine needle aspiration (FNA) cytology, classifies thyroid cytology samples into six grades: Grade I: Undiagnostic; Grade II: Benign; Grade III: Atypical lesion / follicular lesion of unknown significance; Grade IV: Follicular tumor / suspicious follicular tumor; Grade V: Suspicious malignancy; Grade VI: Malignant. Except for Grade I, which represents an unacceptable specimen, the risk of malignancy increases progressively from Grade II to Grade VI.

[0004] Therefore, it is necessary to solve the technical problem of how to retrieve thyroid cancer pathological images according to their grade. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal retrieval system for thyroid pathology images. The system encodes images into feature vectors using a visual encoder. These feature vectors are then connected to different classification heads to perform different classification and diagnostic tasks. The system interacts with text vectors from real-world cases and publicly available pathology data to retrieve the closest image-text case.

[0006] The technical solution adopted by the multimodal retrieval system for thyroid pathology images disclosed in this invention is as follows:

[0007] A multimodal retrieval system for thyroid pathology images includes the following steps:

[0008] Pre-trained visual encoder: Based on the thyroid cytopathology grading report standard for cytological examination, it provides a similarity definition for thyroid pathology images, which not only emphasizes the differences between different grades, but also refers to the similarities between different grades, enabling the visual encoder to complete pre-training on the graded image dataset.

[0009] Aligned Text Encoder: Based on a pre-trained visual encoder, alignment training is completed with the text encoder using clinical diagnostic data and images and text from pathology books, enabling data from different modalities to interact.

[0010] Complete multimodal retrieval and classification tasks: Based on a pre-trained multimodal model, text and image inputs are mapped to the same feature vector space. Multimodal retrieval is completed by comparing similarity, finding similar cases in professional books or clinical diagnoses. Furthermore, the feature vectors from the visual end can also be transferred to downstream classification tasks for various classification tasks.

[0011] As a preferred approach, in the pre-trained visual encoder: the Rank of any two cytopathology images a and b is defined as follows, where Grade(·) represents the grade of the image:

[0012] Rank(a,b)=|Grade(a)-Grade(b)|

[0013] Hierarchical datasets are denoted as Where X is the set of all pathological images, Y represents the set of all graded annotations, and N represents the total number of samples. This dataset... With the Rank relation defined above, a large number of triplet samples can be constructed. A typical triplet sample contains an anchor sample Anc, a positive sample Pos, and a negative sample Neg. The similarity between Anc and Pos is greater than that between Anc and Neg. The final triplet dataset is denoted as...

[0014] Based on these triplet samples, the visual encoder can acquire feature extraction capabilities through training. The visual encoder can encode the thyroid image into a one-dimensional feature vector. In order to align with the subsequent text encoder, a Transformer-based neural network can be used as the visual encoder.

[0015] As a preferred approach, the similarity of feature vectors can be measured using Euclidean distance. The Euclidean distance Distance between any two images a and b can be described by the following formula, where VisionEncoder(·) represents the neural network used to extract features:

[0016] Distance(a,b)=||VisionEncoder(a)-VisionEncoder(b)|| 2

[0017] The similarity is inversely proportional to the distance. During network training, the optimization objective is to reduce the distance between anchor samples and positive samples while increasing the distance between anchor samples and negative samples. The final objective function is shown below, where g is a hyperparameter used to control the interval between positive and negative samples:

[0018]

[0019] As a preferred approach, based on the pre-trained visual encoder described above, after collecting a sufficient number of ranking annotations, the text descriptions in the books are defined as anchor samples, images matching the text are considered positive samples, and those not matching are considered negative samples. Based on these rules, a multimodal triplet ranking dataset can be obtained, denoted as [database name missing].

[0020] Where PI i NI represents the i-th positive sample image. i This represents the corresponding negative sample image.

[0021] As a preferred approach, during training, images and text are encoded into feature vectors by corresponding encoders. The visual encoder is denoted as VisionEncoder, and the image feature vector IF = VisionEncoder(Image) is obtained from the visual encoder. The text encoder is denoted as TextEncoder, and the text feature vector TF = TextEncoder(Text) is obtained from the text encoder. The similarity is measured by the cosine value of the two vectors. This calculation is based on the dot product and Euclidean length of the two vectors, as shown below:

[0022]

[0023] The cosine similarity value ranges from -1 to 1, where 1 indicates that the two vectors have the same direction, -1 indicates that the two vectors have opposite directions, and 0 indicates that they are independent. It is used to complete the training process of multimodal models. The optimization objective is to increase the similarity between the text and positive sample feature vectors (PIF) while reducing the similarity between the text and negative sample feature vectors (NIF). The final objective function is shown below, where g is a hyperparameter used to control the interval between positive and negative samples:

[0024]

[0025] The beneficial effects of the thyroid pathology image multimodal retrieval system disclosed in this invention are as follows: First, the visual encoder is pre-trained, and fine-grained similarity definitions are given for different grades of cancer based on the severity of thyroid tumors and the probability of malignancy. This not only emphasizes the differences between different grades but also fully considers the similarities between different grades. For the text encoder, image-text multimodal data extracted from clinical diagnostic data and publicly available pathology books are used to fine-tune the text encoder based on the visual encoder, so that the text encoder and the visual encoder can fully interact and align. After the model training is completed, an image to be diagnosed is input and encoded into a feature vector by the visual encoder. The feature vector can be connected to different classification heads to complete different classification and diagnostic tasks. It can also interact with the text vectors in real cases and publicly available pathology data to retrieve the closest image and text cases. Attached Figure Description

[0026] Figure 1 This is a flowchart of a multimodal retrieval system for thyroid pathology images according to the present invention.

[0027] Figure 2 This is a flowchart of an image retrieval network based on thyroid cell pathology grading reports. Detailed Implementation

[0028] The present invention will be further described and illustrated below with reference to specific embodiments and the accompanying drawings:

[0029] Please refer to Figure 1 A multimodal retrieval system for thyroid pathology images includes the following steps:

[0030] Pre-trained visual encoder: Based on the thyroid cytopathology grading report standard for cytological examination, it provides a similarity definition for thyroid pathology images, which not only emphasizes the differences between different grades, but also refers to the similarities between different grades, enabling the visual encoder to complete pre-training on the graded image dataset.

[0031] In the pre-trained visual encoder: the Rank of any two cytopathology images a and b is defined as follows, where Grade(·) represents the grade of the image:

[0032] Rank(a,b)=|Grade(a)-Grade(b)|

[0033] Hierarchical datasets are denoted as Where X is the set of all pathological images, Y represents the set of all graded annotations, and N represents the total number of samples. This dataset... With the Rank relation defined above, a large number of triplet samples can be constructed. A typical triplet sample contains an anchor sample Anc, a positive sample Pos, and a negative sample Neg. The similarity between Anc and Pos is greater than that between Anc and Neg. The final triplet dataset is denoted as...

[0034]

[0035] Based on these triplet samples, the visual encoder can acquire feature extraction capabilities through training. The visual encoder can encode the thyroid image into a one-dimensional feature vector. In order to align with the subsequent text encoder, a Transformer-based neural network can be used as the visual encoder.

[0036] The similarity of feature vectors can be measured by Euclidean distance. The Euclidean distance Distance between any two images a and b can be described by the following formula, where VisionEncoder(·) represents the neural network used to extract features:

[0037] Distance(a,b)=||VisionEncoder(a)-VisionEncoder(b)|| 2

[0038] The similarity is inversely proportional to the distance. During network training, the optimization objective is to reduce the distance between anchor samples and positive samples while increasing the distance between anchor samples and negative samples. The final objective function is shown below, where g is a hyperparameter used to control the interval between positive and negative samples:

[0039]

[0040] The ranking relationship was defined with full consideration of the thyroid cancer grading criteria, and malignancy probability was introduced to describe the similarity between images in a more granular way. This stage of the process references... Figure 2 As shown, this ranking relationship can be used to train a visual encoder for image retrieval. Since only static feature vectors are needed to calculate distance, and the size of the feature vectors is much smaller than the original image, the overhead of actual runtime can be saved by running the network in advance and storing the feature vectors.

[0041] Aligned Text Encoder: Based on a pre-trained visual encoder, alignment training is completed with the text encoder using clinical diagnostic data and images and text from pathology books, enabling data from different modalities to interact.

[0042] Based on the pre-trained visual encoder described above, after collecting a sufficient number of ranking annotations, the text descriptions in the books are defined as anchor samples, images matching the text are positive samples, and those not matching are negative samples. Based on these rules, a multimodal triplet ranking dataset can be obtained, denoted as .

[0043] Where PI i NI represents the i-th positive sample image. i This represents the corresponding negative sample image.

[0044] Complete multimodal retrieval and classification tasks: Based on a pre-trained multimodal model, text and image inputs are mapped to the same feature vector space. Multimodal retrieval is completed by comparing similarity, finding similar cases in professional books or clinical diagnoses. Furthermore, the feature vectors from the visual end can also be transferred to downstream classification tasks for various classification tasks.

[0045] During training, images and text are encoded into feature vectors by corresponding encoders. The visual encoder is denoted as VisionEncoder, and the image feature vector IF = VisionEncoder(Image) is obtained from the visual encoder. The text encoder is denoted as TextEncoder, and the text feature vector TF = TextEncoder(Text) is obtained from the text encoder. The similarity is measured by the cosine of the two vectors. This calculation is based on the dot product and Euclidean length of the two vectors, as shown below:

[0046]

[0047] The cosine similarity value ranges from -1 to 1, where 1 indicates that the two vectors have the same direction, -1 indicates that the two vectors have opposite directions, and 0 indicates that they are independent. It is used to complete the training process of multimodal models. The optimization objective is to increase the similarity between the text and positive sample feature vectors (PIF) while reducing the similarity between the text and negative sample feature vectors (NIF). The final objective function is shown below, where g is a hyperparameter used to control the interval between positive and negative samples:

[0048]

[0049] By training on a large number of pathological images and texts, the feature vectors of images and texts gradually align, thus enabling multimodal retrieval tasks. The model can learn more general and robust feature representations in cytopathology images. Through transfer learning, the model can quickly adapt to new data and can be adapted to more custom classification tasks with only a small amount of labeled data.

[0050] This invention provides a multimodal retrieval system for thyroid pathology images. First, a visual encoder is pre-trained. Fine-grained similarity definitions are given for different grades of cancer, considering both the severity of thyroid tumors and the probability of malignancy. This emphasizes the differences between different grades while also taking into full account their similarities. For the text encoder, image-text multimodal data extracted from clinical diagnostic data and publicly available pathology books is fine-tuned based on the visual encoder, ensuring full interaction and alignment between them. After model training, an input image to be diagnosed is encoded into a feature vector by the visual encoder. This feature vector can be connected to different classification heads to perform different classification and diagnostic tasks. It can also interact with text vectors from real-world cases and publicly available pathology data to retrieve the closest image-text case.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A multimodal retrieval system for thyroid pathology images, characterized in that, Includes the following steps: Pre-trained visual encoder: Based on the thyroid cytopathology grading report standard for cytological examination, it provides a similarity definition for thyroid pathology images, which not only emphasizes the differences between different grades, but also refers to the similarities between different grades, enabling the visual encoder to complete pre-training on the graded image dataset. Aligned Text Encoder: Based on a pre-trained visual encoder, alignment training is completed with the text encoder using clinical diagnostic data and images and text from pathology books, enabling data from different modalities to interact. Complete multimodal retrieval and classification tasks: Based on a pre-trained multimodal model, text and image inputs are mapped to the same feature vector space. Multimodal retrieval is completed by comparing similarity, finding similar cases in professional books or clinical diagnoses. Furthermore, the feature vectors from the visual end can also be transferred to downstream classification tasks for various classification tasks. In the pre-trained visual encoder: any two cytopathology images , sorting The definition is as follows: The classification of this image: ； Hierarchical datasets are denoted as ,in It is a collection of all pathological images, and Represents the set of all hierarchical labels. Represents the total number of samples, obtained through the dataset. and the definition above By establishing relationships, a large number of triplet samples can be constructed. A typical triplet sample contains an anchor sample. A positive sample A negative sample , and The similarity is greater than that with The similarity is used to denot the final triplet dataset. ; Based on these triplet samples, the visual encoder can acquire feature extraction capabilities through training. The visual encoder encodes the thyroid image into a one-dimensional feature vector. In order to align with the subsequent text encoder, a Transformer-based neural network is used as the visual encoder. The similarity of feature vectors is measured by Euclidean distance for any two images. , Euclidean distance It is described by the following formula, where Represents a neural network used to extract features: ； Among them, similarity and The distance between anchor samples and positive samples is inversely proportional to the distance between anchor samples and negative samples. During network training, the optimization objective is to reduce the distance between anchor samples and positive samples while increasing the distance between anchor samples and negative samples. The final objective function is shown below, where... These are hyperparameters used to control the interval between positive and negative samples: ； Based on the pre-trained visual encoder described above, after collecting a sufficient number of ranking annotations, the text descriptions in the books are defined as anchor samples, images matching the text are positive samples, and those not matching are negative samples. Based on these rules, a multimodal triplet ranking dataset can be obtained, denoted as . ,in Represents the i-th positive sample image. This represents the corresponding negative sample image; During training, images and text are encoded into feature vectors by corresponding encoders, where the visual encoder is denoted as... Image feature vectors are obtained through the visual end. The encoder at the text end is denoted as Text feature vectors are obtained from the text end. Similarity is measured by the cosine of two vectors, calculated based on the dot product and Euclidean length of the two vectors, as shown below: ； Cosine similarity values ​​range from -1 to 1, where 1 indicates that the two vectors have the same direction, -1 indicates that the two vectors have opposite directions, and 0 indicates that they are independent. It is used to complete the training process of multimodal models, and the optimization goal is to increase the feature vectors of text and positive samples. While reducing similarity, reduce the feature vectors of text and negative samples. The similarity is calculated, and the final objective function is shown below, where... These are hyperparameters used to control the interval between positive and negative samples: 。

Images