Thyroid pathology image multi-mode retrieval system

Abstract

The invention discloses a thyroid pathology image multi-modal retrieval system, which comprises the following steps that based on thyroid cytopathology grading report standards of cytological examination, similar definitions of thyroid pathology images are given, the definitions not only emphasize differences among different grades, but also refer to similarities among the different grades, and the thyroid pathology images can be retrieved according to the similarities among the different grades. Enabling the visual encoder to complete pre-training in the hierarchical image data set; on the basis of a pre-trained visual encoder, alignment training with a text encoder is completed through clinical diagnosis data and images and texts in a pathological book, so that data interaction in different modes is completed; based on a pre-trained multi-modal model, text and image input are mapped to the same feature vector space, and multi-modal retrieval is completed by comparing similarities. The invention provides a thyroid pathology image multi-mode retrieval system which not only can complete image-text retrieval, but also can encode images into feature vectors through a visual encoder, and further completes different classification diagnosis tasks by connecting different classification heads.

Description

Technical Field

[0001] The present invention relates to the technical field of thyroid pathology image retrieval, and in particular to a thyroid pathology image multimodal retrieval system. Background Art

[0002] The increasing number of large-scale institutional and public histopathological image datasets has effectively promoted applications in diagnosis, research, and education. However, diagnostic annotations are extremely scarce compared to pathological image data. How to make full use of the existing small amount of annotations to construct a retrieval library for massive images has become a concern. Specifically, it is to find areas with annotation value from massive unlabeled pathological images, or to find labeled reference cases for newly acquired pathological images.

[0003] For example, the diagnosis of thyroid cancer pathological slides refers to the analysis and detection of the state of thyroid tissue from a given whole-field thyroid pathological slide (wholeslide image, referred to as WSI) image, and then the classification of pathological images, such as benign or malignant. Tumor grading is an important process in pathology, which can be used to assess the malignancy of tumors and improve the consistency of diagnosis. The Bethesda System for Reporting Thyroid Cytopathology (TBSRTC) based on fine needle aspiration (FNA) cytology examination divides thyroid cytology samples into six levels, namely, Grade I: Undiagnosable, Grade II: Benign, Grade III: Atypical lesions of unknown significance / follicular lesions, Grade IV: Follicular tumors / suspected follicular tumors, Grade V: Suspected malignancy, and Grade VI: Malignant. Except for the unqualified specimens of the first level, the remaining qualified specimens are from II to VI, and the risk of malignancy increases step by step.

[0004] Therefore, it is necessary to solve the technical problem of how to retrieve thyroid cancer pathology images according to the grading situation. Summary of the invention

[0005] The purpose of the present invention is to provide a multimodal retrieval system for thyroid pathology images, which is encoded into feature vectors through a visual encoder. The feature vectors complete different classification diagnosis tasks by connecting different classification heads, and interact with text vectors in real cases and public pathology data to retrieve the closest graphic and text cases.

[0006] The technical solution adopted by a multimodal retrieval system for thyroid pathology images disclosed in the present invention is:

[0007] A multimodal retrieval system for thyroid pathology images comprises the following steps:

[0008] Pre-trained visual encoder: Based on the thyroid cytopathology grading reporting standard of cytology examination, a similar definition of thyroid pathology images is given, which not only emphasizes the differences between different grades, but also refers to the similarities between different grades, so that the visual encoder can be pre-trained in the graded image dataset;

[0009] Aligned text encoder: Based on the pre-trained visual encoder, the text encoder is aligned with clinical diagnosis data and pictures and texts in pathology books to enable data in different modalities to interact.

[0010] Complete multimodal retrieval and classification tasks: Based on the pre-trained multimodal model, text and image inputs are mapped to the same feature vector space, and multimodal retrieval is completed by comparing similarities to find similar cases in professional books or clinical diagnoses. The feature vectors on the visual side can also complete the migration of downstream classification tasks for various classification tasks.

[0011] As a preferred solution, in the pre-trained visual encoder: the ranking Rank of any two cytopathological images a and b is defined as follows, where Grade(·) represents the grade of the image:

[0012] Rank(a,b)=|Grade(a)-Grade(b)|

[0013] The hierarchical dataset is denoted as Where X is the set of all pathological images, Y represents the set of all graded annotations, and N represents the total number of samples. And the Rank relationship defined above, a large number of triple samples can be constructed. A typical triple sample contains an anchor sample Anc, a positive sample Pos, and a negative sample Neg. The similarity between Anc and Pos is greater than the similarity between Anc and Neg. The final triple data set is recorded as

[0014] Based on these triplet samples, the visual encoder can acquire feature extraction capabilities through training. The visual encoder can encode the thyroid image into a one-dimensional feature vector. In order to align with the subsequent text encoder, a Transformer-based neural network can be used as the visual encoder.

[0015] As a preferred solution, the similarity of feature vectors can be measured by Euclidean distance. The Euclidean distance Distance between any two images a and b can be described by the following formula, where VisionEncoder(·) represents the neural network used to extract features:

[0016] Distance(a,b)=||VisionEncoder(a)-VisionEncoder(b)|| 2

[0017] The similarity is inversely proportional to the size of the Distance. When training the network, the optimization goal is to reduce the distance between the anchor sample and the positive sample while increasing the distance between the anchor sample and the negative sample. The final objective function is as follows, where g is a hyperparameter used to control the interval between the positive sample and the negative sample:

[0018]

[0019] As a preferred solution, based on the above pre-trained visual encoder, after collecting enough sorting annotations, the text description in the book is defined as the anchor sample, and the image matching the text is the positive sample, otherwise it is the negative sample. Based on the above rules, a multimodal triplet sorting dataset can be obtained, which is denoted as

[0020] Among them, PI i Represents the i-th positive sample image, NI i Represents the corresponding negative sample image.

[0021] As a preferred solution, during the training process, the image and text will be encoded into feature vectors by the corresponding encoders, where the encoder at the visual end is recorded as VisionEncoder, and the image feature vector IF=VisionEncoder(Image) is obtained through the visual end, and the encoder at the text end is recorded as TextEncoder, and the text feature vector TF=TextEncoder(Text) is obtained through the text end. The cosine value of the two vectors is used to measure the similarity, which is based on the dot product and Euclidean length of the two vectors, as shown below:

[0022]

[0023] The value range of cosine similarity is between -1 and 1, where 1 means that the directions of the two vectors are exactly the same, -1 means that the directions of the two vectors are completely opposite, and 0 means that the two are independent. It is used to complete the training process of the multimodal model. The optimization goal is to increase the similarity between the feature vector PIF of the text and the positive sample while reducing the similarity between the feature vector NIF of the text and the negative sample. The final objective function is as follows, where g is a hyperparameter used to control the interval between the positive sample and the negative sample:

[0024]

[0025] The beneficial effects of a thyroid pathology image multimodal retrieval system disclosed in the present invention are: first, the visual encoder is pre-trained, and a fine-grained similarity definition is given for cancers of different grades in combination with the severity of thyroid tumors and the probability of malignant lesions. This not only emphasizes the differences between different grades, but also fully refers to the similarities between different grades. As for the text encoder, the text encoder is fine-tuned based on the visual encoder through clinical diagnosis data and image-text multimodal data extracted from public pathology books, so that the text encoder and the visual encoder fully interact and align. After the model training is completed, a picture to be diagnosed is input, which will be encoded into a feature vector through the visual encoder. The feature vector can complete different classification diagnosis tasks by connecting different classification heads, and can also interact with the text vectors in the public pathology data in combination with real cases to retrieve the closest picture and text cases. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 The present invention is a flowchart of a multimodal retrieval system for thyroid pathology images.

[0027] Figure 2 It is a network flowchart for image retrieval based on thyroid cytopathology grading reports. DETAILED DESCRIPTION

[0028] The present invention will be further described and illustrated below in conjunction with specific embodiments and accompanying drawings:

[0029] Please refer to Figure 1 , a multimodal retrieval system for thyroid pathology images, comprising the following steps:

[0030] Pre-trained visual encoder: Based on the thyroid cytopathology grading reporting standard of cytology examination, a similar definition of thyroid pathology images is given, which not only emphasizes the differences between different grades, but also refers to the similarities between different grades, so that the visual encoder can be pre-trained in the graded image dataset;

[0031] In the pre-trained visual encoder, the ranking of any two cytopathology images a and b is defined as follows, where Grade(·) represents the grade of the image:

[0032] Rank(a,b)=|Grade(a)-Grade(b)|

[0033] The hierarchical dataset is denoted as Where X is the set of all pathological images, Y represents the set of all graded annotations, and N represents the total number of samples. And the Rank relationship defined above, a large number of triple samples can be constructed. A typical triple sample contains an anchor sample Anc, a positive sample Pos, and a negative sample Neg. The similarity between Anc and Pos is greater than the similarity between Anc and Neg. The final triple data set is recorded as

[0034]

[0035] Based on these triplet samples, the visual encoder can acquire feature extraction capabilities through training. The visual encoder can encode the thyroid image into a one-dimensional feature vector. In order to align with the subsequent text encoder, a Transformer-based neural network can be used as the visual encoder.

[0036] The similarity of feature vectors can be measured by the Euclidean distance. The Euclidean distance between any two images a and b can be described by the following formula, where VisionEncoder(·) represents the neural network used to extract features:

[0037] Distance(a,b)=||VisionEncoder(a)-VisionEncoder(b)|| 2

[0038] The similarity is inversely proportional to the size of the Distance. When training the network, the optimization goal is to reduce the distance between the anchor sample and the positive sample while increasing the distance between the anchor sample and the negative sample. The final objective function is as follows, where g is a hyperparameter used to control the interval between the positive sample and the negative sample:

[0039]

[0040] In defining the sorting relationship, the definition of the thyroid cancer grading standard is fully considered, and the malignancy probability is introduced to describe the similarity between images in a more fine-grained manner. The process of this stage refers to Figure 2 As shown in the figure, the visual encoder can be trained through this sorting relationship and used for image retrieval. At the same time, since only static feature vectors are needed to calculate the distance, and the size of the feature vectors is much smaller than the original image, the actual runtime overhead can be saved by running the network in advance and storing the feature vectors.

[0041] Aligned text encoder: Based on the pre-trained visual encoder, the text encoder is aligned with clinical diagnosis data and pictures and texts in pathology books to enable data in different modalities to interact.

[0042] Based on the above pre-trained visual encoder, after collecting enough sorting annotations, the text description in the book is defined as the anchor sample, and the image matching the text is the positive sample, otherwise it is the negative sample. Based on the above rules, a multimodal triplet sorting dataset can be obtained, which is denoted as

[0043] Among them, PI i Represents the i-th positive sample image, NI i Represents the corresponding negative sample image.

[0044] Complete multimodal retrieval and classification tasks: Based on the pre-trained multimodal model, text and image inputs are mapped to the same feature vector space, and multimodal retrieval is completed by comparing similarities to find similar cases in professional books or clinical diagnoses. The feature vectors on the visual side can also complete the migration of downstream classification tasks for various classification tasks.

[0045] During the training process, images and texts are encoded into feature vectors by the corresponding encoders, where the encoder on the visual side is denoted as VisionEncoder, and the image feature vector IF=VisionEncoder(Image) is obtained through the visual side, and the encoder on the text side is denoted as TextEncoder, and the text feature vector TF=TextEncoder(Text) is obtained through the text side. The cosine value of the two vectors is used to measure the similarity, which is based on the dot product and Euclidean length of the two vectors, as shown below:

[0046]

[0047] The value range of cosine similarity is between -1 and 1, where 1 means that the directions of the two vectors are exactly the same, -1 means that the directions of the two vectors are completely opposite, and 0 means that the two are independent. It is used to complete the training process of the multimodal model. The optimization goal is to increase the similarity between the feature vector PIF of the text and the positive sample while reducing the similarity between the feature vector NIF of the text and the negative sample. The final objective function is as follows, where g is a hyperparameter used to control the interval between the positive sample and the negative sample:

[0048]

[0049] By completing training on a large number of pathological images and texts, the feature vectors of images and texts are gradually aligned, thus completing the multimodal retrieval task. The model can learn more general and robust feature representations in cytopathology images. Through transfer learning, the model can quickly adapt to new data and can adapt to more custom classification tasks with only a small amount of annotated data.

[0050] The present invention provides a multimodal retrieval system for thyroid pathology images. First, pre-training is completed on a visual encoder, and a fine-grained similarity definition is given for cancers of different grades in combination with the severity of thyroid tumors and the probability of malignant lesions. This not only emphasizes the differences between different grades, but also fully refers to the similarities between different grades. As for the text encoder, the text encoder is fine-tuned based on the visual encoder through clinical diagnosis data and image-text multimodal data extracted from public pathology books, so that the text encoder and the visual encoder fully interact and align. After the model training is completed, a picture to be diagnosed is input, which will be encoded into a feature vector through the visual encoder. The feature vector can complete different classification diagnosis tasks by connecting different classification heads, and can also interact with the text vectors in the public pathology data in combination with real cases to retrieve the closest picture and text cases.

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, rather than to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the essence and scope of the technical solution of the present invention.

Claims

1. A multimodal retrieval system for thyroid pathology images, characterized in that: The following steps are involved: Pre-trained visual encoder: Based on the thyroid cytopathology grading reporting standard of cytology examination, a similar definition of thyroid pathology images is given, which not only emphasizes the differences between different grades, but also refers to the similarities between different grades, so that the visual encoder can be pre-trained in the graded image dataset; Aligned text encoder: Based on the pre-trained visual encoder, the text encoder is aligned with clinical diagnosis data and pictures and texts in pathology books to enable data in different modalities to interact. Complete multimodal retrieval and classification tasks: Based on the pre-trained multimodal model, text and image inputs are mapped to the same feature vector space, and multimodal retrieval is completed by comparing similarities to find similar cases in professional books or clinical diagnoses. The feature vectors on the visual side can also complete the migration of downstream classification tasks for various classification tasks.

2. A multimodal retrieval system for thyroid pathology images as claimed in claim 1, characterized in that: In the pre-trained visual encoder, the ranking of any two cytopathology images a and b is defined as follows, where Grade(·) represents the grade of the image: Rank*a,b)=|Grade(a)-Grade(b)| The hierarchical dataset is denoted as Where X is the set of all pathological images, Y represents the set of all graded annotations, and N represents the total number of samples. And the Rank relationship defined above, a large number of triple samples can be constructed. A typical triple sample contains an anchor sample Anc, a positive sample Pos, and a negative sample Neg. The similarity between Anc and Pos is greater than the similarity between Anc and Neg. The final triple data set is recorded as Based on these triplet samples, the visual encoder can acquire feature extraction capabilities through training. The visual encoder can encode the thyroid image into a one-dimensional feature vector. In order to align with the subsequent text encoder, a Transformer-based neural network is used as the visual encoder.

3. A multimodal retrieval system for thyroid pathology images as claimed in claim 1, characterized in that: The similarity of feature vectors can be measured by the Euclidean distance. The Euclidean distance between any two images a and b can be described by the following formula, where VisionEncoder(·) represents the neural network used to extract features: Distance(a,b)=||VisionEncoder(a)-VisionEncoder(b)|| 2 The similarity is inversely proportional to the size of the Distance. When training the network, the optimization goal is to reduce the distance between the anchor sample and the positive sample while increasing the distance between the anchor sample and the negative sample. The final objective function is as follows, where g is a hyperparameter used to control the interval between the positive sample and the negative sample:

4. A thyroid pathology image multimodal retrieval system as claimed in claim 1, characterized in that: Based on the above pre-trained visual encoder, after collecting enough sorting annotations, the text description in the book is defined as the anchor sample, and the image matching the text is the positive sample, otherwise it is the negative sample. Based on the above rules, a multimodal triplet sorting dataset can be obtained, which is denoted as Among them, PI i Represents the i-th positive sample image, NI i Represents the corresponding negative sample image.

5. A multimodal retrieval system for thyroid pathology images as claimed in claim 1, characterized in that: During the training process, images and texts are encoded into feature vectors by the corresponding encoders, where the encoder on the visual side is denoted as VisionEncoder, and the image feature vector IF=VisionEncoder(Image) is obtained through the visual side, and the encoder on the text side is denoted as TextEncoder, and the text feature vector TF=TextEncoder(Text) is obtained through the text side. The cosine value of the two vectors is used to measure the similarity, which is based on the dot product and Euclidean length of the two vectors, as shown below: The value range of cosine similarity is between -1 and 1, where 1 means that the directions of the two vectors are exactly the same, -1 means that the directions of the two vectors are completely opposite, and 0 means that the two are independent. It is used to complete the training process of the multimodal model. The optimization goal is to increase the similarity between the feature vector PIF of the text and the positive sample while reducing the similarity between the feature vector NIF of the text and the negative sample. The final objective function is as follows, where g is a hyperparameter used to control the interval between the positive sample and the negative sample:

Images