Multi-modal based similar pathology retrieval method and device, equipment and storage medium

Abstract

The application belongs to the technical field of pathological retrieval, and discloses a similar pathological retrieval method, device and equipment based on multiple modes and a storage medium, wherein the method comprises the following steps: acquiring clinical information, a gene sequence and a digital pathological image of a target patient; encoding the clinical information, the gene sequence and the digital pathological image to obtain a clinical information feature vector, a gene sequence feature vector and a pathological image visual feature vector; and searching for the most similar pathology in a preset pathological database according to the clinical information feature vector, the gene sequence feature vector and the pathological image visual feature vector. The application improves the retrieval accuracy of pathological retrieval.

Description

Technical Field

[0001] This application relates to the field of pathological retrieval technology, and in particular to a method, apparatus, device and storage medium for retrieving similar pathological conditions based on multimodality. Background Technology

[0002] Whole Slide Image (WSI) analysis is an important auxiliary diagnostic method in current cancer diagnosis and is widely considered the gold standard for human cancer diagnosis and prognosis. A histopathological image refers to a digital image obtained by imaging stained pathological sections under a microscope, such as... Figure 1 As shown. Currently, the most common tasks in digital pathology image-assisted diagnosis include image segmentation, image classification, and image retrieval. While image segmentation and classification can provide intuitive diagnostic results, they struggle to offer definitive diagnostic evidence. In contrast, content-based image retrieval (CBIR) can search databases for pathological images and related cases similar to the target patient, providing doctors with richer diagnostic information. CBIR utilizes image processing and pattern recognition methods to extract features from digital pathology images. Based on these features, it retrieves historical cases with similar image content from pathology databases and returns them to doctors as diagnostic references, helping them form more reliable diagnostic opinions and accumulate diagnostic experience. CBIR can assist evidence-based medicine decision-making when pathologists face difficult or special cases, or when newly trained pathologists undergo effective training. For example, Google developed the SMILY (Similar Image Search for Histopathology) system, which retrieves image results with similar features from an input image. This system uses 5 billion natural images to train the model, compressing the images into a highly representative numerical vector—the embedding vector. During the learning process, the network gradually learns to distinguish between similar and different categories of images by calculating and comparing their embedding vectors. This model was then used to create a paired dataset of pathological image slices in the field of cancer and their corresponding embedding vectors. When a search image slice is extracted using the SMILY tool, the model calculates the corresponding embedding vector and compares it with the embeddings in the dataset, subsequently retrieving similar images and returning them to the user.

[0003] Based on the evidence-based medicine decision-making significance of similar pathological images in digital pathological image-assisted diagnosis, many methods currently rely on unimodal visual retrieval of pathological images. This involves mapping a pathological image into a low-dimensional vector and then using the similarity of this vector to measure the similarity between two pathological images. However, due to the ultra-high resolution (megapixels) of pathological images, discriminative information cannot always be effectively encoded into the final feature vector during the mapping process, which significantly limits retrieval accuracy. Although deep learning has demonstrated powerful capabilities in extracting high-order semantic features from digital images, this remains a significant challenge for ultra-high resolution pathological images. Summary of the Invention

[0004] The main objective of this application is to provide a method, device, equipment, and storage medium for similar pathological searches based on multimodality, aiming to solve the technical problem that insufficient semantic discriminativeness of visual monomodality leads to low search accuracy.

[0005] To achieve the aforementioned objectives, the first aspect of this application provides a multimodal similar pathology retrieval method, the method comprising:

[0006] Acquire clinical information, gene sequences, and digital pathology images of the target patient;

[0007] The clinical information is encoded to obtain a clinical information feature vector;

[0008] The gene sequence is encoded to obtain a gene sequence feature vector;

[0009] The digital pathological image is encoded to obtain a pathological image visual feature vector;

[0010] The similarity between the visual feature vector of the pathological image and the visual feature vector of the pathological image corresponding to each sample in the pathological database is calculated, and the samples with the largest similarity values ​​(a) are determined from largest to smallest; where a is a positive integer.

[0011] The similarity between the clinical information feature vector and the clinical information feature vector of each sample in the top a samples is calculated to determine the top b samples in descending order of similarity value; where b is a positive integer.

[0012] The similarity between the gene sequence feature vector and the gene sequence feature vector corresponding to each of the top b samples is calculated, and the top c samples with the largest similarity values ​​are determined. The pathological data corresponding to the top c samples are used as the search results; where c is a positive integer.

[0013] Furthermore, the clinical information includes m clinical features, where m is a positive integer. The step of encoding the clinical information to obtain a clinical information feature vector includes:

[0014] A graph is constructed based on the relationships between the m clinical features; where the vertices of the graph represent the clinical features, and the edges of the graph represent the correlations between the clinical features.

[0015] Based on the graph, construct the feature matrix, adjacency matrix, and degree matrix corresponding to the graph.

[0016] The feature matrix, adjacency matrix, and degree matrix are input into a pre-trained graph convolutional network, which maps the graph into a low-dimensional vector to obtain the first low-dimensional vector.

[0017] The first low-dimensional vector is used as the clinical information feature vector.

[0018] Furthermore, the step of encoding the gene sequence to obtain a gene sequence feature vector includes:

[0019] The gene sequence is input into a pre-trained long short-term memory network for encoding to obtain a gene sequence feature vector.

[0020] Furthermore, the step of encoding the digital pathological image to obtain a pathological image visual feature vector includes:

[0021] The digital pathological images are preprocessed to obtain preprocessed images;

[0022] The preprocessed image is split into multiple image blocks;

[0023] Each image block is encoded into a low-dimensional vector, resulting in multiple second low-dimensional vectors;

[0024] Multiple second low-dimensional vectors are clustered into k clusters to obtain the visual feature vectors of the pathological images.

[0025] Furthermore, the step of preprocessing the digital pathological image to obtain a preprocessed image includes:

[0026] The background and holes in the digital pathology image are removed to obtain the preprocessed image.

[0027] Furthermore, prior to the steps of obtaining the target patient's clinical information, gene sequence, and digital pathology images, the following steps are also included:

[0028] Acquire clinical information, gene sequences, and digital pathological images corresponding to n samples;

[0029] The pathological database is constructed based on the clinical information, gene sequences, and digital pathological images corresponding to the n samples.

[0030] Furthermore, after the step of using the pathological data corresponding to the top c samples as the search results, the method further includes:

[0031] The search results are sent to the doctor's device for display.

[0032] Secondly, embodiments of this application provide a multimodal similar pathology retrieval device, the device comprising:

[0033] The acquisition module is used to acquire clinical information, gene sequences, and digital pathology images of the target patient.

[0034] The first encoding module is used to encode the clinical information to obtain a clinical information feature vector;

[0035] The second encoding module is used to encode the gene sequence to obtain a gene sequence feature vector;

[0036] The third encoding module is used to encode the digital pathological image to obtain a pathological image visual feature vector;

[0037] The first similarity calculation module is used to calculate the similarity between the visual feature vector of the pathological image and the visual feature vector of the pathological image corresponding to each sample in the pathological database, and determine the samples with the largest similarity values ​​in descending order of 'a'; where 'a' is a positive integer.

[0038] The second similarity calculation module is used to calculate the similarity between the clinical information feature vector and the clinical information feature vector corresponding to each of the top a samples, and determine the top b samples in descending order of similarity value; where b is a positive integer.

[0039] The third similarity calculation module is used to calculate the similarity between the gene sequence feature vector and the gene sequence feature vector corresponding to each of the top b samples, determine the top c samples in descending order of similarity value, and use the pathological data corresponding to the top c samples as the search results; where c is a positive integer.

[0040] Thirdly, embodiments of this application provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above.

[0041] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any of the preceding claims.

[0042] This application discloses a multimodal similar pathology retrieval method, apparatus, device, and storage medium. By utilizing not only the target patient's digital pathology image for similar pathology retrieval but also the target patient's clinical information and gene sequence, it avoids the semantic discriminative limitations of visual monomodality. Specifically, by encoding the clinical information, gene sequence, and digital pathology image, clinical information feature vectors, gene sequence feature vectors, and pathology image visual feature vectors are obtained. Then, the pathology image visual feature vectors are compared with the pathology image visual feature vectors corresponding to each sample in the pathology database. Similarity calculation is performed to determine the top 'a' samples with the highest similarity values ​​from largest to smallest. The similarity of the clinical information feature vector with the clinical information feature vector corresponding to each of the top 'a' samples is then calculated to determine the top 'b' samples with the highest similarity values ​​from largest to smallest. The similarity of the gene sequence feature vector with the gene sequence feature vector corresponding to each of the top 'b' samples is then calculated to determine the top 'c' samples with the highest similarity values ​​from largest to smallest. The pathological data corresponding to the top 'c' samples is then used as the search results. This method yields more accurate search results, meaning this application improves the accuracy of similar pathological searches. Attached Figure Description

[0043] Figure 1 Histopathological images refer to digital images obtained by imaging stained pathological sections under a microscope.

[0044] Figure 2 This is a flowchart illustrating the multimodal similar pathology retrieval method provided in this application embodiment;

[0045] Figure 3 This is a flowchart illustrating the multimodal similar pathology retrieval method provided in this application embodiment;

[0046] Figure 4 This is a flowchart illustrating the multimodal similar pathology retrieval method provided in this application embodiment;

[0047] Figure 5 A schematic diagram illustrating the creation of a diagram provided in an embodiment of this application;

[0048] Figure 6 express Figure 5 A schematic diagram of the feature matrix, adjacency matrix, and degree matrix corresponding to the graph in the diagram;

[0049] Figure 7 This is a flowchart of a method for encoding digital pathological images to obtain visual feature vectors of pathological images, provided in an embodiment of this application.

[0050] Figure 8This is a flowchart illustrating a multimodal similar pathology retrieval method provided in another embodiment of this application;

[0051] Figure 9 This is a schematic block diagram of the structure of a multimodal similar pathology retrieval device according to an embodiment of this application;

[0052] Figure 10 This is a schematic block diagram of the structure of a computer device according to an embodiment of this application.

[0053] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] Those skilled in the art will understand that, unless explicitly stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of features, integers, steps, operations, elements, modules, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, modules, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any modules and all combinations of one or more associated listed items.

[0056] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0057] Please see Figure 2 This application provides a multimodal similar pathology retrieval method, including steps S1-S7:

[0058] S1. Obtain clinical information, gene sequences, and digital pathology images of the target patient;

[0059] In this embodiment, the clinical information mainly includes demographic information and treatment information. Demographic information includes age, gender, etc.; treatment information includes TNM stage (tumor staging), treatment plan, etc. Clinical information is stored as structured data. Like clinical information, the gene sequence is also a structured feature. The difference is that gene features have sequence relationships and are floating-point values. In contrast, the feature correlations in clinical information are complex, and the data types are diverse. The digital pathological image, also known as a digital tissue pathological image, such as... Figure 3 As shown, Figure 3 It is an ultra-high resolution image that shows the structure at the cellular and tissue levels.

[0060] S2. Encode the clinical information to obtain a clinical information feature vector;

[0061] S3. Encode the gene sequence to obtain a gene sequence feature vector;

[0062] S4. Encode the digital pathological image to obtain a pathological image visual feature vector;

[0063] S5. Calculate the similarity between the visual feature vector of the pathological image and the visual feature vector of the pathological image corresponding to each sample in the pathological database, and determine the samples with the largest similarity values ​​in descending order of value (a); where a is a positive integer.

[0064] In this embodiment of the application, it should be understood that, assuming there are three samples in the pathological database, namely sample 1, sample 2, sample 3 and sample 4, and the visual feature vector of the pathological image of the target patient is denoted as V I Then, the visual feature vector V of the pathological image I The visual feature vector of the pathological image corresponding to Sample 1 will be used to calculate the similarity and obtain the first similarity value; the visual feature vector of the pathological image V I It will also perform similarity calculation with the visual feature vector of the pathological image corresponding to Sample 2 to obtain a second similarity value; the visual feature vector of the pathological image V I It will also perform similarity calculation with the visual feature vector of the pathological image corresponding to sample 3 to obtain a third similarity value; the visual feature vector of the pathological image V I The similarity between the pathological image visual feature vector corresponding to sample 4 and sample 4 is calculated to obtain the fourth similarity value. Assuming a equals 3, we need to determine the top 3 samples with the highest similarity values, i.e., the top 3 samples with the highest similarity values. Assuming the first similarity value is 0.9, the second is 0.8, the third is 0.85, and the fourth is 0.95, then sample 1, sample 3, and sample 4 are the top 3 samples.

[0065] S6. Calculate the similarity between the clinical information feature vector and the clinical information feature vector corresponding to each of the top a samples, and determine the top b samples with the largest similarity values ​​in descending order; where b is a positive integer.

[0066] In this embodiment of the application, it should be understood that a>b. Continuing with the previous example, the clinical information feature vector V... c The similarity between the clinical information feature vector and sample 1 is calculated to obtain the fifth similarity value; the clinical information feature vector V is then used to calculate the similarity. c The similarity between the clinical information feature vector and sample 3 is calculated to obtain the sixth similarity value; the clinical information feature vector V is then used to calculate the similarity. c The similarity between the clinical information feature vectors corresponding to sample 4 and sample 4 is calculated to obtain the seventh similarity value. Assuming b equals 2, and the fifth similarity value is 0.8, the sixth similarity value is 0.85, and the seventh similarity value is 0.9, then sample 3 and sample 4 are the top two samples.

[0067] S7. Calculate the similarity between the gene sequence feature vector and the gene sequence feature vector corresponding to each of the top b samples, determine the top c samples with the largest similarity value, and use the pathological data corresponding to the top c samples as the search results; where c is a positive integer.

[0068] In this embodiment of the application, it should be understood that b>c. Continuing with the previous example, the gene sequence feature vector V... G The similarity between the sequence feature vectors of sample 3 and sample 3 is calculated to obtain the eighth similarity value; the gene sequence feature vector V is then used to calculate the similarity between the sequence feature vectors of sample 3 and sample 3. G The similarity score is calculated between the gene sequence feature vector corresponding to Sample 4 and Sample 1, resulting in the ninth similarity score. Assuming c equals 1, and the eighth similarity score is 0.96 and the second similarity score is 0.95, then Sample 1 is the ranked sample. Finally, Sample 1 is provided to the doctor as the most similar case. It should be understood that providing Sample 1 as the most similar case allows the doctor to access its clinical information, gene sequence, and digital pathology image.

[0069] This application discloses a multimodal similar pathology retrieval method, apparatus, device, and storage medium. By utilizing not only the target patient's digital pathology image for similar pathology retrieval but also the target patient's clinical information and gene sequence, it avoids the semantic discriminative limitations of visual monomodality. Specifically, by encoding the clinical information, gene sequence, and digital pathology image, clinical information feature vectors, gene sequence feature vectors, and pathology image visual feature vectors are obtained. Then, the pathology image visual feature vectors are compared with the pathology image visual feature vectors corresponding to each sample in the pathology database. Similarity calculation is performed to determine the top 'a' samples with the highest similarity values ​​from largest to smallest. The similarity of the clinical information feature vector with the clinical information feature vector corresponding to each of the top 'a' samples is then calculated to determine the top 'b' samples with the highest similarity values ​​from largest to smallest. The similarity of the gene sequence feature vector with the gene sequence feature vector corresponding to each of the top 'b' samples is then calculated to determine the top 'c' samples with the highest similarity values ​​from largest to smallest. The pathological data corresponding to the top 'c' samples is then used as the search results. This method yields more accurate search results, meaning this application improves the accuracy of similar pathological searches.

[0070] Please see Figure 4 In one embodiment, the clinical information includes m clinical features, where m is a positive integer. The step of encoding the clinical information to obtain a clinical information feature vector includes:

[0071] S21. Construct a graph based on the relationships between the m clinical features; where the vertices of the graph represent the clinical features, and the edges of the graph represent the correlations between the clinical features.

[0072] For example, the graph G = (V, E) is as follows Figure 5 As shown, G represents a graph, V represents a vertex, and E represents an edge. The vertices of the graph represent features, and the edges represent the correlations between clinical features, such as the relationship between age and treatment plan.

[0073] S22. Based on the graph, establish the feature matrix, adjacency matrix, and degree matrix corresponding to the graph;

[0074] In this embodiment of the application, assuming m = 5, we can obtain the following: Figure 6 And the corresponding feature matrix, adjacency matrix, and degree matrix.

[0075] S23. Input the feature matrix, adjacency matrix and degree matrix into a pre-trained graph convolutional network, and map the graph into a low-dimensional vector through the graph convolutional network to obtain the first low-dimensional vector.

[0076] S24. Use the first low-dimensional vector as the clinical information feature vector.

[0077] This application embodiment establishes a graph based on the relationships between m clinical features; wherein the vertices of the graph represent the clinical features, and the edges of the graph represent the correlations between the clinical features. A feature matrix, adjacency matrix, and degree matrix are established based on the graph. The feature matrix, adjacency matrix, and degree matrix are input into a pre-trained graph convolutional network, which maps the graph into a low-dimensional vector to obtain a first low-dimensional vector. This first low-dimensional vector is used as the clinical information feature vector, thus achieving the encoding of clinical information.

[0078] In one embodiment, the step of encoding the gene sequence to obtain a gene sequence feature vector includes:

[0079] The gene sequence is input into a pre-trained long short-term memory network for encoding to obtain a gene sequence feature vector.

[0080] This application's embodiments address the sequential nature of gene data by employing a Long Short-Term Memory (LSTM) network. LSTM is a special type of recurrent neural network that can solve the gradient vanishing problem and easily learn long-term dependencies. It should be noted that the gene sequence data requires no preprocessing; the gene sequence is directly input into the LSTM for encoding, ultimately yielding a low-dimensional vector V. G The low-dimensional vector V G It is the gene sequence feature vector.

[0081] In this embodiment, the gene sequence is encoded by inputting it into a pre-trained long short-term memory network to obtain a gene sequence feature vector, thereby achieving the encoding of the gene sequence.

[0082] Please see Figure 7 In one embodiment, the step of encoding the digital pathological image to obtain a pathological image visual feature vector includes:

[0083] S41. Preprocess the digital pathological image to obtain a preprocessed image;

[0084] S42. Divide the preprocessed image into multiple image blocks;

[0085] S43. Encode each image block into a low-dimensional vector to obtain multiple second low-dimensional vectors;

[0086] S44. Cluster the multiple second low-dimensional vectors into k clusters to obtain the visual feature vectors of the pathological image.

[0087] In this embodiment, due to the extremely high resolution of digital pathology images (e.g., 1 million × 1 million pixels), a single image cannot be directly input into the network for processing. Therefore, preprocessing is performed first, followed by splitting the image into patches (blocks), for example, splitting each image into 1024 patches of size 512 × 512. Next, a convolutional network is used to encode each patch into a low-dimensional vector, resulting in 1024 vectors. Then, these 1024 vectors are clustered to obtain k clusters, as shown in Table 1 below.

[0088] Table 1

[0089] Number of clusters 1 2 ...... k Number of vectors 8 11 ...... 100

[0090] Table 1 shows that 8 patches are clustered into the first cluster, 11 patches are clustered into the second cluster, and 100 patches are clustered into the Kth cluster. Table 1 represents the visual feature vectors of the digital pathology images of the target patients.

[0091] In one embodiment, the step of preprocessing the digital pathological image to obtain a preprocessed image includes:

[0092] The background and holes in the digital pathology image are removed to obtain the preprocessed image.

[0093] In this embodiment, the preprocessing step mainly includes removing the background and holes of the digital pathology image. After preprocessing the image, the subsequent digital pathology image processing effect is better.

[0094] Please see Figure 8 In one embodiment, prior to the steps of acquiring the target patient's clinical information, gene sequence, and digital pathology images, the method further includes:

[0095] S8. Obtain the clinical information, gene sequences, and digital pathology images corresponding to n samples; where n is a positive integer;

[0096] S9. The pathological database is constructed based on the clinical information, gene sequences, and digital pathological images corresponding to the n samples.

[0097] In this embodiment of the application, by pre-constructing a pathological database, it is possible to find similar samples from the pathological database.

[0098] Specifically, the records in the pathology database include at least: sample serial number, clinical information feature vector, gene sequence feature vector, and pathological image visual feature vector.

[0099] The database construction is performed in the following three steps, and the order of these three steps does not matter.

[0100] Step 1: Clinical Information Modeling. Considering the diversity of data types and correlations among the features of clinical information, this embodiment employs a Graph Convolutional Network (GCN) for modeling. Assuming each sample has m clinical features, the first step is to construct a graph G = (V, E) based on the relationships between these m clinical features, as follows: Figure 5 As shown, G represents a graph, V represents a vertex, and E represents an edge. The vertices of the graph represent features, and the edges represent correlations between clinical features, such as the relationship between age and treatment plan. Assuming m = 5, we can obtain the following... Figure 6 And the corresponding feature matrix, adjacency matrix, and degree matrix. Based on the graph G = (V, E), these three matrices can be input into the graph convolutional network GCN to map the graph into a low-dimensional vector V. c The low-dimensional vector V c This refers to the clinical information feature vector. For GCN training, classification labels can be used to train and update the GCN parameters, ensuring that the vector V... c The most discriminative information in the graph is encoded. Furthermore, assuming there are n samples, based on the graph G = (V, E) for each sample and GCN classification training, n vectors V can be obtained. c This implies the distinctiveness of each sample in terms of clinical information.

[0101] Step 2: Gene Sequence Modeling. To address the sequential nature of gene data, this embodiment employs a Long Short-Term Memory (LSTM) network for modeling. LSTM is a special type of recurrent neural network that can solve the gradient vanishing problem and easily learn long-term dependencies. No preprocessing is required for the gene sequence data; the sequence of each sample is directly input into the LSTM for encoding, ultimately yielding a low-dimensional vector V. G The low-dimensional vector V G This refers to the gene sequence feature vector. Similarly, classification labels can be used to train and update the LSTM, making V... G The most discriminative information in the sequence is learned. With n samples, n vectors V are obtained. G .

[0102] Step 3: Digital Pathology Image Modeling. Digital pathology images have extremely high resolution, and a single image cannot be directly input into the network for processing. First, preprocessing is performed. For full-size digital pathology images (e.g., 1 million × 1 million pixels), the main operations are background removal and hole subtraction. Then, the image is split into patches. Assuming each image is split into 1024 patches of size 512 × 512, then after splitting n samples, a total of n * 1024 patches are obtained, each patch being 512 × 512. The overall process is as follows:

[0103] (1) Patch encoding: For n*1024 patches (512×512 in size), a convolutional neural network (CNN) is used to encode each patch into a low-dimensional vector, resulting in n*1024 vectors. In order to train the CNN, the patches need to be classified and labeled first.

[0104] (2) Patch clustering: Cluster these n*1024 vectors to obtain k clusters;

[0105] (3) Vectorization of pathological images: Each sample has 1024 patches, and each patch belongs to one of k clusters. Therefore, each pathological image can be vectorized on k clusters, as shown in Table 2. For sample 1, 8 patches are clustered into the first cluster, 11 patches are clustered into the second cluster, and 100 patches are clustered into the kth cluster. For n samples, the visual k-dimensional feature vector V is obtained. I The k-dimensional feature vector is the visual feature vector of the pathological image, which implicitly contains the discriminatory information between samples in the visual features of the digital pathological image.

[0106] Table 2

[0107]

[0108] In one embodiment, after the step of calculating the similarity between the gene sequence feature vector and the gene sequence feature vector corresponding to each of the top b samples, and determining the top c samples in descending order of similarity value, the method further includes:

[0109] The search results are sent to the doctor's device for display.

[0110] This application embodiment sends the pathological data most similar to that of the target patient to the doctor's end, thus providing the doctor with richer auxiliary information for diagnosis.

[0111] Example 2:

[0112] Please see Figure 9 This application also provides a multimodal similar pathology retrieval device, the device comprising:

[0113] The first acquisition module 1 is used to acquire the target patient's clinical information, gene sequence, and digital pathology images.

[0114] The first encoding module 2 is used to encode the clinical information to obtain a clinical information feature vector;

[0115] The second encoding module 3 is used to encode the gene sequence to obtain a gene sequence feature vector;

[0116] The third encoding module 4 is used to encode the digital pathological image to obtain a pathological image visual feature vector;

[0117] The first similarity calculation module 5 is used to calculate the similarity between the visual feature vector of the pathological image and the visual feature vector of the pathological image corresponding to each sample in the pathological database, and determine the samples with the largest similarity values ​​in descending order of a; where a is a positive integer.

[0118] The second similarity calculation module 6 is used to calculate the similarity between the clinical information feature vector and the clinical information feature vector corresponding to each sample in the top a samples, and determine the top b samples in terms of similarity value from largest to smallest; where b is a positive integer.

[0119] The third similarity calculation module 7 is used to calculate the similarity between the gene sequence feature vector and the gene sequence feature vector corresponding to each of the top b samples, determine the top c samples in descending order of similarity value, and use the pathological data corresponding to the top c samples as the search results; where c is a positive integer.

[0120] In one embodiment, the clinical information includes m clinical features, where m is a positive integer, and the first encoding module 2 includes:

[0121] The graph building unit is used to build a graph based on the relationships between m clinical features; wherein, the vertices of the graph represent the clinical features, and the edges of the graph represent the correlations between the clinical features.

[0122] A matrix building unit is used to build the feature matrix, adjacency matrix, and degree matrix corresponding to the graph based on the graph.

[0123] The input unit is used to input the feature matrix, adjacency matrix and degree matrix into a pre-trained graph convolutional network, and to map the graph into a low-dimensional vector through the graph convolutional network to obtain a first low-dimensional vector.

[0124] An equivalent unit is used to use the first low-dimensional vector as a clinical information feature vector.

[0125] In one embodiment, the second encoding module 3 is specifically used to input the gene sequence into a pre-trained long short-term memory network for encoding to obtain a gene sequence feature vector.

[0126] In one embodiment, the third encoding module 4 includes:

[0127] The preprocessing unit is used to preprocess the digital pathological image to obtain a preprocessed image;

[0128] A splitting unit is used to split the preprocessed image into multiple image blocks;

[0129] The encoding unit is used to encode each image block into a low-dimensional vector, resulting in multiple second low-dimensional vectors;

[0130] Clustering units are used to cluster multiple second low-dimensional vectors into k clusters to obtain pathological image visual feature vectors.

[0131] In one embodiment, the preprocessing unit is specifically used for:

[0132] The background and holes in the digital pathology image are removed to obtain the preprocessed image.

[0133] In one embodiment, the multimodal-based similar pathology retrieval device further includes:

[0134] The second acquisition module is used to acquire clinical information, gene sequences, and digital pathological images corresponding to n samples;

[0135] The pathology database construction module is used to construct the pathology database based on the clinical information, gene sequences, and digital pathology images corresponding to the n samples.

[0136] In one embodiment, the multimodal-based similar pathology retrieval device further includes:

[0137] The sending module is used to send the search results to the doctor's terminal for display.

[0138] Reference Figure 10 This application also provides a computer device whose internal structure can be as follows: Figure 10The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores operating devices, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data such as multimodal similar pathology retrieval methods. The network interface allows communication with external terminals via a network connection. Furthermore, the computer device may also include input devices and a display screen. When the aforementioned computer program is executed by a processor, it implements a multimodal similar pathology retrieval method, comprising the following steps: acquiring the clinical information, gene sequence, and digital pathology image of the target patient; encoding the clinical information to obtain a clinical information feature vector; encoding the gene sequence to obtain a gene sequence feature vector; encoding the digital pathology image to obtain a pathology image visual feature vector; calculating the similarity between the pathology image visual feature vector and the pathology image visual feature vector corresponding to each sample in the pathology database, and determining the samples with the highest similarity values, ranked as 'a', where 'a' is a positive integer; calculating the similarity between the clinical information feature vector and the clinical information feature vector corresponding to each of the top 'a' samples, and determining the samples with the highest similarity values, ranked as 'b', where 'b' is a positive integer; calculating the similarity between the gene sequence feature vector and the gene sequence feature vector corresponding to each of the top 'b' samples, and determining the samples with the highest similarity values, ranked as 'c', and using the pathology data corresponding to the top 'c' samples as the retrieval result, where 'c' is a positive integer. Those skilled in the art will understand that... Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment on which the present application is applied.

[0139] One embodiment of this application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements a multimodal similar pathology retrieval method, including the following steps: acquiring clinical information, gene sequences, and digital pathology images of a target patient; encoding the clinical information to obtain a clinical information feature vector; encoding the gene sequence to obtain a gene sequence feature vector; encoding the digital pathology image to obtain a pathology image visual feature vector; calculating the similarity between the pathology image visual feature vector and the pathology image visual feature vector corresponding to each sample in the pathology database, and determining the samples with the highest similarity values, ranked from largest to smallest, 'a'; where 'a' is a positive integer; calculating the similarity between the clinical information feature vector and the clinical information feature vector corresponding to each of the samples ranked from largest to smallest, and determining the samples with the highest similarity values, ranked from largest to smallest, 'b'; where 'b' is a positive integer; calculating the similarity between the gene sequence feature vector and the gene sequence feature vector corresponding to each of the samples ranked from largest to smallest, and determining the samples with the highest similarity values, ranked from largest to smallest, 'c'; and using the pathology data corresponding to the samples ranked from largest to smallest, as the retrieval result; where 'c' is a positive integer. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0140] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media provided in this application and in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0141] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0142] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A multimodal similar pathology retrieval method, characterized in that, The method includes: Acquire clinical information, gene sequences, and digital pathology images of the target patient; The clinical information is encoded to obtain a clinical information feature vector; The gene sequence is encoded to obtain a gene sequence feature vector; The digital pathological image is encoded to obtain a pathological image visual feature vector; The similarity between the visual feature vector of the pathological image and the visual feature vector of the pathological image corresponding to each sample in the pathological database is calculated, and the samples with the largest similarity values ​​(a) are determined from largest to smallest; where a is a positive integer. The similarity between the clinical information feature vector and the clinical information feature vector of each sample in the top a samples is calculated to determine the top b samples in descending order of similarity value; where b is a positive integer. The similarity between the gene sequence feature vector and the gene sequence feature vector of each sample in the top b samples is calculated, and the top c samples with the largest similarity values ​​are determined. The pathological data corresponding to the top c samples are used as the search results; where c is a positive integer.

2. The multimodal similar pathology retrieval method according to claim 1, characterized in that, The clinical information includes m clinical features, where m is a positive integer. The step of encoding the clinical information to obtain a clinical information feature vector includes: A graph is constructed based on the relationships between the m clinical features; where the vertices of the graph represent the clinical features, and the edges of the graph represent the correlations between the clinical features. Based on the graph, construct the feature matrix, adjacency matrix, and degree matrix corresponding to the graph. The feature matrix, adjacency matrix, and degree matrix are input into a pre-trained graph convolutional network, which maps the graph into a low-dimensional vector to obtain the first low-dimensional vector. The first low-dimensional vector is used as the clinical information feature vector.

3. The multimodal similar pathology retrieval method according to claim 1, characterized in that, The step of encoding the gene sequence to obtain a gene sequence feature vector includes: The gene sequence is input into a pre-trained long short-term memory network for encoding to obtain a gene sequence feature vector.

4. The multimodal similar pathology retrieval method according to claim 1, characterized in that, The step of encoding the digital pathological image to obtain a pathological image visual feature vector includes: The digital pathological images are preprocessed to obtain preprocessed images; The preprocessed image is split into multiple image blocks; Each image block is encoded into a low-dimensional vector, resulting in multiple second low-dimensional vectors; Multiple second low-dimensional vectors are clustered into k clusters to obtain the visual feature vectors of the pathological images.

5. The multimodal similar pathology retrieval method according to claim 4, characterized in that, The step of preprocessing the digital pathological image to obtain a preprocessed image includes: The background and holes in the digital pathology image are removed to obtain the preprocessed image.

6. The multimodal similar pathology retrieval method according to claim 1, characterized in that, Before the steps of obtaining the target patient's clinical information, gene sequence, and digital pathology images, the following are also included: Acquire clinical information, gene sequences, and digital pathological images corresponding to n samples; The pathological database is constructed based on the clinical information, gene sequences, and digital pathological images corresponding to the n samples.

7. The multimodal similar pathology retrieval method according to claim 1, characterized in that, After the step of using the pathological data corresponding to the top c samples as the search results, the method further includes: The search results are sent to the doctor's device for display.

8. A multimodal similar pathology retrieval device, characterized in that, The device includes: The acquisition module is used to acquire clinical information, gene sequences, and digital pathology images of the target patient. The first encoding module is used to encode the clinical information to obtain a clinical information feature vector; The second encoding module is used to encode the gene sequence to obtain a gene sequence feature vector; The third encoding module is used to encode the digital pathological image to obtain a pathological image visual feature vector; The first similarity calculation module is used to calculate the similarity between the visual feature vector of the pathological image and the visual feature vector of the pathological image corresponding to each sample in the pathological database, and determine the samples with the largest similarity values ​​in descending order of 'a'; where 'a' is a positive integer. The second similarity calculation module is used to calculate the similarity between the clinical information feature vector and the clinical information feature vector corresponding to each of the top a samples, and determine the top b samples in descending order of similarity value; where b is a positive integer. The third similarity calculation module is used to calculate the similarity between the gene sequence feature vector and the gene sequence feature vector corresponding to each of the top b samples, determine the top c samples in descending order of similarity value, and use the pathological data corresponding to the top c samples as the search results; where c is a positive integer.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images