Pathological Image Retrieval Method and Device

Abstract

This application relates to the technical field of pathological image retrieval, and particularly to a pathological image retrieval method and apparatus. The method includes: obtaining a pathological image to be queried; generating a hash code of the pathological image to be queried; and using the hash code to perform retrieval in a preset database to obtain a query result of the pathological image to be queried, where the database contains the correspondence between the hash code obtained by converting the high-dimensional feature representation output by the hypergraph neural network and the pathological image data. The embodiments of this application not only realize the extraction of full-size pathological image features that take into account both local and global information, train the feature extraction parameters under unsupervised conditions to achieve better feature representation, and the hash code optimizes the retrieval matching mode, greatly improving the retrieval performance.

Description

Technical Field

[0001] This application relates to the technical field of pathological image retrieval, and particularly to a pathological image retrieval method and device. Background Art

[0002] Currently, the analysis of WSI (Whole Slide Image) of tumor patients has always been an important task in the field of pathology and is of great significance for evaluating the patient's condition.

[0003] In the related art, biopsy sampling of the lesion is performed through puncture / minimally invasive surgery, and then a section of the pathological tissue is made. Finally, a pathologist observes it manually, that is, the pathologist visually analyzes the section of the tumor tissue of a cancer patient under a microscope and judges the tumor situation through cell morphology. With the application of AI, AI analysis has shown its prowess in this link. Through the training of a large number of tumor sections, AI captures important features in the pictures, can quickly infer new samples, and achieve various analyses. For example, detecting the location of tumor cells and judging the tumor stage, etc. This system can use deep learning to train a large number of tumor section samples, capture the features of a single pathological image, and on this basis, complete the hash representation of the pathological image, so as to achieve the efficient retrieval of pathological images.

[0004] However, there are three great challenges in analyzing whole slide images. First, the morphological differences in pathological images interfere greatly with feature extraction. After making the sections, the morphological differences in the pathological images recorded by high-definition scanning are extremely large. And these morphologies basically only depend on the sampling and section-making methods in clinical practice and have nothing to do with the disease situation. Therefore, predicting the disease situation through pathological images cannot rely on morphology. Second, the image has hundreds of millions of pixels, and the detailed information therein is very important for diagnosis. One cannot only focus on the whole and ignore the details. Traditional deep learning image processing architectures, such as convolutional neural networks, are difficult to handle pathological images while taking into account both details and the whole. Finally, the annotation of pathological images is difficult and costly. In the actual production and application process, a large number of unannotated pathological images are often obtained. How to effectively utilize a large number of pathological images without annotation information is a difficult point.

[0005] In summary, in the prior art, due to the large interference of morphological differences in pathological images on feature extraction, and the inability to take into account the feature extraction of whole slide images of local and global information, and the high difficulty and cost of pathological image annotation, it is difficult to represent the features of pathological images, resulting in the problem of low retrieval performance.

[0006] Content of the Application

[0007] The present application provides a method and apparatus for pathological image retrieval to solve the technical problems in the related art that the extraction of full-size pathological image features cannot take into account both local and global information, and the feature representation of pathological images is relatively difficult, resulting in low retrieval performance.

[0008] In a first aspect embodiment of the present application, a method for pathological image retrieval is provided, including the following steps: obtaining a pathological image to be queried; generating a hash code of the pathological image to be queried; using the hash code to perform a retrieval in a preset database to obtain a query result of the pathological image to be queried, where the database contains the correspondence between the hash code obtained by converting the high-dimensional feature representation output by the hypergraph neural network and the pathological image data.

[0009] Optionally, in an embodiment of the present application, before using the hash code to perform a retrieval in the preset database, it further includes: using the hypergraph neural network to extract and fuse the features of the full-size pathological image to obtain a trained hypergraph neural network; inputting the pathological image data into the trained hypergraph neural network to generate a high-dimensional feature representation, converting the high-dimensional feature representation into a corresponding hash code, and storing the hash code corresponding to the pathological image data to establish the database.

[0010] Optionally, in an embodiment of the present application, the using the hypergraph neural network to extract and fuse the features of the full-size pathological image to obtain a trained hypergraph neural network includes: preprocessing each full-size pathological image; for each preprocessed full-size pathological image, constructing a hypergraph between sample points; propagating the information of the entire hypernode in the hypergraph through hyperedges, training the hypergraph neural network, and using a key encoder and a sample queue to record old features, comparing the old features with the new features generated by the query encoder of the hypergraph neural network, and optimizing the hypergraph neural network according to the comparison result to obtain the trained hypergraph neural network.

[0011] Optionally, in an embodiment of the present application, the inputting the pathological image data into the trained hypergraph neural network to generate a high-dimensional feature representation, converting the high-dimensional feature representation into a corresponding hash code, and storing the hash code corresponding to the pathological image data to establish the database includes: converting the high-dimensional feature representation into the hash code stored in binary 0 and 1 using a hash algorithm; using the existing pathological image sample data to establish the database based on hash code retrieval after hash encoding.

[0012] Optionally, in an embodiment of the present application, the retrieving in the preset database using the hash code to obtain the query result of the pathological image to be queried includes: passing the pathological image to be queried through the hypergraph neural network to obtain the corresponding hash code; querying in the database using the hash code corresponding to the pathological image to be queried to obtain at least one best match corresponding to the pathological image to be queried; analyzing the relevant medical data situation of the at least one best match, estimating the relevant information of the new pathological image, and generating the query result.

[0013] An embodiment of the second aspect of the present application provides a pathological image retrieval device, including: an acquisition module, configured to acquire a pathological image to be queried; a generation module, configured to generate a hash code of the pathological image to be queried; a retrieval module, configured to retrieve in a preset database using the hash code to obtain a query result of the pathological image to be queried, where the database contains the correspondence between the hash code obtained by converting the high-dimensional feature representation output by the hypergraph neural network and the pathological image data.

[0014] Optionally, in an embodiment of the present application, it further includes: a building module, configured to, before retrieving in the preset database using the hash code, use the hypergraph neural network to extract and fuse the features of the full-size pathological image to obtain a trained hypergraph neural network, input the pathological image data into the trained hypergraph neural network to generate a high-dimensional feature representation, convert the high-dimensional feature representation into a corresponding hash code, and store the hash code corresponding to the pathological image data to build the database.

[0015] Optionally, in an embodiment of the present application, the building module is further configured to preprocess each full-size pathological image; for each preprocessed full-size pathological image, construct a hypergraph between sample points; propagate the information of the entire hypernode in the hypergraph through hyperedges, train the hypergraph neural network, and use a key encoder and a sample queue to record old features, compare the old features with the new features generated by the query encoder of the hypergraph neural network, and optimize the hypergraph neural network according to the comparison result to obtain the trained hypergraph neural network.

[0016] Optionally, in an embodiment of the present application, the building module is further configured to convert the high-dimensional feature representation into the hash code stored in binary 0 and 1 using a hash algorithm; use the existing pathological image sample data to build the database based on hash code retrieval after hash coding.

[0017] Optionally, in one embodiment of the present application, the retrieval module is further used to pass the pathological image to be queried through the hypergraph neural network to obtain the corresponding hash code; use the hash code corresponding to the pathological image to be queried as an index to query the database to obtain at least one best match corresponding to the pathological image to be queried; analyze the relevant medical data of the at least one best match, estimate the relevant information of the new pathological image, and generate the query result.

[0018] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the pathological image retrieval method as described in the above embodiment.

[0019] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the pathological image retrieval method as described in the above embodiment.

[0020] The pathological image retrieval method of the embodiment of the present application uses the hash code of the new pathological image to search in the database, and the database is established by converting the hash code obtained by the high-dimensional feature representation output by the hypergraph neural network and the corresponding pathological image, so as to obtain the query result of the new pathological image, which not only realizes the full-size pathological image feature extraction that takes into account both local and overall information, but also trains the feature extraction parameters under unsupervised conditions to achieve better feature representation, and the hash code optimizes the retrieval matching mode, greatly improving the retrieval performance. Therefore, it solves the technical problem that the related technology cannot extract the full-size pathological image feature that takes into account both local and overall information, and the feature representation of the pathological image is relatively difficult, resulting in low retrieval performance.

[0021] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0023] Figure 1 A flowchart of a pathological image retrieval method provided according to an embodiment of the present application;

[0024] Figure 2 is a flow chart of a pathological image retrieval method according to a specific embodiment of the present application;

[0025] Figure 3Schematic diagram of a hypergraph neural network for processing pathological images according to an embodiment of the present application;

[0026] Figure 4 Example diagram of a pathological image retrieval device according to an embodiment of the present application;

[0027] Figure 5 Schematic structural diagram of an electronic device according to an embodiment of the present application. Detailed implementation manners

[0028] The embodiments of the present application will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, but should not be construed as limiting the present application.

[0029] The pathological image retrieval method and device according to the embodiments of the present application will be described below with reference to the accompanying drawings. In view of the technical problem in the above-mentioned background art that the related technology cannot take into account the feature extraction of full-size pathological images with both local and global information, and the feature representation of pathological images is relatively difficult, resulting in low retrieval performance, the present application provides a pathological image retrieval method. In this method, the hash code of the new pathological image is used for retrieval in the database, and the database is established by the hash code obtained by converting the high-dimensional feature representation output by the hypergraph neural network and the corresponding pathological images, so as to obtain the query result of the new pathological image. It not only realizes the feature extraction of full-size pathological images with both local and global information, but also trains the feature extraction parameters under unsupervised conditions to achieve better feature representation, and the hash code optimizes the retrieval matching mode, greatly improving the retrieval performance. Thus, the technical problem that the related technology cannot take into account the feature extraction of full-size pathological images with both local and global information, and the feature representation of pathological images is relatively difficult, resulting in low retrieval performance is solved.

[0030] Specifically, Figure 1 Schematic flowchart of a pathological image retrieval method provided by an embodiment of the present application.

[0031] As Figure 1 shown, the pathological image retrieval method includes the following steps:

[0032] In step S101, a pathological image to be queried is obtained.

[0033] In step S102, a hash code of the pathological image to be queried is generated.

[0034] That is to say, for each new pathological image to be queried in the embodiments of the present application, a separate hash code is generated through the same hypergraph neural network. For example, for a new sample found in the actual production process, it is input into the retrieval system, and through the feature extraction and fusion of the hypergraph neural network and the hash algorithm, a unique hash code is obtained.

[0035] In step S103, the hash code is used to retrieve in a preset database to obtain the query result of the pathological image to be queried, where the database contains the corresponding relationship between the hash code converted from the high-dimensional feature representation output by the hypergraph neural network and the pathological image data.

[0036] Those skilled in the art should understand that the embodiments of the present application perform fast retrieval based on the hash code in the known sample database to achieve efficient query. The following will describe in detail how to retrieve and how to establish the database.

[0037] Optionally, in an embodiment of the present application, using the hash code to retrieve in a preset database to obtain the query result of the pathological image to be queried includes: passing the pathological image to be queried through the hypergraph neural network to obtain the corresponding hash code; using the hash code corresponding to the pathological image to be queried as an index to query in the database to obtain at least one best match corresponding to the pathological image to be queried; analyzing the relevant medical data situation of at least one best match, estimating the relevant information of the new pathological image, and generating the query result.

[0038] The following lists embodiments to illustrate how to retrieve. Among them, the embodiments of the present application may include:

[0039] Step S1: For a new sample found in the actual production process, it is input into the retrieval system, and through the feature extraction and fusion of the hypergraph neural network and the hash algorithm, a unique hash code is obtained

[0040] Step S2: Match the hash code of the new sample in the system database. The matching is performed through hash retrieval, and the hash retrieval between hash codes can achieve efficient performance.

[0041] Step S3: The retrieval result will display the best one or more matches of the new sample in the database. By analyzing the relevant medical data situation of the matched samples, the situation of the current new sample can be deduced, so as to estimate the unknown sample with the existing experience.

[0042] In addition, in one embodiment of the present application, before retrieving in a preset database using hash coding, it further includes: using a hypergraph neural network to extract and fuse the features of the full-size pathological image to obtain a trained hypergraph neural network; inputting the pathological image data into the trained hypergraph neural network to generate a high-dimensional feature representation, converting the high-dimensional feature representation into a corresponding hash code, and storing the hash code corresponding to the pathological image data to establish a database.

[0043] In the embodiments of the present application, the embodiments of the present application can use a hypergraph neural network to extract and fuse the features of the full-size pathological image. The neural network parameters are trained by a self-supervised method, making full use of a large amount of unlabeled medical image data. Secondly, a large amount of existing data is input into the trained hypergraph neural network to generate a high-dimensional feature representation. The feature representation is converted into a hash code through a hash algorithm, and the hash codes corresponding to all data are stored in the system database as known sample data.

[0044] Optionally, in one embodiment of the present application, using a hypergraph neural network to extract and fuse the features of the full-size pathological image to obtain a trained hypergraph neural network includes: preprocessing each full-size pathological image; for each preprocessed full-size pathological image, constructing a hypergraph between sample points; propagating the information of the entire hypernode in the hypergraph through hyperedges, training the hypergraph neural network, and using a key encoder and a sample queue to record old features, comparing the old features with the new features generated by the query encoder of the hypergraph neural network, and optimizing the hypergraph neural network according to the comparison result to obtain a trained hypergraph neural network.

[0045] As a possible implementation manner, in the actual execution process, the embodiments of the present application include:

[0046] Step S1: Preprocessing the full-size pathological image, randomly sampling in the effective area of the pathological image with a pixel size of 256x256. The sampled image passes through a pre-trained convolutional residual network to generate original features, and each pathological image generates a feature representation of 2000 sampling points in total.

[0047] Step S2: For each pathological image, construct a hypergraph between sample points. Each hypergraph structure is described by G=(V; E). Each vertex in V represents a sampling point, and E represents the hyperedge connecting the sampling points. The hyperedge is established depending on the distribution of the sampling points in the spatial position. There is a greater chance of forming a connection of the hyperedge between multiple sampling points that are closest to each other.

[0048] Step S3: Through the hypergraph neural network, propagate the information of the entire hypernode through the hyperedge. The propagation is based on hypergraph convolution:

[0049] X (l+1) =σ[((I-L)X(l) +H -1 (I - L)X (λ) )Θ (l) ,

[0050] Among them, Θ is a parameter to be learned. After obtaining the fused features, a unified feature representing the entire pathological image is obtained through a pooling operation.

[0051] Step S4: In order to utilize a large amount of unlabeled medical data, a self - supervised method is adopted to achieve training without relying on picture annotation information. It is implemented using a query encoder, a key encoder, and a sample queue. The key encoder and the queue are used to assist in recording old features, and a comparison is made with the new features generated by the query encoder, thereby optimizing the parameters of the query encoder and achieving parameter update.

[0052] Optionally, in an embodiment of the present application, the pathological image data is input into the trained hypergraph neural network to generate a high - dimensional feature representation, and the high - dimensional feature representation is converted into a corresponding hash code. The hash code is stored corresponding to the pathological image data to establish a database, including: converting the high - dimensional feature representation into a hash code stored in 0, 1 binary using a hash algorithm; using the existing pathological image sample data to establish a database based on hash - code retrieval after hash - coding.

[0053] It can be understood that for how to establish the database, the embodiments of the present application may include:

[0054] Step S1: Convert the high - dimensional pathological image feature representation generated by the system into a hash code using a hash algorithm. The hash code is stored in 01 binary, which can greatly improve the retrieval performance.

[0055] Step S2: Use the existing pathological image sample data to establish a hash - code retrieval database after hash - coding as a standard and basis.

[0056] In summary, the embodiments of the present application utilize a hypergraph neural network to extract and fuse the features of full-size pathological images. The neural network parameters are trained through a self-supervised method to make full use of a large amount of unlabeled medical imaging data; a large amount of existing data is input into the trained hypergraph neural network to generate a high-dimensional feature representation, and the feature representation is converted into a hash code through a hash algorithm, and the hash codes corresponding to all data are saved in the system database as known sample data; for each new pathological image to be queried, a separate hash code is generated through the same hypergraph neural network, and a fast retrieval based on the hash code is performed in the known sample database to achieve efficient query. In the embodiments of the present application, the embodiments of the present application realize the feature extraction of full-size pathological images that takes into account both local and overall information, and trains the feature extraction parameters under unsupervised conditions to achieve a better feature representation. The hash code optimizes the retrieval matching mode and greatly improves the retrieval performance.

[0057] Combination Figure 2 As shown, the following examples are listed to schematically illustrate the principle of the search method of the embodiment of the present application. The method of the embodiment of the present application may include the following steps:

[0058] Step S1: Using the hypergraph neural network, the features of the full-size pathological image are extracted and fused. The neural network parameters are trained through a self-supervised method to make full use of a large amount of unlabeled medical imaging data.

[0059] Step S1.1: Preprocessing of full-size pathological images. The specific steps and parameters of preprocessing are as follows:

[0060] 1) Based on the image contrast, the pure white background and colored effective area of the image are segmented, and the segmentation algorithm uses OTSU ( Nobuyuki, Otsu method or maximum inter-class variance method), is a classic computer vision segmentation algorithm based on contrast, which uses the Gaussian distribution variance σ 2 Controls the extent of the effective area after segmentation.

[0061] 2) Randomly sample the valid area of the pathological image with a pixel size of 256x256. Considering that the valid areas of some pathological images are scattered, the sampling area needs to accommodate a certain degree of white invalid area, which is controlled by the threshold τ. During the random sampling process, when the valid area and the proportion are greater than τ, the current sampling point can be selected.

[0062] 3) Each sampled regional image is passed through the pre-trained convolutional residual network ResNet to generate the pre-processed original feature representation. Each pathological image generates a feature representation of 2000 sampling points, and the dimension of each feature representation is 512. Each full-size pathological image forms a pre-processed feature representation of 2000x512.

[0063] Step S1.2: For each pathological image, construct a hypergraph between sample points. Each hypergraph structure is described by G=(V; E), where each vertex in V represents a sampling point, and E represents the hyperedges connecting the sampling points. The hyperedges are established depending on the distribution of the sampling points in space. There is a greater chance of forming hyperedge connections between multiple sampling points that are closest to each other, as Figure 3 shown.

[0064] The construction process of the hypergraph is carried out according to the following steps. Each time, a fixed point in V is taken as the central fixed point, and the 2K nearest samples are selected from the sampling point position records. K samples are randomly selected from these 2K samples and connected with hyperedges.

[0065] During the construction of the hypergraph, the H adjacency matrix is used to describe the connection relationship of the hypergraph. Each element h in H is defined as follows:

[0066]

[0067] The following are some definitions regarding the hypergraph structure: The degree of a hypergraph node can represent the connectivity of the node, and its definition is:

[0068] d(v)=∑ e∈ε ω(e)h(v,e),

[0069] where ω is used to represent the weight of a certain hyperedge. In the embodiments of the present application, the weight ω can be set to 1.

[0070] The degree of a hyperedge can represent the connectivity of a certain hyperedge, and its definition is:

[0071]

[0072] Introducing randomness during the construction of the hypergraph can reduce the overfitting problem during the training process.

[0073] Step S1.3: Through the hypergraph neural network, the information of the entire hypernode is propagated through the hyperedges. The Laplacian operator L of the hypergraph is used to represent the hypergraph structure, and its definition is:

[0074]

[0075] where H is the adjacency matrix; D v is the diagonal matrix composed of the degrees of the hypergraph nodes; D e is the diagonal matrix composed of the degrees of the hypergraph hyperedges; W is the weight matrix and can be set as the identity matrix.

[0076] Furthermore, after calculating the Laplacian operator, the forward propagation of each layer of the hypergraph neural network is based on hypergraph convolution:

[0077] X(l+1) = σ[((I - L)X (l) + H -1 (I - L)X (λ) )Θ (l) ,

[0078] where Θ is the parameter to be learned; σ is the activation function, and in the embodiments of the present application, LeakyRelu can be adopted.

[0079] After obtaining the fused features, a unified feature representing the entire pathological image is obtained through a pooling operation. The specific steps of pooling are as follows: First, obtain the hypergraph convolution result X (l) , in the embodiments of the present application, the dimension of X (l) is 2000x128, where 2000 is the number of nodes and 128 is the feature dimension of each node. Through the pooling operation, the node dimension is averaged and transformed into a unified feature representation with a dimension of 1x128. After pooling, a fully connected layer can be passed through to generate a final feature representation with a lower dimension.

[0080] Step S1.4: In order to utilize a large amount of unlabeled medical data, a self-supervised method is adopted to implement training without relying on picture annotation information. It is implemented by using a query encoder, a key encoder, and a sample queue. The key encoder and the queue are used to assist in recording old features, and are compared with the new features generated by the query encoder, so as to optimize the parameters of the query encoder and achieve parameter update.

[0081] During the self-supervised process, the loss function selects a commonly used contrastive loss function, the negative example loss is set to positive, and the positive example loss is set to negative. Each group of training samples contains a group of positive examples and multiple groups of negative examples, and their definitions are as follows:

[0082]

[0083] During the self-supervised process, the loss function selects a commonly used contrastive loss function, the negative example loss is set to positive, and the positive example loss is set to negative. Each group of training samples contains a group of positive examples and multiple groups of negative examples, and their definitions are as follows: For each input sample, random processing is performed to generate two different randomized samples x q and x k , the former generates a feature representation through the query encoder, and the latter generates a feature representation through the key encoder:

[0084] q = f q (x q ),

[0085] k = f k (x k ),

[0086] During the update process, the query encoder fq Update directly through backpropagation, the key encoder f k Update slowly relying on momentum, and its update formula is:

[0087] θ k ←mθ k +(1 - m)θ q ,

[0088] where m is a constant very close to 1, generally taking 0.999, to ensure a slow update process, so that the parameter update progress of the key encoder lags far behind that of the query encoder.

[0089] The self - supervised system maintains a queue and saves the recently generated k. When constructing the loss function, the positive example is the difference between the latest generated q and k, and the negative examples are the latest generated q and all the previously generated k in the queue. Then iterative training is carried out.

[0090] Step S2: Input a large amount of existing data into the trained hypergraph neural network to generate high - dimensional feature representations, convert the feature representations into hash codes through a hash algorithm, and save the hash codes corresponding to all data in the system database as known sample data.

[0091] Step S2.1: Convert the high - dimensional pathological image feature representations generated by the system into hash codes using a hash algorithm, and the hash codes are stored in binary 01. Mature hash algorithms are divided into two types: supervised and unsupervised. In this embodiment of the application, a supervised hash algorithm can be adopted.

[0092] During the training process, the objective function is defined as:

[0093]

[0094] where H l is defined as In the formula, the h function represents a function that maps the high - dimensional feature space to values in {1, - 1}. There are a total of r h functions, indicating that the length of the hash code is r. Thus, through r functions, the high - dimensional feature space can be mapped to an r - dimensional hash code. S is the target matrix composed of {1, - 1}, where 1 represents similarity and - 1 represents dissimilarity.

[0095] By training and optimizing the above - mentioned objective function, the parameters of the r h functions are trained, and as a whole, they become the target hash function. After obtaining the trained hash function, this hash function can effectively encode the original feature representation, and the encoded result maximally restores the feature similarity situation before encoding.

[0096] Step S2.2: After obtaining the trained hash encoding function, all the existing pathological image sample data are subjected to the above-mentioned hash encoding, which serves as the known sample data in the full-size pathological image retrieval database and becomes the original data of the system, as well as the basic standard and basis. So far, the pathological image retrieval system of this application embodiment is completely built.

[0097] Step S3: For each new pathological image to be queried, a separate hash encoding is generated through the same hypergraph neural network, and fast retrieval based on the hash encoding is performed in the known sample database to achieve efficient query.

[0098] Step S3.1: For the new data that needs to be analyzed in the actual production process, it is input into the retrieval system. The retrieval system passes the sample through the foregoing steps once: obtaining random sample points and a random hypergraph structure through preprocessing sampling; extracting features through the hypergraph neural network, and achieving feature fusion through pooling operations, and performing hash encoding through the trained hash function to obtain the encoding of this data.

[0099] Step S3.2: The hash encoding of the new data is matched in the system database, and the matching is performed through hash retrieval to achieve hash-based feature matching.

[0100] The hash encoding is a binary encoding composed of 0 and 1, which is naturally suitable for computer storage and calculation. The similarity of binary hash encoding can be represented by the Hamming distance. The definition of the Hamming distance is to perform a bitwise exclusive OR operation on two binary encodings. In the result, the number of bits with a value of 1. The computer can implement the bitwise exclusive OR calculation extremely fast, which can greatly improve the retrieval speed.

[0101] Step S3.3: The retrieval result will display the best one or more matches of the new sample in the database, and the rest of the database can be associated to query the relevant medical information of the best match. By analyzing the relevant medical data of the matched sample, the situation of the current new sample can be deduced, so as to estimate the unknown sample with the existing experience.

[0102] The method for retrieving pathological images based on hypergraphs proposed according to the embodiments of the present application analyzes full-size pathological pictures using hypergraph neural networks, connects randomly sampled sample points with a non-Euclidean hypergraph structure, thereby extracting the features of the entire pathological image under the condition of taking into account both detailed information and overall information. The full utilization of information ensures the accuracy and representativeness of the extraction results; the self-supervised learning method based on queue storage builds a self-supervised training framework on the basis of hypergraph neural networks, can be independent of annotations, and only trains the feature extractor through unannotated samples to optimize the ability of the neural network to extract the features of full-size pathological images; uses the hash algorithm to encode the high-dimensional features generated by the full-size pathological images. Through the obtained hash codes, efficient retrieval can be achieved in the database system, improving the retrieval performance in the actual production and application process.

[0103] Next, a pathological image retrieval device proposed according to the embodiments of the present application will be described with reference to the accompanying drawings.

[0104] Figure 4 It is a block diagram of the pathological image retrieval device according to the embodiments of the present application.

[0105] As Figure 4 shown, the pathological image retrieval device 10 includes: an acquisition module 100, a generation module 200, and a retrieval module 300.

[0106] Specifically, the acquisition module 100 is used to acquire the pathological image to be queried.

[0107] The generation module 200 is used to generate the hash code of the pathological image to be queried.

[0108] The retrieval module 300 is used to perform a retrieval in a preset database using the hash code to obtain the query result of the pathological image to be queried, where the database contains the corresponding relationship between the hash code obtained by converting the high-dimensional feature representation output by the hypergraph neural network and the pathological image data.

[0109] Optionally, in an embodiment of the present application, the embodiments of the present application further include: a building module. The building module is used to, before performing a retrieval in a preset database using the hash code, use the hypergraph neural network to extract and fuse the features of the full-size pathological image to obtain a trained hypergraph neural network, input the pathological image data into the trained hypergraph neural network to generate a high-dimensional feature representation, convert the high-dimensional feature representation into a corresponding hash code, and store the hash code and the pathological image data in correspondence to build a database.

[0110] Optionally, in an embodiment of the present application, the establishment module is further configured to preprocess each full-size pathological image; for each preprocessed full-size pathological image, construct a hypergraph between sample points; propagate the information of the entire hypernode in the hypergraph through hyperedges, train a hypergraph neural network, and use a key encoder and a sample queue to record old features, compare the old features with the new features generated by the query encoder of the hypergraph neural network, and optimize the hypergraph neural network according to the comparison result to obtain a trained hypergraph neural network.

[0111] Optionally, in an embodiment of the present application, the establishment module is further configured to convert the high-dimensional feature representation into a hash code stored in binary 0 and 1 using a hash algorithm; utilize the existing pathological image sample data to establish a database based on hash code retrieval after hash encoding.

[0112] Optionally, in an embodiment of the present application, the retrieval module 300 is further configured to pass the pathological image to be queried through the hypergraph neural network to obtain a corresponding hash code; use the hash code corresponding to the pathological image to be queried as an index to query in the database to obtain at least one best match corresponding to the pathological image to be queried; analyze the relevant medical data situation of at least one best match, estimate the relevant information of the new pathological image, and generate a query result.

[0113] It should be noted that the foregoing explanation of the embodiment of the pathological image retrieval method also applies to the pathological image retrieval device of this embodiment, and will not be repeated here.

[0114] According to the pathological image retrieval device provided by the embodiment of the present application, the full-size pathological picture is analyzed using a hypergraph neural network, and randomly sampled sample points are connected by a non-Euclidean hypergraph structure, so as to extract the features of the entire pathological image under the condition of taking into account both detailed information and overall information. The full utilization of information ensures the accuracy and representativeness of the extraction result; based on the self-supervised learning method of queue storage, a self-supervised training framework is built on the basis of the hypergraph neural network, which can be independent of annotations and only train the feature extractor through unlabeled samples, optimizing the neural network's ability to extract features of full-size pathological images; using the hash algorithm to encode the high-dimensional features generated by the full-size pathological image, through the obtained hash code, efficient retrieval can be achieved in the database system, improving the retrieval performance in the actual production and application process.

[0115] Figure 5 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device may include:

[0116] A memory 501, a processor 502, and a computer program stored on the memory 501 and executable on the processor 502.

[0117] When the processor 502 executes the program, it implements the pathological image retrieval method provided in the above embodiments.

[0118] Furthermore, the electronic device further includes:

[0119] A communication interface 503, which is used for communication between the memory 501 and the processor 502.

[0120] A memory 501, which is used to store a computer program that can run on the processor 502.

[0121] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0122] If the memory 501, the processor 502, and the communication interface 503 are implemented independently, the communication interface 503, the memory 501, and the processor 502 can be interconnected through a bus and complete communication with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity of representation, Figure 5 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0123] Optionally, in a specific implementation, if the memory 501, the processor 502, and the communication interface 503 are integrated on a chip, the memory 501, the processor 502, and the communication interface 503 can complete communication with each other through an internal interface.

[0124] The processor 502 may be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0125] This embodiment also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the above pathological image retrieval method.

[0126] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or N embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0127] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of this application, the meaning of "N" is at least two, such as two, three, etc., unless otherwise specifically defined.

[0128] Any process or method description shown in the flowchart or described in other ways herein can be understood as representing a module, segment, or part of code including one or more N executable instructions for implementing a customized logic function or process, and the scope of the preferred embodiments of this application includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in the reverse order according to the involved functions, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of this application pertain.

[0129] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in conjunction with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection part (electronic device) having one or N wirings, a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then storing it in a computer memory.

[0130] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having suitable combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0131] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the methods of the above-described embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0132] In addition, each functional unit in various embodiments of the present application may be integrated into a processing module, may exist physically alone for each unit, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0133] The above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disc, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A method for retrieving pathological images, characterized in that, It includes the following steps: Obtain the pathological image to be queried; Generate a hash code of the pathological image to be queried; And Use the hash code to retrieve in a preset database to obtain the query result of the pathological image to be queried, where the database contains the corresponding relationship between the hash code obtained by converting the high-dimensional feature representation output by the hypergraph neural network and the pathological image data; Before using the hash code to retrieve in the preset database, it further includes: Use the hypergraph neural network to extract and fuse the features of the full-size pathological image to obtain a trained hypergraph neural network; Input the pathological image data into the trained hypergraph neural network to generate a high-dimensional feature representation, convert the high-dimensional feature representation into a corresponding hash code, and store the hash code corresponding to the pathological image data to establish the database; The using the hypergraph neural network to extract and fuse the features of the full-size pathological image to obtain a trained hypergraph neural network includes: Preprocess each full-size pathological image; For each preprocessed full-size pathological image, construct a hypergraph between sample points; Propagate the information of the entire hypernode in the hypergraph through hyperedges, train the hypergraph neural network, and use a key encoder and a sample queue to record old features, compare the old features with the new features generated by the query encoder of the hypergraph neural network, and optimize the hypergraph neural network according to the comparison result to obtain the trained hypergraph neural network; The preprocessing each full-size pathological image includes: Based on the image contrast, use a segmentation algorithm to segment the pure white background and the colored effective area of the image; randomly sample in the effective area of the pathological image; respectively pass each sampled regional image through a pre-trained convolutional residual network to generate a preprocessed original feature representation.

2. The method according to claim 1, characterized in that, The inputting the pathological image data into the trained hypergraph neural network to generate a high-dimensional feature representation, converting the high-dimensional feature representation into a corresponding hash code, and storing the hash code corresponding to the pathological image data to establish the database includes: Convert the high-dimensional feature representation into the hash code stored in binary 0 and 1 using a hash algorithm; Use the existing pathological image sample data to establish the database based on hash code retrieval after hash encoding.

3. The method according to claim 1, characterized in that The using the hash code to retrieve in a preset database to obtain the query result of the pathological image to be queried includes: Pass the pathological image to be queried through the hypergraph neural network to obtain the corresponding hash code; Use the hash code corresponding to the pathological image to be queried as an index to query in the database to obtain at least one best match corresponding to the pathological image to be queried; Analyze the relevant medical data situation of the at least one best match, estimate the relevant information of the pathological image, and generate the query result.

4. A pathological image retrieval device, characterized in that, It includes: An acquisition module for acquiring the pathological image to be queried; A generation module for generating a hash code of the pathological image to be queried; And A retrieval module, configured to retrieve in a preset database by using the hash code to obtain a query result of the pathological image to be queried, wherein the database contains the corresponding relationship between the hash code obtained by converting the high-dimensional feature representation output by the hypergraph neural network and the pathological image data; A building module, configured to, before retrieving in the preset database by using the hash code, use the hypergraph neural network to extract and fuse the features of the full-size pathological image to obtain a trained hypergraph neural network, input the pathological image data into the trained hypergraph neural network to generate a high-dimensional feature representation, convert the high-dimensional feature representation into a corresponding hash code, store the hash code and the pathological image data in correspondence, and build the database; The building module is further configured to preprocess each full-size pathological image; for each preprocessed full-size pathological image, construct a hypergraph between sample points; propagate the information of the entire hypernode in the hypergraph through hyperedges, train the hypergraph neural network, record old features by using a key encoder and a sample queue, compare the old features with the new features generated by the query encoder of the hypergraph neural network, and optimize the hypergraph neural network according to the comparison result to obtain the trained hypergraph neural network; Specifically, the building module is configured to: based on the image contrast, use a segmentation algorithm to segment the pure white background and the colored effective area of the image; randomly sample in the effective area of the pathological image; respectively pass each sampled regional image through a pre-trained convolutional residual network to generate a preprocessed original feature representation.

5. The device according to claim 4, characterized in that, The building module is further configured to convert the high-dimensional feature representation into the hash code stored in binary 0 and 1 by using a hash algorithm; use the existing pathological image sample data to build the database based on hash code retrieval after hash coding.

6. The device according to claim 4, characterized in that, The retrieval module is further configured to pass the pathological image to be queried through the hypergraph neural network to obtain the corresponding hash code; query in the database by using the hash code corresponding to the pathological image to be queried as an index to obtain at least one best match corresponding to the pathological image to be queried; analyze the relevant medical data situation of the at least one best match, estimate the relevant information of the pathological image, and generate the query result.

7. An electronic device, characterized in that, Comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the pathological image retrieval method according to any one of claims 1-3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to be used for implementing the pathological image retrieval method according to any one of claims 1-3.

Images