Construction method, retrieval method and device of digital pathological section storage retrieval library

Abstract

The invention provides a construction method of a digital pathological section storage retrieval library and a retrieval method and device, and the method comprises the following steps: obtaining a pathological section image, and extracting the microscopic morphological features, the texture periodic features and the pathological semantic features of the pathological section image; constructing a first-level retrieval library, a second-level retrieval library and a third-level retrieval library on the basis of the micromorphological features, the texture periodic features and the pathological semantic features, obtaining retrieval conditions, and performing feature extraction on the retrieval conditions to obtain retrieval features; and retrieving in the first-level retrieval library, the second-level retrieval library and the third-level retrieval library layer by layer based on the retrieval features to obtain a final retrieval result. According to the scheme, multiple types of features are extracted from the pathological section image to construct the retrieval libraries of different levels, and retrieval is performed in the retrieval libraries step by step in sequence according to the retrieval conditions, so that the pathological section image meeting the requirements can be accurately and efficiently retrieved, and powerful support is provided for related work such as pathological diagnosis.

Description

Technical Field

[0001] The present application relates to the field of image retrieval, and in particular to a method for constructing a digital pathology slice storage and retrieval library, a retrieval method and a device. Background Art

[0002] In the field of modern medicine, digital pathology slide technology has become an important tool for pathological diagnosis. With the widespread use of digital scanners, pathology slides can be scanned with high resolution and stored as digital images. However, the current digital pathology slide management system usually relies on the hospital information system (HIS) to associate digital slides with patient information. Once the system fails to connect to the hospital information system, the matching between slides and patients cannot be achieved, which will result in the inability to effectively retrieve and manage pathology slides.

[0003] The inability to associate pathology sections with patients will make it difficult for pathologists or researchers to find specific pathology sections, increase their workload, and may even delay diagnosis and treatment. Therefore, how to quickly retrieve digital pathology sections independently of the hospital information system is an urgent problem that needs to be solved. Summary of the invention

[0004] The embodiments of the present application provide a method for constructing a digital pathology section storage and retrieval library, a retrieval method and a device, which extract multiple types of features from pathology section images and construct a digital pathology section storage and retrieval library composed of multiple retrieval libraries at different levels, and then search the retrieval libraries at each level in turn according to the retrieval conditions, which helps to accurately and efficiently retrieve pathology section images that meet the requirements, and provide strong support for pathology diagnosis and other related work.

[0005] In a first aspect, an embodiment of the present application provides a method for constructing a digital pathology slide storage and retrieval library, the method comprising: Acquire at least one pathological slice image, and perform feature extraction on each of the pathological slice images to obtain microscopic morphological features, texture periodic features, and pathological semantic features; The microscopic morphological features, texture periodicity features and pathological semantic features of each pathological section image are converted into binary fingerprints respectively, and a hash fingerprint library is constructed as the primary retrieval library; a spatial decision tree is constructed as the secondary retrieval library according to the feature discreteness of the microscopic morphological features, texture periodicity features and pathological semantic features of each pathological section image; the microscopic morphological features, texture periodicity features and pathological semantic features of each pathological section image are semantically associated with pathological diagnosis keywords to obtain a semantic association network as the tertiary retrieval library, among which the primary retrieval library, the secondary retrieval library and the tertiary retrieval library constitute the digital pathological section storage retrieval library.

[0006] In a second aspect, an embodiment of the present application provides a method for searching a digital pathology slide storage and retrieval library, comprising the following steps: Acquiring search conditions, wherein the search conditions at least include pathological description text or pathological reference images; Based on the retrieval conditions, the corresponding pathological slice images are retrieved from the digital pathological slice storage retrieval library constructed in the first aspect, wherein the retrieval conditions are subjected to feature extraction to obtain retrieval features, the retrieval features are converted into binary fingerprints, a search is performed in a primary retrieval library based on the binary fingerprint to obtain a first retrieval range, and then within the first retrieval range, a search is performed in a secondary retrieval library based on the discrete degree of the retrieval features to obtain a second retrieval range, and finally, the retrieval features are converted into retrieval semantic features, and within the second retrieval range, a search is performed in a tertiary retrieval library based on the retrieval semantic features to obtain a third retrieval range, and pathological slice images within the third retrieval range are obtained as final retrieval results.

[0007] In a third aspect, an embodiment of the present application provides a device for constructing a digital pathology slide retrieval library, comprising: An acquisition module, used for acquiring at least one pathological slice image, and performing feature extraction on each of the pathological slice images to obtain microscopic morphological features, texture periodicity features and pathological semantic features; The retrieval library construction module is used to convert the microscopic morphological features, texture periodicity features and pathological semantic features of each pathological section image into binary fingerprints and construct a hash fingerprint library as the first-level retrieval library; construct a spatial decision tree as the second-level retrieval library according to the feature discreteness of the microscopic morphological features, texture periodicity features and pathological semantic features of each pathological section image; and semantically associate the microscopic morphological features, texture periodicity features and pathological semantic features of each pathological section image with pathological diagnosis keywords to obtain a semantic association network as the third-level retrieval library.

[0008] In a fourth aspect, an embodiment of the present application provides a retrieval device for a digital pathology slide retrieval library, comprising: An acquisition module, used for acquiring a search condition, wherein the search condition at least includes a pathology description text or a pathology reference image; A retrieval module retrieves corresponding pathological slice images from the digital pathological slice storage retrieval library constructed in the first aspect based on the retrieval conditions, wherein the retrieval conditions are subjected to feature extraction to obtain retrieval features, the retrieval features are converted into binary fingerprints, a search is performed in a primary retrieval library based on the binary fingerprint to obtain a first retrieval range, a search is performed in a secondary retrieval library based on the discrete degree of the retrieval features within the first retrieval range to obtain a second retrieval range, and finally the retrieval features are converted into retrieval semantic features, and a search is performed in a tertiary retrieval library based on the retrieval semantic features within the second retrieval range to obtain a third retrieval range, and pathological slice images within the third retrieval range are obtained as final retrieval results.

[0009] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute a method for constructing or retrieving a digital pathology section retrieval library.

[0010] In a sixth aspect, an embodiment of the present application provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute a process, and the process includes a method for constructing or retrieving a digital pathology section retrieval library.

[0011] The main contributions and innovations of the present invention are as follows: The embodiment of the present application extracts microscopic morphology, texture periodicity and pathological semantic features from the pathological slice image, constructs a three-level retrieval library of hash fingerprint library, spatial decision tree and semantic association network, and screens step by step during retrieval. For example, a large number of non-similar slices are excluded in the first-level retrieval library, the second-level retrieval library screens the spatial neighbor slices, and the third-level retrieval library is optimized and sorted according to the semantic features, which is progressive layer by layer to improve the accuracy and efficiency of retrieval; the embodiment of the present application considers rotation robustness when calculating the LBP value, and comprehensively analyzes the periodicity characteristics of textures at different scales; enhances the weight of the first-layer filter of the convolutional neural network for the purple system to better capture local details; constructs a feature relationship map, integrates spatial distance and feature similarity, and more carefully depicts the internal connection of image features to improve retrieval accuracy; the embodiment of the present application establishes a spatial coordinate mapping relationship for the same patient's continuous layer thickness digital pathological slices in the second-level and third-level retrieval libraries, breaking the limitations of traditional single-slice two-dimensional information analysis, supporting three-dimensional retrieval, and facilitating the observation and analysis of pathological features from a three-dimensional perspective, and understanding the development and distribution of lesions.

[0012] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a flow chart of a method for retrieving digital pathological sections according to an embodiment of the present application; Figure 2 is a structural block diagram of a digital pathology section retrieval device according to an embodiment of the present application; Figure 3 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0014] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0015] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0016] Embodiment 1 The embodiment of the present application provides a method for constructing a digital pathology section storage and retrieval library, which extracts multiple types of features from pathology section images and constructs multiple retrieval libraries of different levels, and then searches in the retrieval libraries of each level in turn according to the retrieval conditions, which helps to accurately and efficiently retrieve pathology section images that meet the requirements, and provides strong support for pathology diagnosis and other related work. Specifically, reference Figure 1 , the method comprising: Acquire at least one pathological slice image, and perform feature extraction on each of the pathological slice images to obtain microscopic morphological features, texture periodic features, and pathological semantic features; The microscopic morphological features, texture periodicity features and pathological semantic features of each pathological section image are converted into binary fingerprints respectively, and a hash fingerprint library is constructed as the primary retrieval library; a spatial decision tree is constructed as the secondary retrieval library according to the feature discreteness of the microscopic morphological features, texture periodicity features and pathological semantic features of each pathological section image; the microscopic morphological features, texture periodicity features and pathological semantic features of each pathological section image are semantically associated with pathological diagnosis keywords to obtain a semantic association network as the tertiary retrieval library, among which the primary retrieval library, the secondary retrieval library and the tertiary retrieval library constitute the digital pathological section storage retrieval library.

[0017] In some specific embodiments, in order to ensure that the characteristic details on the pathological slice can be accurately extracted, the pathological slice image used in this solution is a high-resolution slice image.

[0018] In some specific embodiments, before feature extraction is performed on the pathological slice image, each pathological slice image is first subjected to binarization processing, multimodal denoising, and contrast enhancement operations in sequence to improve the detail expression in the pathological slice image.

[0019] Specifically, dynamic area segmentation is adopted during the binarization processing, that is, the binarization threshold is automatically adjusted according to the cell density in different areas of the pathological section image, a high-sensitivity mode is used for the cell-dense area to retain the weakly stained area, and a high-specificity model is used for the cell-sparse area to suppress background noise. In the process of binarization processing, the H&E staining characteristics are analyzed to establish a staining intensity-tissue type mapping table to automatically filter out overexposed areas such as reflections on the edge of the slide.

[0020] Specifically, spatial-frequency domain collaborative denoising is adopted in multimodal denoising. That is, the noise points are first located through a high-frequency vibration detection algorithm, and then verification filtering is performed in combination with cell morphological features to retain the true cell structure. In addition, adaptive structural elements are designed according to the typical cell nucleus size (5μm-15μm), and an expansion-then-erosion operation is performed to eliminate isolated noise points.

[0021] Specifically, when contrast enhancement is performed, the image is divided into dynamically variable grids, that is, fine grids are divided for cell-dense areas, and coarse grids are divided for background areas. Each grid is independently histogram equalized, and color deconvolution technology is used to separately enhance the hematoxylin (cell nucleus) and eosin (cytoplasm) staining channels.

[0022] In some specific embodiments, the pathological section image is input into a pre-trained deep learning model to obtain a cell nucleus probability map, and then each cell nucleus is segmented in the cell nucleus probability map based on a watershed algorithm, and the morphological features of each cell nucleus are extracted to obtain microscopic morphological features.

[0023] Specifically, a large number of digital pathology slice images with precise cell nucleus annotations are used as training sets to train the deep learning model to obtain a trained deep learning model. The trained deep learning model predicts the position of the cell nucleus in the digital pathology slice to obtain a cell nucleus probability map. The cell nucleus probability map is in pixels, and the value of each pixel represents the probability that the position belongs to the cell nucleus. The higher the probability, the greater the possibility that the position is the cell nucleus.

[0024] Specifically, the watershed algorithm sets a probability threshold first, takes the pixels in the cell nucleus probability map with probability values ​​higher than the probability threshold as potential markers, and screens and optimizes the potential markers according to the density or sparseness of the potential markers to obtain each cell nucleus in the cell nucleus probability map. It is precisely because of the particularity of the cell structure that the watershed algorithm can better segment each cell nucleus.

[0025] Specifically, the morphological feature extraction is performed by calculating the multi-dimensional microscopic morphological data of each cell nucleus, for example, the ellipticity of the cell nucleus is obtained by calculating the ratio of the major axis and the minor axis length of the cell nucleus contour; the fractal dimension is calculated by the box dimension method, etc.; the cell nucleus region is divided into multiple sub-regions, and the color value in each sub-region is counted to calculate the staining uniformity of the cell nucleus, etc.

[0026] In some specific embodiments, a three-dimensional space is determined based on a pathological slice image, a neighborhood of each pixel in the three-dimensional space is determined with each pixel in the three-dimensional space as the center, and the LBP value of each pixel is calculated. An LBP value matrix is ​​formed by the LBP value of each pixel in the three-dimensional space and the LBP value matrix is ​​encoded to obtain a texture feature code corresponding to the pathological slice image. The texture feature code is then filtered through filters of different frequencies to obtain a multi-scale texture. Finally, feature extraction is performed on the multi-scale texture to obtain a texture periodic feature.

[0027] Specifically, the neighborhood of a pixel point is a 3×3 area, that is, a 3×3 nine-square grid centered on each pixel point is the neighborhood of the central pixel point.

[0028] Furthermore, in the process of calculating the LBP value of each pixel in the three-dimensional space, the grayscale value of the central pixel is used as the grayscale threshold, and the grayscale values ​​of other pixels in the neighborhood are compared with the grayscale threshold. If the grayscale values ​​of other pixels in the field are greater than or equal to the grayscale threshold, they are assigned a value of 1; if they are less than the grayscale threshold, they are assigned a value of 0. All possible sorting of the grayscale values ​​of all pixels in the neighborhood except the central pixel is performed clockwise or counterclockwise to obtain multiple binary sequences. Each binary sequence is converted to decimal and compared, and the smallest value is selected as the LBP value of the corresponding pixel.

[0029] In practical applications, the direction of pathological slice images is not fixed, so this scheme will fully consider rotation robustness when calculating the LBP value, that is, to ensure that the characteristics of each pixel value can be guaranteed to be stable in any direction. Specifically, when calculating the LBP value, this scheme will obtain all possible grayscale values ​​of all pixels except the center pixel in the neighborhood in clockwise or counterclockwise order, such as 11001100, 10011001, 00110011..., and then convert these binary sequences into decimals for comparison, and select the smallest value as the LBP value.

[0030] Furthermore, the filter used in this scheme is an adjustable Gabor filter of different frequencies. This scheme filters the texture feature code through filters of different scales to obtain texture periodicity features at different scales. The texture periodicity features at different scales in the pathological section image can be more comprehensively analyzed, such as the periodicity of cell arrangement, the texture regularity of tissue structure, etc.

[0031] In some specific embodiments, a convolutional neural network is used to extract semantic features from the pathological section image to obtain high-dimensional semantic features, and then a pre-trained teacher-student network is used to perform feature compression on the high-dimensional semantic features to obtain pathological semantic features, wherein the weight of the first-layer filter in the convolutional neural network for the purple system is enhanced.

[0032] Specifically, since digital pathology sections are generally stained with hematoxylin-eosin (H&E), enhancing the weight of the first-layer filter for purple in the convolutional neural network can better capture local details in high-resolution pathology sections, thereby better capturing semantic features.

[0033] Specifically, the teacher-student network architecture consists of a teacher network and a student network. The teacher network is usually a larger, higher-performance network with high complexity and expressiveness, and can learn rich pathological image features. The student network is relatively small and simpler in structure. Its goal is to imitate the behavior of the teacher network while achieving lightweight features. A pre-trained large convolutional neural network is selected as the teacher network, such as a ResNet or VGG network pre-trained on a large-scale image dataset. Based on the structure of the teacher network, a simplified student network is designed to reduce the number of layers and parameters of the network, but maintain the basic structure and function of the network. In the student network, the number of channels of the convolutional layer can be reduced, and a smaller convolution kernel can be used. During the pre-training process of the teacher-student network, the teacher network forward propagates the input image to extract high-dimensional features. The student network also forward propagates the same input image to try to learn the feature representation of the teacher network. The difference between the output features of the student network and the teacher network is measured by the loss function. Commonly used loss functions include mean square error (MSE) loss or KL divergence loss. Taking the mean square error loss as an example, the sum of the squares of the differences between the output features of the student network and the output features of the teacher network in each channel and each spatial position is calculated, and then the loss values ​​of all samples are averaged. Through the back propagation algorithm, the weight parameters of the student network are adjusted according to the loss value, so that the output features of the student network gradually approach the output features of the teacher network. During the training process, some hyperparameters can also be set, such as the temperature parameter (used to adjust the calculation method of the loss function), to optimize the training effect.

[0034] In the application stage, the student network gradually learns feature representations similar to those of the teacher network, but due to its simple structure, the output feature dimension is low. At this time, the features output by the student network are compact codes containing pathological semantics after lightweight processing, and their size is usually 1 / 8 of the original high-dimensional features. In practical applications, new pathological images are input into the trained student network, and the compact codes output by the student network can be used for subsequent tasks, such as retrieval and classification of pathological sections. Since the compact code has a low dimension, it not only reduces the cost of data storage and transmission, but also improves computational efficiency, while retaining the key semantic information of pathological images, which can meet the needs of practical applications.

[0035] In some specific embodiments, in the process of converting micromorphological features, texture periodic features, and pathological semantic features into binary fingerprints, for the micromorphological features, the indicators of each dimension are first normalized, and then the content of each indicator is generated into a binary code to obtain the binary fingerprint of the micromorphological features.

[0036] For texture periodicity features, the texture periodicity features at different scales are first quantified, such as calculating the length and intensity of the texture period. The quantized texture features are discretized and divided into multiple intervals, and binary codes are generated according to the intervals where the feature values ​​are located to obtain the binary fingerprint of the texture periodicity features.

[0037] For pathological semantic features, since the compact encoding containing pathological semantics obtained by deep learning feature distillation has a low dimension itself, the pathological semantic features can be directly mapped into a binary string of fixed length through a hash function to obtain the binary fingerprint of the pathological semantic features.

[0038] Then, the binary fingerprint of the microscopic morphological features, the binary fingerprint of the texture periodic features, and the binary fingerprint of the pathological semantic features of each pathological slice image are merged to form a complete binary fingerprint of the corresponding pathological slice image and stored in the constructed hash fingerprint library.

[0039] In some specific embodiments, in the process of constructing a spatial decision tree based on the feature discreteness of the micromorphological features, texture periodic features, and pathological semantic features of each pathological section image, their discreteness in the data set is calculated respectively. For numerical micromorphological and texture periodic features, the discreteness can be measured by variance or standard deviation; for pathological semantic features (such as category labels, etc.), the discreteness is evaluated by information gain or Gini index, and then a spatial decision tree is constructed based on the discreteness of all features of each pathological section image.

[0040] In some specific embodiments, in the process of semantically associating the microscopic morphological features, texture periodic features, and pathological semantic features of each pathological slice image with pathological diagnostic keywords, for microscopic morphological features, such as uneven cell nucleus staining, which is often related to "cytopathic disease", the microscopic morphological features of uneven cell nucleus staining are associated with the diagnostic keyword "cytopathic disease".

[0041] For texture periodic features, if some specific texture periodic changes represent "tumor growth", the specific texture periodic changes are associated with the diagnostic keyword "tumor growth".

[0042] Since the pathological semantic features themselves contain semantic information, the corresponding pathological semantic features can be directly associated with the corresponding diagnostic keywords.

[0043] Finally, a semantic association network is constructed based on the degree of association between each feature in each pathological slice image and the diagnostic keyword.

[0044] In some specific embodiments, the primary retrieval library, the secondary retrieval library, and the tertiary retrieval library in this scheme can be dynamically constructed and used according to the complexity of the case. That is to say, if the case is not complicated, only any one retrieval library can be constructed for storage and retrieval. If the case is very complicated, then the three retrieval libraries are used for step-by-step retrieval.

[0045] In some specific embodiments, static weights are set in the secondary retrieval library, that is, high retrieval weights are set for features with high discreteness, and low retrieval weights are set for features with low discreteness; dynamic weights are set in the tertiary retrieval library, that is, the retrieval weights of pathological semantic features related to the retrieval semantic features are increased.

[0046] Exemplarily, this scheme uses a neural network to set static rights and dynamic weights. In the secondary retrieval library, the feature discreteness reflects the degree of variation of the feature in the data set. The higher the discreteness, the more important the feature is in distinguishing different samples. Based on this feature, a high retrieval weight is set for features with high discreteness, and a low retrieval weight is set for features with low discreteness to ensure the accuracy of the retrieval.

[0047] Since the third-level retrieval library stores semantic information, dynamic weights can be set directly based on the semantic information. For example, when a user searches for "metastasis", the system automatically increases the weight of the tumor edge infiltration feature.

[0048] In some specific embodiments, a spatial coordinate mapping relationship is established for digital pathological slices of continuous thickness of the same patient in the secondary retrieval library and the tertiary retrieval library.

[0049] Specifically, in the digital pathology section scenario, spatiotemporal feature binding refers to the automatic establishment of a spatial coordinate mapping relationship between continuous pathology sections through technical means, so as to support the retrieval of sections from a three-dimensional perspective. Traditional pathology section retrieval is mostly limited to the two-dimensional information analysis of a single section, while spatiotemporal feature binding breaks this limitation. In tumor pathology research, tumor cells are distributed three-dimensionally in tissues, and continuous sections can show the morphology of tumors at different levels. Spatiotemporal feature binding is to integrate these continuous sections to construct a three-dimensional information model, which makes it easier for researchers to observe and analyze pathological features from a three-dimensional perspective and have a more comprehensive understanding of the development and distribution of lesions.

[0050] Embodiment 2 A retrieval method for a digital pathology slide storage retrieval library comprises the following steps: Acquiring search conditions, wherein the search conditions at least include pathological description text or pathological reference images; Based on the retrieval conditions, the corresponding pathological slice images are retrieved in the digital pathological slice storage retrieval library constructed in the first embodiment, wherein the retrieval conditions are feature extracted to obtain retrieval features, the retrieval features are converted into binary fingerprints, and a search is performed in the primary retrieval library based on the binary fingerprint to obtain a first retrieval range, and then a search is performed in the secondary retrieval library based on the discrete degree of the retrieval features within the first retrieval range to obtain a second retrieval range, and finally the retrieval features are converted into retrieval semantic features, and a search is performed in the tertiary retrieval library based on the retrieval semantic features within the second retrieval range to obtain a third retrieval range, and the pathological slice images within the third retrieval range are obtained as the final retrieval results.

[0051] In some specific embodiments, when the retrieval condition is a pathological reference image, the feature extraction of the pathological reference image may be performed directly first, and then the binary fingerprint, discrete degree and semantic feature extraction may be performed.

[0052] In some specific embodiments, when the search condition is a pathology description text, a pre-trained natural language processing model is used to extract features of the pathology description text to obtain high-dimensional semantic features, and then a lightweight network is used to compress the high-dimensional semantic features to obtain a low-dimensional compact code, and finally the low-dimensional compact code is mapped to a binary fingerprint of a fixed length through a hash function. The binary fingerprint converted from the pathology description text is format-aligned with the binary fingerprint in the primary search library to ensure compatibility with the matching mechanism of the primary search library.

[0053] Furthermore, when the search condition is a pathology description text, if the pathology description text is a numerical feature, the variance or standard deviation of the pathology description text is used as the discrete degree of the search feature; if the pathology description text is a categorical feature, the classification discrimination is evaluated using information gain or Gini index as the discrete degree of the search feature; if the pathology description text includes both numerical features and categorical features, the variance or standard deviation of the numerical features and the classification discrimination of the categorical features are calculated respectively as the discrete degree of the search feature.

[0054] Furthermore, when the search condition is a pathology description text, the pre-trained natural language processing model is directly used to extract features of the pathology description text to obtain high-dimensional semantic features, and then the high-dimensional semantic features are used to search in the three-level search library.

[0055] In some specific embodiments, in the process of searching using retrieval features, a composite distance is constructed by constructing a joint score of the retrieval features and each pathological slice image in the primary retrieval library / secondary retrieval library / tertiary retrieval library in terms of spatial distribution similarity, color histogram matching, and texture correlation. The closer the composite distance, the more similar the retrieval feature is to the corresponding pathological slice image.

[0056] In some specific embodiments, the first search range is larger than the second search range, and the second search range is larger than the third search range. In this scheme, when the search feature is used to search in the primary search library, the first search range obtained is all pathological section images whose similarity with the search feature is greater than 60%; when the search feature is used to search in the secondary search library, all pathological section images in the first search range are sorted by similarity with the search feature, and the first 100 pathological section images are taken to form the second search range; when the search feature is used to search in the tertiary search library, all pathological section images in the second search range are sorted by similarity with the search feature, and the first 10 pathological section images are taken to form the third search range.

[0057] Specifically, this solution uses the primary retrieval library, the secondary retrieval library, and the tertiary retrieval library to perform a hierarchical progressive search. When searching in the primary retrieval library, a large number of non-similar digital pathology sections will be directly excluded. When searching in the secondary retrieval library, spatial neighbors will be screened to screen out digital pathology sections that are too far away in space. When searching in the tertiary retrieval library, pathology keywords are directly optimized and sorted based on semantic features, forming a progressive filtering, which improves the accuracy and efficiency of the search.

[0058] In some specific embodiments, histological prior knowledge is introduced to modify the search results.

[0059] Specifically, when searching for cancerous regions, the search results are corrected based on the spatial distribution law of the cancerous regions to delete digital pathological sections that do not conform to the spatial distribution law of the cancerous regions.

[0060] In some specific embodiments, the retrieval is performed in a structure graph matching manner during the retrieval process, in which the nodes represent feature points and the edge weights include spatial distance and feature similarity.

[0061] Specifically, in this scheme, the feature point represented by the node is the center of the cell nucleus, etc., and the edge weight = feature similarity × the inverse of the spatial distance.

[0062] Furthermore, abnormal connections in the initial matching are automatically corrected through multiple rounds of information transmission. Abnormal connections refer to matching pairs with spatial distance exceeding the threshold (such as >50μm) but similar features caused by algorithm errors.

[0063] Specifically, in traditional image matching, it is often just a simple comparison of the overall features or partial features of the image, which makes it difficult to fully capture the complex relationship between the features. The feature relationship graph is an innovative method that transforms the matching problem into a graph structure optimization problem. In this graph structure, various feature points such as microscopic morphological features and texture features extracted from pathological slice images are used as nodes. These feature points cover information such as the ellipticity of the cell nucleus and the periodicity of the texture. The edge weight combines two key factors: spatial distance and feature similarity. Spatial distance describes the proximity of different feature points in the image space. If two feature points are adjacent in the slice image, the spatial distance weight of the edge between them may be small; feature similarity reflects the similarity of the two feature points in feature attributes. For example, if the contour fractal dimensions of two cell nuclei are similar, the feature similarity weight of the edge between them is large. In this way, the image matching problem is transformed into an analysis of the relationship between nodes and edges in the graph structure, which can more carefully and comprehensively characterize the intrinsic connection between image features and provide richer information for subsequent matching optimization.

[0064] Specifically, iterative attention propagation automatically corrects these abnormal connections through multiple rounds of information transmission. In the first round of information transmission, each node will pass its own feature information to the adjacent nodes along the edge. The adjacent nodes update their feature representations based on the received information and the weight of the edge. In this process, the information transmission corresponding to the edge with a larger weight is more important, that is, it is more "concerned". For example, if the feature similarity weight of an edge is very high, then the feature information transmitted through this edge will have a greater impact on the receiving node. After the first round of information transmission, the feature representation of the node will be more accurate. Then, multiple rounds of information transmission such as the second and third rounds are carried out, and each round is updated based on the results of the previous round. With the continuous transmission and updating of information, abnormal connections will gradually be discovered and corrected. For example, in the initial matching, two unrelated feature points may be mistakenly connected due to noise or local feature similarity, but after multiple rounds of information transmission, the node will find that this connection is unreasonable based on the more comprehensive information transmitted by the surrounding nodes, thereby reducing the weight of this edge or disconnecting the connection, making the final matching result more accurate and reliable, and improving the accuracy of digital pathology section retrieval.

[0065] In some specific embodiments, after obtaining the final search results, each search result is accurately verified, and the verification method is: 1. Multi-scale registration verification Feature-guided registration: First, coarse registration is performed using key feature points, followed by pixel-level elastic deformation compensation, and then verification is performed by comparing the organelle distribution pattern (such as the number and arrangement of nucleoli) at 40x magnification.

[0066] 2. Visual verification system Difference heat map generation: Use gradient visualization technology to intuitively display the difference areas of two slices, and automatically generate a report containing indicators such as spatial overlap, feature matching, and staining consistency for verification.

[0067] Embodiment 3 Based on the same idea, refer to Figure 2 The present application also proposes a device for constructing a digital pathology slide storage and retrieval library, including: An acquisition module, used for acquiring at least one pathological slice image, and performing feature extraction on each of the pathological slice images to obtain microscopic morphological features, texture periodicity features and pathological semantic features; The retrieval library construction module is used to convert the microscopic morphological features, texture periodicity features and pathological semantic features of each pathological section image into binary fingerprints and construct a hash fingerprint library as the first-level retrieval library; construct a spatial decision tree as the second-level retrieval library according to the feature discreteness of the microscopic morphological features, texture periodicity features and pathological semantic features of each pathological section image; and semantically associate the microscopic morphological features, texture periodicity features and pathological semantic features of each pathological section image with pathological diagnosis keywords to obtain a semantic association network as the third-level retrieval library.

[0068] Embodiment 4 The present application also proposes a retrieval device for a digital pathology slide storage and retrieval library, comprising: An acquisition module, used for acquiring a search condition, wherein the search condition at least includes a pathology description text or a pathology reference image; A retrieval module retrieves corresponding pathological slice images from the digital pathological slice storage retrieval library constructed in the first embodiment based on the retrieval conditions, wherein the retrieval conditions are feature extracted to obtain retrieval features, the retrieval features are converted into binary fingerprints, a search is performed in a primary retrieval library based on the binary fingerprint to obtain a first retrieval range, and then a search is performed in a secondary retrieval library based on the discrete degree of the retrieval features within the first retrieval range to obtain a second retrieval range, and finally the retrieval features are converted into retrieval semantic features, and a search is performed in a tertiary retrieval library based on the retrieval semantic features within the second retrieval range to obtain a third retrieval range, and the pathological slice images within the third retrieval range are obtained as the final retrieval results.

[0069] Embodiment 3 This embodiment also provides an electronic device, referring to Figure 3 , comprises a memory 404 and a processor 402, wherein the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to execute the steps in any of the above method embodiments.

[0070] Specifically, the processor 402 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0071] Among them, the memory 404 may include a large capacity memory 404 for data or instructions. For example, but not limitation, the memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In appropriate cases, the memory 404 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 404 may be inside or outside the data processing device. In a specific embodiment, the memory 404 is a non-volatile memory. In a specific embodiment, the memory 404 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable Programmable Read-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable Programmable Read-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. In appropriate circumstances, the RAM may be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM may be a fast page mode dynamic random access memory 404 (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0072] The memory 404 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402 .

[0073] The processor 402 implements any one of the digital pathology section retrieval methods in the above embodiments by reading and executing the computer program instructions stored in the memory 404 .

[0074] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408 , wherein the transmission device 406 is connected to the processor 402 , and the input / output device 408 is connected to the processor 402 .

[0075] The transmission device 406 can be used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired or wireless network provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0076] The input / output device 408 is used to input or output information. In this embodiment, the input information may be search conditions, pathological slice images, etc., and the output information may be final search results, etc.

[0077] Optionally, in this embodiment, the processor 402 may be configured to perform the following steps through a computer program: Acquire at least one pathological slice image, and perform feature extraction on each of the pathological slice images to obtain microscopic morphological features, texture periodic features, and pathological semantic features; The microscopic morphological features, texture periodicity features and pathological semantic features of each pathological slice image are converted into binary fingerprints and a hash fingerprint library is constructed as a primary retrieval library; a spatial decision tree is constructed as a secondary retrieval library according to the feature discreteness of the microscopic morphological features, texture periodicity features and pathological semantic features of each pathological slice image; the microscopic morphological features, texture periodicity features and pathological semantic features of each pathological slice image are semantically associated with pathological diagnosis keywords to obtain a semantic association network as a tertiary retrieval library; Acquire search conditions, perform feature extraction on the search conditions to obtain search features, convert the search features into binary fingerprints and then perform a search in a primary search library to obtain a first search range, then perform a search in a secondary search library based on the discreteness of the search features within the first search range to obtain a second search range, finally convert the search features into search semantic features, perform a search in a tertiary search library based on the search semantic features within the second search range to obtain a third search range, and obtain pathological section images within the third search range as final search results.

[0078] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0079] In general, various embodiments may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0080] Embodiments of the present invention may be implemented by computer software that is executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and / or macros may be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. A computer program product may include one or more computer executable components configured to perform an embodiment when the program is run. One or more computer executable components may be at least one software code or a portion thereof. In addition, at this point, it should be noted that, for example, Figure 3 Any block of the logic flow in the program may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media such as memory chips or storage blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs, etc. Physical media are non-transitory media.

[0081] Those skilled in the art should understand that the technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0082] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for constructing a digital pathology slide storage and retrieval library, characterized in that: The following steps are involved: Acquire at least one pathological slice image, and perform feature extraction on each of the pathological slice images to obtain microscopic morphological features, texture periodic features, and pathological semantic features; The microscopic morphological features, texture periodicity features and pathological semantic features of each pathological section image are converted into binary fingerprints respectively, and a hash fingerprint library is constructed as the primary retrieval library; a spatial decision tree is constructed as the secondary retrieval library according to the feature discreteness of the microscopic morphological features, texture periodicity features and pathological semantic features of each pathological section image; the microscopic morphological features, texture periodicity features and pathological semantic features of each pathological section image are semantically associated with pathological diagnosis keywords to obtain a semantic association network as the tertiary retrieval library, among which the primary retrieval library, the secondary retrieval library and the tertiary retrieval library constitute the digital pathological section storage retrieval library.

2. The method for constructing a digital pathology slide storage and retrieval library according to claim 1, characterized in that: The method comprises the following steps: inputting the pathological section image into a pre-trained deep learning model to obtain a cell nucleus probability map, then segmenting each cell nucleus in the cell nucleus probability map based on a watershed algorithm, and extracting the morphological features of each cell nucleus to obtain microscopic morphological features.

3. The method for constructing a digital pathology slide storage and retrieval library according to claim 1, characterized in that: The method comprises the following steps: determining a three-dimensional space with a pathological slice image as a reference, determining a neighborhood of each pixel point with each pixel point in the three-dimensional space as the center, and calculating the LBP value of each pixel point; forming an LBP value matrix with the LBP value of each pixel point in the three-dimensional space and encoding the LBP value matrix to obtain a texture feature code corresponding to the pathological slice image; filtering the texture feature code through filters of different frequencies to obtain a multi-scale texture; and finally extracting features from the multi-scale texture to obtain a texture periodic feature.

4. The method for constructing a digital pathology slide storage and retrieval library according to claim 3, characterized in that: The method includes the following steps: in the process of calculating the LBP value of each pixel point in the three-dimensional space, the grayscale value of the central pixel point is used as the grayscale threshold, and the grayscale values ​​of other pixel points in the neighborhood are compared with the grayscale threshold; if the grayscale values ​​of other pixel points in the domain are greater than or equal to the grayscale threshold, they are assigned a value of 1; if they are less than the grayscale threshold, they are assigned a value of 0; all possible sorting of the grayscale values ​​of all pixel points in the neighborhood except the central pixel point is performed clockwise or counterclockwise to obtain multiple binary sequences; each binary sequence is converted into decimal and compared, and the smallest value is selected as the LBP value of the corresponding pixel point.

5. The method for constructing a digital pathology slide storage and retrieval library according to claim 1, characterized in that: The method comprises the following steps: using a convolutional neural network to extract semantic features from the pathological slice image to obtain high-dimensional semantic features, and then using a pre-trained teacher-student network to perform feature compression on the high-dimensional semantic features to obtain pathological semantic features, wherein the weight of the first-layer filter in the convolutional neural network for the purple system is enhanced.

6. A retrieval method for a digital pathology slide storage retrieval library, characterized in that: The following steps are involved: Acquiring search conditions, wherein the search conditions at least include pathological description text or pathological reference images; Based on the retrieval conditions, the corresponding pathological slice images are retrieved in the digital pathological slice storage retrieval library constructed in claim 1, wherein the retrieval conditions are subjected to feature extraction to obtain retrieval features, the retrieval features are converted into binary fingerprints, a search is performed in a primary retrieval library based on the binary fingerprint to obtain a first retrieval range, and then a search is performed in a secondary retrieval library based on the discrete degree of the retrieval features within the first retrieval range to obtain a second retrieval range, and finally the retrieval features are converted into retrieval semantic features, and a search is performed in a tertiary retrieval library based on the retrieval semantic features within the second retrieval range to obtain a third retrieval range, and the pathological slice images within the third retrieval range are obtained as the final retrieval results.

7. A method for retrieving a digital pathology slide storage and retrieval library according to claim 6, characterized in that: Static weights are set in the secondary retrieval library, that is, high retrieval weights are set for features with high discreteness, and low retrieval weights are set for features with low discreteness; dynamic weights are set in the tertiary retrieval library, that is, the retrieval weights of pathological semantic features related to the retrieval semantic features are increased.

8. A device for constructing a digital pathology slide storage and retrieval library, characterized in that: include: An acquisition module, used for acquiring at least one pathological slice image, and performing feature extraction on each of the pathological slice images to obtain microscopic morphological features, texture periodicity features and pathological semantic features; The retrieval library construction module is used to convert the microscopic morphological features, texture periodicity features and pathological semantic features of each pathological section image into binary fingerprints and construct a hash fingerprint library as the first-level retrieval library; construct a spatial decision tree as the second-level retrieval library according to the feature discreteness of the microscopic morphological features, texture periodicity features and pathological semantic features of each pathological section image; and semantically associate the microscopic morphological features, texture periodicity features and pathological semantic features of each pathological section image with pathological diagnosis keywords to obtain a semantic association network as the third-level retrieval library.

9. A retrieval device for a digital pathology slide storage and retrieval library, characterized in that: include: An acquisition module, used for acquiring a search condition, wherein the search condition at least includes a pathology description text or a pathology reference image; A retrieval module retrieves corresponding pathological slice images from the digital pathological slice storage retrieval library constructed in the first aspect based on the retrieval conditions, wherein the retrieval conditions are subjected to feature extraction to obtain retrieval features, the retrieval features are converted into binary fingerprints, a search is performed in a primary retrieval library based on the binary fingerprint to obtain a first retrieval range, a search is performed in a secondary retrieval library based on the discrete degree of the retrieval features within the first retrieval range to obtain a second retrieval range, and finally the retrieval features are converted into retrieval semantic features, and a search is performed in a tertiary retrieval library based on the retrieval semantic features within the second retrieval range to obtain a third retrieval range, and pathological slice images within the third retrieval range are obtained as final retrieval results.

10. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to run the computer program to execute a method for constructing a digital pathology section storage and retrieval library as described in any one of claims 1-5 or a method for retrieving a digital pathology section storage and retrieval library as described in any one of claims 6-7.

Images