Method and apparatus for multiscale analysis of puncture pathology images

Abstract

This disclosure provides a multi-scale analysis method and apparatus for biopsy pathology images. The method includes: acquiring biopsy pathology images of a target patient and preprocessing the biopsy pathology images to obtain tissue-level image patches and cell-level image patches; extracting and aggregating features from the tissue-level image patches to obtain tissue-level features, and determining tissue-level prediction probabilities based on the tissue-level features; parsing cell nucleus instances from the cell-level image patches to obtain cell sets, and generating tumor infiltration regions based on tumor cells in the cell sets; constructing a cell map within the tumor infiltration regions, extracting image patch features of corresponding image patches based on the cell map, and clustering the features of each image patch to obtain multiple functional region clusters; constructing a hypergraph based on the functional region clusters, and aggregating node features in the hypergraph using a reinforcement learning-driven hypergraph neural network to obtain cell-level prediction probabilities; and obtaining prediction results based on the tissue-level prediction probabilities and cell-level prediction probabilities.

Description

Technical Field

[0001] This disclosure relates to the field of medical imaging technology, specifically to a multi-scale analysis method and apparatus for puncture pathology images. Background Technology

[0002] Biopsy slide images contain rich information about tumors and their microenvironment, making them an important data source for pathological image analysis. However, existing computational pathology methods still have the following shortcomings when using biopsy images for state analysis. First, most methods only model at a single tissue or cellular scale, making it difficult to simultaneously characterize the overall tissue morphology and local cellular microenvironment. This results in insufficient exploration of the spatial relationships between tumor cells and other cells in the microenvironment, leading to inadequate utilization of predictive information. Second, traditional multi-instance learning methods typically perform global averaging aggregation of all image blocks within the slide. This causes the discriminative signals carried by key areas of tumor infiltration to be easily diluted by large areas of normal tissue or non-critical areas, making it difficult to highlight the local areas most valuable for state discrimination. Furthermore, this uniform aggregation process is inconsistent with the actual slide reading process of pathologists who first skim and then focus on important areas, thus limiting the accuracy and generalization ability of the model in state analysis tasks. Summary of the Invention

[0003] (a) Technical problems to be solved

[0004] In view of the above problems, this disclosure provides a multi-scale analysis method and apparatus for puncture pathology images, so as to at least solve the technical problems of insufficient multi-scale information fusion, easy dilution of signals in key areas, and insufficient evaluation accuracy caused by inconsistency between model decision-making mechanism and expert diagnostic logic in the prior art.

[0005] (II) Technical Solution

[0006] This disclosure provides a multi-scale analysis method for biopsy pathology images, comprising: acquiring biopsy pathology images of a target patient and preprocessing the biopsy pathology images to obtain tissue-level image patches and cell-level image patches; extracting and aggregating features from the tissue-level image patches to obtain tissue-level features, and determining tissue-level prediction probabilities based on the tissue-level features; parsing cell nucleus instances from the cell-level image patches to obtain cell sets, and generating tumor infiltration regions based on tumor cells in the cell sets; constructing a cell map within the tumor infiltration regions, extracting image patch features of corresponding image patches based on the cell map, and clustering the features of each image patch to obtain multiple functional region clusters; constructing a hypergraph based on the functional region clusters, and aggregating node features in the hypergraph using a reinforcement learning-driven hypergraph neural network to obtain cell-level prediction probabilities; and obtaining prediction results based on the tissue-level prediction probabilities and cell-level prediction probabilities.

[0007] According to embodiments of this disclosure, preprocessing of a biopsy pathology image to obtain tissue-level image blocks and cell-level image blocks includes: performing foreground tissue segmentation and color regularization on the biopsy pathology image to obtain a standardized pathology image; and slicing the standardized pathology image into blocks at a first magnification and a second magnification to obtain tissue-level image blocks and cell-level image blocks, wherein the second magnification is greater than the first magnification.

[0008] According to embodiments of this disclosure, feature extraction and aggregation are performed on tissue-level image blocks to obtain tissue-level features, and tissue-level prediction probabilities are determined based on the tissue-level features. This includes: extracting features from tissue-level image blocks using a pre-trained pathological baseline model to obtain local features of each tissue-level image block; aggregating the local features of all tissue-level image blocks in the puncture pathology image using a pre-trained multi-instance learning model to obtain tissue-level features; and performing linear mapping and normalization processing on the tissue-level features to obtain tissue-level prediction probabilities.

[0009] According to embodiments of this disclosure, cell nucleus instance parsing is performed on a cell-level image block to obtain a cell set, including: detecting cell nuclei in the cell-level image block and determining the location information of each cell nucleus; performing contour segmentation on each detected cell nucleus to obtain the morphological information of each cell nucleus; performing category identification on each detected cell nucleus to determine the category information of each cell nucleus; and obtaining the cell set based on the location information, morphological information, and category information.

[0010] According to embodiments of this disclosure, generating a tumor infiltration region based on tumor cells in a cell set includes: screening tumor cells from the cell set; estimating the nuclear density based on the center coordinates of the tumor cells to generate a tumor cell density field; binarizing the tumor cell density field to determine high-density regions and extracting the outer contour of the high-density regions to obtain the tumor infiltration region.

[0011] According to embodiments of this disclosure, a cell map is constructed within the tumor infiltration area, and image patch features of corresponding image blocks are extracted based on the cell map. This includes: using preset category cells contained in each image block within the tumor infiltration area as nodes, and using the spatial proximity relationship between preset category cells as edges to construct a cell map; extracting multi-dimensional features from the cell map, and stitching the multi-dimensional features together to form the image patch features of the image block.

[0012] According to embodiments of this disclosure, constructing a hypergraph based on functional region clusters includes: selecting image blocks from each functional region cluster as hypergraph nodes; for each functional region cluster, constructing intra-cluster hyperedges that simultaneously connect all hypergraph nodes within the cluster, and constructing cross-cluster hyperedges that simultaneously connect hypergraph nodes between all functional region clusters, thereby forming a hypergraph.

[0013] According to embodiments of this disclosure, a reinforcement learning-driven hypergraph neural network is used to aggregate node features in a hypergraph to obtain cell-level prediction probabilities. This includes: sampling hypergraph nodes in multiple rounds using a reinforcement learning sampler, wherein the reinforcement learning sampler determines the sampling strategy for the current round based on the state of the hypergraph nodes output by the hypergraph neural network in the previous round; in each round, the hypergraph neural network is used to aggregate the node features of the currently sampled hypergraph nodes to obtain round-level representations; the round-level representations from each round are accumulated across rounds to obtain an accumulated result; and the cell-level prediction probabilities are determined based on the accumulated result.

[0014] According to embodiments of this disclosure, the method further includes: in each round, determining the predicted probability of the current round based on the round-level representation output by the hypergraph neural network, and extracting the target probability value corresponding to the true category from the predicted probability; calculating the increment between the target probability value extracted in the current round and the target probability value extracted in the previous round as a reward signal; and adjusting the sampling strategy of the reinforcement learning sampler based on the reward signal.

[0015] This disclosure also provides a multi-scale analysis device for biopsy pathology images, comprising: a preprocessing module for acquiring biopsy pathology images of a target patient and preprocessing the biopsy pathology images to obtain tissue-level image blocks and cell-level image blocks; a first prediction module for extracting and aggregating features from the tissue-level image blocks to obtain tissue-level features and determining tissue-level prediction probabilities based on the tissue-level features; a localization module for parsing cell nucleus instances from the cell-level image blocks to obtain cell sets and generating tumor infiltration regions based on tumor cells in the cell sets; a clustering module for constructing a cell map within the tumor infiltration regions, extracting image block features of corresponding image blocks based on the cell map, and clustering the features of each image block to obtain multiple functional region clusters; a second prediction module for constructing a hypergraph based on the functional region clusters, aggregating node features in the hypergraph using a reinforcement learning-driven hypergraph neural network to obtain cell-level prediction probabilities; and a fusion module for obtaining prediction results based on the tissue-level prediction probabilities and cell-level prediction probabilities.

[0016] (III) Beneficial Effects

[0017] The multi-scale analysis method and apparatus for puncture pathology images disclosed herein have at least the following technical advantages:

[0018] (1) Simultaneously, multi-scale analysis is performed based on tissue-level image patches and cell-level image patches, which overcomes the shortcomings of existing methods that only model from a single scale and are difficult to simultaneously characterize tissue morphology and cell microenvironment, thus improving the stability and accuracy of prediction.

[0019] (2) By limiting cell-level analysis to the tumor infiltration area and using the infiltration area automatically generated based on tumor cell density as a spatial prior, the system explored the spatial relationships between preset cell types (such as tumors, lymphocytes and fibroblasts).

[0020] (3) A cell map was constructed in the tumor infiltration area and a hypermap was constructed between image blocks to explicitly characterize the spatial topological relationship of the tumor microenvironment at both the local and global levels.

[0021] (4) A multi-round hypergraph neural network driven by a reinforcement learning sampler is introduced to simulate the image analysis logic of first overviewing and then focusing on key areas. The sampling strategy for each functional area cluster is dynamically adjusted between multiple rounds and complementary information is accumulated, which effectively alleviates the problem that the key discrimination signal is diluted by a large area of ​​irrelevant tissue due to the uniform aggregation of traditional multi-instance learning. Attached Figure Description

[0022] To gain a more complete understanding of this disclosure and its advantages, reference will now be made to the following description taken in conjunction with the accompanying drawings, wherein:

[0023] Figure 1 A flowchart illustrating the multi-scale analysis method for puncture pathology images provided in embodiments of this disclosure is shown in the schematic diagram.

[0024] Figure 2 This illustration schematically shows a cell-tile-hypergraph aggregation analysis diagram provided in an embodiment of the present disclosure;

[0025] Figure 3 This illustration schematically shows a structural diagram of the aggregation of reinforcement learning sampler and hypergraph neural network provided in an embodiment of the present disclosure;

[0026] Figure 4 This schematic diagram illustrates the structural block diagram of the multi-scale analysis device for puncture pathology images provided in an embodiment of the present disclosure;

[0027] Figure 5 A block diagram schematically illustrates an electronic device suitable for implementing a multi-scale analysis method for puncture pathology images, as provided in embodiments of this disclosure. Detailed Implementation

[0028] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0029] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0031] The accompanying drawings show some block diagrams and / or flowcharts. It should be understood that some blocks or combinations thereof in the block diagrams and / or flowcharts can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing system, so that when executed by the processor, these instructions can create a system for implementing the functions / operations described in these block diagrams and / or flowcharts.

[0032] like Figure 1 As shown, the flowchart of the multi-scale analysis method for puncture pathology images includes S1~S6.

[0033] In operation S1, the puncture pathology images of the target patient are acquired and preprocessed to obtain tissue-level image blocks and cell-level image blocks.

[0034] It should be noted that the puncture pathology images obtained by the method described in this disclosure are pre-acquired and stored static image data, and do not include any steps involving puncture or other interventional procedures on a living organism. The output of the method in this disclosure is a multi-scale analysis index calculated based on the image data, and does not include any steps for diagnosis, treatment, or prognosis based on this index. Any use of this index for clinical diagnosis, treatment decisions, or prognostic assessment is not within the scope of protection claimed in this disclosure.

[0035] Target patients can be patients with specific types of tumors (such as those scheduled for neoadjuvant therapy), such as breast cancer patients. The biopsy images can come from multiple data sources or pathology image storage systems, specifically digital pathology images of preoperative biopsy sections, such as whole-section scans of hematoxylin-eosin stained sections.

[0036] By preprocessing the original biopsy pathology images, the images can be divided into image patches of different scales. Among them, tissue-level image patches can be used for macroscopic tissue structure analysis, while cell-level image patches can be used for microscopic cell morphology analysis, thus providing a data foundation for subsequent multi-scale feature extraction.

[0037] In operation S2, feature extraction and aggregation are performed on the tissue-level image patch to obtain tissue-level features, and the tissue-level prediction probability is determined based on the tissue-level features.

[0038] Tissue-level image patches can be input into a pathological baseline model for feature extraction. The pathological baseline model can employ an open-source, large-scale pathological language-visual model. This model can map tissue-level image patches to a high-dimensional feature space and output tissue-level feature vectors with rich semantic information, which are used to characterize the pathological histological morphological information of the region corresponding to the image patch.

[0039] Since different image patches in biopsy pathology images contribute differently to state analysis, a multi-instance learning approach can be used to aggregate tissue-level features. For example, a multi-instance learning model can be used to weighted aggregate features from all tissue-level image patches of the same patient, outputting a tissue-level prediction probability. This probability can be used to characterize the likelihood of tumor state analysis based on tissue structure assessment.

[0040] In operation S3, cell nucleus instances are parsed from cell-level image blocks to obtain cell sets, and tumor infiltration areas are generated based on tumor cells in the cell sets.

[0041] Cell-level image patches can be input into a cell nucleus instance segmentation and classification network. For example, a cell nucleus instance segmentation and classification model can be used to accurately locate and classify each cell nucleus in the image patch, resulting in a cell set N. This cell set can contain the location coordinates, morphological features, and cell type labels (such as tumor cells, immune cells, mesenchymal cells, etc.) of various cell nuclei.

[0042] After obtaining the cell set, a tumor infiltration zone can be generated based on the spatial distribution density of tumor cells within it. The tumor infiltration zone refers to the main area where tumor cells invade and proliferate. It is the core functional region in the tumor microenvironment where intercellular interactions are most active, and it can provide spatial constraints for subsequent cell map construction and feature extraction.

[0043] In operation S4, a cell map is constructed within the tumor infiltration area. Based on the cell map, image patch features of corresponding image patches are extracted, and the features of each image patch are clustered to obtain multiple functional region clusters.

[0044] Within the tumor infiltration zone, a spatial cell map can be constructed for each image patch. Specifically, the k-nearest neighbor algorithm can be used, treating each cell nucleus as a node in the graph, and connecting edges based on the spatial distance between cell nuclei to construct a k-nearest neighbor spatial cell map. Then, cell features (such as cell nucleus morphology features, texture features, etc.) of each node can be extracted, and the features of neighboring nodes can be aggregated to the central node using graph neural networks or similar methods to obtain the image patch features of each image patch.

[0045] Furthermore, cluster analysis (such as K-means clustering) can be performed on all image patch features to group image patches with similar cellular microenvironment features into the same category, resulting in a slice-level functional region cluster set Cs. Each functional region cluster can represent a specific tumor microenvironment functional region, such as the tumor proliferation zone, immune infiltration zone, matrix reaction zone, etc.

[0046] In operation S5, a hypergraph is constructed based on functional region clusters, and a reinforcement learning-driven hypergraph neural network is used to aggregate the node features in the hypergraph to obtain cell-level prediction probabilities.

[0047] Based on functional region clusters, a hypergraph structure can be constructed. Each node in the hypergraph represents an image patch sampled from a functional region cluster, and each hyperedge can connect multiple nodes to represent higher-order relationships between multiple image patches. Nodes within the same functional region cluster are connected via intra-cluster hyperedges, while nodes from different functional region clusters are connected via cross-cluster hyperedges. Using a reinforcement learning-driven hypergraph neural network, node features in the hypergraph can be iteratively aggregated to learn complex interaction patterns between different functional region clusters, ultimately outputting a cell-level prediction probability. This probability can be used to characterize the likelihood of tumor state analysis based on cellular microenvironment assessment.

[0048] In operation S6, the prediction results are obtained based on the tissue-level prediction probability and the cell-level prediction probability.

[0049] Fusing tissue-level and cell-level predicted probabilities yields a comprehensive predicted probability of the overall response state. The fusion method can employ weighted averaging, product fusion, or an adaptive fusion strategy based on meta-learning. The final output prediction result can comprehensively assess the pathological characteristics of the tumor.

[0050] Understandably, by simultaneously extracting tissue-level macroscopic structural features and cellular-level microscopic morphological features, multi-scale information in pathological images can be effectively mined, improving the accuracy of state analysis. Furthermore, by constructing cell maps within the tumor infiltration zone and hypergraphs between image patches, the spatial topological relationships of the tumor microenvironment at both the local and global levels are explicitly characterized. In addition, by introducing a multi-round hypergraph neural network driven by a reinforcement learning sampler, the image analysis logic of first surveying and then focusing on key areas is simulated, effectively alleviating the problem of key discriminative signals being diluted by large areas of irrelevant tissue due to uniform aggregation in traditional multi-instance learning.

[0051] Based on the above embodiments, in this embodiment, the puncture pathology image is preprocessed to obtain tissue-level image blocks and cell-level image blocks, including: performing foreground tissue segmentation and color regularization processing on the puncture pathology image to obtain a standardized pathology image; and slicing the standardized pathology image into blocks at a first magnification and a second magnification to obtain tissue-level image blocks and cell-level image blocks, wherein the second magnification is greater than the first magnification.

[0052] First, the input biopsy pathology image can be downsampled to a preset low magnification (e.g., 10x magnification) and segmented into image blocks of a preset size (e.g., 512×512), which are then fed into a foreground segmentation network. This foreground segmentation network outputs the probability that each pixel belongs to the tissue foreground. After threshold binarization (e.g., a 0.5 threshold), morphological closing operations, and small-area connected component removal, a polygonal region covering the entire tissue foreground can be obtained. Through foreground tissue segmentation, the background region in the pathology image can be removed, retaining only the effective tissue region, reducing the computational load of subsequent processing.

[0053] Then, color regularization can be applied to images falling within the foreground region. Specifically, using a high-quality hematoxylin-eosin stained slide selected by experts as a reference template, the hematoxylin and eosin staining distributions of the pathological image are aligned to this template, resulting in a standardized pathological image with consistent color, reducing the interference of cross-center staining batches on subsequent deep learning models.

[0054] Finally, based on the standardized pathological images, segmentation can be performed at a first magnification and a second magnification, where the second magnification is greater than the first. For example, tissue-level image blocks of 224×224 pixels can be segmented at 20x resolution. (224×224) was used to capture macroscopic structural information of tumor tissue; cell-level image patches of 1024×1024 pixels were segmented at 40x resolution. (1024×1024), used to capture the microscopic morphological details of the cell nucleus.

[0055] Understandably, removing invalid background areas through foreground tissue segmentation, eliminating batch-to-batch staining differences through color regularization, and obtaining image patches of different resolutions through multi-scale slicing can effectively improve the quality and consistency of pathological images, thereby enhancing the accuracy and robustness of subsequent multi-scale analysis.

[0056] Based on the above embodiments, in this embodiment, feature extraction and aggregation are performed on tissue-level image blocks to obtain tissue-level features, and tissue-level prediction probabilities are determined based on the tissue-level features. This includes: extracting features from tissue-level image blocks using a pre-trained pathological model to obtain local features of each tissue-level image block; aggregating the local features of all tissue-level image blocks in the puncture pathology image using a pre-trained multi-instance learning model to obtain tissue-level features; and performing linear mapping and normalization on the tissue-level features to obtain tissue-level prediction probabilities.

[0057] For the preprocessed tissue-level image patches (e.g., 224×224 pixel images at 20x resolution), they can first be input into a pre-trained pathological model for feature extraction to obtain the corresponding local feature vectors. Then, the local features of all tissue-level image patches within the same biopsy pathological image can be used as input. The importance weights of each image patch are adaptively learned through the attention mechanism within the multi-instance learning model, and the local features are weighted and aggregated accordingly to obtain the tissue-level features of the entire slice. These tissue-level features can comprehensively represent the overall structural information of the tumor tissue.

[0058] After obtaining the organizational-level features, they can be input into the classification head for linear mapping and normalization. Specifically, the organizational-level features can be passed sequentially through a linear layer and a Softmax activation function to output the organizational-level prediction probability p. tissue (s).

[0059] Understandably, by extracting local features from each image patch using a pathological baseline model and then aggregating features from the entire slice using a multi-instance learning model, the problem of uneven contribution from different regions in puncture pathology images can be effectively addressed, thereby improving the accuracy and robustness of tissue-level prediction.

[0060] Based on the above embodiments, in this embodiment, cell nucleus instance parsing is performed on cell-level image blocks to obtain a cell set, including: detecting cell nuclei in cell-level image blocks to determine the location information of each cell nucleus; performing contour segmentation on each detected cell nucleus to obtain the morphological information of each cell nucleus; performing category identification on each detected cell nucleus to determine the category information of each cell nucleus; and obtaining a cell set based on the location information, morphological information, and category information.

[0061] The preprocessed cell-level image patches can be input into a cell nucleus instance segmentation and classification network for further processing. This network can employ a deep learning architecture to detect, classify, and segment cell nuclei, resulting in a complete cell slice set N.

[0062] First, cell nuclei can be detected in cell-level image patches to determine the location information of each nucleus. Specifically, the backbone of the network can use a computer vision backbone network as the feature extraction backbone network, combined with a feature pyramid network as the feature fusion module. Candidate boxes are generated through a region proposal network, and then the presence and location coordinates of cell nuclei in each candidate box are predicted, outputting the localization box information of each cell nucleus.

[0063] Secondly, contour segmentation can be performed on each detected cell nucleus to obtain its morphological information. Specifically, a three-level cascaded detection mechanism can be used to perform refined instance mask prediction on each candidate box in sequence, and finally output the pixel-level instance mask of each cell nucleus to characterize the contour boundary and morphological features of the cell nucleus.

[0064] Then, each detected cell nucleus can be categorized to determine its type. Categories can include different types such as tumor cells (T), lymphocytes (I), and fibroblasts (C). For example, categories corresponding to tumor cells, lymphocytes, and fibroblasts can be retained from the original classifications, while other categories can be discarded as needed.

[0065] Finally, based on the acquired location, morphology, and category information, the center coordinates, category label, and pixel-level instance mask of each preserved cell nucleus can be recorded to form a whole-slice cell set. For example, the cell set can be represented as... ,in, The coordinates of the center of the nth cell nucleus, category , This is a pixel-level instance mask.

[0066] Understandably, by using the analysis process of cell nucleus detection, contour segmentation, and category recognition, the precise location, morphological contour, and cell type information of each cell nucleus within the entire slice can be obtained, thus providing basic data for the subsequent generation of tumor infiltration areas and the construction of cell maps.

[0067] Based on the above embodiments, in this embodiment, generating a tumor infiltration zone based on tumor cells in a cell set includes: screening tumor cells from the cell set; estimating the nuclear density based on the center coordinates of the tumor cells to generate a tumor cell density field; binarizing the tumor cell density field to determine high-density regions and extracting the outer contour of the high-density regions to obtain the tumor infiltration zone.

[0068] By selecting all cells of the tumor category from the cell set N and recording their center coordinates, the tumor cell center set can be obtained. Based on the set of tumor cell centers, a tumor cell density field can be generated using a kernel density estimation method. Specifically, a Gaussian kernel is used as the kernel function for density estimation, resulting in the tumor cell density field. The calculation formula is:

[0069]

[0070] Where h is the density estimation radius, Let h be the center coordinates of the nth tumor cell, and h be the density estimation radius (bandwidth parameter). These represent the spatial coordinates within the density field. Through kernel density estimation, discrete tumor cell centers can be transformed into a continuous density distribution field; regions with higher density values ​​represent denser clusters of tumor cells.

[0071] Based on preset threshold It can affect the density field of tumor cells. Binarization is performed, marking regions with density values ​​above a threshold as high-density regions and regions below the threshold as background. Then, the outer contour of the high-density regions can be extracted to obtain the tumor infiltration area. This infiltrative area represents the core region where tumor cells are densely distributed and the adjacent stroma.

[0072] In some exemplary embodiments, after binarization but before outer contour extraction, a morphological closing operation (dilation followed by erosion) can be performed on the binarization result to fill the small pores inside the high-density region and connect adjacent high-density regions, thereby obtaining a more continuous tumor infiltration area. Furthermore, connected regions with excessively small areas can be removed to prevent noisy regions from being misidentified as infiltration areas.

[0073] by As a spatial mask, it can be used for the original set of cell-level image patches. With cell collection The image patches and cells falling within the infiltration zone were selected and retained, resulting in separate sets of image patches within the infiltration zone. With cell collection .

[0074] Understandably, locating tumor infiltration areas by using nuclear density estimation methods based on the spatial density of tumor cells can reduce interference from background areas and improve computational efficiency and model accuracy.

[0075] Based on the above embodiments, in this embodiment, a cell map is constructed within the tumor infiltration area, and image patch features of corresponding image blocks are extracted based on the cell map, including: using the preset category cells contained in each image block within the tumor infiltration area as nodes, and using the spatial proximity relationship between the preset category cells as edges to construct a cell map; extracting multi-dimensional features on the cell map, and stitching the multi-dimensional features into the image patch features of the image block.

[0076] For each image patch m within the tumor infiltration area, cells of a predetermined category (e.g., tumor cells, lymphocytes, fibroblasts) falling within that patch are used as nodes. A k-nearest neighbor spatial graph can be constructed based on the spatial proximity relationships between cells. On this cell graph, three types of multi-dimensional features can be extracted: morphological features, spatial network features, and spatial interaction features, respectively, as image patch features.

[0077] In some exemplary embodiments, attribute vectors can be extracted for each cell, including morphological features (such as area, perimeter, major axis length, minor axis length, eccentricity, solidity, and roundness), intensity features (such as mean and standard deviation of staining intensity), and texture features (such as gray-level co-occurrence matrix texture features). Statistical summarization of these attribute vectors within the image patch allows for the calculation of sixth-order statistics for each attribute (including maximum, minimum, mean, standard deviation, skewness, and kurtosis), forming morphological features characterizing the heterogeneity in the number and morphology of the three cell types. Simultaneously, multiple graph theory measures, including but not limited to degree, clustering coefficient, proximity, and degree centrality, can be calculated on the constructed k-nearest neighbor spatial graph. Sixth-order statistics can also be summed for these graph measures to form spatial network analysis features characterizing the local connectivity topology of cells. Furthermore, a spatial interaction analysis toolkit can be invoked to calculate multi-scale spatial interaction features, including but not limited to multi-scale Ripley K-function, modularity, global information entropy, local information entropy, and infiltration score, to characterize the multi-scale spatial co-distribution patterns of different cell types within the image patch.

[0078] By concatenating the feature vectors of the three types of image patches (morphological features, spatial network features, and spatial interaction features) according to coordinates, we can obtain the image patch features of the image patch, i.e., the image patch feature matrix. .

[0079] After obtaining the image patch features of all image patches, cluster analysis (such as K-means clustering) can be performed on the features of all image patches within the same patient slice to obtain several functional region clusters. For example, the number of clusters can be set to eight to obtain a set of functional region clusters. Each cluster corresponds to a type of tumor microenvironment field of view, such as the tumor parenchyma, tumor-stromal junction, and lymphatic infiltration zone. Let z m Let m be the cluster label of image patch m. .

[0080] Understandably, by extracting multi-dimensional features from cell maps, it is possible to comprehensively characterize the cellular composition, spatial topology, and co-distribution patterns of the tumor microenvironment from multiple dimensions. Simultaneously, by combining cluster analysis, image patches with similar microenvironmental characteristics are grouped into the same functional region cluster, thus providing a structured functional region division basis for subsequent hypermap construction and cell-level prediction.

[0081] Based on the above embodiments, in this embodiment, the construction of a hypergraph based on functional area clusters includes: selecting image blocks from each functional area cluster as hypergraph nodes; for each functional area cluster, constructing intra-cluster hyperedges that simultaneously connect all hypergraph nodes within the cluster, and constructing cross-cluster hyperedges that simultaneously connect hypergraph nodes between all functional area clusters, thereby forming a hypergraph.

[0082] Figure 2 The illustration shows a schematic diagram of cell-tile-hypergraph aggregation analysis provided in an embodiment of the present disclosure.

[0083] like Figure 2 As shown, during the construction process, image patches can first be sampled from each functional region cluster as candidate nodes for hypergraph construction using a sampler. Sampling strategies can employ random sampling, center sampling, or feature-representation-based sampling methods to ensure that the selected nodes in each cluster represent the typical microenvironment features of that cluster. For each functional region cluster... An intra-cluster hyperedge can be constructed that connects all sampled hypergraph nodes within a cluster. Since image patches within the same functional region cluster share similar tumor microenvironment characteristics (e.g., all are tumor parenchyma or lymphatic infiltration zones), connecting them via intra-cluster hyperedges captures semantically shared information within the same functional region. A cross-cluster hyperedge can be constructed that connects hypergraph nodes from different clusters across all functional region clusters. The cross-cluster hyperedge is designed because different functional region clusters (e.g., between tumor parenchyma and tumor-stromal junction) exhibit synergistic interactions that jointly influence tumor biological behavior; therefore, connecting them via cross-cluster hyperedges captures higher-order synergistic relationships between different functional regions. These two types of hyperedges (intra-cluster and cross-cluster hyperedges) together constitute the hypergraph association matrix H. In the hypergraph association matrix, rows correspond to hypergraph nodes, and columns correspond to hyperedges. If a node belongs to a hyperedge, the corresponding position in the matrix is ​​set to 1; otherwise, it is 0.

[0084] Using the above construction method, the image patch feature matrix For the set of nodes, with the set of functional area clusters C STo provide a priori information for grouping, a two-layer hypergraph structure can be constructed. This hypergraph structure can capture both the semantic consistency within the same functional region (through intra-cluster hyperedges) and the collaborative interaction relationships between different functional regions (through cross-cluster hyperedges), thus providing a structured graph topology foundation for subsequent hypergraph neural network aggregation.

[0085] It is understandable that by constructing a two-layer hypergraph that includes intra-cluster hyperedges and cross-cluster hyperedges, it is possible to simultaneously characterize the semantic sharing relationships within the same functional region as well as the higher-order collaborative relationships between different functional regions, thereby more comprehensively representing the complex interaction patterns between multiple functional regions in the tumor microenvironment.

[0086] Based on the above embodiments, in this embodiment, a reinforcement learning-driven hypergraph neural network is used to aggregate node features in the hypergraph to obtain cell-level prediction probabilities. This includes: sampling hypergraph nodes in multiple rounds using a reinforcement learning sampler, wherein the reinforcement learning sampler determines the sampling strategy for the current round based on the state of the hypergraph nodes output by the hypergraph neural network in the previous round; in each round, the node features of the currently sampled hypergraph nodes are aggregated using the hypergraph neural network to obtain round-level representations; the round-level representations of each round are accumulated across rounds to obtain an accumulated result; and the cell-level prediction probabilities are determined based on the accumulated result.

[0087] Figure 3 The schematic diagram illustrates a structure based on the aggregation of a reinforcement learning sampler and a hypergraph neural network provided in an embodiment of this disclosure.

[0088] like Figure 3 As shown, a reinforcement learning-driven hypergraph neural network aggregation mechanism can achieve accurate output of cell-level prediction probabilities through multiple rounds of dynamic sampling and information accumulation. Specifically, an actor-critic sampler based on proximal policy optimization (PPO) can be used as the reinforcement learning sampler. In the first round (initial round), random actions can be used as a cold start, randomly sampling image patches from each functional region cluster as hypergraph nodes. From the second to the Tth round, the sampler can generate action vectors based on the state of the hypergraph nodes output by the hypergraph neural network in the previous round. The sampling seeds are used as the sampling seeds for each of the K functional region clusters, and image patches are extracted from each cluster as hypergraph nodes for this round. In each round, the node features of the currently sampled hypergraph nodes are aggregated; that is, a two-layer hypergraph convolution operation can be performed using the hypergraph normalized convolution operator. The formula for calculating hypergraph convolution is:

[0089]

[0090] in, Here, H is the hypergraph normalized convolution operator, H is the hypergraph incidence matrix, W is the hyperedge weight matrix, and D is the hypergraph normalized convolution operator. v and D c These are the node degree matrix and the hyperedge degree matrix, respectively. Let be the learnable parameters of the l-th layer, and σ be the activation function. Through two layers of hypergraph convolution, node features can be aggregated within clusters and propagated between clusters, thereby achieving a full modeling of the hypergraph structure information. Furthermore, global average pooling can be performed on the output node representations to obtain round-level representations, i.e., the slice-level vectors of this round. .

[0091] Representing each round at the round level The data is sequentially fed into a recurrent neural network for accumulation across iterations. For example, a gated recurrent unit (GRU) can be used for accumulation, with the following state update formula:

[0092]

[0093] in, Let t be the hidden state in round t. Through the sequential processing of GRU, information from multiple rounds is accumulated sequentially, enabling the network to synthesize information sampled from different rounds to make a final decision.

[0094] Extracting the hidden states of the GRU in round T (the last round), and mapping them sequentially through a linear classifier head and a softmax activation function, yields cell-level prediction probabilities. .

[0095] In embodiments of this disclosure, the method further includes: in each round, determining the predicted probability of the current round based on the round-level representation output by the hypergraph neural network, and extracting the target probability value corresponding to the true category from the predicted probability; calculating the increment between the target probability value extracted in the current round and the target probability value extracted in the previous round as a reward signal; and adjusting the sampling strategy of the reinforcement learning sampler based on the reward signal.

[0096] In each round, the slice-level vector obtained in this round can be... The classification head is mapped to the prediction probability for this round. This allows us to extract the target probability value corresponding to the true category. The increment between the target probability value in the current round and the target probability value in the previous round is calculated as the reward signal.

[0097]

[0098] Among them, y sThis represents the true state label of patient s. The reward signal reflects the improvement in prediction confidence in this round of sampling compared to the previous round. Using this reward signal as feedback, the reinforcement learning sampler can update its policy by using a pruning objective optimized for proximal strategies. This allows the sampler to dynamically adjust its sampling strategy across multiple rounds, enabling it to gradually focus on the functional areas most beneficial for state discrimination.

[0099] Understandably, by using a reinforcement learning sampler to drive a hypergraph neural network to dynamically focus on key functional areas in multiple rounds, and combining it with a recurrent neural network to accumulate information across rounds, it is possible to adaptively mine the most discriminative tumor microenvironment functional regions for state discrimination, thereby improving the accuracy and robustness of cell-level prediction.

[0100] In obtaining the organizational-level prediction probability p tissue With cell-level prediction probability p cell Then, the two can be fused to obtain the final prediction result. For example, p tissue With p cell By concatenating the input logistic regression fusion unit, the fusion probability p(s) is obtained:

[0101]

[0102] Where σ is the Sigmoid activation function, w tissue and w cell Let be the learnable fusion weights, and b be the bias term. The decision threshold can be determined on the training set using the Yoden exponent. :

[0103]

[0104] Finally, the prediction results can be output. The prediction results can be used to characterize whether a target patient has specific pathological features.

[0105] Figure 4 The schematic diagram illustrates the structural block diagram of the multi-scale analysis device for puncture pathology images provided in the embodiments of this disclosure.

[0106] like Figure 4 As shown, the multi-scale analysis device 400 for puncture pathology images in this embodiment includes a preprocessing module 401, a first prediction module 402, a localization module 403, a clustering module 404, a second prediction module 405, and a fusion module 406.

[0107] According to some embodiments of this disclosure, the multi-scale analysis device 400 for puncture pathology images can be used to implement a reference... Figures 1-3 The method for multi-scale analysis of puncture pathology images according to embodiments of the present disclosure is described.

[0108] The preprocessing module 401 is used to acquire the puncture pathology images of the target patient and preprocess the puncture pathology images to obtain tissue-level image blocks and cell-level image blocks.

[0109] The first prediction module 402 is used to extract and aggregate features from tissue-level image blocks to obtain tissue-level features, and determine the tissue-level prediction probability based on the tissue-level features.

[0110] The positioning module 403 is used to parse cell nucleus instances from cell-level image blocks to obtain cell sets, and to generate tumor infiltration areas based on tumor cells in the cell sets.

[0111] The clustering module 404 is used to construct a cell map within the tumor infiltration area, extract image patch features of corresponding image patches based on the cell map, and cluster the features of each image patch to obtain multiple functional region clusters.

[0112] The second prediction module 405 is used to construct a hypergraph based on functional region clusters, and to aggregate the node features in the hypergraph using a reinforcement learning-driven hypergraph neural network to obtain cell-level prediction probabilities.

[0113] The fusion module 406 is used to obtain prediction results based on tissue-level prediction probabilities and cell-level prediction probabilities.

[0114] It is understood that the preprocessing module 401, the first prediction module 402, the localization module 403, the clustering module 404, the second prediction module 405, and the fusion module 406 can be implemented in one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the preprocessing module 401, the first prediction module 402, the localization module 403, the clustering module 404, the second prediction module 405, and the fusion module 406 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or can be implemented in hardware or firmware in any other reasonable manner of integrating or packaging circuitry, or in a suitable combination of software, hardware, and firmware implementations. Alternatively, at least one of the preprocessing module 401, the first prediction module 402, the localization module 403, the clustering module 404, the second prediction module 405, and the fusion module 406 can be implemented at least partially as a computer program module, which can perform the functions of the corresponding module when the program is run by a computer.

[0115] It should be understood that the multi-scale analysis device for puncture pathology images in the embodiments of this disclosure corresponds to the multi-scale analysis method for puncture pathology images in the embodiments of this disclosure, and their specific implementation details are the same, which will not be repeated here.

[0116] Figure 5 A block diagram schematically illustrates an electronic device suitable for implementing a multi-scale analysis method for puncture pathology images according to embodiments of the present disclosure.

[0117] like Figure 5 As shown, an electronic device 500 according to an embodiment of the present disclosure includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory 502 or a program loaded from a storage portion 508 into a random access memory 503. The processor 501 may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a dedicated microprocessor. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different steps of the method flow according to an embodiment of the present disclosure.

[0118] Random access memory 503 stores various programs and data required for the operation of electronic device 500. Processor 501, read-only memory 502, and random access memory 503 are interconnected via bus 504. Processor 501 executes various steps of the method flow according to embodiments of the present disclosure by executing programs in read-only memory 502 and / or random access memory 503. It should be noted that programs may also be stored in one or more memories other than read-only memory 502 and random access memory 503. Processor 501 may also execute various steps of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0119] According to embodiments of this disclosure, the electronic device 500 may further include an input / output interface 505, which is also connected to a bus 504. The electronic device 500 may also include one or more of the following components connected to the input / output interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube, liquid crystal display, etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card, such as a local area network card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0120] Embodiments of this disclosure also provide a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0121] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In embodiments of this disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include the read-only memory 502 described above, and / or random access memory 503, and / or one or more memories other than read-only memory 502 and random access memory 503.

[0122] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0123] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0124] In embodiments of this disclosure, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0125] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. The program code can execute entirely on a user computing device, partially on a user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0126] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0127] Those skilled in the art will understand that the features described in the various embodiments of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

Claims

1. A multi-scale analysis method for puncture pathology images, characterized in that, include: Acquire biopsy pathology images of the target patient and preprocess the biopsy pathology images to obtain tissue-level image blocks and cell-level image blocks; The tissue-level image patch is subjected to feature extraction and aggregation to obtain tissue-level features, and the tissue-level prediction probability is determined based on the tissue-level features; Cell nucleus instance parsing is performed on the cell-level image block to obtain a cell set, and a tumor infiltration area is generated based on the tumor cells in the cell set; A cell map is constructed within the tumor infiltration area. Based on the cell map, image patch features of corresponding image patches are extracted, and the image patch features are clustered to obtain multiple functional region clusters. A hypergraph is constructed based on the functional region clusters, and the node features in the hypergraph are aggregated using a reinforcement learning-driven hypergraph neural network to obtain cell-level prediction probabilities. The prediction result is obtained based on the tissue-level prediction probability and the cell-level prediction probability.

2. The method according to claim 1, characterized in that, The preprocessing of the biopsy pathological images to obtain tissue-level image blocks and cell-level image blocks includes: The puncture pathology images are segmented into foreground tissue and color regularized to obtain standardized pathology images. The standardized pathological image is segmented at a first magnification and a second magnification to obtain the tissue-level image block and the cell-level image block, respectively, wherein the second magnification is greater than the first magnification.

3. The method according to claim 1, characterized in that, The step of extracting and aggregating features from the tissue-level image patch to obtain tissue-level features, and determining the tissue-level prediction probability based on the tissue-level features, includes: The tissue-level image blocks are feature extracted using a pre-trained pathological model to obtain the local features of each tissue-level image block; The local features of all tissue-level image blocks within the puncture pathology image are aggregated using a pre-trained multi-instance learning model to obtain the tissue-level features; The organizational-level features are linearly mapped and normalized to obtain the organizational-level predicted probabilities.

4. The method according to claim 1, characterized in that, The cell-level image patch is parsed to obtain a cell set, including: The cell nuclei in the cell-level image block are detected to determine the location information of each cell nucleus; Each detected cell nucleus is segmented by contour to obtain the morphological information of each cell nucleus; Each detected cell nucleus is classified to determine its category information; The cell set is obtained based on the location information, morphological information, and category information.

5. The method according to claim 1, characterized in that, The generation of a tumor infiltration zone based on tumor cells in the cell set includes: The tumor cells were selected from the cell collection; Based on the center coordinates of the tumor cells, the nuclear density is estimated to generate a tumor cell density field. The tumor cell density field is binarized to identify high-density regions, and the outer contour of the high-density regions is extracted to obtain the tumor infiltration area.

6. The method according to claim 1, characterized in that, The process of constructing a cell map within the tumor infiltration area and extracting image patch features of corresponding image patches based on the cell map includes: A cell graph is constructed by using the cells of a predetermined category contained in each image block within the tumor infiltration area as nodes and the spatial proximity relationships between the cells of the predetermined category as edges. Multi-dimensional features are extracted from the cell map and then stitched together to form the image patch features of the image patch.

7. The method according to claim 1, characterized in that, The construction of the hypergraph based on the functional area clusters includes: Image blocks are selected from each of the aforementioned functional area clusters as hypergraph nodes; For each functional area cluster, an intra-cluster hyperedge is constructed that connects all hypergraph nodes within the cluster, and a cross-cluster hyperedge is constructed that connects all hypergraph nodes between all functional area clusters, thus forming the hypergraph.

8. The method according to claim 7, characterized in that, The aggregation of node features in the hypergraph using a reinforcement learning-driven hypergraph neural network to obtain cell-level prediction probabilities includes: The hypergraph nodes are sampled in multiple rounds by a reinforcement learning sampler, wherein the reinforcement learning sampler determines the sampling strategy for the current round based on the state of the hypergraph nodes output by the hypergraph neural network in the previous round. In each round, the node features of the currently sampled hypergraph nodes are aggregated using the hypergraph neural network to obtain round-level representations; The round-level representations of each round are accumulated across rounds to obtain the cumulative result; Based on the accumulated results, the cell-level prediction probability is determined.

9. The method according to claim 8, characterized in that, The method further includes: In each round, based on the round-level representation output by the hypergraph neural network, the prediction probability of this round is determined, and the target probability value corresponding to the true category is extracted from the prediction probability. The increment between the target probability value extracted in this round and the target probability value extracted in the previous round is calculated and used as a reward signal; Based on the reward signal, the sampling strategy of the reinforcement learning sampler is adjusted.

10. A multi-scale analysis device for puncture pathology images, characterized in that, include: The preprocessing module is used to acquire puncture pathology images of the target patient and preprocess the puncture pathology images to obtain tissue-level image blocks and cell-level image blocks. The first prediction module is used to extract and aggregate features from the tissue-level image patch to obtain tissue-level features, and determine the tissue-level prediction probability based on the tissue-level features. The localization module is used to parse cell nucleus instances from the cell-level image block to obtain a cell set, and to generate a tumor infiltration area based on the tumor cells in the cell set; A clustering module is used to construct a cell map within the tumor infiltration area, extract image patch features of corresponding image patches based on the cell map, and cluster the image patch features to obtain multiple functional region clusters; The second prediction module is used to construct a hypergraph based on the functional area clusters, and to aggregate the node features in the hypergraph using a reinforcement learning-driven hypergraph neural network to obtain cell-level prediction probabilities. The fusion module is used to obtain the prediction result based on the tissue-level prediction probability and the cell-level prediction probability.

Images