Pathological image spatial feature extraction method and electronic equipment

Abstract

The invention relates to a pathological image spatial feature extraction method and an electronic device, and the method comprises the steps: respectively obtaining image block features and cell features from a region of interest, carrying out the spatial modeling of the image block features and the cell features, and respectively extracting tissue pathological features and cell pathological features based on a cell map and a tissue map; and fusing the two to obtain a final pathological feature. A cell map and a tissue map are respectively constructed from two perspectives of cells and tissues, the cell map properly contains distribution and interaction of the cells, the tissue map properly encodes a tissue microenvironment including tissue topology distribution which cannot be summarized by the cell map, and finally spatial information contained in the cell map and the tissue map is fused. Therefore, cell and tissue information in the pathological image is fully utilized, compared with a network which only uses a cell map or a tissue map, rich information in the pathological image can be better captured by simultaneously using the two types of maps, and the purpose of fully utilizing multi-scale space information in the pathological image can be achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing and applications, and particularly to a method for extracting spatial features of pathological images and an electronic device. Background Technique

[0002] Pathological image examination is the gold standard for tumor diagnosis and prognosis evaluation. Pathological images contain rich spatial interaction information of cells and tissues. The original pathological examination is realized by doctors observing pathological sections under a microscope. The emergence of high-resolution pathological section scanners has achieved high-quality digitization of pathological images, providing convenience for doctors to read images. The digitized tissue pathological sections are called Whole slide images (WSI). Although the emergence of WSI has facilitated doctors to read images, the process for doctors to extract useful information from the huge high-resolution information is still cumbersome. A good artificial intelligence-based pathological image feature extraction method can achieve good results in downstream tasks based on pathological images (tumor subtype classification, survival prediction), is expected to provide a more standardized diagnosis, and at the same time reduce the burden on doctors. CNN-based methods are difficult to capture the inherent complex spatial dependence relationships in pathological images, while graph neural network-based methods can often extract richer pathological image features because they can effectively capture the spatial topological information between tissues and cells.

[0003] Currently, among the pathological image feature extraction methods based on graph neural networks, PatchGCN constructs a graph based on the positions of image patches and uses a GCN with residual connections for pathological image feature extraction; the transformer, which has achieved success in various tasks in recent years, has also been combined with graph neural networks and used for pathological image feature extraction; PeiL et al. proposed an anchor box-based graph structure learning method for survival prediction, which constructs a graph based on an adaptive method to extract features applicable to the survival prediction task in pathological images. However, these methods only obtain information from one scale and do not fully utilize the information in pathological images.

[0004] Most of the pathological image feature extraction methods based on graph neural networks do not consider the spatial position perception of nodes, and the spatial information in pathological images is not fully utilized. Therefore, the rich cell and tissue interaction information in pathological images is not fully utilized, resulting in poor performance in downstream tasks (survival prediction, subtype classification). Summary of the Invention

[0005] In order to solve the defect that the multi-scale spatial information in pathological images is not fully utilized in the pathological image feature extraction method based on graph neural networks, the present invention proposes a method for extracting spatial features of pathological images.

[0006] The technical solution adopted by the present invention is a method for extracting spatial features of pathological images, including,

[0007] S100. Obtain a pathological image;

[0008] S200. Outline the region of interest and select the region of interest from the pathological image;

[0009] S310. Spatial modeling of tissue. Obtain the image patch features from the region of interest and construct a tissue map based on the image patch features for spatial modeling of the tissue;

[0010] S320. Spatial modeling of cells. Obtain the cell features from the region of interest and construct a cell map based on the cell features for spatial modeling of the cells;

[0011] S400. Extract pathological features. Extract tissue pathological features containing inter-tissue spatial information according to the tissue map, and extract cell pathological features containing inter-cell spatial information according to the cell map;

[0012] S500. Feature fusion. Fuse the tissue pathological features and the cell pathological features to obtain the final pathological features.

[0013] Preferably, in step S310, it includes using an image patch node compression scheme when constructing the tissue map.

[0014] Preferably, in step S320, it includes using a cell node compression scheme when constructing the cell map.

[0015] Preferably, in step S310, it includes:

[0016] S311. Calculate the Euclidean distance between the image patches where the image patch features are located, and the cosine similarity and / or mutual information between the image patch features;

[0017] S312. Generate super image patches according to the Euclidean distance and the cosine similarity and / or mutual information;

[0018] S313. Each super image patch serves as a separate node in the tissue map, and obtain the tissue map according to the super image patches for spatial modeling of the tissue.

[0019] Preferably, in step S320, it includes:

[0020] S321. Use the farthest point sampling method to sample the cell features to obtain the farthest point cell nuclei;

[0021] S322. Use the random sampling method to sample the cell features to obtain random cell nuclei;

[0022] S323. Use the farthest point cell nuclei and the random cell nuclei as separate nodes in the cell map, and obtain the cell map according to the farthest point cell nuclei and the random cell nuclei for spatial modeling of the cells.

[0023] Preferably, in step S312, it includes:

[0024] S3121. Exclude similar image patches according to cosine similarity to obtain super image patches;

[0025] S3122. Judge the distance values of adjacent super image patches according to Euclidean distance. When the Euclidean distance is less than the set threshold range, create an edge between the adjacent super image patches.

[0026] Preferably, in step S321, it includes:

[0027] S311. Randomly sample a cell nucleus and put it into the sampling point set;

[0028] S312. Calculate the Euclidean distance between all unsampled cell nuclei and the sampled cell nuclei, and put the cell nucleus with the largest distance into the sampling point set;

[0029] S313. Calculate the distance between each unsampled cell nucleus and all cell nuclei in the sampling point set, and retain the minimum distance. After traversing all unsampled cell nuclei, select the unsampled cell nucleus corresponding to the maximum value among the previously retained minimum distances and add it to the sampling point set;

[0030] S314. Repeat step S313 until the number of cell nuclei in the sampling point set meets the requirements, and then construct an edge using the K-nearest neighbor method based on the centroid position of the cell nuclei.

[0031] Preferably, in steps S310 and S320, it includes cutting the region of interest into image patches of a fixed size at a low magnification and extracting image patch features therefrom; performing cell segmentation on the region of interest at a high magnification and extracting cell features therefrom.

[0032] Preferably, in step S400, it includes inputting the tissue map and the cell map into a position-aware graph neural network respectively, using the input position-aware graph neural network to extract tissue pathological features and cell pathological features respectively. The position-aware graph neural network is implemented based on a Transformer encoder, and the Transformer encoder is composed of multiple Graph Transformer layers. Each Graph Transformer layer includes a multi-head attention module and a feed-forward neural network.

[0033] In order to solve the defect that multi-scale spatial information in a pathological image is not fully utilized in an electronic device for extracting pathological image features based on a graph neural network, the present invention proposes an electronic device.

[0034] The technical solution adopted by the present invention is an electronic device, including: a processor and a memory. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method as described above.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present application discloses a method for extracting spatial features of pathological images and an electronic device. Image patch features and cell features are respectively obtained from the region of interest, and spatial modeling is respectively performed on the two. After tissue pathological features and cell pathological features are respectively extracted based on the cell graph and the tissue graph, the two are fused to obtain the final pathological features. From the perspectives of cells and tissues, a cell graph and a tissue graph are respectively constructed. The cell graph appropriately includes the distribution and interaction of cells, and the tissue graph appropriately encodes the tissue microenvironment, including the tissue topological distribution that cannot be generalized by the cell graph. Finally, the spatial information contained in the cell graph and the tissue graph is fused. Thus, the cell and tissue information in the pathological image is fully utilized. Compared with the network that only uses the cell graph or the tissue graph, using both types of graphs can better capture the rich information in the pathological image.

[0037] Compared with the prior art, a method for extracting spatial features of pathological images and an electronic device disclosed in the present application can achieve the purpose of fully utilizing the multi-scale spatial information in the pathological image. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The present invention will be described in detail below with reference to the embodiments and the drawings, where:

[0039] Figure 1 FIG. shows a schematic flowchart of a method for extracting spatial features of pathological images according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0040] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the drawings. Examples of the embodiments are shown in the drawings, where the same or similar reference numerals represent the same or similar components or components with the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present invention and should not be construed as limiting the present invention.

[0041] Most of the pathological image feature extraction methods based on graph neural networks do not consider the spatial location perception of nodes, and the spatial information in pathological images is not fully utilized. Although some methods consider spatial location perception, they do not perform pathological feature fusion for pathological images at the tissue and cell scales. Therefore, the rich cell and tissue interaction information in pathological images is not fully utilized, resulting in poor performance in downstream tasks (survival prediction, subtype classification).

[0042] The present invention discloses a method for extracting spatial features of pathological images, including

[0043] S100. Obtain a pathological image;

[0044] S200. Draw a region of interest and select a region of interest from the pathological image;

[0045] S310. Spatial modeling of tissues. Obtain image patch features from the region of interest, and construct a tissue graph based on the image patch features for spatial modeling of tissues;

[0046] S320. Spatial modeling of cells. Obtain cell features from the region of interest, and construct a cell graph based on the cell features for spatial modeling of cells;

[0047] S400. Extract pathological features. Extract tissue pathological features containing inter-tissue spatial information according to the tissue graph, and extract cell pathological features containing inter-cell spatial information according to the cell graph;

[0048] S500. Feature fusion. Fuse the tissue pathological features and the cell pathological features to obtain the final pathological features.

[0049] In step S100 of obtaining a pathological image, a dataset to be processed is obtained. Digital pathological images are usually saved in a pyramid structure, which saves pathological images at different resolutions. This application does not limit the types of pathological images. In addition to whole-slide images, other section images, etc. can also obtain good feature extraction effects for such pathological images.

[0050] In step S200 of drawing a region of interest, a pathologist uses professional annotation software such as QuPath or other professional annotation software to draw a region of interest; in addition to manual drawing by a pathologist, it is also possible to automatically draw a region of interest through a trained learning model, avoiding the need for manual drawing and improving the operation efficiency of the method.

[0051] Perform data preprocessing on pathological images, construct a graph based on patch features for tissue spatial modeling, and construct a graph based on cell features for cell spatial modeling. Pathologists usually observe tissue information at a lower magnification and cell information at a higher magnification. After delineating the region of interest, different magnifications and / or regions of different sizes can be selected according to the resolution image quality and pathological target symptoms (such as diseases like tumors or nodules) for feature extraction, and targeted spatial modeling can be performed to obtain better feature extraction results.

[0052] Due to the high resolution of pathological images, graph neural networks with positional encoding often require a large amount of computational effort.

[0053] Therefore, in steps S310 and S320, a node compression scheme is adopted to compress the nodes of the cell graph and the patch graph input to the position-aware graph neural network, reducing the computational effort of the position-aware graph neural network, significantly reducing the number of nodes in the cell graph and the patch graph, preserving important spatial information and feature relationships, and maintaining the representativeness of the data and the integrity of the structure.

[0054] Step S300 is to perform spatial modeling on tissues and cells respectively, specifically including step S310 tissue spatial modeling and step S320 cell spatial modeling.

[0055] For step S310 tissue spatial modeling, the method for constructing a graph based on patch features to describe the spatial position relationship between tissues is as follows. At a low magnification (such as 10x), the selected region of interest is cut into patches of a fixed size (such as 512 pixels x 512 pixels). A pre-trained feature extractor (such as ResNet50, CTransPath, UNI, etc.) is used to extract the features of the patches. At the same time, the physical space coordinates of the upper left corner of the patch are saved. The construction of the graph based on the patches is specifically described as follows:

[0056] Node compression is performed by using the Euclidean distance between the physical positions of image patches and the cosine similarity between the image patch features. This method only compares the spatial correlation of image patches within a distance of two image patches (1024 pixels when the image patch size is 512 pixels) to maximize the compression of similar image patches. First, the cosine similarity of the features between image patches is calculated. When the cosine similarity between two image patches is higher than the threshold T, these two image patches are defined as similar image patches. The image patches are sorted according to the number of similar image patches for each image patch. Then, they are processed in sequence, starting from the image patch with the most similar image patches. The similar image patches belonging to this image patch are removed, and the remaining image patches are called super image patches. After generating the super image patches, each super image patch is regarded as a single node in the graph to be constructed. When creating the edges of the graph, only the spatial distance between the super image patches is considered. If the distance between the super image patches is less than the length of five image patch distances (2560 pixels when the image patch size is 512 pixels), an edge is created between the nodes. The node features of the super image patches are obtained by averaging all the image patch features integrated into the super image patches.

[0057] When calculating the cosine similarity of the features between image patches, the mutual information between the two feature vectors can be combined at the same time. When both the cosine similarity and the mutual information are higher than the threshold, they are defined as similar, that is, the cosine similarity is higher than the threshold of the cosine similarity, and the mutual information is higher than the threshold of the mutual information. Whether the image patches are similar is measured from two perspectives. The specific selection of the threshold needs to be determined through experiments, or it can be directly set to the median of all the calculated cosine similarities / mutual information.

[0058] When creating the edges of the graph, only considering the spatial distance between the image patches rather than the feature similarity between the graphs when creating the edges of the graph is to model the tissue spatial structure through edge connection. Compared with the traditional method of using the K-nearest neighbor method based on image patches to obtain graph data, constructing graph data based on the spatial distance of super graph patches can well represent the spatial relationship between dissimilar nodes while reducing the number of nodes, and such relationships happen to be the key points to be focused on in pathological diagnosis. The size of the spatial distance can be determined according to the specific number of the obtained super image patches and the preset calculation amount, or it can also be determined by empirical values, such as five image patch distances.

[0059] It should be noted that compared with input images in other fields, pathological images usually contain rich details and complex structures, which increases the difficulty of feature extraction. For the spatial modeling of tissues, for the special image features of tissues in pathological images, in order to prevent image patches with similar textures and edges from repeatedly appearing in the feature extraction calculation, it is necessary to first screen out most of the similar image patches and only select the super-image patch that can best represent the set of similar image patches. Calculating with this super-image patch can enable the position perception model to directly learn the spatial position relationship between dissimilar tissues in pathological images, which is the key point to be concerned about in pathological diagnosis. And optimizing the nodes by removing similarities is the most suitable operation at the tissue level, which can maintain the information of pathological images to the greatest extent and prevent data loss compared with methods such as random removal and central removal.

[0060] The spatial modeling of cells in step S320, the method for constructing a graph based on cell features used to describe the spatial position relationship between cells is as follows. The selected region is segmented into cells at a high magnification (such as 40x), and the cytoplasmic centroid coordinates are saved at the same time. The cell segmentation method can use a deep learning-based nucleus segmentation method. Then, the features of the nucleus are extracted, including morphological features and texture features. Similarly, to reduce the computational amount, as described in detail below, the farthest point sampling method is used to sample cells at a ratio of a. This method effectively alleviates the problem that nuclei in sparse regions are removed. In addition, to prevent overfitting, a part of the nuclei are randomly sampled at a ratio of b and are used together with the cells sampled by the farthest point sampling method for cell graph construction. The farthest point sampling method is divided into four steps. The first step is to randomly sample a point and put it into the initial point sampling set. The second step is to calculate the Euclidean distance between all unsampled points and the sampled points, and select the point with the largest distance and put it into the sampling set. The third step is to calculate the distance between each unsampled point and all points in the sampling set and retain the minimum distance. After traversing all unsampled points, select the unsampled point corresponding to the maximum value among the previously retained minimum distances and add it to the sampling set. The fourth step is to repeat the third step until the number of points in the sampling set meets the requirements. Then, the edges are constructed using the K-nearest neighbor method based on the cytoplasmic centroid positions.

[0061] Among them, for the spatial modeling of cells, a mixed method of the farthest point sampling method and random sampling is selected for node compression optimization. The purpose of selecting the farthest point sampling method is that at the cell level, when the nucleus is not selected, the obtained cell graph will lose the key functional information provided by the nucleus. Using the farthest point sampling method can effectively alleviate the problem that nuclei in sparse regions are removed. The farthest point sampling method can ensure that the nuclei at the distal end in the region where cell features are located are collected, which can improve the sample representativeness compared with the subsequent random sampling method.

[0062] Step S400 extracts pathological features. After that, the graph based on cell features is input into the position-aware graph neural network to extract pathological features containing intercellular spatial information; the graph based on patch features is input into the position-aware graph neural network to extract pathological features containing inter-tissue spatial information. Compared with traditional graph neural networks, the position-aware graph neural network can better capture the differences between the input graph structures, and thus can better simulate the spatial information interaction of tissue cells in pathological images.

[0063] For the position-aware graph neural network, one implementation scheme is as follows. This method is implemented based on the classical Transformer encoder and is called CGT. It consists of L Graph Transformer Layers (GTLs), and each GTL includes a multi-head attention module (MHA) and a feed-forward neural network (FFN). The calculation process of GTL is as follows:

[0064] h' (l) = MHA(LN(h (l-1) )) + h (l-1)

[0065] h (l) = FFN(LN(h' (l) )) + h' (e)

[0066] where LN is the layer normalization function. h (0) is the input of the first GTL (which can be cell graph node features or patch graph node features). h (e) represents the output of the e-th GTL.

[0067] Further detailed description is as follows. To capture the spatial position relationship between cells and tissues, topological connection features are added to the node features. First, a learnable connection vector e i (0) is initialized as shown in the following formula. This vector is encoded based on the degree of each node. Deg represents the function to calculate the node degree, and g represents an encoder, and v i represents the i-th node.

[0068]

[0069] The formula for adding connection features to node features is as follows:

[0070]

[0071] where λ is a penalty parameter used to balance the feature scales between h i and ; h i represents the feature of the i-th node, and fh A function that represents the addition of the original node features and connection features.

[0072] In addition, the distance between nodes is calculated for subsequent encoding into the attention calculation to further capture the spatial positional relationships between cells and between tissues. Let p i and p j represent the centroid positions of node i and node j respectively (for cells, it is the centroid position; for image patches, the physical coordinates of the upper left corner are used). The Euclidean distance between node i and node j is calculated as follows:

[0073] (v i , v j ) = ∥p i - p j ∥2

[0074] After completing the above basic information calculation, node information aggregation is performed. First, the connection vectors are aggregated After that, the aggregated connection vectors are added to the node features. The specific operation is as follows:

[0075]

[0076]

[0077] where f e is a function used to aggregate local connection vectors, and Ni is the set of neighbor nodes of node i. The set V is the set of nodes in the input graph. In the formula, V represents all other nodes in the set V except node i, and GTL (l) represents the l-th GTL.

[0078] Among them, in the MHA module of GTL, the self-attention Attention(h i , h j ) between node i and node j with position bias is calculated in the following way, specifically as follows:

[0079]

[0080] Attention(h i , h j ) = softmax(A ij )h j W V ,

[0081] W Q , W K , W V represent the weighted matrices of the query vector, key vector, and value vector, and softmax is the normalized exponential function.

[0082] Step S500 Feature Fusion. Finally, fuse the pathological features at the cellular level and the pathological features at the tissue level to obtain the finally extracted pathological image features. The available methods include fusing through simple splicing or fusing features through a cross-attention module.

[0083] Among them, it is preferred to use a cross-attention module for feature fusion, which can effectively integrate complementary information in multiple scales; alternatively, the method of direct splicing can also be considered for fusion, which is relatively simple and can reduce the computational complexity of the model.

[0084] In some embodiments, in step S310, it includes:

[0085] Use an image patch node compression scheme when constructing the tissue map.

[0086] In some embodiments, in step S320, it includes:

[0087] Use a cell node compression scheme when constructing the cell map.

[0088] In some embodiments, in step S310, it includes:

[0089] S311. Calculate the Euclidean distance between the image patches where the image patch features are located, and the cosine similarity and / or mutual information between the image patch features;

[0090] S312. Generate super image patches according to the Euclidean distance and the cosine similarity and / or mutual information;

[0091] S313. Each super image patch serves as a separate node in the tissue map, and obtain the tissue map according to the super image patches for spatial modeling of the tissue.

[0092] In some embodiments, in step S320, it includes:

[0093] S321. Use the farthest point sampling method to sample the cell features to obtain the farthest point cell nuclei;

[0094] S322. Use the random sampling method to sample the cell features to obtain random cell nuclei;

[0095] S323. Take the farthest point cell nuclei and the random cell nuclei as separate nodes in the cell map, and obtain the cell map according to the farthest point cell nuclei and the random cell nuclei for spatial modeling of the cells.

[0096] In some embodiments, in step S312, it includes:

[0097] S3121. Exclude similar image patches according to the cosine similarity to obtain super image patches;

[0098] S3122. Determine the distance value between adjacent super image patches according to the Euclidean distance. When the Euclidean distance is less than the set threshold range, create an edge between the adjacent super image patches.

[0099] In some embodiments, in step S321, it includes:

[0100] S311. Randomly sample a cell nucleus and put it into the sampling point set;

[0101] S312. Calculate the Euclidean distance between all unsampled cell nuclei and the sampled cell nuclei, and put the cell nucleus with the largest distance into the sampling point set;

[0102] S313. Calculate the distance between each unsampled cell nucleus and all the cell nuclei in the sampling point set, and retain the minimum distance. After traversing all the unsampled cell nuclei, select the unsampled cell nucleus corresponding to the maximum value among the previously retained minimum distances and add it to the sampling point set;

[0103] S314. Repeat step S313 until the number of cell nuclei in the sampling point set meets the requirements, and then construct an edge using the K-nearest neighbor method based on the centroid position of the cell nuclei.

[0104] In some embodiments, in step S310 and step S320, it includes:

[0105] At low magnification, cut the region of interest into image patches of a fixed size, and extract image patch features from them;

[0106] At high magnification, perform cell segmentation on the region of interest, and extract cell features from it.

[0107] In some embodiments, in step S400, it includes:

[0108] Input the tissue map and the cell map into the position-aware graph neural network respectively, and use the input position-aware graph neural network to extract tissue pathological features and cell pathological features respectively. The position-aware graph neural network is implemented based on the Transformer encoder, and the Transformer encoder is composed of multiple Graph Transformer layers. Each Graph Transformer layer includes a multi-head attention module and a feed-forward neural network.

[0109] In some embodiments, the steps of inputting the position-aware graph neural network include:

[0110] S410. Obtain the topological connection features and node features between the nodes in the tissue map and the cell map;

[0111] S420. Add topological connection features to the node features to capture the spatial position relationships between tissue graphs and between cell graphs;

[0112] S430. Aggregate the node information of the spatial position relationships and the node features to obtain tissue pathological features and cell pathological features.

[0113] In some embodiments, in step S500, it includes:

[0114] Feature fusion. Fuse the tissue pathological features and the cell pathological features through simple splicing and / or a cross-attention module to obtain the final pathological features.

[0115] The present invention also discloses an electronic device, including: a processor and a memory. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method as described above.

[0116] The electronic device includes a processor and a memory. Optionally, the electronic device further includes an input device and an output device. The processor, the memory, the input device, and the output device are coupled through a connector. The connector includes various interfaces, transmission lines, or buses, etc. The embodiments of the present application do not limit this. It should be understood that in various embodiments of the present application, coupling means being interconnected in a specific manner, including being directly connected or indirectly connected through other devices. For example, they can be connected through various interfaces, transmission lines, buses, etc.

[0117] The processor may include one or more processors. For example, it includes one or more central processing units (CPUs). When the processor is a single CPU, the CPU can be a single-core CPU or a multi-core CPU. Optionally, the processor can be a processor group composed of multiple CPUs, and the multiple processors are coupled to each other through one or more buses. Optionally, the processor can also be other types of processors, etc. The embodiments of the present application do not limit this.

[0118] The memory can be used to store computer program instructions and various computer program codes including the program codes for executing the solution of this application. Optionally, the memory includes but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or compact disc read-only memory (CD-ROM), and this memory is used for relevant instructions and data.

[0119] The input device is used to input data and / or signals, and the output device is used to output data and / or signals. The input device and the output device can be independent devices or an integrated device.

[0120] It can be understood that in the embodiments of this application, the memory can not only be used to store relevant instructions, but also be used to store relevant data. For example, the memory can be used to store the target ultrasonic image, the first foreground image, and the background image obtained through the input device, or the memory can also be used to store the first target image obtained through the processor, etc. The embodiments of this application do not limit the specific data stored in this memory.

[0121] In the description of this specification, if terms such as "Embodiment 1", "this embodiment", "in one embodiment", etc. appear, it means that the specific features, structures, materials, or characteristics described in connection with this embodiment or example are included in at least one embodiment or example of the invention or the invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example; moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in an appropriate manner.

[0122] In the description of this specification, terms such as "connection", "installation", "fixation", "setting", "having", etc. are all understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0123] In the description of this specification, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising said element.

[0124] The above description of the embodiments is to enable those of ordinary skill in the art to understand and apply the technology of this case. Those familiar with the technology in this field can obviously make various modifications to these examples easily and apply the general principles described herein to other embodiments without creative labor. Therefore, this case is not limited to the above embodiments. For the following several types of modifications, they should all be within the protection scope of this case: ① A new technical solution implemented based on the technical solution of the present invention in combination with the existing common general knowledge, and the technical effect produced by this new technical solution does not exceed the technical effect of the present invention; ② An equivalent replacement of some features of the technical solution of the present invention using well-known technologies, and the technical effect produced is the same as the technical effect of the present invention; ③ Expansion based on the technical solution of the present invention, and the substantial content of the expanded technical solution does not exceed the technical solution of the present invention; ④ An equivalent transformation made using the content of the specification and drawings of the present invention, directly or indirectly applied to other related technical fields.

Claims

1. A method for extracting spatial features of pathological images, characterized in that: include, S100, obtaining a pathological image; S200, delineating a region of interest, selecting a region of interest from the pathological image; S310, spatial modeling of tissues, obtaining image block features from the region of interest, and constructing a tissue map based on the image block features to perform spatial modeling of tissues; S320, spatial modeling of cells, obtaining cell features from the region of interest, and constructing a cell map based on the cell features to perform spatial modeling of cells; S400, extracting pathological features, extracting tissue pathological features containing inter-tissue spatial information according to the tissue map, and extracting cell pathological features containing inter-cellular spatial information according to the cell map; S500: feature fusion, fusing tissue pathology features and cell pathology features to obtain final pathology features.

2. A pathological image spatial feature extraction method according to claim 1, characterized in that: In the step S310, it includes: An image block node compression scheme is used when constructing the organizational graph.

3. A pathological image spatial feature extraction method according to claim 1, characterized in that: In the step S320, it includes: A cell node compression scheme is used when constructing the cell graph.

4. A pathological image spatial feature extraction method according to claim 2, characterized in that: In the step S310, it includes: S311, calculating the Euclidean distance between the image blocks where the image block features are located, and the cosine similarity and / or mutual information between the image block features; S312, generating a super image block according to the Euclidean distance, the cosine similarity and / or the mutual information; S313: Each of the super image blocks is used as a separate node in the organization diagram, and the organization diagram is obtained according to the super image block to perform spatial modeling of the organization.

5. A pathological image spatial feature extraction method according to claim 3, characterized in that: In the step S320, it includes: S321, sampling the cell feature using the farthest point sampling method to obtain the farthest point cell nucleus; S322, sampling random cell nuclei for the cell characteristics using a random sampling method; S323, taking the farthest point cell nucleus and the random cell nucleus as separate nodes in the cell graph, and obtaining the cell graph according to the farthest point cell nucleus and the random cell nucleus to perform spatial modeling of cells.

6. A pathological image spatial feature extraction method according to claim 4, characterized in that: In the step S312, it includes: S3121, eliminating similar image blocks according to the cosine similarity to obtain the super image block; S3122: Determine the distance value between adjacent super-image blocks according to the Euclidean distance, and when the Euclidean distance is less than a set threshold range, create an edge between adjacent super-image blocks.

7. A pathological image spatial feature extraction method according to claim 5, characterized in that: In the step S321, it includes: S311, randomly sample a cell nucleus and put it into the sampling point set; S312, calculating the Euclidean distances between all unsampled cell nuclei and sampled cell nuclei, and taking the cell nucleus with the largest distance and putting it into the sampling point set; S313, calculating the distance between each unsampled cell nucleus and all the cell nuclei in the sampling point set, retaining the minimum distance, and after traversing all the unsampled cell nuclei, selecting the unsampled cell nucleus corresponding to the maximum value among the previously retained minimum distances, and adding it to the sampling point set; S314, repeating step S313 until the number of cell nuclei in the sampling point set meets the requirement, and constructing the edge based on the centroid position of the cell nucleus using the K nearest neighbor method.

8. A pathological image spatial feature extraction method according to any one of claims 1 to 7, characterized in that: In the step S310 and the step S320, it includes: Cutting the region of interest into image blocks of fixed size at a low magnification, and extracting the image block features therefrom; The region of interest is segmented into cells at high magnification to extract the cell features.

9. A method for extracting spatial features of pathological images according to any one of claims 1 to 7, characterized in that: In the step S400, it includes: The tissue map and the cell map are respectively input into a position-aware graph neural network, and the tissue pathology features and the cell pathology features are respectively extracted using the input position-aware graph neural network. The position-aware graph neural network is implemented based on a Transformer encoder, and the Transformer encoder is composed of multiple Graph Transformer layers, each of which includes a multi-head attention module and a feedforward neural network.

10. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and when the processor executes the computer instructions, the electronic device executes the method according to any one of claims 1 to 9.

Images