A method for identifying microsatellite instability states based on graph convolutional neural network

Abstract

The present invention provides a method for identifying microsatellite instability state based on graph convolutional neural network, wherein small pictures containing tumor areas are screened out from target pathological slices; coordinates of N recorded small pictures are randomly selected, and small tumor pictures are cut downward and rightward in the target pathological slices; for each small tumor picture, the cell nuclei are detected, the center of each cell nucleus is used as a node, feature extraction is performed on each node, node features of the small tumor picture are generated, and node resampling is performed on the small tumor picture and graph edges are constructed to generate an adjacency matrix of the small tumor picture; the MSI score of each small tumor picture is obtained by using a graph convolution model; and the average value of the MSI scores of N small tumor pictures is used as the MSI score of the target pathological slice. The present invention predicts microsatellite instability based on the graph convolution model, thereby improving the accuracy of MSI prediction.

Description

Technical Field

[0001] The present invention relates to the field of computer medical image information processing technology, and in particular to a method for identifying microsatellite instability status from pathological images based on a graph convolutional neural network (a GNN method to predict the MSI status of stained pathological images, abbreviated as WiseMSI). @ G), electronic device and computer readable storage medium. Background Art

[0002] Microsatellite instability (MSI) refers to the increase or loss of simple repeat sequences caused by mismatch repair or replication errors, resulting in changes in the length of microsatellites. The main reason for MSI is that the gene function involved in pairing error repair is defective, so replication errors cannot be corrected normally, causing changes in the DNA of microsatellites, making them unable to play a normal regulatory role. MSI can easily lead to abnormal cell proliferation and differentiation, and even promote the formation of malignant tumors. Compared with normal tissues, microsatellites in tumors are more likely to cause changes in the length of microsatellites. Studies have shown that MSI is closely related to the occurrence of tumors such as colorectal cancer, gastric cancer, and endometrial cancer. Among them, about 15% of sporadic colorectal cancer is associated with MSI, and more than 90% of hereditary non-polyposis colorectal cancer (HNPCC, also known as Lynch syndrome) is associated with MSI, so MSI detection is of great clinical significance. MSI can be divided into three categories according to the degree: microsatellite high instability (MSI-H), microsatellite low instability (MSI-L), and microsatellite stable (MSS). The MSI score is obtained by performing MSI analysis on the pathological sections of the patient's tissue samples, and the MSI category is further determined.

[0003] Most of the current methods for processing pathological slice images are to divide the pathological slice images into small images, input the small images into a convolutional neural network to extract features, obtain the classification results of the small images, and then aggregate all the small images of the entire pathological slice image to obtain the classification results of the pathological slice image. Alternatively, the small images are encoded using a convolutional neural network, and then input into an attention network or a recurrent neural network to directly obtain the classification results of the pathological slice image. These methods require convolution operations on small images. However, one problem with neural networks is that it is difficult for data modelers or pathologists to explain the relationship between input features and their responses (outputs). The more complex the neural network is, the more neurons and layers are added, and it becomes increasingly difficult to explain and determine the significance of each layer and each feature map in the network in pathology.

[0004] In addition, the research object of convolutional neural network is still limited to data in Euclidean domains. The most notable feature is the regular spatial structure, such as pictures are regular square grids, and speech is a regular one-dimensional sequence. These data structures can be represented by one-dimensional and two-dimensional matrices, and convolutional neural networks are very efficient in processing. In the field of pathology, in order to apply convolutional neural networks, pathological slices need to be divided into regular squares, but this is different from the original cell structure. For example, if we use cells as nodes to study the interaction between each cell, the affected range of each cell is different, and the corresponding connection of each node is different. Some nodes have three connections, and some nodes have two connections. This is an irregular data structure, called Non Euclidean data, which cannot be applied to traditional convolutional neural networks. Summary of the invention

[0005] The purpose of the present invention is to provide a method, electronic device and computer-readable storage medium for identifying microsatellite instability states based on a graph convolutional neural network, segmenting a small tumor image from a pathological slice image, using the cell nucleus in the small tumor image as a node, using the cell nucleus image information as a node feature, connecting adjacent nodes of the cell nucleus to generate a graph, and then predicting MSI based on the graph to improve the accuracy of MSI prediction.

[0006] In order to achieve the above objectives, the present invention provides a method for identifying microsatellite instability states based on a graph convolutional neural network, comprising the following steps:

[0007] S1. Obtaining a target pathological section image after cell staining of a patient's tissue sample;

[0008] S2, selecting tissue regions from the target pathological slice image, and dividing all tissue regions into small images with a first preset resolution;

[0009] S3, determining the probability that each small image contains a tumor area, screening out all small images whose probabilities meet preset conditions, and recording the coordinates of the upper left corner of the screened small images in the target pathological slice image;

[0010] S4, randomly selecting coordinates of N recorded small pictures, and cutting downward and rightward in the target pathological slice image a small tumor picture of a second preset resolution according to each coordinate;

[0011] S5. For each tumor image, detect the cell nuclei therein, take the center of each cell nucleus as a node, extract features from each node, generate node features of the tumor image, and use farthest point sampling and random sampling to resample the nodes of the tumor image and use the KNN algorithm to construct graph edges, and generate an adjacency matrix of the tumor image;

[0012] S6. Input the node features and adjacency matrix of each tumor image into the pre-trained graph convolution model to obtain the MSI score of the tumor image.

[0013] S7. Calculate the average value of the MSI scores of the N small tumor images as the MSI score of the target pathological section image.

[0014] Further, in step S4, the step of cutting downward and rightward in the target pathological slice image according to each coordinate into a small tumor image of a second preset resolution includes:

[0015] According to each coordinate, a small intermediate tumor image with a resolution greater than the second preset resolution is cut downward and rightward in the target pathological slice image;

[0016] The resolution of each intermediate tumor small picture is compressed to the second preset resolution to obtain the tumor small picture of the second preset resolution.

[0017] Furthermore, if there are overlapping parts between the segmented small tumor images of the second preset resolution, the coordinates of the N recorded small images are randomly selected again until there are no overlapping parts between the segmented small tumor images of the second preset resolution.

[0018] Furthermore, in step S5, for each tumor small picture, the method of detecting the cell nucleus therein includes:

[0019] Each small tumor picture is input into a pre-trained cell nucleus segmentation model to identify the cell nuclei in each small tumor picture. The cell nucleus segmentation model is a convolutional neural network model.

[0020] Furthermore, the cell nucleus segmentation model is a HoVer-Net pre-trained model, and the model parameters are CoNSepcheckpoint.

[0021] Furthermore, in step S5, the features extracted for each node include: average nuclear intensity of the cell nucleus, average front and back background difference, standard deviation of nuclear intensity, skewness of nuclear intensity, average entropy of nuclear intensity, heterogeneous gray-level co-occurrence matrix, homogeneous gray-level co-occurrence matrix, gray-level co-occurrence matrix energy, texture properties of gray-level co-occurrence matrix, eccentricity, area, maximum axis length, minimum axis length, perimeter, solidity, direction, and core coordinates.

[0022] Furthermore, in step S5, the adjacency matrix A of each tumor small picture is generated using the following formula:

[0023]

[0024] Wherein, KNN(i) represents a set of a preset number of nodes whose Euclidean distance to node i is closest, D(i,j) represents the Euclidean distance between node i and node j, and d is a preset value.

[0025] Further, in step S6, the graph convolution model includes k graph convolution layers and a bidirectional LSTM module;

[0026] The vector embedding of each vertex in the graph output by each graph convolutional layer is composed of Sequence, input into the bidirectional LSTM module, and get the forward hidden embedding and backward hidden embedding

[0027] For each Linking forward and backward hidden embeddings Input into the linear mapping function and perform softmax to get the importance score of each layer Then, weighted summation is performed through multi-level features to obtain the representation M of each node;

[0028] For each embedding dimension, select the maximum value from the embeddings of all nodes as the representation of the graph G, G = MAX(M, dim = node);

[0029] The representation G of the graph is input into the subsequent classification network, and after passing through two fully connected layers, it is input into the softmax function to obtain the probability value p of the MSI classification.

[0030] Furthermore, the vector embedding of the k-th layer vertex v is expressed as follows:

[0031]

[0032] In the formula, represents the k-th layer embedding of node v, f k (·) is an aggregate function, represents a random sampling of neighbor nodes of node v, W (k) is a learnable weight parameter and σ represents a nonlinear function.

[0033] Furthermore, the k-th layer importance score of node v is It is expressed as follows:

[0034]

[0035] Where W s represents the weight parameter of the linear mapping, b s Represents the bias parameter of the linear mapping.

[0036] Furthermore, the representation M of each node is as follows:

[0037]

[0038] Furthermore, the probability value p of MSI classification is obtained using the following expression:

[0039] p=softmax(W 2 (Dropout(ReLU(W 1 G+b 1 )))+b 2 )

[0040] Where W 1 , W 2 represents the weight parameters of the two fully connected layers, b 1 , b 2 Represents the bias parameters of the two fully connected layers.

[0041] Furthermore, the graph convolution layer adopts SAGEConv, GCNConv, GATConv or GINConv.

[0042] Furthermore, in step S3, the method for determining the probability that each small image contains a tumor region includes:

[0043] Each small image obtained in step S2 is input into a pre-trained classification model to obtain the probability that each small image output by the classification model contains a tumor area; the classification model is a convolutional neural network model.

[0044] Furthermore, the classification model adopts the ResNet18 model.

[0045] To achieve the above object, the present invention further provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus; wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0046] Memory, used to store computer programs;

[0047] The processor is used to implement the steps of the above-mentioned method for identifying the instability state of microsatellites based on a graph convolutional neural network when executing the program stored in the memory.

[0048] To achieve the above objectives, the present invention also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the method for identifying the instability state of microsatellites based on a graph convolutional neural network are implemented.

[0049] Compared with the prior art, the present invention has the following advantages:

[0050] The present invention obtains a target pathological slice image after cell staining of a patient tissue sample, screens out a tissue region and divides it into small images, determines a small image containing a tumor region, and records coordinates, divides N tumor small images from the target pathological slice image according to the coordinates, detects the cell nucleus for each tumor small image, uses the cell nucleus as a node, extracts node features, and generates an adjacency matrix, thereby using a graph convolution model to obtain the MSI score of each tumor small image, and then obtains the MSI score of the entire target pathological slice image. It can be seen that the present invention divides the pathological slice image into small images, uses the cell nucleus in the small image as a node, uses the cell nucleus image information as a node feature, connects the adjacent nodes of the cell nucleus to generate a graph, and then predicts MSI based on the graph. Since the present invention establishes a model of interaction between adjacent cell nuclei, the characteristics of each tumor cell nucleus are finally taken into account, the features are deaggregated in a more microscopic cell field, and then the category of the entire pathological slice image is inferred, thereby improving the accuracy of MSI prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for the description. Obviously, the drawings described below are an embodiment of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work:

[0052] Figure 1 A schematic diagram of a process for identifying microsatellite instability states based on a graph convolutional neural network provided by an embodiment of the present invention;

[0053] Figure 2 A flow chart of the overall solution provided for one embodiment of the present invention;

[0054] Figure 3 A diagram of a small picture classification model training process provided by an embodiment of the present invention;

[0055] Figure 4 A schematic diagram of a graph convolution model provided by an embodiment of the present invention;

[0056] Figure 5 A diagram of the training process of a graph convolutional model provided by an embodiment of the present invention;

[0057] Figure 6 A structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The scheme proposed by the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. According to the following description, the advantages and features of the present invention will be clearer. It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the implementation method of the present invention. In order to make the purposes, features and advantages of the present invention more obvious and easy to understand, please refer to the accompanying drawings. It should be noted that the structures, proportions, sizes, etc. illustrated in the accompanying drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limiting conditions for the implementation of the present invention, so they have no technical substantive significance. Any structural modification, change in proportional relationship or adjustment of size, without affecting the effects that the present invention can produce and the purposes that can be achieved, should still fall within the scope of the technical content disclosed by the present invention.

[0059] The core idea of ​​the present invention is to segment the pathological slice image to obtain a small tumor image, use the cell nucleus in the small tumor image as a node, use the cell nucleus image information as the node feature, connect the adjacent nodes of the cell nucleus to generate a graph, and then predict MSI based on the graph. The present invention establishes a model of the interaction between adjacent cell nuclei, ultimately taking into account the characteristics of each tumor cell nucleus, de-aggregating features in a more microscopic cell field, and then inferring the category of the entire pathological slice image to improve the accuracy of MSI prediction.

[0060] The following combination Figure 1 , Figure 2 The present invention is described in detail. The present invention provides a method for identifying microsatellite instability state based on graph convolutional neural network, comprising the following steps:

[0061] S1. Obtain target pathological section images after cell staining of patient tissue samples.

[0062] In the present embodiment, the pathological section diagram is a picture obtained by making a tissue sample taken from a specific part of the patient into a pathological section and then staining the cells, and then placing it under a microscope for imaging (e.g., 40 times the microscope). The patient can be a patient to be diagnosed, or a cancer patient who has been diagnosed, not limited to colorectal cancer patients, but also cancer patients such as gastric cancer and pancreatic cancer. The cell staining method can select hematoxylin-eosin staining (hemotoxylin eosin, HE), Papanicolaou staining, Wright-Giemsa staining, etc. It is preferred to use HE staining, which has good staining transparency, a clear contrast between the nucleus and the cytoplasm, and a stable effect. In order to obtain a better pathological section diagram, in addition to cell staining, other pretreatment operations can also be performed on the patient's tissue sample, and the present invention does not limit this. In addition, the pathological section diagram can also be pretreated, such as adjusting the clarity, contrast enhancement, etc.

[0063] S2. Filter out tissue regions from the target pathological slice image, and divide all tissue regions into small images with a first preset resolution.

[0064] In this embodiment, the tissue area is first marked in the target pathological slice image. The specific marking method can refer to the existing technology, and then the marked tissue area is segmented according to a first preset resolution to obtain multiple non-overlapping small images (also called patches). The first preset resolution is, for example, 512×512 pixels, that is, the tissue area is segmented into small images of 512×512 size.

[0065] S3. Determine the probability that each small image contains a tumor area, select all small images whose probabilities meet preset conditions, and record the coordinates of the upper left corner of the selected small images in the target pathological slice image.

[0066] In the aforementioned step S2, all the small images obtained by segmenting the tissue area may contain tumor areas or may not contain tumor areas (that is, all are normal areas). Therefore, it is necessary to filter out the small images containing tumor areas to facilitate subsequent processing of these small images containing tumor areas.

[0067] Specifically, one implementation method is to use a convolutional neural network (CNN) model to determine the probability that each small picture contains a tumor region, that is, input each small picture obtained in step S2 into a pre-trained classification model to obtain the probability that each tissue small picture output by the classification model contains a tumor region. After obtaining the probability that each small picture contains a tumor region, small pictures whose probability of containing a tumor region is greater than or equal to a preset threshold can be screened out. The preset threshold can be set according to actual conditions, for example, set to 50%, that is, all small pictures whose probability of containing a tumor region is greater than or equal to 50%. Other conditions can also be used for screening.

[0068] Assuming that the total number of the screened small pictures is A, the coordinates of the upper left corners of the screened A small pictures in the target pathological slice image are recorded.

[0069] Optionally, the classification model may adopt a ResNet18 model, see Figure 3 As shown, the training process of the classification model is as follows:

[0070] a. Select several sample pathological sections (40x magnification), extract tissue regions and segment them into small images of preset size. For example, segment the tissue regions of 280 pathological sections of colorectal cancer (40x magnification) into small images of 512×512, for a total of 181,328 images;

[0071] b. Pathologists classify all small images into tumor areas / normal areas;

[0072] c. Divide the small images into 70% training set, 10% validation set and 20% test set at the level of pathological sections;

[0073] d. Use the ResNet18 model to classify the small images of the tumor area and the small images of the normal area in the training set (100 small images of the tumor area and 100 small images of the normal area are randomly selected from the pathological sections in each round);

[0074] e. Optimize the model through the validation set and save the model weights when the loss value is minimized;

[0075] f. When the validation set loss value does not decrease for consecutive preset rounds (e.g., 5 rounds) and the training round is greater than the preset round (e.g., 10 rounds), stop training;

[0076] g. The test set was used to calculate the model's sensitivity, specificity, and AUC. The final test set AUC was 98.8%.

[0077] It is understandable that the classification model is not limited to the ResNet18 model, and other models that can achieve classification may also be used, and the present invention is not limited to this.

[0078] S4. Randomly select coordinates of N recorded small pictures, and cut downward and rightward in the target pathological slice image to obtain a small tumor picture of a second preset resolution according to each coordinate.

[0079] Among them, N can be 10, or any other value. That is, randomly select N recorded coordinates, and cut out small pictures of the second preset resolution from the original target pathological slice downward and rightward. The cut out small pictures all contain tumor areas, so these small pictures are called tumor small pictures. The second resolution is, for example, 1000×1000 resolution, or other resolution sizes. If there are overlapping parts between the cut out tumor small pictures of the second preset resolution, then randomly select the coordinates of the N recorded small pictures again until there are no more overlapping parts between the cut out tumor small pictures of the second preset resolution.

[0080] In one implementation, the target pathological slice image can be first cut downward and rightward according to each coordinate to obtain an intermediate tumor small picture with a resolution greater than the second preset resolution; and then the resolution of each intermediate tumor small picture is compressed to the second preset resolution to obtain a tumor small picture of the second preset resolution. After actual testing, by first cutting out the intermediate tumor small picture with a large resolution and then compressing it to obtain the tumor small picture of the second preset resolution, the subsequent cell nucleus division effect can be improved.

[0081] S5. For each small tumor image, detect the cell nuclei therein, take the center of each cell nucleus as a node, extract features from each node, generate node features of the small tumor image, use farthest point sampling and random sampling to resample the nodes of the small tumor image, and use the KNN algorithm to construct graph edges to generate an adjacency matrix of the small tumor image.

[0082] Specifically, each small tumor image is input into a pre-trained cell nucleus segmentation model to identify the cell nuclei in each small tumor image, and the cell nucleus segmentation model is a convolutional neural network model.

[0083] For example, the cell nucleus segmentation model is a HoVer-Net pre-trained model, the model parameters are CoNSepcheckpoint, and the output is a matrix of a size corresponding to the second preset resolution (for example, a 1000×1000 matrix), and each point in the matrix records the pixel type (cell nucleus, background) of the tumor small picture. The cell nucleus segmentation model can also be other types of convolutional neural network models.

[0084] Then, each cell nucleus of each small tumor image is used as a node of the Graph, and the position of the node is located at the center of the cell nucleus. Then, feature extraction is performed on each node. The extracted features include, for example, average nuclear intensity, average front and back background difference, nuclear intensity standard deviation, nuclear intensity skewness, nuclear intensity average entropy, heterogeneous gray level co-occurrence matrix, homogeneous gray level co-occurrence matrix, gray level co-occurrence matrix energy, texture properties of gray level co-occurrence matrix, eccentricity, area, maximum axis length, minimum axis length, perimeter, solidity, direction, and core coordinates.

[0085] Furthermore, the nodes of each tumor image are resampled using farthest point sampling and random sampling (the ratio of farthest point sampling to random sampling is 1:1), the graph edges are constructed using the KNN algorithm, and the adjacency matrix of each tumor image is generated. Specifically, the adjacency matrix A of each tumor image can be generated using the following formula:

[0086]

[0087] Among them, KNN(i) represents the set of a preset number of nodes (set to 8 nodes in this embodiment) whose Euclidean distance to node i is closest, D(i,j) represents the Euclidean distance between node i and node j, and d is a preset value (set to 100 in this embodiment).

[0088] S6. Input the node features and adjacency matrix of each small tumor image into the pre-trained graph convolution model to obtain the MSI score of the small tumor image.

[0089] This embodiment adopts a graph convolution model based on the spatial domain, such as Figure 4 As shown, it includes k GraphSage graph convolution layers. In this embodiment, the SAGEConv graph convolution layer is used. Of course, other types of graph convolution layers such as GCNConv, GATConv, and GINConv can be used. The process of the graph convolution layer is divided into the following three steps:

[0090] 1. Sample the neighbor vertices of each vertex in the graph;

[0091] 2. Aggregate the information contained in neighbor vertices according to the aggregation function;

[0092] 3. Get the vector embedding of each vertex in the graph for use in downstream tasks.

[0093] The vector embedding of the k-th layer vertex v is expressed as follows:

[0094]

[0095] In the formula, represents the k-th level embedding of node v, x v Represents the node features and adjacency matrix input to the graph convolutional model;

[0096] f k (·) is an aggregation function, which can be an average, a sum, a maximum, etc. This embodiment uses the SAGEConv module in the pytorch-geometric library, so the aggregation function is a sum;

[0097] represents a random sampling of neighbor nodes of node v;

[0098] W (k) is a learnable weight parameter, σ represents a nonlinear function, and ReLU is used in this embodiment.

[0099] like Figure 4 As shown in the figure, the graph convolution model also includes a bidirectional LSTM module, which embeds the vectors of each vertex in the graph output by each graph convolution layer to form Sequence, input into the bidirectional LSTM module, and get the forward hidden embedding and backward hidden embedding

[0100] For each Linking forward and backward hidden embeddings Input into the linear mapping function and perform softmax to get the importance score of each layer Then, weighted summation is performed through multi-level features to obtain the representation M of each node (shape is n0 ×d 1 );

[0101]

[0102]

[0103] In the formula, represents the importance score of node v at the kth layer, They represent the forward hidden embedding and backward hidden embedding of node v after the bidirectional LSTM module, W s represents the weight parameter of the linear mapping, b s Represents the bias parameter of the linear mapping.

[0104] For each embedding dimension, the maximum value is selected from the embeddings of all nodes as the representation of the graph G (shape d 1 ), G = MAX(M, dim = node);

[0105] The representation G of the graph is input into the subsequent classification network, and then input into the softmax function after two fully connected layers to obtain the probability value p of the MSI classification. The MSI score of the tumor image can be known based on the probability value p.

[0106] p=softmax(W 2 (Dropout(ReLU(W 1 G+b 1 )))+b 2 )

[0107] W 1 , W 2 Represent the weight parameters of the two fully connected layers, b 1 , b 2 They represent the deviation parameters of the two fully connected layers respectively; the value of the Dropout layer is 0.2 in this embodiment;

[0108] In this embodiment, the node input dimension is 18, and the output dimension of the graph convolution layer and the bidirectional LSTM module is d 1 is 20, and the output dimensions of the two fully connected layers are 10 and 2 respectively.

[0109] like Figure 5 As shown, the training process of the graph convolution model is as follows:

[0110] a. Processing several sample pathology slice images by the method of steps S1-S5 to obtain node features and adjacency matrices of N tumor small pictures corresponding to each sample pathology slice image, wherein the sample pathology slice images include pathology slice images of tissue samples of several MSS patients and several MSI-H patients, and each type of sample pathology slice image has, for example, 1000 each, and each sample pathology slice image is also annotated with its own MSI score, so that the node features and adjacency matrix of the N tumor small pictures corresponding to each sample pathology slice image and the corresponding relationship with the MSI score can be established;

[0111] b. Divide the node features and adjacency matrix into training set, validation set and test set according to the patient's level. For example, divide 70% of all sample feature matrices into training set, 10% into validation set and 20% into test set.

[0112] c. Use Figure 4 The graph convolution model is used to classify the training set;

[0113] d. Optimize the model through the validation set to minimize the loss value and save the model weight;

[0114] e. When the validation set loss value does not decrease for consecutive preset rounds (e.g., 40 rounds) and the training round is greater than the preset round (e.g., 100 rounds), stop training;

[0115] f. The sensitivity, specificity and AUC of the model were calculated using the test set, and the final test set AUC was 95.8%.

[0116] S7. Calculate the average value of the MSI scores of the N small tumor images as the MSI score of the target pathological section image.

[0117] After obtaining the MSI score, the MSI category of the target pathological slice image can be determined. For example, a threshold can be set in advance. When the MSI score is higher than the threshold, the category is MSI-H, and when the MSI score is lower than the threshold, the category is MSS. The specific value of the threshold can be set based on experience or actual conditions.

[0118] At this point, the prediction of microsatellite instability of the target pathological slice map is completed. The present invention divides the pathological slice map into small pictures, takes the cell nucleus in the small picture as a node, extracts the cell nucleus image information as the node feature, connects the adjacent nodes of the cell nucleus to generate a graph (Graph) and obtains the adjacency matrix, and then uses the node features and the adjacency matrix as the reasoning data to predict MSI. Since the present invention establishes a model of interaction between adjacent cell nuclei, the characteristics of each tumor cell nucleus are finally taken into account, the aggregation features are removed in a more microscopic cell field, the classification results of the small pictures are inferred, and the average value of all small pictures is finally taken as the category of the entire pathological slice map.

[0119] Taking the cell nucleus as the base node, after passing through the multi-layer graph convolution layer, each node will contain multi-level neighborhood information, but the ordinary graph convolution cannot adaptively allocate the weight of the structure around the target node, and can only fuse the multi-layer embedding features of all cell nucleus nodes in the graph under the same weight, which is not applicable in the pathological cell graph. In order to obtain the glandular structure at multiple scales, the present invention makes each vertex aggregate the embedding of neighbor vertices at different levels as the subsequent node representation, and the proportion of each level is calculated by obtaining the representation of the nodes of the previous level and the next level through the bidirectional LSTM module, and then weighted summation is performed, so that the weight of the node embedding at each scale can be automatically allocated, and the node features can be better aggregated to make the final classification. Therefore, the present invention improves the accuracy of MSI prediction.

[0120] Based on the same inventive concept, according to the above-mentioned microsatellite instability prediction method embodiment, the present invention also provides an electronic device, such as Figure 6 As shown, it includes a processor 301, a communication interface 302, a memory 303 and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.

[0121] Memory 303, used for storing computer programs;

[0122] The processor 301 is used to implement the steps of the method for identifying microsatellite instability states based on a graph convolutional neural network in the above embodiment when executing the program stored in the memory 303.

[0123] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0124] The communication interface is used for communication between the above electronic device and other devices.

[0125] The memory may include a random access memory (RAM) or a non-volatile memory (NVM). Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0126] The above-mentioned processor can be a general-purpose processor, or it can be a digital signal processor (Digital Signal Processing, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0127] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of the method for identifying the instability state of microsatellites based on a graph convolutional neural network in the above embodiment are implemented.

[0128] In another embodiment provided by the present invention, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute the steps of the method for identifying microsatellite instability states based on a graph convolutional neural network in the above-mentioned embodiment.

[0129] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiment for identifying microsatellite instability based on a graph convolutional neural network, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment for identifying microsatellite instability based on a graph convolutional neural network.

[0130] Although the content of the present invention has been described in detail through the above preferred embodiments, it should be appreciated that the above description should not be considered as a limitation of the present invention. After reading the above content, it will be apparent to those skilled in the art that various modifications and substitutions of the present invention will occur. Therefore, the protection scope of the present invention should be limited by the appended claims.

Claims

1. A method for identifying microsatellite instability states based on graph convolutional neural networks, characterized in that: The steps include: S1. Obtaining a target pathological section image after cell staining of a patient's tissue sample; S2, selecting tissue regions from the target pathological slice image, and dividing all tissue regions into small images with a first preset resolution; S3, determining the probability that each small image contains a tumor area, screening out all small images whose probabilities meet preset conditions, and recording the coordinates of the upper left corner of the screened small images in the target pathological slice image; S4, randomly selecting coordinates of N recorded small pictures, and cutting downward and rightward in the target pathological slice image a small tumor picture of a second preset resolution according to each coordinate; S5. For each tumor image, detect the cell nuclei therein, take the center of each cell nucleus as a node, extract features from each node, generate node features of the tumor image, and use farthest point sampling and random sampling to resample the nodes of the tumor image and use the KNN algorithm to construct graph edges, and generate an adjacency matrix of the tumor image; S6. Input the node features and adjacency matrix of each tumor image into the pre-trained graph convolution model to obtain the MSI score of the tumor image. S7. Calculate the average value of the MSI scores of the N small tumor images as the MSI score of the target pathological section image.

2. The method for identifying microsatellite instability state based on graph convolutional neural network as claimed in claim 1, characterized in that: In step S4, the step of cutting downward and rightward from the target pathological slice image into a small tumor image of a second preset resolution according to each coordinate includes: According to each coordinate, a small intermediate tumor image with a resolution greater than the second preset resolution is cut downward and rightward in the target pathological slice image; The resolution of each intermediate tumor small picture is compressed to the second preset resolution to obtain the tumor small picture of the second preset resolution.

3. The method for identifying microsatellite instability state based on graph convolutional neural network as described in claim 1 or 2, characterized in that: If there are overlapping parts between the segmented small tumor images of the second preset resolution, the coordinates of the N recorded small images are randomly selected again until there are no overlapping parts between the segmented small tumor images of the second preset resolution.

4. The method for identifying microsatellite instability state based on graph convolutional neural network as claimed in claim 1, characterized in that: In step S5, for each tumor small picture, the method of detecting the cell nucleus therein includes: Each small tumor picture is input into a pre-trained cell nucleus segmentation model to identify the cell nuclei in each small tumor picture. The cell nucleus segmentation model is a convolutional neural network model.

5. The method for identifying microsatellite instability state based on graph convolutional neural network as claimed in claim 4, characterized in that: The cell nucleus segmentation model is a HoVer-Net pre-trained model, and the model parameters are CoNSep checkpoint.

6. The method for identifying microsatellite instability state based on graph convolutional neural network as claimed in claim 1, characterized in that: In step S5, the features extracted for each node include: average nuclear intensity of the cell nucleus, average front and back background difference, standard deviation of nuclear intensity, skewness of nuclear intensity, average entropy of nuclear intensity, heterogeneous gray level co-occurrence matrix, homogeneous gray level co-occurrence matrix, gray level co-occurrence matrix energy, texture properties of gray level co-occurrence matrix, eccentricity, area, maximum axis length, minimum axis length, perimeter, solidity, direction, and core coordinates.

7. The method for identifying microsatellite instability state based on graph convolutional neural network as claimed in claim 1, characterized in that: In step S5, the adjacency matrix A of each tumor image is generated using the following formula: Wherein, KNN(i) represents a set of a preset number of nodes whose Euclidean distance to node i is closest, D(i,j) represents the Euclidean distance between node i and node j, and d is a preset value.

8. The method for identifying microsatellite instability state based on graph convolutional neural network as claimed in claim 1, characterized in that: In step S6, the graph convolution model includes k graph convolution layers and a bidirectional LSTM module; The vector embedding of each vertex in the graph output by each graph convolutional layer is composed of Sequence, input into the bidirectional LSTM module, and get the forward hidden embedding and backward hidden embedding For each Linking forward and backward hidden embeddings Input into the linear mapping function and perform softmax to get the importance score of each layer Then, weighted summation is performed through multi-level features to obtain the representation M of each node; For each embedding dimension, select the maximum value from the embeddings of all nodes as the representation of the graph G, G = MAX(M, dim = node); The representation G of the graph is input into the subsequent classification network, and after passing through two fully connected layers, it is input into the softmax function to obtain the probability value p of the MSI classification.

9. The method for identifying microsatellite instability state based on graph convolutional neural network as claimed in claim 8, characterized in that: The vector embedding of the k-th layer vertex v is expressed as follows: In the formula, represents the k-th layer embedding of node v, f k (·) is an aggregate function, represents a random sampling of neighbor nodes of node v, W (k) is a learnable weight parameter and σ represents a nonlinear function.

10. The method for identifying microsatellite instability state based on graph convolutional neural network as claimed in claim 8, characterized in that: The representation M of each node is as follows:

11. The method for identifying microsatellite instability state based on graph convolutional neural network as claimed in claim 8, characterized in that: The probability value p of MSI classification is obtained using the following expression: p=softmax.W2.Dropout(ReLU(W1G+b1)) / +b2 / Where W1 and W2 represent the weight parameters of the two fully connected layers, and b1 and b2 represent the bias parameters of the two fully connected layers.

12. The method for identifying microsatellite instability state based on graph convolutional neural network as claimed in claim 8, characterized in that: The k-th layer importance score of node v It is expressed as follows: Where W s represents the weight parameter of the linear mapping, b s Represents the bias parameter of the linear mapping.

13. The method for identifying microsatellite instability state based on graph convolutional neural network as claimed in claim 8, characterized in that: The graph convolution layer uses SAGEConv, GCNConv, GATConv or GINConv.

14. The method for identifying microsatellite instability state based on graph convolutional neural network according to claim 1, characterized in that: In step S3, the method for determining the probability that each small image contains a tumor region includes: Each small image obtained in step S2 is input into a pre-trained classification model to obtain the probability that each small image output by the classification model contains a tumor area; the classification model is a convolutional neural network model.

15. The method for identifying microsatellite instability state based on graph convolutional neural network according to claim 14, characterized in that: The classification model adopts the ResNet18 model.

16. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the steps of the method for identifying microsatellite instability states based on a graph convolutional neural network as described in any one of claims 1 to 15 when executing a program stored in a memory.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for identifying microsatellite instability states based on a graph convolutional neural network according to any one of claims 1 to 15 are implemented.

Images