A Gastric Cancer Pathology Image Processing Method and System Based on Multi-Task Learning

Abstract

The present invention relates to the technical field of medical data processing, and particularly relates to a gastric cancer pathological image processing method and system based on multi-task learning. A gastric cancer pathological image processing system based on multi-task learning includes: a gastric cancer pathological image acquisition module and a gastric cancer pathological condition analysis module. The gastric cancer pathological condition analysis model set in the present invention segments gastric cancer pathological images, and regards the different segmented structures as feature nodes to construct a spatial adjacency graph, a connectivity graph and a pathological association graph. It specifically learns the features in gastric cancer pathological images from the adjacency relationship, connectivity relationship and pathological association between structures, and strengthens the features of different structures and the connections between different structure features in gastric cancer pathological images through a gastric cancer pathological knowledge graph, thereby improving the analysis ability of gastric cancer pathological images.

Description

Technical Field

[0001] The present invention relates to the technical field of medical data processing, and particularly relates to a gastric cancer pathological image processing method and system based on multi-task learning. Background Art

[0002] The analysis of gastric cancer pathological images is a key link in disease diagnosis and precise treatment. However, gastric cancer pathological images themselves have extremely high heterogeneity, which is reflected in multiple levels such as tissue morphology, cell structure, and staining patterns. This heterogeneity stems from the complex biological mechanisms of tumor occurrence and development, posing great challenges to traditional machine learning models. Specifically, in the face of the complexity and heterogeneity of gastric cancer pathological images, traditional machine learning models often exhibit problems such as insufficient feature representation ability, limited analysis accuracy, and poor robustness in key tasks such as tumor segmentation and typing and grading, severely restricting the clinical application value of computer-aided gastric cancer pathological diagnosis. There is an urgent need for more advanced methods to improve the model's analysis ability for highly heterogeneous gastric cancer pathological images. Summary of the Invention

[0003] The gastric cancer pathological condition analysis model set in the present invention segments gastric cancer pathological images, and regards the different segmented structures as feature nodes to construct a spatial adjacency graph, a connectivity graph, and a pathological association graph. It specifically learns the features in gastric cancer pathological images from the adjacency relationship, connectivity relationship, and pathological association between structures, and strengthens the features of different structures and the connections between different structural features in gastric cancer pathological images through a gastric cancer pathological knowledge graph, thereby improving the analysis ability for gastric cancer pathological images.

[0004] The present invention provides a gastric cancer pathological image processing method based on multi-task learning, including:

[0005] Obtain the gastric cancer pathological image of a patient, and then send the gastric cancer pathological image into the gastric cancer pathological condition analysis model for processing to output a corresponding gastric cancer pathological condition report;

[0006] Set a structure segmentation layer, a shape convolution kernel mask parameter construction layer, a graph structure feature extraction layer, and a gastric cancer pathological feature fusion layer in the gastric cancer pathological condition analysis model. The structure segmentation layer segments different structures in the gastric cancer pathological image, the shape convolution kernel mask parameter construction layer sets corresponding convolution kernels for the segmented different structures for subsequent feature extraction, the graph structure feature extraction layer extracts features based on the adjacency relationship, connectivity relationship, and pathological association between different structures, and the gastric cancer pathological feature fusion layer strengthens the features of different structures and the connections between different structural features in the gastric cancer pathological image based on the gastric cancer pathological knowledge graph.

[0007] As a preferred aspect, the gastric cancer pathological condition analysis model includes a structure segmentation layer, a shape convolution kernel mask parameter construction layer, a structure feature extraction layer, a graph structure feature extraction layer, a gastric cancer pathological feature fusion layer, a gastric cancer condition multi-task analysis layer, and a gastric cancer pathological condition report output layer. Among them, the structure segmentation layer is used to segment different structures in the gastric cancer pathological image, output a number of structure region detection frames and corresponding structure type feature encodings, and map the structure region detection frames to the gastric cancer pathological image to generate structure region images; the shape convolution kernel mask parameter construction layer is used to process the structure region detection frames and corresponding structure type feature encodings, and output the shape convolution kernel mask parameters corresponding to the structure region detection frames; the structure feature extraction layer is used to extract features from the structure region images and output corresponding structure features. During the feature extraction process, the convolution kernel is dot-product processed through the shape convolution kernel mask parameters corresponding to the structure region images; the graph structure feature extraction layer is used to splice the structure features and corresponding structure type feature encodings to construct pathological feature nodes, and based on the pathological feature nodes, construct a spatial adjacency graph, a connectivity graph, and a pathological association graph. Graph structure feature extraction is performed respectively according to the spatial adjacency graph, the connectivity graph, and the pathological association graph to construct corresponding spatial feature graphs, connectivity feature graphs, and pathological association feature graphs; the gastric cancer pathological feature fusion layer is used to perform weighted fusion on the spatial feature graph, the connectivity feature graph, and the pathological association feature graph to construct an aggregated feature graph, and query the gastric cancer pathological knowledge graph based on the pathological feature nodes to construct a pathological knowledge feature graph. An self-attention mechanism operation is performed based on the aggregated feature graph and the pathological knowledge feature graph to construct a pathological fusion feature graph; the gastric cancer condition multi-task analysis layer includes a Lauren classification unit and a pathological staging unit, which are used to process the pathological fusion feature graph and output corresponding Lauren classification labels and pathological staging labels; the gastric cancer pathological condition report output layer is used to form a gastric cancer pathological condition report and output it with the Lauren classification labels and pathological staging labels.

[0008] As a preferred aspect, the structure region detection frames and corresponding structure type feature encodings are processed by the shape convolution kernel mask parameter construction layer to output the shape convolution kernel mask parameters corresponding to the structure region detection frames, which specifically include the following steps: extract the length and width of the structure region detection frames, splice the length and width of the structure region detection frames and the structure type feature encodings corresponding to the structure region detection frames to construct structure region analysis data, and then send the structure region analysis data into the shape convolution kernel mask parameter construction network built in the shape convolution kernel mask parameter construction layer for processing to output the shape convolution kernel mask parameters corresponding to the structure region detection frames.

[0009] As a preferred aspect, the structural features and the corresponding structural type features are encoded and spliced through the graph structure feature extraction layer to construct pathological feature nodes, and a spatial adjacency graph, a connectivity graph, and a pathological association graph are constructed based on the pathological feature nodes. Graph structure feature extraction is performed respectively according to the spatial adjacency graph, the connectivity graph, and the pathological association graph to construct corresponding spatial feature graphs, connectivity feature graphs, and pathological association feature graphs, which specifically include the following steps:

[0010] Splice each structural feature and the corresponding structural type feature encoding to construct the corresponding pathological feature node, and splice all the pathological nodes from top to bottom to construct a pathological feature graph;

[0011] Traverse all the pathological feature nodes, construct adjacent feature edges between two pathological feature nodes with an adjacency relationship. The way to determine the adjacency relationship is as follows: construct a Voronoi diagram for all the pathological feature nodes, and the pathological feature nodes in the same region in the Voronoi diagram have an adjacency relationship. Construct a spatial adjacency graph based on all the pathological feature nodes and all the adjacent feature edges, and construct a spatial adjacency matrix based on the spatial adjacency graph;

[0012] Traverse all the pathological feature nodes, construct connectivity feature edges between two pathological feature nodes with a connectivity relationship. The way to determine the connectivity relationship is as follows: the structural region images corresponding to two pathological feature nodes share boundary pixels, which is regarded as having a connectivity relationship. Construct a connectivity graph based on all the pathological feature nodes and all the connectivity feature edges, and construct a connectivity adjacency matrix based on the connectivity graph;

[0013] Traverse all the pathological feature nodes, construct pathological association feature edges between two pathological feature nodes with a pathological association. The way to determine the pathological association is as follows: query in the gastric cancer pathological knowledge graph based on the structural type names corresponding to the two pathological feature nodes. If the structural type names corresponding to the two pathological feature nodes have an entity relationship in the gastric cancer pathological knowledge graph, it is regarded as having a pathological association between the two pathological feature nodes. Construct a pathological association graph based on all the pathological feature nodes and all the pathological association feature edges, and construct a pathological association adjacency matrix based on the pathological association graph;

[0014] Perform graph convolution operations on the pathological feature graph respectively based on the spatial adjacency matrix, the connectivity adjacency matrix, and the pathological association adjacency matrix to construct corresponding spatial feature graphs, connectivity feature graphs, and pathological association feature graphs.

[0015] As a preferred aspect, the spatial feature map, the connectivity feature map, and the pathological association feature map are weighted and fused through the gastric cancer pathological feature fusion layer to construct an aggregated feature map. The gastric cancer pathological knowledge graph is queried based on the pathological feature nodes to construct a pathological knowledge feature map. The self-attention mechanism operation is performed based on the aggregated feature map and the pathological knowledge feature map to construct a pathological fusion feature map, which specifically includes the following steps:

[0016] The spatial feature map, the connectivity feature map, and the pathological association feature map are sent to a weight adjustment network for processing to output corresponding feature weights. The spatial feature map, the connectivity feature map, the pathological association feature map, and the corresponding feature weights are weighted and fused to construct an aggregated feature map;

[0017] For each pathological feature node, a query is made in the gastric cancer pathological knowledge graph based on the structure type name corresponding to the pathological feature node. If the structure type names corresponding to any two pathological feature nodes have an entity relationship in the gastric cancer pathological knowledge graph, a pathological knowledge vector is constructed in the form of pathological feature node - entity relationship - pathological feature node. All pathological knowledge vectors are concatenated from top to bottom to form a pathological knowledge feature map. Based on the aggregated feature map, a corresponding value vector V and key vector K are constructed. Based on the pathological knowledge feature map, a corresponding query vector Q is constructed. The self-attention matrix ATT = softmax(QK T / D 0.5 ) is calculated, where T is the matrix transpose operation and D is the dimension size of the key vector K. Then, the value vector V and the self-attention matrix ATT are multiplied matrix-wise to obtain the pathological fusion feature map.

[0018] As a preferred aspect, the gastric cancer pathological condition analysis model is trained, which specifically includes the following steps:

[0019] Obtain a number of gastric cancer pathological condition analysis training samples, which include gastric cancer pathological images. The gastric cancer pathological condition analysis training samples are labeled with a structural region detection box and the corresponding structural type feature encoding, and are also labeled with a Lauren classification label and a pathological stage label. All the labeled gastric cancer pathological condition analysis training samples are combined to form a gastric cancer pathological condition analysis training set;

[0020] The structure segmentation layer is pre-trained with the gastric cancer pathological condition analysis training set. Taking the structural region detection box and the corresponding structural type feature encoding as the target, the first loss value is calculated. It is determined whether the first loss value is within the first preset range. If the first loss value is within the first preset range, the pre-trained structure segmentation layer is output; otherwise, the structure segmentation layer is continuously pre-trained with the gastric cancer pathological condition analysis training set;

[0021] The gastric cancer pathological condition analysis model is trained with a gastric cancer pathological condition analysis training set, using the Lauren classification label and the pathological stage label as multi-task objectives. The Lauren classification loss value corresponding to the Lauren classification label and the pathological stage loss value corresponding to the pathological stage label are calculated respectively. The jointly optimized Lauren classification loss value and pathological stage loss value are used to calculate a second loss value. It is determined whether the second loss value is within a second preset range. If the second loss value is within the second preset range, the pre-trained gastric cancer pathological condition analysis model is output; otherwise, the gastric cancer pathological condition analysis model is continuously trained with the gastric cancer pathological condition analysis training set.

[0022] The present invention also provides a gastric cancer pathological image processing system based on multi-task learning, including:

[0023] A gastric cancer pathological image acquisition module for acquiring the gastric cancer pathological images of patients;

[0024] A gastric cancer pathological condition analysis module for sending the gastric cancer pathological images into the gastric cancer pathological condition analysis model for processing and outputting the corresponding gastric cancer pathological condition report;

[0025] The gastric cancer pathological condition analysis model includes a structure segmentation layer, a shape convolution kernel mask parameter construction layer, a structure feature extraction layer, a graph structure feature extraction layer, a gastric cancer pathological feature fusion layer, a gastric cancer condition multi-task analysis layer, and a gastric cancer pathological condition report output layer. Among them, the structure segmentation layer is used to segment different structures in the gastric cancer pathological image, output several structure region detection frames and corresponding structure type feature encodings, and map the structure region detection frames to the gastric cancer pathological image to generate structure region images; the shape convolution kernel mask parameter construction layer is used to process the structure region detection frames and corresponding structure type feature encodings, and output the shape convolution kernel mask parameters corresponding to the structure region detection frames; the structure feature extraction layer is used to extract features from the structure region images and output corresponding structure features. During the feature extraction process, the convolution kernel is dot-product processed through the shape convolution kernel mask parameters corresponding to the structure region images; the graph structure feature extraction layer is used to splice the structure features and corresponding structure type feature encodings to construct pathological feature nodes, and based on the pathological feature nodes, construct a spatial adjacency graph, a connectivity graph, and a pathological association graph, and respectively perform graph structure feature extraction according to the spatial adjacency graph, the connectivity graph, and the pathological association graph to construct corresponding spatial feature graphs, connectivity feature graphs, and pathological association feature graphs; the gastric cancer pathological feature fusion layer is used to perform weighted fusion on the spatial feature graph, the connectivity feature graph, and the pathological association feature graph to construct an aggregated feature graph, and query the gastric cancer pathological knowledge graph based on the pathological feature nodes to construct a pathological knowledge feature graph, and perform a self-attention mechanism operation based on the aggregated feature graph and the pathological knowledge feature graph to construct a pathological fusion feature graph; the gastric cancer condition multi-task analysis layer includes a Lauren classification unit and a pathological staging unit, which are used to process the pathological fusion feature graph and output corresponding Lauren classification labels and pathological staging labels; the gastric cancer pathological condition report output layer is used to form a gastric cancer pathological condition report and output the Lauren classification labels and pathological staging labels.

[0026] The present invention has the following advantages:

[0027] The gastric cancer pathological condition analysis model set by the present invention segments the gastric cancer pathological image, and regards the different segmented structures as feature nodes to construct a spatial adjacency graph, a connectivity graph, and a pathological association graph, and specifically learns the features in the gastric cancer pathological image from the adjacency relationship, connectivity relationship, and pathological association between the structures, and strengthens the features of different structures and the connections between different structure features in the gastric cancer pathological image through the gastric cancer pathological knowledge graph, thereby improving the analysis ability of the gastric cancer pathological image. Brief Description of the Drawings

[0028] Figure 1 It is a schematic structural diagram of the gastric cancer pathological condition analysis model adopted by the embodiment of the present invention.

[0029] Figure 2It is a schematic structural diagram of a gastric cancer pathological image processing system based on multi-task learning adopted in the embodiments of the present invention. Detailed implementation manners

[0030] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0031] Embodiment 1, a gastric cancer pathological image processing method based on multi-task learning, includes:

[0032] When analyzing the pathological conditions of gastric cancer, generally, the pathological images of gastric cancer of patients are obtained through gastroscopy. Here, the pathological images of gastric cancer can also be processed pictures of gastric tissue samples. Then, the pathological images of gastric cancer are sent to the gastric cancer pathological condition analysis model for processing, and the corresponding gastric cancer pathological condition report is output. Here, the gastric cancer pathological condition report includes the Lauren classification and pathological stage of the gastric cancer pathological image. The Lauren classification includes intestinal-type gastric cancer, diffuse gastric cancer, and mixed gastric cancer, and the pathological stage is the TNM stage;

[0033] See Figure 1 , the gastric cancer pathological condition analysis model includes a structure segmentation layer, a shape convolution kernel mask parameter construction layer, a structure feature extraction layer, a graph structure feature extraction layer, a gastric cancer pathological feature fusion layer, a gastric cancer condition multi-task analysis layer, and a gastric cancer pathological condition report output layer. Each part of the gastric cancer pathological condition analysis model will be described separately:

[0034] The structure segmentation layer is established based on the U-net model and is pre-trained through a segmentation task. It is used to segment different structures in the gastric cancer pathological image. Here, the structures in the gastric cancer pathological image include glandular ducts, tumor cells, and stromal components, etc. These structures have corresponding structural prior associations with the gastric cancer conditions. For example, intestinal-type gastric cancer has a higher density of glandular duct structures, and diffuse gastric cancer usually has more stromal components. It outputs several structure region detection frames and corresponding structure type feature encodings, and maps the structure region detection frames to the gastric cancer pathological image to generate a structure region image. Here, the structure type feature encodings are set in advance by means of word embedding, and corresponding structure type feature encodings are set for glandular ducts, tumor cells, and stromal components;

[0035] The shape convolution kernel mask parameter construction layer is used to process the structural region detection box and the corresponding structural type feature encoding, and output the shape convolution kernel mask parameters corresponding to the structural region detection box. It should be noted that since different structures have different morphological features, such as the number of branches and the branching angle in the glandular duct structure, and the degree of cytoplasmic vacuolization and the tightness of cell nest arrangement in the stromal component, therefore, different-shaped convolution kernels need to be set for different structural regions to adapt to the feature extraction of different structures. Different-shaped convolution kernels are realized through the shape convolution kernel mask parameters. The shape convolution kernel mask parameters store data between 0 and 1, which are used for dot product operations with the convolution kernels in the structural feature extraction layer to control the shape and size of the convolution kernels;

[0036] The structural feature extraction layer is used to extract features from the structural region image and output the corresponding structural features. During the feature extraction process, the convolution kernel is processed by the dot product of the shape convolution kernel mask parameters corresponding to the structural region image, so that the feature extraction process can better adapt to the shape corresponding to the structural region and extract features in a targeted manner;

[0037] The graph structural feature extraction layer is used to splice the structural features and the corresponding structural type feature encoding, construct pathological feature nodes, and construct a spatial adjacency graph, a connectivity graph, and a pathological association graph based on the pathological feature nodes. Graph structural feature extraction is performed respectively according to the spatial adjacency graph, the connectivity graph, and the pathological association graph to construct the corresponding spatial feature graph, connectivity feature graph, and pathological association feature graph; The spatial adjacency graph here refers to the mutual proximity relationship between different structures in the gastric cancer pathological image. If two structures are close to each other, it means there is an interaction between them. For example, the proximity relationship between tumor cells and stromal components will affect the tumor microenvironment. The connectivity graph refers to the fact that different structures in the gastric cancer pathological image are connected to each other or share boundaries. The connectivity relationship reflects the continuity and integrity of the tissue structure. For example, normal glandular ducts are usually continuous structures, while cancerous glandular ducts may show breaks or irregular hyperplasia. The connectivity relationship can capture these structural changes. The pathological association graph represents the association between different structures in terms of pathological relationships. For example, the interaction between tumor cells and stromal components plays an important role in tumor growth, invasion, and metastasis;

[0038] The gastric cancer pathological feature fusion layer is used to perform weighted fusion on the spatial feature graph, the connectivity feature graph, and the pathological association feature graph, construct an aggregated feature graph, and query the gastric cancer pathological knowledge graph based on the pathological feature nodes to construct a pathological knowledge feature graph. An self-attention mechanism operation is performed based on the aggregated feature graph and the pathological knowledge feature graph to construct a pathological fusion feature graph; The gastric cancer pathological knowledge graph is constructed based on professional gastric cancer pathological knowledge. Here, the professional gastric cancer pathological knowledge can be sourced from pathology textbooks, literature, databases, and expert knowledge, etc. The storage form is generally an entity-relationship-entity triple;

[0039] The multi-task analysis layer for gastric cancer conditions includes a Lauren classification unit and a pathological staging unit, which are used to process the pathological fusion feature map and output the corresponding Lauren classification label and pathological staging label;

[0040] The gastric cancer pathological condition report output layer is used to output the gastric cancer pathological condition report by combining the Lauren classification label and the pathological staging label.

[0041] The gastric cancer pathological condition analysis model set in this application segments the gastric cancer pathological image, regards the different segmented structures as feature nodes to construct a spatial adjacency graph, a connectivity graph and a pathological association graph, learns the features in the gastric cancer pathological image specifically from the adjacency relationship, connectivity relationship and pathological association between the structures, and strengthens the features of different structures and the connections between different structure features in the gastric cancer pathological image through the gastric cancer pathological knowledge graph, thereby improving the analysis ability of the gastric cancer pathological image.

[0042] The shape convolution kernel mask parameter construction layer processes the structure region detection box and the corresponding structure type feature encoding, and outputs the shape convolution kernel mask parameter corresponding to the structure region detection box. The specific steps are as follows: extract the length and width of the structure region detection box, splice the length and width of the structure region detection box and the structure type feature encoding corresponding to the structure region detection box to construct the structure region analysis data, and then send the structure region analysis data into the shape convolution kernel mask parameter construction network built in the shape convolution kernel mask parameter construction layer for processing, and output the shape convolution kernel mask parameter corresponding to the structure region detection box. The shape convolution kernel mask parameter construction network is established based on a multi-layer perceptron, and the built-in parameters are adjusted along with the end-to-end training;

[0043] The graph structure feature extraction layer splices the structure feature and the corresponding structure type feature encoding to construct a pathological feature node, and constructs a spatial adjacency graph, a connectivity graph and a pathological association graph based on the pathological feature node. The graph structure features are extracted respectively according to the spatial adjacency graph, the connectivity graph and the pathological association graph to construct the corresponding spatial feature graph, connectivity feature graph and pathological association feature graph. The specific steps are as follows:

[0044] Splice each structure feature and the corresponding structure type feature encoding at the head and tail to construct the corresponding pathological feature node, and splice all the pathological nodes from top to bottom to construct a pathological feature graph;

[0045] Traverse all pathological feature nodes, construct adjacent feature edges between two pathological feature nodes with an adjacency relationship, and determine the way to have an adjacency relationship as follows: construct a Voronoi diagram for all pathological feature nodes, and the pathological feature nodes within the same region in the Voronoi diagram have an adjacency relationship. The construction method uses the SciPy library in the Python library. Based on all pathological feature nodes and all adjacent feature edges, construct a spatial adjacency graph, and based on the spatial adjacency graph, construct a spatial adjacency matrix. The spatial adjacency matrix is of size N×N. The element stored in the i-th row and j-th column is 1, indicating that there is an adjacent feature edge between the i-th pathological feature node and the j-th pathological feature node; the element stored in the i-th row and j-th column is 0, indicating that there is no adjacent feature edge between the i-th pathological feature node and the j-th pathological feature node, where i, j = 1, 2, 3, …, N, and N is the total number of pathological feature nodes;

[0046] Traverse all pathological feature nodes, construct connectivity feature edges between two pathological feature nodes with a connectivity relationship, and determine the way to have a connectivity relationship as follows: if the structural region images corresponding to two pathological feature nodes share boundary pixels, it is regarded as having a connectivity relationship. Based on all pathological feature nodes and all connectivity feature edges, construct a connectivity graph, and based on the connectivity graph, construct a connectivity adjacency matrix. The connectivity adjacency matrix is of size N×N. The element stored in the i-th row and j-th column is 1, indicating that there is a connectivity feature edge between the i-th pathological feature node and the j-th pathological feature node; the element stored in the i-th row and j-th column is 0, indicating that there is no connectivity feature edge between the i-th pathological feature node and the j-th pathological feature node;

[0047] Traverse all pathological feature nodes, construct pathological association feature edges between two pathological feature nodes with a pathological association, and determine the way to have a pathological association as follows: query in the gastric cancer pathological knowledge graph based on the structural type names corresponding to the two pathological feature nodes. The structural type names refer to glandular ducts, tumor cells, and stromal components, etc. If the structural type names corresponding to the two pathological feature nodes have an entity relationship in the gastric cancer pathological knowledge graph, it is regarded as having a pathological association between the two pathological feature nodes. Based on all pathological feature nodes and all pathological association feature edges, construct a pathological association graph, and based on the pathological association graph, construct a pathological association adjacency matrix. The pathological association adjacency matrix is of size N×N. The element stored in the i-th row and j-th column is 1, indicating that there is a pathological association feature edge between the i-th pathological feature node and the j-th pathological feature node; the element stored in the i-th row and j-th column is 0, indicating that there is no pathological association feature edge between the i-th pathological feature node and the j-th pathological feature node;

[0048] Perform graph convolution operations on the pathological feature map based on the spatial adjacency matrix, connectivity adjacency matrix, and pathological association adjacency matrix respectively to construct the corresponding spatial feature map, connectivity feature map, and pathological association feature map. The graph convolution operation here refers to the GCN model to achieve message passing and feature learning of feature information, and further realize the fusion of feature information corresponding to adjacency relationships, connectivity relationships, and pathological associations respectively;

[0049] Through the gastric cancer pathological feature fusion layer, the spatial feature map, connectivity feature map, and pathological association feature map are weighted and fused to construct an aggregated feature map, and the gastric cancer pathological knowledge graph is queried based on the pathological feature nodes to construct a pathological knowledge feature map. An self-attention mechanism operation is performed based on the aggregated feature map and the pathological knowledge feature map to construct a pathological fusion feature map, which specifically includes the following steps:

[0050] Send the spatial feature map, connectivity feature map, and pathological association feature map into the weight adjustment network for processing, and output the corresponding feature weights. The weight adjustment network here is also established based on a multi-layer perceptron, and the built-in parameters are adjusted with end-to-end training. The spatial feature map, connectivity feature map, and pathological association feature map and the corresponding feature weights are weighted and fused to construct an aggregated feature map;

[0051] For each pathological feature node, query in the gastric cancer pathological knowledge graph based on the structural type name corresponding to the pathological feature node. If the structural type names corresponding to any two pathological feature nodes have an entity relationship in the gastric cancer pathological knowledge graph, construct a pathological knowledge vector in the form of pathological feature node - entity relationship - pathological feature node, and splice all the pathological knowledge vectors from top to bottom to form a pathological knowledge feature map. Based on the aggregated feature map, construct the corresponding value vector V and key vector K, and based on the pathological knowledge feature map, construct the corresponding query vector Q. The construction of the value vector V, key vector K, and query vector Q refers to the self-attention mechanism in the Transformer model. The specific method is: perform matrix multiplication operations on the aggregated feature map with the value weight matrix and the key weight matrix respectively to construct the corresponding value vector V and key vector K, perform matrix multiplication operation on the pathological knowledge feature map with the query weight matrix to construct the corresponding query vector Q. The value weight matrix, key weight matrix, and query weight matrix here are adjusted with end-to-end training, and calculate the self-attention matrix ATT = softmax(QK T / D 0.5 )), T is the matrix transpose operation, D is the dimension size of the key vector K, and then perform matrix multiplication operation on the value vector V and the self-attention matrix ATT to obtain the pathological fusion feature map; The pathological knowledge graph can provide prior professional gastric cancer pathological knowledge to guide the gastric cancer pathological situation analysis model to be more in line with the doctor's pathological reasoning;

[0052] The gastric cancer pathology knowledge graph is constructed as follows: collect and organize knowledge related to gastric cancer pathology, determine entities and entity relationships through named entity recognition technology and relationship extraction technology, construct triple data based on the form of entity-entity relationship-entity, and then form the gastric cancer pathology knowledge graph with all the triple data;

[0053] Train the gastric cancer pathology situation analysis model, which specifically includes the following steps:

[0054] Obtain several gastric cancer pathology situation analysis training samples. The gastric cancer pathology situation analysis training samples include gastric cancer pathology images, annotate the gastric cancer pathology situation analysis training samples with structural region detection frames and corresponding structural type feature codes, and annotate the gastric cancer pathology situation analysis training samples with Lauren classification labels and pathological stage labels. Combine all the annotated gastric cancer pathology situation analysis training samples to form a gastric cancer pathology situation analysis training set;

[0055] Pre-train the structure segmentation layer with the gastric cancer pathology situation analysis training set. Taking the structural region detection frame and the corresponding structural type feature code as the target, calculate the first loss value, and judge whether the first loss value is within the first preset range, which is determined by the developer. If the first loss value is within the first preset range, output the pre-trained structure segmentation layer; otherwise, continue to pre-train the structure segmentation layer with the gastric cancer pathology situation analysis training set;

[0056] Train the gastric cancer pathology situation analysis model with the gastric cancer pathology situation analysis training set. Taking the Lauren classification label and the pathological stage label as multi-task targets, calculate the Lauren classification loss value corresponding to the Lauren classification label and the pathological stage loss value corresponding to the pathological stage label respectively. Combine and optimize the Lauren classification loss value and the pathological stage loss value, calculate the second loss value, and judge whether the second loss value is within the second preset range, which is determined by the developer. If the second loss value is within the second preset range, output the pre-trained gastric cancer pathology situation analysis model; otherwise, continue to train the gastric cancer pathology situation analysis model with the gastric cancer pathology situation analysis training set.

[0057] Example 2, a gastric cancer pathology image processing system based on multi-task learning, see Figure 2 , including:

[0058] A gastric cancer pathology image acquisition module, which is used to obtain the gastric cancer pathology image of the patient through gastroscopy when analyzing the gastric cancer pathology situation. Here, the gastric cancer pathology image can also be a processed picture of the gastric tissue sample;

[0059] The gastric cancer pathological condition analysis module is used to send gastric cancer pathological images into the gastric cancer pathological condition analysis model for processing, and output the corresponding gastric cancer pathological condition report. Here, the gastric cancer pathological condition report includes the Lauren classification and pathological stage of the gastric cancer pathological image. The Lauren classification includes intestinal-type gastric cancer, diffuse gastric cancer, and mixed gastric cancer, and the pathological stage is the TNM stage;

[0060] The gastric cancer pathological condition analysis model includes a structure segmentation layer, a shape convolution kernel mask parameter construction layer, a structure feature extraction layer, a graph structure feature extraction layer, a gastric cancer pathological feature fusion layer, a gastric cancer condition multi-task analysis layer, and a gastric cancer pathological condition report output layer. Among them, the structure segmentation layer is used to segment different structures in the gastric cancer pathological image, output several structure region detection frames and corresponding structure type feature encodings, and map the structure region detection frames to the gastric cancer pathological image to generate structure region images; the shape convolution kernel mask parameter construction layer is used to process the structure region detection frames and corresponding structure type feature encodings, and output the shape convolution kernel mask parameters corresponding to the structure region detection frames; the structure feature extraction layer is used to extract features from the structure region images and output the corresponding structure features. During the feature extraction process, the convolution kernel is dot-product processed by the shape convolution kernel mask parameters corresponding to the structure region images; the graph structure feature extraction layer is used to splice the structure features and corresponding structure type feature encodings to construct pathological feature nodes, and based on the pathological feature nodes, construct a spatial adjacency graph, a connectivity graph, and a pathological association graph, and respectively perform graph structure feature extraction according to the spatial adjacency graph, the connectivity graph, and the pathological association graph to construct corresponding spatial feature graphs, connectivity feature graphs, and pathological association feature graphs; the gastric cancer pathological feature fusion layer is used to perform weighted fusion on the spatial feature graph, the connectivity feature graph, and the pathological association feature graph to construct an aggregated feature graph, and query the gastric cancer pathological knowledge graph based on the pathological feature nodes to construct a pathological knowledge feature graph, and perform a self-attention mechanism operation based on the aggregated feature graph and the pathological knowledge feature graph to construct a pathological fusion feature graph; the gastric cancer condition multi-task analysis layer includes a Lauren classification unit and a pathological stage unit, which are used to process the pathological fusion feature graph and output the corresponding Lauren classification label and pathological stage label; the gastric cancer pathological condition report output layer is used to form a gastric cancer pathological condition report by combining the Lauren classification label and the pathological stage label and output it.

[0061] It should be understood that for those of ordinary skill in the art, improvements or transformations can be made according to the above description, and all such improvements and transformations should fall within the protection scope of the appended claims of the present invention. The parts not described in detail in this specification belong to the prior art well-known to those skilled in the art.

Claims

1. A gastric cancer pathological image processing method based on multi-task learning, characterized in that Including: Obtain the gastric cancer pathological image of the patient, and then send the gastric cancer pathological image into the gastric cancer pathological condition analysis model for processing, and output the corresponding gastric cancer pathological condition report; Set up a structure segmentation layer, a shape convolution kernel mask parameter construction layer, a graph structure feature extraction layer, and a gastric cancer pathological feature fusion layer in the gastric cancer pathological condition analysis model. The structure segmentation layer segments different structures in the gastric cancer pathological image. The shape convolution kernel mask parameter construction layer sets corresponding convolution kernels for the segmented different structures for subsequent feature extraction. The graph structure feature extraction layer extracts features based on the adjacency relationship, connectivity relationship, and pathological association between different structures. The gastric cancer pathological feature fusion layer strengthens the features of different structures in the gastric cancer pathological image and the connection between different structure features based on the gastric cancer pathological knowledge graph; The gastric cancer pathological condition analysis model includes a structure segmentation layer, a shape convolution kernel mask parameter construction layer, a structure feature extraction layer, a graph structure feature extraction layer, a gastric cancer pathological feature fusion layer, a gastric cancer condition multi-task analysis layer, and a gastric cancer pathological condition report output layer. Among them, the structure segmentation layer is used to segment different structures in the gastric cancer pathological image, output several structure region detection frames and corresponding structure type feature encodings, and map the structure region detection frames to the gastric cancer pathological image to generate structure region images; the shape convolution kernel mask parameter construction layer is used to process the structure region detection frames and corresponding structure type feature encodings, and output the shape convolution kernel mask parameters corresponding to the structure region detection frames; The structure feature extraction layer is used to extract features from the structure region image and output the corresponding structure features. During the feature extraction process, the convolution kernel is dot-product processed through the shape convolution kernel mask parameter corresponding to the structure region image; The graph structure feature extraction layer is used to splice the structure features and the corresponding structure type feature encodings to construct pathological feature nodes, and construct a spatial adjacency graph, a connectivity graph, and a pathological association graph based on the pathological feature nodes. Graph structure feature extraction is performed according to the spatial adjacency graph, the connectivity graph, and the pathological association graph respectively to construct corresponding spatial feature graphs, connectivity feature graphs, and pathological association feature graphs; the gastric cancer pathological feature fusion layer is used to perform weighted fusion on the spatial feature graph, the connectivity feature graph, and the pathological association feature graph to construct an aggregated feature graph, and query the gastric cancer pathological knowledge graph based on the pathological feature nodes to construct a pathological knowledge feature graph, and perform a self-attention mechanism operation based on the aggregated feature graph and the pathological knowledge feature graph to construct a pathological fusion feature graph; the gastric cancer condition multi-task analysis layer includes a Lauren classification unit and a pathological staging unit, which are used to process the pathological fusion feature graph and output the corresponding Lauren classification label and pathological staging label; the gastric cancer pathological condition report output layer is used to form a gastric cancer pathological condition report with the Lauren classification label and the pathological staging label and output it.

2. The gastric cancer pathological image processing method based on multi-task learning according to claim 1, wherein, Construct the shape convolution kernel mask parameters corresponding to the layer output structure region detection boxes through the shape convolution kernel mask parameter construction layer, which specifically includes the following steps: Extract the length and width of the structure region detection boxes, splice the length and width of the structure region detection boxes and the structure type feature encoding corresponding to the structure region detection boxes to construct structure region analysis data, and then send the structure region analysis data into the shape convolution kernel mask parameter construction network built in the shape convolution kernel mask parameter construction layer for processing, and output the shape convolution kernel mask parameters corresponding to the structure region detection boxes.

3. The gastric cancer pathological image processing method based on multi-task learning according to claim 2, wherein, Construct the corresponding spatial feature map, connectivity feature map and pathological association feature map through the graph structure feature extraction layer, which specifically includes the following steps: Splice each structure feature and the corresponding structure type feature encoding to construct the corresponding pathological feature nodes, and splice all the pathological nodes from top to bottom to construct a pathological feature map; Traverse all the pathological feature nodes, construct adjacency feature edges between two pathological feature nodes with an adjacency relationship, and determine the way to have an adjacency relationship as follows: Construct a Voronoi diagram for all the pathological feature nodes, and the pathological feature nodes in the same region in the Voronoi diagram have an adjacency relationship. Based on all the pathological feature nodes and all the adjacency feature edges, construct a spatial adjacency graph, and construct a spatial adjacency matrix based on the spatial adjacency graph; Traverse all the pathological feature nodes, construct connectivity feature edges between two pathological feature nodes with a connectivity relationship, and determine the way to have a connectivity relationship as follows: The structure region images corresponding to two pathological feature nodes share boundary pixels, which is regarded as having a connectivity relationship. Based on all the pathological feature nodes and all the connectivity feature edges, construct a connectivity graph, and construct a connectivity adjacency matrix based on the connectivity graph; Traverse all the pathological feature nodes, construct pathological association feature edges between two pathological feature nodes with a pathological association, and determine the way to have a pathological association as follows: Query in the gastric cancer pathological knowledge graph based on the structure type names corresponding to the two pathological feature nodes. If the structure type names corresponding to the two pathological feature nodes have an entity relationship in the gastric cancer pathological knowledge graph, it is regarded as having a pathological association between the two pathological feature nodes. Based on all the pathological feature nodes and all the pathological association feature edges, construct a pathological association graph, and construct a pathological association adjacency matrix based on the pathological association graph; Perform graph convolution operations on the pathological feature map respectively based on the spatial adjacency matrix, connectivity adjacency matrix and pathological association adjacency matrix to construct the corresponding spatial feature map, connectivity feature map and pathological association feature map.

4. The gastric cancer pathological image processing method based on multi-task learning according to claim 3, wherein, Construct a pathological fusion feature map through the gastric cancer pathological feature fusion layer, which specifically includes the following steps: Send the spatial feature map, connectivity feature map and pathological association feature map into the weight adjustment network for processing, output the corresponding feature weights, and perform weighted fusion on the spatial feature map, connectivity feature map and pathological association feature map and the corresponding feature weights to construct an aggregated feature map; For each pathological feature node, query in the gastric cancer pathological knowledge graph based on the structure type name corresponding to the pathological feature node. If there is an entity relationship between the structure type names corresponding to any two pathological feature nodes in the gastric cancer pathological knowledge graph, construct a pathological knowledge vector in the form of pathological feature node - entity relationship - pathological feature node, and splice all the pathological knowledge vectors from top to bottom to form a pathological knowledge feature map. Perform a self-attention mechanism operation based on the aggregated feature map and the pathological knowledge feature map to obtain a pathological fusion feature map.

5. A gastric cancer pathological image processing method based on multi-task learning according to claim 4, characterized in that Train the gastric cancer pathological condition analysis model, which specifically includes the following steps: Obtain a number of gastric cancer pathological condition analysis training samples, where the gastric cancer pathological condition analysis training samples include gastric cancer pathological images. Label the gastric cancer pathological condition analysis training samples through the structural region detection frame and the corresponding structural type feature encoding, and label the gastric cancer pathological condition analysis training samples through the Lauren classification label and the pathological stage label. Combine all the labeled gastric cancer pathological condition analysis training samples to form a gastric cancer pathological condition analysis training set; Pre-train the structure segmentation layer through the gastric cancer pathological condition analysis training set, with the structural region detection frame and the corresponding structural type feature encoding as the targets; Train the gastric cancer pathological condition analysis model through the gastric cancer pathological condition analysis training set, using the Lauren classification label and the pathological stage label as multi-task targets.

6. A gastric cancer pathological image processing system based on multi-task learning, characterized in that, The system applies the method for processing gastric cancer pathological images based on multi-task learning described in any one of claims 1-5 above, including: A gastric cancer pathological image acquisition module, configured to acquire the gastric cancer pathological images of a patient; A gastric cancer pathological condition analysis module, configured to send the gastric cancer pathological images into the gastric cancer pathological condition analysis model for processing and output the corresponding gastric cancer pathological condition report.

Images