Gastric cancer pathological image processing method and system based on multi-task learning

Abstract

The invention relates to the technical field of medical data processing, in particular to a gastric cancer pathological image processing method and system based on multi-task learning. A stomach cancer pathological image processing system based on multi-task learning comprises a stomach cancer pathological image acquisition module and a stomach cancer pathological condition analysis module. According to the gastric cancer pathological condition analysis model provided by the invention, the gastric cancer pathological image is segmented, and the segmented different structures are regarded as feature nodes to construct the spatial adjacency graph, the connectivity graph and the pathological association graph, so that the features in the gastric cancer pathological image are learned in a targeted manner from the adjacency relationship, the connectivity relationship and the pathological association between the structures; and through the gastric cancer pathology knowledge graph, the features of different structures in the gastric cancer pathology image and the relation between the features of the different structures are enhanced, so that the analysis ability of the gastric cancer pathology image is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical data processing, and particularly relates to a gastric cancer pathological image processing method and system based on multi-task learning. Background Art

[0002] The analysis of gastric cancer pathological images is a key link in disease diagnosis and precise treatment. However, gastric cancer pathological images themselves have extremely high heterogeneity, which is reflected in multiple levels such as tissue morphology, cell structure, and staining patterns. This heterogeneity stems from the complex biological mechanisms of tumor occurrence and development, posing great challenges to traditional machine learning models. Specifically, in the face of the complexity and heterogeneity of gastric cancer pathological images, traditional machine learning models often exhibit problems such as insufficient feature representation ability, limited analysis accuracy, and poor robustness in key tasks such as tumor segmentation and typing and grading, severely restricting the clinical application value of computer-aided gastric cancer pathological diagnosis. There is an urgent need for more advanced methods to improve the model's analysis ability for highly heterogeneous gastric cancer pathological images. Summary of the Invention

[0003] The gastric cancer pathological condition analysis model set up in the present invention segments gastric cancer pathological images, and regards the different segmented structures as feature nodes to construct a spatial adjacency graph, a connectivity graph, and a pathological association graph, and specifically learns the features in gastric cancer pathological images from the adjacency relationship, connectivity relationship, and pathological association between the structures. Moreover, through the gastric cancer pathological knowledge graph, the features of different structures in gastric cancer pathological images and the connections between different structural features are strengthened, thereby improving the analysis ability for gastric cancer pathological images.

[0004] The present invention provides a gastric cancer pathological image processing method based on multi-task learning, including: Obtain the gastric cancer pathological image of a patient, and then send the gastric cancer pathological image into the gastric cancer pathological condition analysis model for processing to output a corresponding gastric cancer pathological condition report; Set a structure segmentation layer, a shape convolution kernel mask parameter construction layer, a graph structure feature extraction layer, and a gastric cancer pathological feature fusion layer in the gastric cancer pathological condition analysis model. The structure segmentation layer segments different structures in the gastric cancer pathological image. The shape convolution kernel mask parameter construction layer sets corresponding convolution kernels for the segmented different structures for subsequent feature extraction. The graph structure feature extraction layer extracts features based on the adjacency relationship, connectivity relationship, and pathological association between different structures. The gastric cancer pathological feature fusion layer strengthens the features of different structures in the gastric cancer pathological image and the connections between different structural features based on the gastric cancer pathological knowledge graph.

[0005] As a preferred aspect, the gastric cancer pathological condition analysis model includes a structure segmentation layer, a shape convolution kernel mask parameter construction layer, a structure feature extraction layer, a graph structure feature extraction layer, a gastric cancer pathological feature fusion layer, a gastric cancer condition multi-task analysis layer, and a gastric cancer pathological condition report output layer. The structure segmentation layer is used to segment different structures in the gastric cancer pathological image, output a number of structure region detection frames and corresponding structure type feature encodings, and map the structure region detection frames to the gastric cancer pathological image to generate structure region images; the shape convolution kernel mask parameter construction layer is used to process the structure region detection frames and corresponding structure type feature encodings, and output the shape convolution kernel mask parameters corresponding to the structure region detection frames; the structure feature extraction layer is used to extract features from the structure region images and output corresponding structure features. During the feature extraction process, the convolution kernel is dot-product processed by the shape convolution kernel mask parameters corresponding to the structure region images; the graph structure feature extraction layer is used to splice the structure features and corresponding structure type feature encodings to construct pathological feature nodes, and based on the pathological feature nodes, construct a spatial adjacency graph, a connectivity graph, and a pathological association graph, and perform graph structure feature extraction according to the spatial adjacency graph, the connectivity graph, and the pathological association graph respectively to construct corresponding spatial feature graphs, connectivity feature graphs, and pathological association feature graphs; the gastric cancer pathological feature fusion layer is used to perform weighted fusion on the spatial feature graph, the connectivity feature graph, and the pathological association feature graph to construct an aggregated feature graph, and query the gastric cancer pathological knowledge graph based on the pathological feature nodes to construct a pathological knowledge feature graph, and perform a self-attention mechanism operation based on the aggregated feature graph and the pathological knowledge feature graph to construct a pathological fusion feature graph; the gastric cancer condition multi-task analysis layer includes a Lauren classification unit and a pathological staging unit, which are used to process the pathological fusion feature graph and output corresponding Lauren classification labels and pathological staging labels; the gastric cancer pathological condition report output layer is used to form a gastric cancer pathological condition report and output it with the Lauren classification labels and pathological staging labels.

[0006] As a preferred aspect, the shape convolution kernel mask parameter construction layer processes the structure region detection frames and corresponding structure type feature encodings to output the shape convolution kernel mask parameters corresponding to the structure region detection frames, specifically including the following steps: extracting the length and width of the structure region detection frames, splicing the length and width of the structure region detection frames and the structure type feature encodings corresponding to the structure region detection frames to construct structure region analysis data, and then sending the structure region analysis data into the shape convolution kernel mask parameter construction network built in the shape convolution kernel mask parameter construction layer for processing to output the shape convolution kernel mask parameters corresponding to the structure region detection frames.

[0007] As a preferred aspect, the structural features and the corresponding structural type features are encoded and spliced through the graph structure feature extraction layer to construct pathological feature nodes, and a spatial adjacency graph, a connectivity graph, and a pathological association graph are constructed based on the pathological feature nodes. Graph structure feature extraction is performed respectively according to the spatial adjacency graph, the connectivity graph, and the pathological association graph to construct corresponding spatial feature graphs, connectivity feature graphs, and pathological association feature graphs, which specifically include the following steps: Splice each structural feature and the corresponding structural type feature encoding to construct the corresponding pathological feature node, and splice all pathological nodes from top to bottom to construct a pathological feature graph; Traverse all pathological feature nodes, construct adjacent feature edges between two pathological feature nodes with an adjacency relationship. The way to determine the adjacency relationship is as follows: construct a Voronoi diagram for all pathological feature nodes, and the pathological feature nodes in the same region in the Voronoi diagram have an adjacency relationship. Construct a spatial adjacency graph based on all pathological feature nodes and all adjacent feature edges, and construct a spatial adjacency matrix based on the spatial adjacency graph; Traverse all pathological feature nodes, construct connectivity feature edges between two pathological feature nodes with a connectivity relationship. The way to determine the connectivity relationship is as follows: the structural region images corresponding to two pathological feature nodes share boundary pixels, which is regarded as having a connectivity relationship. Construct a connectivity graph based on all pathological feature nodes and all connectivity feature edges, and construct a connectivity adjacency matrix based on the connectivity graph; Traverse all pathological feature nodes, construct pathological association feature edges between two pathological feature nodes with a pathological association. The way to determine the pathological association is as follows: query in the gastric cancer pathological knowledge graph based on the structural type names corresponding to the two pathological feature nodes. If the structural type names corresponding to the two pathological feature nodes have an entity relationship in the gastric cancer pathological knowledge graph, it is regarded as having a pathological association between the two pathological feature nodes. Construct a pathological association graph based on all pathological feature nodes and all pathological association feature edges, and construct a pathological association adjacency matrix based on the pathological association graph; Perform graph convolution operations on the pathological feature graph respectively based on the spatial adjacency matrix, the connectivity adjacency matrix, and the pathological association adjacency matrix to construct corresponding spatial feature graphs, connectivity feature graphs, and pathological association feature graphs.

[0008] As a preferred aspect, the spatial feature graph, the connectivity feature graph, and the pathological association feature graph are weighted and fused through the gastric cancer pathological feature fusion layer to construct an aggregated feature graph, and the gastric cancer pathological knowledge graph is queried based on the pathological feature nodes to construct a pathological knowledge feature graph. Perform a self-attention mechanism operation based on the aggregated feature graph and the pathological knowledge feature graph to construct a pathological fusion feature graph, which specifically includes the following steps: The spatial feature map, connectivity feature map, and pathological correlation feature map are fed into the weight adjustment network for processing, and the corresponding feature weights are output. The spatial feature map, connectivity feature map, pathological correlation feature map, and the corresponding feature weights are weighted and fused to construct an aggregated feature map; For each pathological feature node, query in the gastric cancer pathological knowledge graph based on the structural type name corresponding to the pathological feature node. If the structural type names corresponding to any two pathological feature nodes have an entity relationship in the gastric cancer pathological knowledge graph, construct a pathological knowledge vector in the form of pathological feature node - entity relationship - pathological feature node, and splice all the pathological knowledge vectors from top to bottom to form a pathological knowledge feature map. Based on the aggregated feature map, construct the corresponding value vector V and key vector K, and based on the pathological knowledge feature map, construct the corresponding query vector Q. Calculate the self-attention matrix ATT = softmax(QK T / D 0.5 ), where T is the matrix transpose operation and D is the dimension size of the key vector K. Then, perform a matrix multiplication operation on the value vector V and the self-attention matrix ATT to obtain the pathological fusion feature map.

[0009] As a preferred aspect, train the gastric cancer pathological condition analysis model, which specifically includes the following steps: Obtain a number of gastric cancer pathological condition analysis training samples, which include gastric cancer pathological images. Label the gastric cancer pathological condition analysis training samples through the structural region detection frame and the corresponding structural type feature encoding, and label the gastric cancer pathological condition analysis training samples through the Lauren classification label and pathological stage label. Combine all the labeled gastric cancer pathological condition analysis training samples to form a gastric cancer pathological condition analysis training set; Pre-train the structure segmentation layer through the gastric cancer pathological condition analysis training set. Taking the structural region detection frame and the corresponding structural type feature encoding as the target, calculate the first loss value, and determine whether the first loss value is within the first preset range. If the first loss value is within the first preset range, output the pre-trained structure segmentation layer; otherwise, continue to pre-train the structure segmentation layer through the gastric cancer pathological condition analysis training set; The gastric cancer pathological condition analysis model is trained using the gastric cancer pathological condition analysis training set, with the Lauren classification label and the pathological stage label as multi-task objectives. The Lauren classification loss value corresponding to the Lauren classification label and the pathological stage loss value corresponding to the pathological stage label are calculated respectively. The jointly optimized Lauren classification loss value and pathological stage loss value are used to calculate the second loss value. It is determined whether the second loss value is within the second preset range. If the second loss value is within the second preset range, the pre-trained gastric cancer pathological condition analysis model is output; otherwise, the gastric cancer pathological condition analysis model is continuously trained using the gastric cancer pathological condition analysis training set.

[0010] The present invention also provides a gastric cancer pathological image processing system based on multi-task learning, including: A gastric cancer pathological image acquisition module for acquiring the gastric cancer pathological image of a patient; A gastric cancer pathological condition analysis module for sending the gastric cancer pathological image into the gastric cancer pathological condition analysis model for processing and outputting the corresponding gastric cancer pathological condition report; The gastric cancer pathological condition analysis model includes a structure segmentation layer, a shape convolution kernel mask parameter construction layer, a structure feature extraction layer, a graph structure feature extraction layer, a gastric cancer pathological feature fusion layer, a gastric cancer condition multi-task analysis layer, and a gastric cancer pathological condition report output layer. Among them, the structure segmentation layer is used to segment different structures in the gastric cancer pathological image, output several structure region detection frames and corresponding structure type feature encodings, and map the structure region detection frames to the gastric cancer pathological image to generate structure region images; the shape convolution kernel mask parameter construction layer is used to process the structure region detection frames and corresponding structure type feature encodings, and output the shape convolution kernel mask parameters corresponding to the structure region detection frames; the structure feature extraction layer is used to extract features from the structure region images and output corresponding structure features. During the feature extraction process, the convolution kernel is subjected to dot product processing through the shape convolution kernel mask parameters corresponding to the structure region images; the graph structure feature extraction layer is used to splice the structure features and corresponding structure type feature encodings to construct pathological feature nodes, and based on the pathological feature nodes, construct a spatial adjacency graph, a connectivity graph, and a pathological association graph, and perform graph structure feature extraction according to the spatial adjacency graph, the connectivity graph, and the pathological association graph respectively to construct corresponding spatial feature graphs, connectivity feature graphs, and pathological association feature graphs; the gastric cancer pathological feature fusion layer is used to perform weighted fusion on the spatial feature graph, the connectivity feature graph, and the pathological association feature graph to construct an aggregated feature graph, and query the gastric cancer pathological knowledge graph based on the pathological feature nodes to construct a pathological knowledge feature graph, and perform a self-attention mechanism operation based on the aggregated feature graph and the pathological knowledge feature graph to construct a pathological fusion feature graph; the gastric cancer condition multi-task analysis layer includes a Lauren classification unit and a pathological staging unit, which are used to process the pathological fusion feature graph and output corresponding Lauren classification labels and pathological staging labels; the gastric cancer pathological condition report output layer is used to form a gastric cancer pathological condition report and output the Lauren classification labels and pathological staging labels.

[0011] The present invention has the following advantages: The gastric cancer pathological condition analysis model set by the present invention segments the gastric cancer pathological image, and regards the different segmented structures as feature nodes to construct a spatial adjacency graph, a connectivity graph, and a pathological association graph, and learns the features in the gastric cancer pathological image specifically from the adjacency relationship, connectivity relationship, and pathological association between the structures. Moreover, through the gastric cancer pathological knowledge graph, the features of different structures in the gastric cancer pathological image and the connections between different structure features are strengthened, thereby improving the analysis ability of the gastric cancer pathological image. Description of the Drawings

[0012] Figure 1 It is a schematic structural diagram of the gastric cancer pathological condition analysis model adopted by the embodiment of the present invention.

[0013] Figure 2Schematic diagram of the gastric cancer pathological image processing system based on multi-task learning adopted in the embodiments of the present invention. Detailed implementation manners

[0014] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0015] Embodiment 1, a gastric cancer pathological image processing method based on multi-task learning, includes: When analyzing the pathological conditions of gastric cancer, generally, the gastric cancer pathological images of patients are obtained through gastroscopy. Here, the gastric cancer pathological images can also be processed pictures of gastric tissue samples. Then, the gastric cancer pathological images are sent to the gastric cancer pathological condition analysis model for processing, and the corresponding gastric cancer pathological condition reports are output. Here, the gastric cancer pathological condition reports include the Lauren classification and pathological stage of the gastric cancer pathological images. The Lauren classification includes intestinal-type gastric cancer, diffuse gastric cancer, and mixed gastric cancer, and the pathological stage is the TNM stage; See Figure 1 , the gastric cancer pathological condition analysis model includes a structure segmentation layer, a shape convolution kernel mask parameter construction layer, a structure feature extraction layer, a graph structure feature extraction layer, a gastric cancer pathological feature fusion layer, a gastric cancer condition multi-task analysis layer, and a gastric cancer pathological condition report output layer. Each part of the gastric cancer pathological condition analysis model will be described separately: The structure segmentation layer is established based on the U-net model and is pre-trained through a segmentation task. It is used to segment different structures in the gastric cancer pathological images. Here, the structures in the gastric cancer pathological images include glandular ducts, tumor cells, and stromal components, etc. These structures have corresponding structure prior associations with the gastric cancer conditions. For example, intestinal-type gastric cancer has a higher density of glandular duct structures, and diffuse gastric cancer usually has more stromal components. It outputs several structure region detection frames and corresponding structure type feature encodings, and maps the structure region detection frames to the gastric cancer pathological images to generate structure region images. Here, the structure type feature encodings are set in advance by means of word embedding, and corresponding structure type feature encodings are set for glandular ducts, tumor cells, and stromal components; The shape convolution kernel mask parameter construction layer is used to process the structural region detection box and the corresponding structural type feature encoding, and output the shape convolution kernel mask parameters corresponding to the structural region detection box. It should be noted that since different structures have different morphological features, such as the number of branches and branch angles in glandular duct structures, and the degree of cytoplasmic vacuolization and the tightness of cell nest arrangement in stromal components, different shapes of convolution kernels need to be set for different structural regions to adapt to the feature extraction of different structures. Different shapes of convolution kernels are realized through shape convolution kernel mask parameters. The shape convolution kernel mask parameters store data between 0 and 1, which are used for dot product operations with the convolution kernels in the structural feature extraction layer to control the shape and size of the convolution kernels; The structural feature extraction layer is used to extract features from the structural region image and output the corresponding structural features. During the feature extraction process, dot product processing is performed on the convolution kernel through the shape convolution kernel mask parameters corresponding to the structural region image, so that the feature extraction process can better adapt to the shape corresponding to the structural region and extract features in a targeted manner; The graph structure feature extraction layer is used to splice the structural features and the corresponding structural type feature encoding, construct pathological feature nodes, and construct a spatial adjacency graph, a connectivity graph, and a pathological association graph based on the pathological feature nodes. Graph structure feature extraction is performed respectively according to the spatial adjacency graph, the connectivity graph, and the pathological association graph to construct the corresponding spatial feature graph, connectivity feature graph, and pathological association feature graph; The spatial adjacency graph here refers to the mutual proximity relationship between different structures in gastric cancer pathological images. If two structures are close to each other, it means there is an interaction between them. For example, the proximity relationship between tumor cells and stromal components will affect the tumor microenvironment. The connectivity graph refers to the fact that different structures in gastric cancer pathological images are connected to each other or share boundaries. The connectivity relationship reflects the continuity and integrity of tissue structure. For example, normal glandular ducts are usually continuous structures, while cancerous glandular ducts may show breaks or irregular hyperplasia. The connectivity relationship can capture these structural changes. The pathological association graph represents the association between different structures in pathological relationships. For example, the interaction between tumor cells and stromal components plays an important role in tumor growth, invasion, and metastasis; The gastric cancer pathological feature fusion layer is used to perform weighted fusion on the spatial feature graph, the connectivity feature graph, and the pathological association feature graph to construct an aggregated feature graph, and query the gastric cancer pathological knowledge graph based on the pathological feature nodes to construct a pathological knowledge feature graph. An self-attention mechanism operation is performed based on the aggregated feature graph and the pathological knowledge feature graph to construct a pathological fusion feature graph; The gastric cancer pathological knowledge graph is constructed based on professional gastric cancer pathological knowledge. Here, the professional gastric cancer pathological knowledge can be sourced from pathology textbooks, literature, databases, and expert knowledge, etc. The storage form is generally entity-relationship-entity triples; The multi-task analysis layer for gastric cancer conditions includes a Lauren classification unit and a pathological staging unit, which are used to process the pathological fusion feature map and output the corresponding Lauren classification label and pathological staging label; The gastric cancer pathological condition report output layer is used to output the gastric cancer pathological condition report by combining the Lauren classification label and the pathological staging label.

[0016] The gastric cancer pathological condition analysis model set in this application segments gastric cancer pathological images, regards different segmented structures as feature nodes to construct a spatial adjacency graph, a connectivity graph, and a pathological association graph, learns the features in gastric cancer pathological images specifically from the adjacency relationship, connectivity relationship, and pathological association between structures, and strengthens the features of different structures and the connections between different structure features in gastric cancer pathological images through a gastric cancer pathological knowledge graph, thereby improving the analysis ability of gastric cancer pathological images.

[0017] The shape convolution kernel mask parameter construction layer processes the structure region detection box and the corresponding structure type feature encoding, and outputs the shape convolution kernel mask parameter corresponding to the structure region detection box. The specific steps are as follows: extract the length and width of the structure region detection box, splice the length and width of the structure region detection box and the structure type feature encoding corresponding to the structure region detection box to construct structure region analysis data, and then send the structure region analysis data into the shape convolution kernel mask parameter construction network built in the shape convolution kernel mask parameter construction layer for processing, and output the shape convolution kernel mask parameter corresponding to the structure region detection box. The shape convolution kernel mask parameter construction network is established based on a multi-layer perceptron, and the built-in parameters are adjusted along with the end-to-end training; The graph structure feature extraction layer splices the structure feature and the corresponding structure type feature encoding to construct pathological feature nodes, and constructs a spatial adjacency graph, a connectivity graph, and a pathological association graph based on the pathological feature nodes. Graph structure feature extraction is performed respectively according to the spatial adjacency graph, the connectivity graph, and the pathological association graph to construct corresponding spatial feature maps, connectivity feature maps, and pathological association feature maps. The specific steps are as follows: Splice each structure feature and the corresponding structure type feature encoding at the head and tail to construct the corresponding pathological feature node, and splice all pathological nodes from top to bottom to construct a pathological feature map; Traverse all pathological feature nodes, construct adjacent feature edges between two pathological feature nodes with an adjacency relationship, and determine the way to have an adjacency relationship as follows: construct a Voronoi diagram for all pathological feature nodes, and the pathological feature nodes within the same region in the Voronoi diagram have an adjacency relationship. The construction method uses the SciPy library in the Python library. Based on all pathological feature nodes and all adjacent feature edges, construct a spatial adjacency graph, and based on the spatial adjacency graph, construct a spatial adjacency matrix. The spatial adjacency matrix is of size N×N. The element stored in the i-th row and j-th column is 1, indicating that there is an adjacent feature edge between the i-th pathological feature node and the j-th pathological feature node; the element stored in the i-th row and j-th column is 0, indicating that there is no adjacent feature edge between the i-th pathological feature node and the j-th pathological feature node, where i, j = 1, 2, 3, …, N, and N is the total number of pathological feature nodes; Traverse all pathological feature nodes, construct connectivity feature edges between two pathological feature nodes with a connectivity relationship, and determine the way to have a connectivity relationship as follows: if the structural region images corresponding to two pathological feature nodes share boundary pixels, it is considered to have a connectivity relationship. Based on all pathological feature nodes and all connectivity feature edges, construct a connectivity graph, and based on the connectivity graph, construct a connectivity adjacency matrix. The connectivity adjacency matrix is of size N×N. The element stored in the i-th row and j-th column is 1, indicating that there is a connectivity feature edge between the i-th pathological feature node and the j-th pathological feature node; the element stored in the i-th row and j-th column is 0, indicating that there is no connectivity feature edge between the i-th pathological feature node and the j-th pathological feature node; Traverse all pathological feature nodes, construct pathological association feature edges between two pathological feature nodes with a pathological association, and determine the way to have a pathological association as follows: query in the gastric cancer pathological knowledge graph based on the structural type names corresponding to the two pathological feature nodes. The structural type names refer to glandular ducts, tumor cells, stromal components, etc. If the structural type names corresponding to the two pathological feature nodes have an entity relationship in the gastric cancer pathological knowledge graph, it is considered that there is a pathological association between the two pathological feature nodes. Based on all pathological feature nodes and all pathological association feature edges, construct a pathological association graph, and based on the pathological association graph, construct a pathological association adjacency matrix. The pathological association adjacency matrix is of size N×N. The element stored in the i-th row and j-th column is 1, indicating that there is a pathological association feature edge between the i-th pathological feature node and the j-th pathological feature node; the element stored in the i-th row and j-th column is 0, indicating that there is no pathological association feature edge between the i-th pathological feature node and the j-th pathological feature node; Perform graph convolution operations on the pathological feature map based on the spatial adjacency matrix, connectivity adjacency matrix, and pathological association adjacency matrix respectively to construct the corresponding spatial feature map, connectivity feature map, and pathological association feature map. The graph convolution operation here refers to the GCN model to realize the message passing and feature learning of feature information, and then realize the fusion of feature information corresponding to the adjacency relationship, connectivity relationship, and pathological association respectively; Through the gastric cancer pathological feature fusion layer, the spatial feature map, connectivity feature map, and pathological association feature map are weighted and fused to construct an aggregated feature map, and the gastric cancer pathological knowledge graph is queried based on the pathological feature nodes to construct a pathological knowledge feature map. Perform self-attention mechanism operations based on the aggregated feature map and pathological knowledge feature map to construct a pathological fusion feature map, which specifically includes the following steps: Send the spatial feature map, connectivity feature map, and pathological association feature map into the weight adjustment network for processing, and output the corresponding feature weights. The weight adjustment network here is also established based on a multi-layer perceptron, and the built-in parameters are adjusted with end-to-end training. The spatial feature map, connectivity feature map, and pathological association feature map and the corresponding feature weights are weighted and fused to construct an aggregated feature map; For each pathological feature node, query in the gastric cancer pathological knowledge graph based on the structure type name corresponding to the pathological feature node. If the structure type names corresponding to any two pathological feature nodes have an entity relationship in the gastric cancer pathological knowledge graph, construct a pathological knowledge vector in the form of pathological feature node - entity relationship - pathological feature node, and splice all the pathological knowledge vectors from top to bottom to form a pathological knowledge feature map. Based on the aggregated feature map, construct the corresponding value vector V and key vector K, and based on the pathological knowledge feature map, construct the corresponding query vector Q. The construction of the value vector V, key vector K, and query vector Q refers to the self-attention mechanism in the Transformer model. The specific method is: perform matrix multiplication operations on the aggregated feature map with the value weight matrix and key weight matrix respectively to construct the corresponding value vector V and key vector K, perform matrix multiplication operations on the pathological knowledge feature map with the query weight matrix to construct the corresponding query vector Q. The value weight matrix, key weight matrix, and query weight matrix here are adjusted with end-to-end training. Calculate the self-attention matrix ATT = softmax(QK T / D 0.5 ), where T is the matrix transpose operation, D is the dimension size of the key vector K, and then perform matrix multiplication operations on the value vector V and the self-attention matrix ATT to obtain the pathological fusion feature map; The pathological knowledge graph can provide prior professional gastric cancer pathological knowledge to guide the gastric cancer pathological situation analysis model to be more in line with the doctor's pathological reasoning; The gastric cancer pathology knowledge graph is constructed as follows: collect and organize knowledge related to gastric cancer pathology, determine entities and entity relationships through named entity recognition technology and relationship extraction technology, construct triple data in the form of entity-entity relationship-entity, and then form the gastric cancer pathology knowledge graph with all triple data; Train the gastric cancer pathology situation analysis model, which specifically includes the following steps: Obtain several gastric cancer pathology situation analysis training samples. The gastric cancer pathology situation analysis training samples include gastric cancer pathology images. Label the gastric cancer pathology situation analysis training samples with structure region detection frames and corresponding structure type feature encodings, and label the gastric cancer pathology situation analysis training samples with Lauren classification labels and pathological stage labels. Form the gastric cancer pathology situation analysis training set with all the labeled gastric cancer pathology situation analysis training samples; Pre-train the structure segmentation layer with the gastric cancer pathology situation analysis training set. Taking the structure region detection frame and the corresponding structure type feature encoding as the target, calculate the first loss value, and judge whether the first loss value is within the first preset range, which is determined by the developer. If the first loss value is within the first preset range, output the pre-trained structure segmentation layer; otherwise, continue to pre-train the structure segmentation layer with the gastric cancer pathology situation analysis training set; Train the gastric cancer pathology situation analysis model with the gastric cancer pathology situation analysis training set. Taking the Lauren classification label and the pathological stage label as multi-task targets, calculate the Lauren classification loss value corresponding to the Lauren classification label and the pathological stage loss value corresponding to the pathological stage label respectively. Combine and optimize the Lauren classification loss value and the pathological stage loss value, calculate the second loss value, and judge whether the second loss value is within the second preset range, which is determined by the developer. If the second loss value is within the second preset range, output the pre-trained gastric cancer pathology situation analysis model; otherwise, continue to train the gastric cancer pathology situation analysis model with the gastric cancer pathology situation analysis training set.

[0018] Example 2, a gastric cancer pathology image processing system based on multi-task learning, see Figure 2 , including: A gastric cancer pathology image acquisition module, which is used to generally obtain the gastric cancer pathology image of the patient through gastroscopy when analyzing the gastric cancer pathology situation. Here, the gastric cancer pathology image can also be a processed picture of the gastric tissue sample; The gastric cancer pathological condition analysis module is used to send gastric cancer pathological images into the gastric cancer pathological condition analysis model for processing and output the corresponding gastric cancer pathological condition report. Here, the gastric cancer pathological condition report includes the Lauren classification and pathological stage of the gastric cancer pathological image. The Lauren classification includes intestinal-type gastric cancer, diffuse gastric cancer, and mixed gastric cancer, and the pathological stage is the TNM stage; The gastric cancer pathological condition analysis model includes a structure segmentation layer, a shape convolution kernel mask parameter construction layer, a structure feature extraction layer, a graph structure feature extraction layer, a gastric cancer pathological feature fusion layer, a gastric cancer condition multi-task analysis layer, and a gastric cancer pathological condition report output layer. Among them, the structure segmentation layer is used to segment different structures in the gastric cancer pathological image, output several structure region detection frames and corresponding structure type feature encodings, and map the structure region detection frames to the gastric cancer pathological image to generate structure region images; the shape convolution kernel mask parameter construction layer is used to process the structure region detection frames and corresponding structure type feature encodings and output the shape convolution kernel mask parameters corresponding to the structure region detection frames; the structure feature extraction layer is used to extract features from the structure region images and output the corresponding structure features. During the feature extraction process, the convolution kernel is dot-product processed by the shape convolution kernel mask parameters corresponding to the structure region images; the graph structure feature extraction layer is used to splice the structure features and corresponding structure type feature encodings to construct pathological feature nodes, and based on the pathological feature nodes, construct a spatial adjacency graph, a connectivity graph, and a pathological association graph, and perform graph structure feature extraction according to the spatial adjacency graph, the connectivity graph, and the pathological association graph respectively to construct corresponding spatial feature graphs, connectivity feature graphs, and pathological association feature graphs; the gastric cancer pathological feature fusion layer is used to perform weighted fusion on the spatial feature graph, the connectivity feature graph, and the pathological association feature graph to construct an aggregated feature graph, and query the gastric cancer pathological knowledge graph based on the pathological feature nodes to construct a pathological knowledge feature graph, and perform a self-attention mechanism operation based on the aggregated feature graph and the pathological knowledge feature graph to construct a pathological fusion feature graph; the gastric cancer condition multi-task analysis layer includes a Lauren classification unit and a pathological stage unit, which are used to process the pathological fusion feature graph and output the corresponding Lauren classification label and pathological stage label; the gastric cancer pathological condition report output layer is used to form the gastric cancer pathological condition report by combining the Lauren classification label and the pathological stage label and output it.

[0019] It should be understood that for those of ordinary skill in the art, improvements or transformations can be made according to the above description, and all such improvements and transformations should fall within the protection scope of the appended claims of the present invention. The parts not described in detail in this specification belong to the prior art well-known to those of ordinary skill in the art.

Claims

1. A gastric cancer pathology image processing method based on multi-task learning, characterized in that: include: Obtain the patient's gastric cancer pathology image, then send the gastric cancer pathology image to the gastric cancer pathology analysis model for processing, and output the corresponding gastric cancer pathology report; In the gastric cancer pathology analysis model, a structure segmentation layer, a shape convolution kernel mask parameter construction layer, a graph structure feature extraction layer and a gastric cancer pathology feature fusion layer are set. The structure segmentation layer segments different structures in the gastric cancer pathology image. The shape convolution kernel mask parameter construction layer sets corresponding convolution kernels for the segmented different structures for subsequent feature extraction. The graph structure feature extraction layer extracts features based on the adjacency relationship, connectivity relationship and pathological association between different structures. The gastric cancer pathology feature fusion layer strengthens the features of different structures in the gastric cancer pathology image and the connections between different structural features based on the gastric cancer pathology knowledge graph.

2. The gastric cancer pathology image processing method based on multi-task learning according to claim 1, characterized in that: The gastric cancer pathology analysis model includes a structure segmentation layer, a shape convolution kernel mask parameter construction layer, a structure feature extraction layer, a graph structure feature extraction layer, a gastric cancer pathology feature fusion layer, a gastric cancer situation multi-task analysis layer and a gastric cancer pathology situation report output layer, wherein the structure segmentation layer is used to segment different structures in the gastric cancer pathology image, output a number of structure region detection frames and corresponding structure type feature codes, and map the structure region detection frames to the gastric cancer pathology image to generate a structure region image; the shape convolution kernel mask parameter construction layer is used to process the structure region detection frame and the corresponding structure type feature code, and output the shape convolution kernel mask parameters corresponding to the structure region detection frame; The structural feature extraction layer is used to extract features from the structural region image and output the corresponding structural features. During the feature extraction process, the convolution kernel is dot-product processed by the shape convolution kernel mask parameters corresponding to the structural region image. The graph structure feature extraction layer is used to concatenate the structural features and the corresponding structural type feature codes to construct pathological feature nodes, and to construct a spatial adjacency graph, a connectivity graph and a pathological association graph based on the pathological feature nodes. The graph structure features are extracted according to the spatial adjacency graph, the connectivity graph and the pathological association graph respectively to construct the corresponding spatial feature graph, the connectivity feature graph and the pathological association feature graph; the gastric cancer pathological feature fusion layer is used to weightedly fuse the spatial feature graph, the connectivity feature graph and the pathological association feature graph to construct an aggregate feature graph, and to query the gastric cancer pathological knowledge graph based on the pathological feature nodes to construct a pathological knowledge feature graph, and to perform self-attention mechanism operations based on the aggregate feature graph and the pathological knowledge feature graph to construct a pathological fusion feature graph; the gastric cancer situation multi-task analysis layer includes a Lauren classification unit and a pathological staging unit, which are used to process the pathological fusion feature graph and output the corresponding Lauren classification label and pathological staging label; the gastric cancer pathological situation report output layer is used to combine the Lauren classification label and the pathological staging label to form a gastric cancer pathological situation report output.

3. The gastric cancer pathology image processing method based on multi-task learning according to claim 2, characterized in that: The shape convolution kernel mask parameter construction layer outputs the shape convolution kernel mask parameters corresponding to the structure area detection frame, which specifically includes the following steps: extracting the length and width of the structure area detection frame, and splicing the length and width of the structure area detection frame with the structure type feature coding corresponding to the structure area detection frame to construct the structure area analysis data, and then sending the structure area analysis data to the shape convolution kernel mask parameter construction network built in the shape convolution kernel mask parameter construction layer for processing, and outputting the shape convolution kernel mask parameters corresponding to the structure area detection frame.

4. The gastric cancer pathology image processing method based on multi-task learning according to claim 3 is characterized in that: The corresponding spatial feature map, connectivity feature map and pathology association feature map are constructed through the graph structure feature extraction layer, which specifically includes the following steps: Each structural feature is spliced ​​with the corresponding structural type feature code to construct the corresponding pathological feature node, and all pathological nodes are spliced ​​from top to bottom to construct a pathological feature map; Traversing all pathological feature nodes, constructing adjacent feature edges between two pathological feature nodes with an adjacent relationship, and determining the adjacent relationship by: constructing a Voronoi diagram for all pathological feature nodes, pathological feature nodes in the same area of ​​the Voronoi diagram have an adjacent relationship, constructing a spatial adjacency graph based on all pathological feature nodes and all adjacent feature edges, and constructing a spatial adjacency matrix based on the spatial adjacency graph; Traversing all pathological feature nodes, constructing a connectivity feature edge between two pathological feature nodes with connectivity relationship, and determining the connectivity relationship in the following manner: the structural region images corresponding to the two pathological feature nodes share boundary pixels, which are considered to have connectivity relationship, constructing a connectivity graph based on all pathological feature nodes and all connectivity feature edges, and constructing a connectivity adjacency matrix based on the connectivity graph; Traverse all pathological feature nodes, build pathological association feature edges between two pathological feature nodes with pathological association, and determine whether there is pathological association by querying the gastric cancer pathology knowledge graph based on the structural type names corresponding to the two pathological feature nodes. If the structural type names corresponding to the two pathological feature nodes have an entity relationship in the gastric cancer pathology knowledge graph, it is considered that there is pathological association between the two pathological feature nodes. Build a pathological association graph based on all pathological feature nodes and all pathological association feature edges, and build a pathological association adjacency matrix based on the pathological association graph; Based on the spatial adjacency matrix, connectivity adjacency matrix and pathology association adjacency matrix, graph convolution operations are performed on the pathology feature map to construct the corresponding spatial feature map, connectivity feature map and pathology association feature map.

5. The gastric cancer pathology image processing method based on multi-task learning according to claim 4, characterized in that: The pathology fusion feature map is constructed through the gastric cancer pathology feature fusion layer, which specifically includes the following steps: The spatial feature map, the connectivity feature map and the pathology-related feature map are sent to the weight adjustment network for processing, and the corresponding feature weights are outputted. The spatial feature map, the connectivity feature map and the pathology-related feature map are weightedly fused with the corresponding feature weights to construct an aggregate feature map. For each pathological feature node, a query is performed in the gastric cancer pathology knowledge graph based on the structural type name corresponding to the pathological feature node. If the structural type names corresponding to any two pathological feature nodes have an entity relationship in the gastric cancer pathology knowledge graph, a pathological knowledge vector is constructed in the form of pathological feature node-entity relationship-pathological feature node, and all pathological knowledge vectors are spliced ​​from top to bottom to form a pathological knowledge feature graph. A self-attention mechanism operation is performed based on the aggregated feature graph and the pathological knowledge feature graph to obtain a pathological fusion feature graph.

6. The gastric cancer pathology image processing method based on multi-task learning according to claim 5, characterized in that: Training the gastric cancer pathology analysis model includes the following steps: Obtaining a number of gastric cancer pathology analysis training samples, the gastric cancer pathology analysis training samples include gastric cancer pathology images, annotating the gastric cancer pathology analysis training samples by using structure region detection frames and corresponding structure type feature codes, annotating the gastric cancer pathology analysis training samples by using Lauren classification labels and pathology stage labels, and forming a gastric cancer pathology analysis training set with all annotated gastric cancer pathology analysis training samples; The structure segmentation layer is pre-trained using the gastric cancer pathology analysis training set, with the structure region detection box and the corresponding structure type feature encoding as the goal; The gastric cancer pathology analysis model was trained using the gastric cancer pathology analysis training set, with Lauren classification labels and pathology staging labels as multi-task objectives.

7. A gastric cancer pathology image processing system based on multi-task learning, characterized in that: The system applies a gastric cancer pathology image processing method based on multi-task learning as described in any one of claims 1 to 6, comprising: A gastric cancer pathology image acquisition module, used to acquire gastric cancer pathology images of patients; The gastric cancer pathology analysis module is used to send the gastric cancer pathology image to the gastric cancer pathology analysis model for processing and output the corresponding gastric cancer pathology report.

Images