Pathological image multi-task analysis method and system integrating medical priori knowledge

Abstract

The invention discloses a pathological image multi-task analysis method and system integrating medical priori knowledge, and the method comprises the steps: obtaining pathological image blocks of a plurality of patients, the pathological image blocks being associated with corresponding space coordinates, pathological labels and patient identifiers; respectively extracting visual features of each pathological image block; based on the space coordinates, the pathological labels and the visual features, constructing a graph structure fused with medical priori knowledge; taking the visual features as initial embedding of graph nodes of the graph structure, and inputting the visual features into a graph neural network for message passing so as to obtain updated graph node feature representation; and based on the updated graph node feature representation, executing multi-task learning so as to realize image block-level pathological classification and patient-level survival risk prediction. By constructing a medical priori knowledge graph structure fusing space coordinates, pathological labels and visual features and executing end-to-end multi-task learning, collaborative optimization of image block-level pathological classification and patient-level survival risk prediction is realized.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for multi-task analysis of pathological images that integrates prior medical knowledge. Background Technology

[0002] In the field of digital pathology analysis, traditional deep learning methods typically segment whole-slice images into independent image patches and then process them using convolutional neural networks. However, such methods often neglect the spatial topological relationships and medical semantic connections between image patches, making it difficult for the model to capture the tissue microenvironment structure with prognostic significance. While existing graph neural network models attempt to establish relationships between modules, they largely rely on purely data-driven connections, failing to effectively integrate prior medical knowledge. This results in graph structures lacking clinical rationality, affecting diagnostic consistency and interpretability. Furthermore, existing methods often model image patch classification and patient survival prediction as isolated tasks, leading to computational redundancy and severing the intrinsic connection between microscopic morphological features and macroscopic prognosis, making end-to-end collaborative optimization and cross-granularity reasoning difficult. Therefore, there is an urgent need for a multi-task analysis method that can systematically integrate prior medical knowledge, construct graph structures consistent with pathological logic, and simultaneously achieve image patch-level classification and patient-level survival prediction within the same framework, thereby improving the model's accuracy, efficiency, and clinical interpretability. Summary of the Invention

[0003] In view of this, the present invention proposes a multi-task analysis method and system for pathological images that integrates prior medical knowledge. This method effectively incorporates prior medical knowledge and improves diagnostic accuracy, computational efficiency, and clinical interpretability through graph structure. The present invention provides the following technical solution: A multi-task analysis method for pathological images integrating prior medical knowledge, the method comprising: Multiple pathological image blocks from different patients are acquired, and each pathological image block is associated with corresponding spatial coordinates, pathological labels, and patient identifiers. Visual features were extracted from each of the aforementioned pathological image blocks; Based on the aforementioned spatial coordinates, pathological labels, and visual features, a graph structure integrating prior medical knowledge is constructed. The visual features are used as the initial embeddings of the graph nodes in the graph structure and input into the graph neural network for message passing to obtain the updated graph node feature representations. Based on the updated graph node feature representation, multi-task learning is performed to achieve image block-level pathological classification and patient-level survival risk prediction.

[0004] Optionally, the construction of the graph structure integrating prior medical knowledge includes: Spatial adjacency relationships are constructed based on the spatial coordinates, and semantic association relationships are constructed based on the pathological labels. The spatial adjacency relationships and semantic association relationships are then weighted and fused to generate intra-patient connections.

[0005] Optionally, the semantic association of the tags includes: Establish connections between image patches with the same pathological label and assign them a first weight; A connection is established between the image patch of the lesion area and the image patch of the boundary area, and a second weight is assigned, which is less than the first weight.

[0006] Optionally, the construction of the graph structure integrating prior medical knowledge further includes: Based on the visual features of the pathological image blocks, the feature similarity between graph nodes corresponding to different patients is calculated, and based on the feature similarity, cross-patient connections are established between each graph node and multiple graph nodes of other patients.

[0007] Optionally, the construction of the graph structure integrating prior medical knowledge further includes: Configure a self-loop connection for each graph node to preserve the original feature information of the graph node itself during message passing in the graph neural network and ensure the stability of message passing.

[0008] Optionally, the multi-task learning process includes: The graph attention mechanism is used to model the interaction relationships between graph nodes in the local neighborhood to output the pathological classification results of each pathological image patch; The updated graph node feature representations corresponding to each graph node for the same patient are weighted and aggregated using the attention pooling mechanism to generate a patient-level embedding, and the patient's survival risk score is output based on the patient-level embedding.

[0009] This invention further discloses a multi-task analysis system for pathological images that integrates prior medical knowledge, comprising: The image acquisition module is used to acquire pathological image blocks of multiple patients respectively. Each pathological image block is associated with corresponding spatial coordinates, pathological labels and patient identifiers. A visual feature extraction module is used to extract visual features from each of the pathological image blocks; The medical prior graph construction module is used to construct a graph structure that integrates medical prior knowledge based on the spatial coordinates, pathological labels, and visual features. The graph neural network processing module is used to input the visual features as the initial embeddings of the graph nodes of the graph structure into the graph neural network for message passing, so as to obtain the updated graph node feature representations. The multi-task analysis module is used to perform multi-task learning based on the updated node feature representation to simultaneously achieve image patch-level pathological classification and patient-level survival risk prediction.

[0010] The present invention further discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0011] The present invention further discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described method.

[0012] The present invention further discloses a computer program product, including a computer program that implements the above-described method when executed by a processor.

[0013] According to the technical solution of this invention, by constructing a medical prior knowledge graph structure that integrates spatial coordinates, pathological labels, and visual features, and performing end-to-end multi-task learning on this basis, the collaborative optimization of image block-level pathological classification and patient-level survival risk prediction is achieved. On the one hand, the medical prior-guided graph structure enables the model to capture tissue region associations that conform to clinical logic, significantly improving diagnostic consistency and interpretability. On the other hand, the multi-task mechanism of graph node feature representation avoids redundant computation, reducing computational overhead while improving classification accuracy and survival prediction discriminability through semantic enhancement between tasks, thereby significantly improving diagnostic accuracy, computational efficiency, and clinical interpretability. Attached Figure Description

[0014] For illustrative purposes and not limiting, the present invention will now be described in conjunction with embodiments and accompanying drawings, wherein: Figure 1 This is a flowchart illustrating the multi-task analysis method for pathological images that integrates prior medical knowledge, as described in this invention. Figure 2 This is a schematic diagram of the system structure of the multi-task analysis method for pathological images that integrates prior medical knowledge according to an embodiment of the present invention. Figure 3 This is a schematic diagram of the structure of the electronic device in an embodiment of the present invention; Figure 4 This is a schematic diagram of the construction process of the medical prior graph structure in the embodiments of the present invention; Figure 5 This is a schematic diagram of the process flow for pathological classification and survival risk prediction in an embodiment of the present invention. Detailed Implementation

[0015] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application.

[0016] It should be noted that, where there is no conflict, the embodiments and features of the embodiments in this application can be combined with each other. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0017] refer to Figure 1 This embodiment discloses a multi-task analysis method for pathological images that integrates prior medical knowledge. The method includes the following steps: S100: Acquire pathological image blocks of multiple patients respectively, and each pathological image block is associated with corresponding spatial coordinates, pathological labels and patient identifiers.

[0018] Specifically, refer to Figure 4 Whole-slide images (WSIs) of multiple patients were obtained from hospital pathology departments or digital pathology platforms, each WSI corresponding to a unique patient identifier. Subsequently, a sliding window strategy was used to uniformly divide each WSI into several non-overlapping pathological image tiles of a fixed size. The two-dimensional spatial coordinates (x, y) of each image tile were recorded based on its position in the original WSI; these coordinates were used to subsequently construct spatial adjacency relationships. Simultaneously, based on existing pathology expert annotations, corresponding pathological labels were assigned to each image tile. If an image patch is entirely within the tumor area, it is labeled as "tumor"; If it is located entirely within normal tissue, it is marked as "non-tumor"; If it contains both tumor and normal tissue, it is marked as "boundary".

[0019] The aforementioned spatial coordinates, pathological labels, and patient identifiers together constitute the metadata for each pathological image patch, which is bound and stored individually to the image patch itself, forming the input dataset required for subsequent graph structure construction and multi-task learning. This step ensures that each pathological image patch possesses clear medical contextual information in the three dimensions of space, semantics, and patient affiliation, laying a data foundation for integrating prior medical knowledge.

[0020] S200: Extract visual features from each of the pathological image blocks.

[0021] Visual feature extraction is achieved by batch inputting the pathological image patches acquired and labeled in step S100 into a pre-trained convolutional neural network as a shared feature extractor. This network employs a ResNet18 model pre-trained on the ImageNet dataset, with its original classification layer removed, retaining only the backbone network for feature extraction. Each pathological image patch, exemplarily 256×256 pixels with RGB three channels, outputs a 1000-dimensional high-dimensional feature vector after forward propagation through ResNet18. To further adapt to the input dimension of the graph neural network and reduce computational complexity, this 1000-dimensional feature vector is then reduced to 512 dimensions through a linear projection layer (i.e., a fully connected layer), forming the final visual feature representation. This 512-dimensional feature vector serves as the initial embedding for the corresponding graph nodes, used for subsequent graph structure construction and message passing within the graph neural network.

[0022] This feature extraction process is performed independently and in parallel on all pathological image blocks, ensuring that each image block obtains discriminative semantic features corresponding to its morphological content, providing a high-quality underlying representation foundation for graph modeling and multi-task learning that integrates medical prior knowledge.

[0023] The aforementioned visual feature extraction module is jointly optimized with the subsequent graph neural network and multi-task head-to-end during the training process, so that the extracted features not only retain general visual semantics, but also specifically serve pathological classification and survival prediction tasks.

[0024] S300: Based on the aforementioned spatial coordinates, pathological labels, and visual features, construct a graph structure that integrates prior medical knowledge.

[0025] First, for each patient, the K-nearest neighbor algorithm is used to construct a spatial adjacency matrix using the spatial coordinates of all their pathological image blocks. For each image block node, the K nearest neighbor nodes in the original whole slice image are connected. For example, K=8. This matrix reflects the medical prior that "the histologically adjacent regions are more functionally related" and is given a fusion weight of 70%. Secondly, a label adjacency matrix is ​​constructed based on pathological labels: image block pairs with the same pathological label (e.g., both are "tumor") are assigned a first weight, i.e., a connection weight of 1.0; image block pairs with the labels "tumor" and "boundary" are assigned a second weight, i.e., a connection weight of 0.8; and no special connections are established for other label combinations. This matrix reflects the clinical experience that "specific lesion areas have semantic associations" and is assigned a fusion weight of 30%. Subsequently, the two matrices are weighted, summed, and binarized to generate the final intra-patient connections. Based on this, inter-patient connections are further constructed: using the 512-dimensional visual features extracted in step S200, the cosine similarity between any two different patient image patch nodes is calculated, and for each node, the five nodes with the highest similarity from other patients are selected to establish cross-patient edges, but repeated connections between nodes within the same patient are strictly prohibited. Finally, self-loop connections are added to each node in the graph, i.e., each node establishes an edge with itself, to ensure that the graph neural network retains original feature information and maintains numerical stability during message passing. These three types of connections together constitute a heterogeneous graph structure that integrates spatial structure, pathological semantics, and cross-patient similarity priors. Each pathological image patch corresponds to a graph node, and the edges between nodes are explicitly guided by medical prior knowledge, providing a topological foundation consistent with clinical logic for subsequent message passing and multi-task learning in the graph neural network.

[0026] S400: The visual features are used as the initial embeddings of the graph nodes in the graph structure and input into the graph neural network for message passing to obtain the updated graph node feature representations.

[0027] For details, please refer to Figure 5 The 512-dimensional feature vector extracted in step S200 is used as the initial embedding of the corresponding graph node and input into a two-layer graph attention convolutional network (GATConv) for end-to-end message passing and feature updating. In the first layer of GAT, each node aggregates the information of its neighboring nodes through four parallel attention heads. Each attention head linearly maps the 512-dimensional input features to 256 dimensions and dynamically weights the importance of neighboring nodes through learnable attention coefficients. Then, the outputs of the four heads are concatenated to form a 1024-dimensional intermediate node representation. This representation is then nonlinearly transformed by the ReLU activation function and fed into the second layer of GAT. The second layer of GAT uses a single-head attention mechanism to further aggregate the 1024-dimensional features into a final 256-dimensional node feature representation, which is also activated by ReLU. Throughout the message passing process, the patient-internal connections, patient-interchange connections, and self-loop connections defined in the graph structure collectively determine the topological path of information propagation. This allows each node to not only integrate semantic information from its local spatial neighborhood but also absorb cross-patient similarity pattern knowledge, thereby obtaining a high-order feature representation that combines discriminativity, context awareness, and prognostic relevance. This updated 256-dimensional graph node feature representation will serve as the unified input for subsequent multi-task learning, used for image block-level pathological classification and patient-level survival risk prediction, respectively.

[0028] S500: Based on the updated node feature representation, it performs multi-task learning to simultaneously achieve image block-level pathological classification and patient-level survival risk prediction.

[0029] For details, please refer to Figure 5 The 256-dimensional updated graph node feature representation output by the graph neural network in step S500 is input into two parallel task heads to achieve end-to-end multi-task collaborative learning.

[0030] On the one hand, image block-level pathological classification is completed by the Tile classification head: the classification head consists of two fully connected network layers. The first layer maps the 256-dimensional graph node feature representation to 128 dimensions and introduces nonlinearity through the ReLU activation function. The second layer further maps to 3-dimensional output, corresponding to the three pathological labels of "tumor", "non-tumor" and "boundary", respectively. The class probability distribution is generated through the Softmax function, thereby realizing fine semantic discrimination for each pathological image block.

[0031] On the other hand, patient-level survival risk prediction is accomplished by the survival analysis head: First, attention pooling is performed on the 256-dimensional graph node feature representations of all graph nodes corresponding to the same patient, introducing a learnable context vector. The similarity between each node and this vector is calculated and normalized by Softmax to obtain the attention weight of each image patch for the patient's prognosis. Then, all node features are weighted and summed using this weight to generate a 256-dimensional patient-level feature representation, which centrally expresses the most discriminative regional information for survival prediction. Finally, this patient-level embedding is input into the risk prediction network, which also consists of two fully connected layers and outputs a continuous risk score for survival analysis under the risk model, thereby quantifying the patient's risk of death or recurrence.

[0032] The two tasks mentioned above share the underlying graph neural network feature representations and are jointly optimized during training. This design allows the semantically clear local lesion features provided by the classification task to effectively guide survival prediction to focus on areas with pathological significance; while the weak supervision signal of the survival task (requiring only patient-level labels) in turn prompts the model to learn node representations with greater prognostic discriminative power, forming a virtuous cycle mechanism of "micro-classification supporting macro-prognosis, and macro-targets optimizing micro-representations". Ultimately, high-precision image patch classification and high-discrimination patient risk stratification are achieved simultaneously within a single framework.

[0033] In summary, this specific implementation constructs a medical prior graph structure that integrates spatial coordinates, pathological labels, and cross-patient visual feature similarity. Based on this, a two-layer graph attention network is used for feature enhancement. Furthermore, a multi-task learning process—combining image block-level pathological classification and patient-level survival risk prediction—is executed collaboratively through graph attention and attention pooling mechanisms, achieving an organic unity between microscopic morphological analysis and macroscopic prognostic assessment. This approach significantly improves the model's diagnostic accuracy, computational efficiency, and clinical interpretability: on the one hand, the medical prior-guided graph connectivity ensures that model decisions conform to pathological logic, avoiding the blindness of purely data-driven graph construction; on the other hand, the end-to-end joint optimization mechanism enables the classification and survival tasks to mutually reinforce each other at the semantic level. With only weakly supervised patient-level labels, it can automatically focus on key regions with prognostic value, effectively reducing reliance on fine-grained annotation and providing an efficient, robust, and reliable technical path for intelligent pathological diagnosis.

[0034] refer to Figure 2 This embodiment further discloses a multi-task analysis system for pathological images that integrates medical prior knowledge, including an image acquisition module 21, a visual feature extraction module 22, a medical prior graph construction module 23, a graph neural network processing module 24, and a multi-task analysis module 25. Image acquisition module 21 is used to acquire pathological image blocks of multiple patients respectively, and the pathological image blocks are associated with corresponding spatial coordinates, pathological labels and patient identifiers respectively; The visual feature extraction module 22 is used to extract visual features from each of the pathological image blocks. The medical prior graph construction module 23 is used to construct a graph structure that integrates medical prior knowledge based on the spatial coordinates, pathological labels, and visual features. This includes: constructing spatial adjacency relationships based on the spatial coordinates; constructing label semantic association relationships based on the pathological labels; and weightedly fusing the spatial adjacency relationships and label semantic association relationships to generate patient-internal connections. The label semantic association relationships include: establishing connections between image blocks with the same pathological label and assigning a first weight; establishing connections between image blocks in the lesion region and image blocks in the boundary region and assigning a second weight, where the second weight is less than the first weight; and not establishing connections between other image blocks.

[0035] The graph neural network processing module 24 is used to input the visual features as the initial embeddings of the graph nodes of the graph structure into the graph neural network for message passing, so as to obtain the updated graph node feature representations. The multi-task analysis module 25 is used to perform multi-task learning based on the updated node feature representation to simultaneously achieve pathological classification at the image patch level and survival risk prediction at the patient level. The multi-task learning includes: using a graph attention mechanism to model the interaction relationship between graph nodes in the local neighborhood to output the pathological classification results of each pathological image patch; using an attention pooling mechanism to perform weighted aggregation of the updated graph node feature representations of each graph node corresponding to the same patient to generate a patient-level embedding, and outputting the patient's survival risk score based on the patient-level embedding.

[0036] Figure 3 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 3 As shown, the electronic device 50 includes: a processor 501, a memory 502, and a bus 503; The processor 501 and the memory 502 communicate with each other via the bus 503; the processor 501 is used to call the program instructions in the memory 502 to execute the methods provided in the above-described embodiments.

[0037] This embodiment provides a non-transitory computer-readable storage medium that stores computer instructions that cause a computer to execute the methods provided in the above-described embodiments.

[0038] Those skilled in the art will understand that all or part of the steps of the above-described method implementation can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above-described method implementation. The aforementioned storage medium includes various storage media capable of storing program code, such as ROM, RAM, magnetic disk, or optical disk.

[0039] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0040] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of each embodiment or some parts of the embodiments.

[0041] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A multi-task analysis method for pathological images integrating prior medical knowledge, characterized in that, The method includes: Multiple pathological image blocks from different patients are acquired, and each pathological image block is associated with corresponding spatial coordinates, pathological labels, and patient identifiers. Visual features were extracted from each of the aforementioned pathological image blocks; Based on the aforementioned spatial coordinates, pathological labels, and visual features, a graph structure integrating prior medical knowledge is constructed. The visual features are used as the initial embeddings of the graph nodes in the graph structure and input into the graph neural network for message passing to obtain the updated graph node feature representations. Based on the updated graph node feature representation, multi-task learning is performed to achieve image block-level pathological classification and patient-level survival risk prediction.

2. The multi-task analysis method for pathological images according to claim 1, characterized in that, The construction of the graph structure integrating prior medical knowledge includes: Spatial adjacency relationships are constructed based on the spatial coordinates, and semantic association relationships are constructed based on the pathological labels. The spatial adjacency relationships and semantic association relationships are then weighted and fused to generate patient-internal connections in the graph structure.

3. The multi-task analysis method for pathological images according to claim 2, characterized in that, The construction of semantic associations for tags includes: Establish connections between image patches with the same pathological label and assign them a first weight; A connection is established between the image patch of the lesion area and the image patch of the boundary area, and a second weight is assigned, which is less than the first weight.

4. The multi-task analysis method for pathological images according to claim 1, characterized in that, The construction of the graph structure integrating prior medical knowledge also includes: Based on the visual features of the pathological image blocks, the feature similarity between graph nodes corresponding to different patients is calculated, and based on the feature similarity, cross-patient connections are established between each graph node and multiple graph nodes of other patients.

5. The multi-task analysis method for pathological images according to claim 1, characterized in that, The construction of the graph structure integrating prior medical knowledge also includes: Configure a self-loop connection for each graph node to preserve the original feature information of the graph node itself during message passing in the graph neural network and ensure the stability of message passing.

6. The multi-task analysis method for pathological images according to claim 1, characterized in that, The multi-task learning process includes: The graph attention mechanism is used to model the interaction relationships between graph nodes in the local neighborhood to output the pathological classification results of each pathological image patch; The updated graph node feature representations corresponding to each graph node for the same patient are weighted and aggregated using the attention pooling mechanism to generate a patient-level embedding, and the patient's survival risk score is output based on the patient-level embedding.

7. A multi-task analysis system for pathological images integrating prior medical knowledge, characterized in that, include: The image acquisition module is used to acquire pathological image blocks of multiple patients respectively. Each pathological image block is associated with corresponding spatial coordinates, pathological labels and patient identifiers. A visual feature extraction module is used to extract visual features from each of the pathological image blocks; The medical prior graph construction module is used to construct a graph structure that integrates medical prior knowledge based on the spatial coordinates, pathological labels, and visual features. The graph neural network processing module is used to input the visual features as the initial embeddings of the graph nodes of the graph structure into the graph neural network for message passing, so as to obtain the updated graph node feature representations. The multi-task analysis module is used to perform multi-task learning based on the updated node feature representation to simultaneously achieve image patch-level pathological classification and patient-level survival risk prediction.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1-6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method of any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-6.

Images