A method, device, medium and program product for learning and analyzing morphological features of pathological images in thyroid nodule surgery

Abstract

The embodiment of the specification provides a method, device, medium and program product for learning and analyzing morphology of pathological images in thyroid nodule surgery, and relates to the field of intelligent medical treatment. The method comprises the following steps: acquiring a frozen section image of the thyroid nodule surgery to be analyzed; inputting the intraoperative frozen section image into a morphology learning network trained by the application to extract a high-dimensional morphology feature vector; using a mapping relationship library established by the application to map the high-dimensional morphology feature vector into a corresponding tissue morphology mode description, and generating a pathological analysis report containing the tissue morphology mode description. The application realizes the reduction of dependence on labeled data, the overcoming of cross-domain slicing differences, and the high interpretability by constructing a whole-chain technical solution from bottom feature extraction to upper report generation.

Description

Technical Field

[0001] This invention relates to the field of intelligent healthcare, and more specifically, to a method, device, medium, and program product for learning and analyzing the morphological characterization of pathological images during thyroid nodule surgery. Background Technology

[0002] Frozen tissue pathology during thyroid nodule surgery is a crucial step in guiding decisions regarding the extent of surgical resection and lymph node dissection. Its core lies in rapidly and accurately determining the benign or malignant nature of the lesion and the extent of lymph node metastasis. However, existing AI-based pathological image analysis technologies face the following challenges when applied to intraoperative frozen tissue scenarios: First, there is a significant "domain shift" problem between intraoperative frozen tissue sections (frozen sections) and postoperative paraffin sections (paraffin sections). Due to the special preparation process of frozen sections, including rapid freezing and short staining time, the images of frozen sections differ greatly from those of postoperative paraffin sections in terms of cell morphology, tissue structure, and staining effect. Most existing deep learning models are trained on high-quality paraffin sections. When directly applied to frozen sections, they cannot adapt to the changes in feature distribution caused by the differences in slide preparation, resulting in decreased generalization ability and a high misdiagnosis rate. Although some existing technologies attempt to introduce domain adaptation methods, they often require a large amount of pixel-level annotated frozen section data. However, obtaining such fine annotations in intraoperative scenarios is extremely costly and time-consuming, making it difficult to meet the real-time clinical needs.

[0003] Secondly, existing models lack interpretability, making it difficult to gain the trust of clinicians. Most mainstream deep learning models operate as "black boxes," only outputting classification results or probability values, failing to provide morphological evidence similar to that used by pathologists when reviewing slides, such as specific descriptions like "tumor cell nests" or "stromal invasion." In high-risk intraoperative decision-making scenarios, doctors need not only to know "what" but also "why." This lack of interpretability in auxiliary diagnosis cannot effectively assist doctors in determining surgical boundaries. Summary of the Invention

[0004] This invention aims to address at least one of the technical problems existing in the prior art. To this end, this invention proposes a method for analyzing and generating reports of pathological images during thyroid nodule surgery that reduces dependence on labeled data, overcomes cross-domain slide preparation differences, and possesses high interpretability.

[0005] The first aspect of this application discloses a method for training a morphological representation network, comprising: Acquire intraoperative frozen pathology slide images and segment them into multiple frozen pathology image blocks; For each frozen pathological image patch, multiple enhanced views are generated and input into the initial feature extraction network to extract the morphological representation vector of each enhanced view; The contrast loss is calculated based on the representation vectors of different enhanced views of the same frozen pathological image patch, and the parameters of the initial feature extraction network are updated. After updating the parameters through the contrast loss, the morphological standard vectors of reference pathological images from different preparation conditions are input into the domain discriminator. Through adversarial training, the source of the preparation condition of the morphological representation vector output by the initial feature extraction network cannot be distinguished, so as to obtain a morphological representation learning network that eliminates preparation differences. Once the morphological representation learning network is trained, for any input frozen pathological slide image, the output morphological representation vector is a high-dimensional morphological representation that is robust to imaging changes and reflects the inherent morphological structure of pathological tissue. Moreover, the extraction process of the high-dimensional morphological representation does not depend on pixel-level labeled data.

[0006] In some embodiments, the second aspect of this application discloses a training method for an initial feature extraction network, wherein the initial feature extraction network is a pathological baseline model pre-trained on a reference pathological image dataset; prior to performing the adversarial training, the method further includes: Freeze the original weights of the pathological model; A low-rank adapter is inserted into the basic pathological model as a trainable model; The adversarial training is performed on the feature output of the low-rank adapter. By updating only the parameters of the low-rank adapter, the morphological feature capture capability learned by the pathological base model from the reference pathological image is losslessly generalized to the frozen pathological slide image.

[0007] A third aspect of this application discloses a method for constructing interpretable semantic mapping relationships, comprising: Using the morphological representation learning network trained by the method disclosed in the first or second aspect of this application, high-dimensional morphological representation vectors of frozen pathological slide images are extracted. A graph structure is constructed using the high-dimensional morphological representation vectors as nodes and the similarity between representation vectors as edges. Unsupervised semantic clustering is performed on the graph structure to obtain multiple feature clusters; Obtain semantic annotations for the organizational morphological pattern descriptions of the multiple feature clusters, and establish a mapping relationship library between the feature decisions and the organizational morphological pattern descriptions; The mapping relation library is used in pathological analysis tasks to transform extracted high-dimensional features into clinically interpretable morphological descriptions.

[0008] The fourth aspect of this application discloses a training method for an interpretable pathology report generation model, including: Obtain a training dataset containing frozen section images from thyroid nodule surgery and corresponding pathology report texts; Using the mapping database established by the method disclosed in the third aspect of this application, semantic enhancement annotation is performed on image patches in the training dataset; Construct an initial generative model comprising a visual encoder and a text decoder. The visual encoder is a morphological representation learning network trained using the method disclosed in the first aspect of this application, or a pathological basic model fine-tuned across domains using the method disclosed in the second aspect of this application. The intraoperative frozen section image is input into the visual encoder to extract feature sequences, and the pathology report text is subjected to text embedding processing to obtain text embedding features; The feature sequence and text embedding features are input into the text encoder to generate predicted text; Key image patches are selected based on their diagnostic contribution. These key image patches are then used to impose directional constraints on the attention distribution of the text decoder. The generation loss is calculated in conjunction with the pathology report text. A training objective containing supervised loss is constructed, and the parameters of the initial generation model are jointly optimized to obtain an interpretable pathology report generation model. The fifth aspect of this application discloses a method for learning and analyzing the morphological characterization of pathological images during thyroid nodule surgery, including: Obtain intraoperative frozen section images of the thyroid nodules to be analyzed during surgery; The intraoperative frozen section image is input into the morphological representation learning network trained by the method disclosed in the first aspect of this application to extract high-dimensional morphological representation vectors. Using the mapping relationship library established by the method disclosed in the third aspect of this application, the high-dimensional morphological representation vector is mapped to the corresponding tissue morphological pattern description, and a pathological analysis report containing the tissue morphological pattern description is generated.

[0009] The sixth aspect of this application discloses a computer device, the device comprising: a memory and a processor; the memory for storing a computer program; and the processor for executing the computer program to implement the steps of the above-described method.

[0010] The seventh aspect of this application discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0011] The eighth aspect of this application discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0012] This application constructs a complete technical solution from low-level feature extraction to high-level report generation, which has the following beneficial effects: First, a self-supervised morphological representation learning mechanism resistant to domain shift was constructed, which significantly reduced the dependence on labeled data and improved the robustness of the model.

[0013] This application employs a strategy combining contrastive learning and domain adversarial training. On one hand, it utilizes multi-view enhancement of the same image patch for self-supervised contrastive learning, enabling the network to learn robust intrinsic morphological features to imaging changes without pixel-level annotation. On the other hand, it introduces paraffin sections as auxiliary data, and through the adversarial training mechanism of the gradient inversion layer, the network ignores the differences in slide preparation between frozen and paraffin sections, focusing on the morphological structure of the pathological tissue itself. This design not only solves the problem of scarce frozen section annotation data but also enables lossless generalization of rich pre-trained knowledge from paraffin sections to intraoperative frozen scenarios, improving the model's generalization ability and diagnostic accuracy across different hospitals and equipment.

[0014] Second, this application provides an interpretable semantic mapping construction method based on "unsupervised clustering and expert annotation," addressing the "black box" problem of AI diagnosis. This application establishes a semantic mapping mechanism that primarily utilizes a pre-trained feature network to extract high-dimensional representations. Through a hierarchical iterative clustering algorithm that maximizes modularity, it automatically discovers feature clusters with similar morphological patterns. Pathologists then provide semantic annotations to these clusters, such as labeling them as "papillary structures" or "nuclear grooves." This method transforms the abstract high-dimensional vectors extracted by the machine into morphological terms understandable to clinicians, constructing a high-quality morphological pattern description library. This makes the AI ​​diagnostic process transparent, allowing doctors to clearly trace the diagnostic basis and enhancing the trustworthiness of clinical applications.

[0015] Third, this application achieves automatic generation of interpretable pathology reports guided by diagnostic contributions and possesses the ability to identify high-risk samples. This application proposes a report generation framework that deeply integrates morphological semantics. This framework utilizes a constructed semantic mapping library to enhance training data and guides the attention mechanism of the Transformer decoder to focus on key diagnostic regions through a supervised loss function. This makes the generated pathology reports no longer simple textual compilations, but rather accurately describes the tissue morphological features corresponding to key image patches, achieving interpretable output with "image-text alignment." Furthermore, this application can quantify the consistency of the contribution direction of various morphological patterns to lymph node metastasis prediction. When multiple key patterns are found to have conflicting contribution directions, they are automatically identified and marked as high-risk uncertain samples, providing doctors with additional safety warnings and reliable evidence for surgical decisions in complex cases. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the training method for a morphological representation network provided in the first aspect of the present invention; Figure 2 This is a schematic diagram of the method for constructing interpretable semantic mapping relationships provided in the third aspect of the present invention; Figure 3 This is a schematic diagram of the training method for the interpretable pathology report generation model provided in the fourth aspect of the present invention; Figure 4 This is a schematic diagram of the morphological characterization learning and analysis method for pathological images during thyroid nodule surgery provided in the fifth aspect of the present invention; Figure 5 This is a flowchart of the morphological representation learning method that integrates contrastive learning and domain adversarial training provided in an embodiment of the present invention. Figure 6 This is a schematic diagram of the semantic clustering algorithm provided in an embodiment of the present invention; Figure 7 This is a flowchart illustrating the method for fine-tuning a pathological baseline model based on slide adversarial training provided in an embodiment of the present invention. Figure 8 This is a diagram of a multi-instance learning pathology report generation framework based on Transformer provided in an embodiment of the present invention. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0019] In some of the processes described in the specification, claims, and accompanying drawings of this invention, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] like Figure 1 As shown in the diagram, this embodiment of the invention provides a flowchart of a training method for a morphological representation network. Specifically, the method includes the following steps: S101: Acquire intraoperative frozen pathology slide images and segment them into multiple frozen pathology image blocks; In some embodiments, the initial feature extraction network is a neural network to be trained for extracting high-dimensional morphological features from pathological image patches.

[0022] In some embodiments, the second aspect of this application discloses a training method for an initial feature extraction network, wherein the initial feature extraction network is a pathological baseline model pre-trained on a reference pathological image dataset; prior to performing the adversarial training, the method further includes: Freeze the original weights of the pathological model; A low-rank adapter is inserted into the basic pathological model as a trainable model; The adversarial training is performed on the feature output of the low-rank adapter. By updating only the parameters of the low-rank adapter, the morphological feature capture capability learned by the pathological base model from the reference pathological image is losslessly generalized to the frozen pathological slide image.

[0023] S102: Generate multiple enhanced views for each frozen pathological image patch, input them into the initial feature extraction network, and extract the morphological representation vector of each enhanced view; In some embodiments, generating multiple enhanced views for each frozen pathology image block includes performing at least two of the following enhancement processes on the frozen pathology image block: color transformation, contrast adjustment, geometric transformation, and staining style transfer. S103: Calculate the contrast loss based on the representation vectors of different enhanced views of the same frozen pathological image patch, and update the parameters of the initial feature extraction network; S104: After updating the parameters through the contrast loss, the morphological standard vectors of reference pathological images from different preparation conditions are input into the domain discriminator. Through adversarial training, the source of the preparation condition of the morphological representation vector output by the initial feature extraction network cannot be distinguished, so as to obtain a morphological representation learning network that eliminates preparation differences. Once the morphological representation learning network is trained, for any input frozen pathological slide image, the output morphological representation vector is a high-dimensional morphological representation that is robust to imaging changes and reflects the inherent morphological structure of pathological tissue. Moreover, the extraction process of the high-dimensional morphological representation does not depend on pixel-level labeled data.

[0024] In this embodiment, to more clearly describe how the morphological representation learning method uses contrastive learning and domain adversarial training, a specific pathological slide processing procedure is used as an example for illustrative description. However, this description is only one use case and is not intended to limit the scope of protection of this processing method. Figure 5 As shown, the complete process can be divided into four core stages. The main steps are to align the representations of different enhanced views of the same pathological patch and minimize the redundancy between feature dimensions, learn high-dimensional representations with consistent morphological features, and construct a feature space that is conducive to subsequent clustering and dimensionality reduction analysis.

[0025] A domain discriminator and gradient inversion layer are introduced. During the training phase, multi-center data from both the Chinese Society of Tumor and TCGA are incorporated. Adversarial training ensures that the learned representations are invariant to technical differences such as slide preparation and staining, focusing on pathological morphological features and addressing the multi-center domain bias problem. The four core stages are as follows: 1. Input Preprocessing: From Pathological Sections to Pathological Patches First, the entire pathological slide (digital whole-slice image, WSI) is cut into multiple pathological patches of uniform size to solve the problem that the full slide size is too large to be directly input into the model.

[0026] 2. Data Augmentation: Generate multiple views for the same patch. For each pathological patch obtained from the cut, multiple enhanced views are generated using various image enhancement techniques (such as color transformation, contrast adjustment, rotation, flipping, staining style transfer, etc.) (corresponding to two sets of image patches with different styles and colors in the upper and lower parts of the image), providing positive sample pairs for subsequent comparative learning.

[0027] 3. Contrastive learning: Learning morphologically consistent high-dimensional representations Different enhanced views from the same pathology patch are input into two feature extractors with shared weights.

[0028] 3.1. The feature extractor is responsible for extracting high-dimensional tissue morphology representation vectors from the pixels of the original image.

[0029] 3.2. Simultaneously, the alignment loss function L_align is introduced: it forces the representation vectors of different augmented views of the same patch to be as close as possible (feature alignment) and minimizes redundant information between feature dimensions.

[0030] 3.3. Through this step, the model can learn a high-dimensional representation that is robust to image enhancement and has highly consistent morphological features, thus constructing a feature space suitable for subsequent clustering and dimensionality reduction analysis.

[0031] 4. Domain adversarial training: Eliminating multi-center technical differences To address the "domain shift" problem caused by differences in slide preparation and staining techniques between different hospitals / datasets, and to simultaneously reduce the "domain shift" problem between frozen and paraffin section features, a domain adversarial training module is superimposed on contrastive learning: 4.1. Input the tissue morphology representation output by the feature extractor into the domain discriminator.

[0032] 4.2. The task of the domain discriminator is to predict the source domain label of the input representation (such as "China Cancer Dataset" or "TCGA Dataset"), and the corresponding optimization objective is the domain classification loss function L_domain.

[0033] 4.3. Adversarial mechanism implemented during training through gradient reversal layer: During backpropagation, the gradient is reversed and passed to the feature extractor. This causes the feature extractor to update its parameters in a way that prevents the domain discriminator from correctly distinguishing the domain origin, while the domain discriminator optimizes in a way that makes the domain label as accurate as possible.

[0034] Final result: By simultaneously optimizing L_align (contrast alignment loss) and L_domain (domain adversarial loss), the model ultimately learns a tissue morphological representation that possesses two core capabilities: Different imaging enhancements of the same pathological tissue show consistency.

[0035] The technical differences in slide preparation, staining, etc., between different centers remain unchanged.

[0036] This allows us to truly focus on the characteristics of pathological morphology itself, providing a high-quality and highly generalizable feature base for subsequent downstream tasks such as clustering, dimensionality reduction, and prognosis prediction.

[0037] In some embodiments, after updating the parameters using the contrast loss, the morphological standard vectors of reference pathological images from different slide preparation conditions are input into the domain discriminator, and slide preparation differences are eliminated through adversarial training, including: The intraoperative frozen pathology slide image block and the reference pathology image block are respectively input into the initial feature extraction network after the contrast loss update parameters to obtain their respective morphological representation vectors. The respective morphological representation vectors are input into the domain discriminator, which determines the source domain of each morphological representation vector and calculates the domain classification loss. The gradient of the domain classification loss is inverted by the gradient inversion layer and then passed to the initial feature extraction network, so that the initial feature extraction network updates its parameters in a way that confuses the domain discriminator in distinguishing the source of frozen pathological sections from reference pathological images. In some embodiments, the reference pathological image is a paraffin-embedded pathological section image; In a more specific embodiment, after updating the parameters through the contrast loss, the morphological standard vectors of reference pathological images from different slide preparation conditions are input into the domain discriminator. Adversarial training is used to eliminate slide preparation differences, including: combining adversarial training and low-rank adaptive techniques to guide the pathological baseline model to ignore the differences in slide preparation between paraffin and frozen sections when capturing morphological representations, without disrupting the original weights of the model. This allows the morphological capture capability learned by the pathological baseline model in paraffin sections to be generalized seamlessly to frozen sections, improving the accuracy of intraoperative report generation. Figure 7 The diagram shows the detailed process of a method for fine-tuning a pathological baseline model based on slide adversarial training. This method addresses the cross-domain pathological AI adaptation problem from paraffin sections (source domain) to frozen sections (target domain). Its core employs a dual-technique approach: low-rank adaptive training and domain adversarial training. This process preserves the original pre-trained weights of the underlying pathological model, achieving lossless generalization of morphological representation capabilities and ultimately improving the accuracy of intraoperative frozen section pathology reports. The complete workflow is as follows: 1. Initialization and Input Preparation 1.1 Model initialization: Load the pre-trained pathological basic model on a large number of paraffin sections and completely freeze all its original weights; insert only a lightweight low-rank adapter at the model output as the only trainable new module to avoid destroying the general morphological representation capabilities already learned by the model.

[0038] 1.2 Dual-field data input: Simultaneously inputting two types of pathological slide data: Source Domain: Conventional formalin-fixed paraffin-embedded slides.

[0039] Target Domain: Intraoperative rapid frozen sections.

[0040] 2. Dual-domain morphological feature extraction The two types of slides are respectively input into a combined network of "frozen pathological baseline model + low-rank adapter" to complete feature extraction: 2.1 Paraffin sections are processed by a network to output source domain morphological features.

[0041] 2.2 Frozen sections are processed through the same network to output the morphological features of the target domain.

[0042] 3. Joint optimization training with dual losses The training process optimizes two loss functions simultaneously, achieving the dual objectives of "preserving original capabilities + resolving alignment domain differences": 3.1 Source Loss The task loss (such as classification / segmentation loss) is calculated only for the morphological features of the source domain. This is used to force the preservation of the model's original morphological recognition performance on paraffin sections, ensuring that the original capabilities are not degraded after fine-tuning.

[0043] 3.2 Adversarial Loss The adversarial game achieved through a domain discriminator is essentially about making the model "ignore production differences and focus only on morphology itself": 3.2.1 The domain discriminator is a binary classifier whose task is to determine, as accurately as possible, whether the input feature comes from a "paraffin section" or a "frozen section".

[0044] 3.2.2 The task of the low-rank adapter is to generate morphological features that the domain discriminator cannot distinguish, i.e., to align the feature distributions of paraffin and frozen sections.

[0045] 3.2.3 The two form a closed-loop adversarial relationship: During backpropagation, the adversarial loss will update the parameters of the low-rank adapter and the domain discriminator at the same time. The better the domain discriminator can distinguish domain differences, the more the low-rank adapter will optimize features to confuse the discriminator, and eventually the morphological feature distributions of the two domains will become consistent.

[0046] 4. Training Completion and Results Output Training ends when the "source domain loss + adversarial loss" converges. At this point, the morphological capture capabilities learned by the pathology model on paraffin sections have been losslessly generalized to frozen sections through a low-rank adapter, and can be directly used for rapid diagnosis and report generation of intraoperative frozen pathology.

[0047] like Figure 2 As shown in the figure, this application embodiment provides a flowchart of a method for constructing interpretable semantic mapping relationships. Specifically, the method includes the following steps: S201: Using the morphological representation learning network trained by the method disclosed in the first or second aspect of this application, extract high-dimensional morphological representation vectors from frozen pathological slide images. S202: Construct a graph structure using the high-dimensional morphological representation vectors as nodes and the similarity between representation vectors as edges; S203: Perform unsupervised semantic clustering on the graph structure to obtain multiple feature clusters; In some embodiments, performing unsupervised semantic clustering on the graph structure to obtain multiple feature clusters includes: Phase 1: Perform local node movement and calculate the change in modularity after each movement. Retain the movement results that improve the overall modularity to form an initial community of nodes. Refine the boundaries of the initial community and aggregate the nodes into multiple fine-grained communities. The second stage involves aggregating each fine-grained community into a super node, preserving the connections between super nodes, and retaining the node structure within the initial community to construct a coarse-grained community. Based on the coarse-grained community, node movement and community boundary refinement are performed again to obtain multi-level feature clusters.

[0048] S204: Obtain semantic annotations for the tissue morphology pattern descriptions of the multiple feature clusters, and establish a mapping relationship library between the feature decisions and the tissue morphology pattern descriptions; the mapping relationship library is used in pathological analysis tasks to transform the extracted high-dimensional features into clinically interpretable morphological descriptions. In this embodiment, in order to more clearly describe the process of the expert pre-interpretation method based on semantic clustering, the semantic clustering algorithm is used as an example for exemplary description. However, this description is only one use case and is not intended to limit the scope of protection of this processing method. With morphological representation as nodes and maximizing community modularity as the goal, the clustering community structure is adjusted through node movement and community refinement operations to obtain feature clusters with organizational morphological semantics.

[0049] Pathologists annotate clusters based on semantic clustering results, associating high-dimensional cluster representations with tissue morphological patterns, providing semantically understandable feature representations for subsequent migration prediction and feature attribution. Specifically, for example... Figure 6 The diagram shown is a hierarchical iterative semantic clustering flowchart. The core objective is to maximize the modularity of the community, ultimately obtaining feature clusters with organizational morphological semantics. The complete process can be divided into two stages: Level 1 fine-grained node clustering and Level 2 coarse-grained community aggregation, with a total of 5 core steps: Phase 1: Level 1 fine-grained node clustering (for individual morphological representation nodes) 1. Initial state ( Figure 6In (a): all extracted tissue morphological features are treated as independent nodes in the graph, and the edges between nodes represent similarity associations between features. At this time, the nodes are randomly distributed and no clear community structure is formed.

[0050] 2. Step 1: Move nodes Figure 6 a→ Figure 6 (b) With the goal of maximizing the modularity formula H, nodes are moved one by one to their neighboring communities. The change in modularity after each move is calculated, and only the moves that improve the overall modularity are retained. After this step, the nodes begin to cluster, forming multiple initial small communities distinguished by color.

[0051] 3. Second step: Community refinement (Refine, Figure 6 b→ Figure 6 c): Fine-tune the initially formed community boundaries, merge loosely connected communities, and split communities with excessive internal heterogeneity, ultimately obtaining multiple semantic clusters with clear boundaries and optimal modularity at a fine-grained level (marked by semi-transparent areas of different colors in the figure).

[0052] Phase 2: Level 2 coarse-grained community aggregation (for fine-grained clustering) 4. Third step: Community aggregation (Aggregate, Figure 6 c→ Figure 6 In d): Each fine-grained cluster obtained from Level 1 is regarded as a "super node". The connection relationship between super nodes (i.e. the association between the original fine-grained communities) is preserved, while the node structure inside the original cluster is preserved as a subgraph to construct a hierarchical coarse-grained graph.

[0053] 5. Fourth step: Move nodes again. Figure 6 d→ Figure 6 e): On the coarse-grained supernode graph, repeat the node movement logic of Level 1 to adjust the ownership of supernodes with the goal of maximizing modularity, thereby optimizing the overall structure of the coarse-grained community.

[0054] 6. Fifth step: Further refine the community (Refine, Figure 6 e→ Figure 6 f): Perform final boundary optimization and structural adjustment on the coarse-grained supernode community to obtain the final semantic clustering results with multiple levels and multiple granularities.

[0055] Process output: After the above hierarchical iterative operation, the different levels of clusters obtained will be handed over to pathologists for semantic annotation, so that the high-dimensional morphological characterization clusters are matched one by one with interpretable tissue morphological patterns (such as tumor cell nests, stromal infiltration, necrotic areas, etc.), providing a semantically understandable feature basis for subsequent metastasis prediction and feature attribution.

[0056] like Figure 3 As shown in the figure, this application provides a schematic flowchart of a training method for an interpretable pathology report generation model. Specifically, the method includes the following steps: S301: Obtain a training dataset containing frozen section images and corresponding pathology report texts from thyroid nodule surgery; S302: Using the mapping database established by the method disclosed in the third aspect of this application, semantic enhancement annotation is performed on the image patches in the training dataset; S303: Construct an initial generative model including a visual encoder and a text decoder, wherein the visual encoder is a morphological representation learning network trained using the method disclosed in the first aspect of this application, or a pathological basic model fine-tuned across domains using the method disclosed in the second aspect of this application. S304: Input the intraoperative frozen section image into the visual encoder to extract feature sequences, and perform text embedding processing on the pathology report text to obtain text embedding features; S305: Input the feature sequence and text embedding features into the text encoder to generate predicted text; S306: Select key image patches based on diagnostic contribution, apply directional constraints to the attention distribution of the text decoder using the key image patches, calculate the generation loss in conjunction with the pathology report text, construct a training objective including supervised loss, jointly optimize the parameters of the initial generation model, and obtain an interpretable pathology report generation model; In some embodiments, the interpretable pathology report generation model can output a pathology report containing a histological description and analyze thyroid-related diseases.

[0057] In some embodiments, the method for training the supervised loss includes: Based on the selected key image blocks, the attention distribution of the text decoder during the generation process is supervised, guiding the text decoder to focus its attention weight on the image region corresponding to the key image block, so that the histomorphological description of the generated pathological report text is consistent with the pathological morphological features reflected by the key image block. Optionally, the supervised loss is also used to quantify the consistency of the contribution direction of each tissue morphological pattern to the prediction of lymph node metastasis. When the directions of multiple key morphological patterns are inconsistent, the loss value is increased to train the model to identify high-risk uncertain samples.

[0058] In this embodiment, to more clearly describe how the process of generating a pathology report is guided by diagnostic contributions, a multi-instance learning pathology report generation framework based on Transformer is used as an example for illustrative description. However, this description is only one use case and is not intended to limit the scope of protection of this processing method. Figure 8 As shown, this complete workflow uses the UNI base model, which has been fine-tuned for production adaptation, as the core of the visual encoder. The overall workflow is divided into 6 core stages, and the complete workflow is as follows: 1. Initial Extraction of Input and Visual Features The input to the process is whole-slice images (WSI) and their segmented image fragments. These pathological images are first fed into the Visual Extractor, which uses a UNI model that has been fine-tuned for slide adaptation as the core visual encoder to extract features from the input pathological image fragments and output the initial image patch feature sequence E0.

[0059] 2. Transformer encoder deep feature encoding The initial feature E0 is fed into N cascaded Encoder Blocks (Transformer encoder blocks) to complete the deep feature encoding.

[0060] The internal execution logic of a single encoder block is as follows: the features first pass through a multi-head self-attention layer to extract the correlation information between features of different image blocks; then they are processed by layer normalization and input into the feedforward network through residual connections; after another layer normalization and residual connections, the computation of a single encoder block is completed.

[0061] Each encoder block outputs the feature E of the corresponding level. i (i ranges from 1 to N, and the final output is the deep feature E_N).

[0062] 3. Multi-scale location-aware feature enhancement The hierarchical feature E output by each encoder block i All of these will be synchronously input into the corresponding Position-awareModule.

[0063] This module extracts multi-scale spatial location information from the input features using three convolutional layers with different receptive fields (kernel sizes of 3, 7, and 13, respectively), capturing local and global spatial relationships at different ranges. The outputs of the multi-scale convolutions are then fused to enhance the location information of the encoder features. Finally, the outputs of all location-aware modules are fused with the final output of the encoder to obtain an aggregated feature H that simultaneously considers global semantics and spatial location information.

[0064] 4. Screening of key diagnostic patches In the parallel process of feature extraction by the encoder, the method automatically filters the input WSI image fragments based on the diagnostic quantification contribution, and accurately selects key image patches that are highly relevant to pathological diagnosis. These key patches will serve as the core basis for subsequent training supervision.

[0065] 5. Transformer Decoder Text Generation The aggregated feature H of the image-text fusion is fed into N concatenated Decoder Blocks (Transformer decoder blocks), and the input of the decoder also includes the target text sequence processed by Text Embedding.

[0066] The internal execution logic of a single decoder block is as follows: First, the input text embedding features are processed by masked multi-head self-attention to avoid leakage of future information during text generation; after layer normalization, the input is fed into the multi-head attention layer and cross-attention is calculated with the aggregated features H from the encoder to complete the alignment of pathological image features and text features; then, it goes through layer normalization, feedforward network, and final layer normalization in sequence to complete the calculation of a single decoder block, and finally the decoder outputs the complete pathological report text.

[0067] 6. Supervised Loss and Model Optimization During the training phase of image-text alignment, the method constructs a Supervised Loss based on the selected diagnostic key patches and the text generation output of the decoder. This loss applies targeted supervision to the model's attention distribution, guiding the model to focus on key image patches related to pathological diagnosis during report generation, ultimately achieving the core goal of improving the accuracy of pathology report generation.

[0068] like Figure 4 As shown in the figure, this application provides a flowchart of a method for learning and analyzing the morphological characterization of pathological images during thyroid nodule surgery. Specifically, the method includes the following steps: S401: Obtain frozen section images of the thyroid nodule during surgery for analysis; In some embodiments, the frozen section images of the thyroid nodules to be analyzed in this application are obtained from a subject. The terms "subject," "test subject," or "sample to be tested" as used herein refer to any animal (e.g., a mammal), including but not limited to humans, non-human primates, rodents, etc., who will become the recipient of a particular treatment. Generally, the terms "subject" and "patient" are used interchangeably herein when referring to human subjects. Preferably, the subject is a human.

[0069] S402: Input the intraoperative frozen section image into the morphological representation learning network trained by the method disclosed in the first aspect of this application, and extract high-dimensional morphological representation vectors; S403: Using the mapping relationship library established by the method disclosed in the third aspect of this application, the high-dimensional morphological representation vector is mapped to the corresponding tissue morphological pattern description, and a pathological analysis report containing the tissue morphological pattern description is generated. In some embodiments, the morphological representation learning network may also be trained using the method disclosed in the second aspect of this application, namely by inserting a low-rank adapter into a pre-trained pathological baseline model and training only the low-rank adapter to adapt the pathological baseline model to frozen sections.

[0070] This application also discloses a computer device, which may include: one or more processors and one or more memories; wherein the memories store computer-readable code that, when run by the one or more processors, can perform the methods described above.

[0071] The processor in this embodiment can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, operations, and logic block diagrams disclosed in this embodiment. The general-purpose processor can be a microprocessor or any conventional processor, and can be based on an x86 or ARM architecture.

[0072] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0073] For example, the methods or apparatus according to embodiments of this disclosure can also be implemented using the architecture of a computing device. A computing device may include a bus, one or more CPUs, read-only memory (ROM), random access memory (RAM), a communication port connected to a network, input / output components, a hard disk, etc. Storage devices in the computing device, such as ROM or hard disk, may store various data or files used for processing and / or communication of the methods provided in this disclosure, as well as program instructions executed by the CPU. The computing device may also include a user interface. Of course, the architecture described above is merely exemplary, and one or more components of the computing device described above may be omitted as needed when implementing different devices.

[0074] This invention also provides a computer-readable storage medium storing computer-readable instructions. When the computer-readable instructions are executed by a processor, the methods disclosed in this embodiment can be performed. The computer-readable storage medium in this embodiment can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous interconnected dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0075] This disclosure also provides a computer program product or system, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0076] This embodiment provides a training system for a morphological representation network, used to implement the steps of the method described in the first aspect of this application, including: The pathology slide image acquisition module is used or configured to acquire intraoperative frozen pathology slide images and segment them into multiple frozen pathology image blocks. An enhancement processing module is used or configured to generate multiple enhanced views for each frozen pathology image patch, input to an initial feature extraction network, and extract morphological representation vectors for each enhanced view; The parameter update module is used or configured to calculate the contrast loss based on the representation vectors of different enhanced views of the same frozen pathological image patch and update the parameters of the initial feature extraction network. The morphological representation learning network training module is used or configured to update the parameters through the contrast loss, input the morphological standard vectors of reference pathological images from different preparation conditions into the domain discriminator, and through adversarial training, make it impossible to distinguish the preparation condition source of the morphological representation vectors output by the initial feature extraction network, so as to obtain a morphological representation learning network that eliminates preparation differences. Once the morphological representation learning network is trained, for any input frozen pathological slide image, the output morphological representation vector is a high-dimensional morphological representation that is robust to imaging changes and reflects the inherent morphological structure of pathological tissue. Moreover, the extraction process of the high-dimensional morphological representation does not depend on pixel-level labeled data.

[0077] This embodiment provides a system for constructing interpretable semantic mapping relationships, used to implement the steps of the method described in the third aspect of this application, including: A high-dimensional morphological representation vector acquisition module is used or configured to extract high-dimensional morphological representation vectors from frozen pathological slide images using a morphological representation learning network trained by the method disclosed in the first or second aspect of this application. The graph structure construction module is used or configured to construct a graph structure using the high-dimensional morphological representation vectors as nodes and the similarity between representation vectors as edges. A clustering module is used or configured to perform unsupervised semantic clustering on the graph structure to obtain multiple feature clusters; The mapping relationship library construction module is used or configured to obtain semantic annotations of the organizational morphological pattern descriptions of the multiple feature clusters and establish a mapping relationship library between the feature decisions and the organizational morphological pattern descriptions. The mapping relation library is used in pathological analysis tasks to transform extracted high-dimensional features into clinically interpretable morphological descriptions.

[0078] This embodiment provides a training system for an interpretable pathology report generation model, used to implement the steps of the method described in the fourth aspect of this application, including: The training dataset acquisition module is used or configured to acquire a training dataset containing frozen section images and corresponding pathology report texts from thyroid nodule surgeries. A semantic annotation module is used or configured to perform semantically enhanced annotation on image patches in the training dataset using a mapping database established by the method disclosed in the third aspect of this application; An initial generative model building module is used or configured to build an initial generative model containing a visual encoder and a text decoder, wherein the visual encoder is a morphological representation learning network trained using the method disclosed in the first aspect of this application, or a pathological basic model fine-tuned across domains using the method disclosed in the second aspect of this application. A text embedding processing module is used or configured to input the intraoperative frozen section image into the visual encoder to extract feature sequences, perform text embedding processing on the pathology report text, and obtain text embedding features. A predicted text generation module is used or configured to input the feature sequence and text embedding features into the text encoder to generate predicted text; The pathology report generation model module is used or configured to filter key image patches based on diagnostic contribution, apply directional constraints to the attention distribution of the text decoder using the key image patches, calculate the generation loss in conjunction with the pathology report text, construct a training objective including supervised loss, jointly optimize the parameters of the initial generation model, and obtain an interpretable pathology report generation model.

[0079] This embodiment provides a training system for a morphological representation network, used to implement the steps of the method described in the fifth aspect of this application, including: The image acquisition module is used or configured to acquire frozen section images of thyroid nodules during surgery. A high-dimensional morphological representation vector extraction module is used or configured to input the intraoperative frozen section image into the morphological representation learning network trained by the method disclosed in the first aspect of this application to extract high-dimensional morphological representation vectors. The pathology analysis report generation module is used or configured to use a mapping relationship library established by the method disclosed in the third aspect of this application to map the high-dimensional morphological representation vector to the corresponding tissue morphology pattern description, and generate a pathology analysis report containing the tissue morphology pattern description.

[0080] The exemplary embodiments of this disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art will understand that various modifications and combinations can be made to these embodiments or their features without departing from the principles and spirit of this disclosure, and such modifications should fall within the scope of this disclosure.

Claims

1. A training method for a morphological representation network, characterized in that, include: Acquire intraoperative frozen pathology slide images and segment them into multiple frozen pathology image blocks; For each frozen pathological image patch, multiple enhanced views are generated and input into the initial feature extraction network to extract the morphological representation vector of each enhanced view; The contrast loss is calculated based on the representation vectors of different enhanced views of the same frozen pathological image patch, and the parameters of the initial feature extraction network are updated. After updating the parameters through the contrast loss, the morphological standard vectors of reference pathological images from different preparation conditions are input into the domain discriminator. Through adversarial training, the source of the preparation condition of the morphological representation vector output by the initial feature extraction network cannot be distinguished, so as to obtain a morphological representation learning network that eliminates preparation differences. Once the morphological representation learning network is trained, for any input frozen pathological slide image, the output morphological representation vector is a high-dimensional morphological representation that is robust to imaging changes and reflects the inherent morphological structure of pathological tissue. Moreover, the extraction process of the high-dimensional morphological representation does not depend on pixel-level labeled data.

2. The training method for the morphological representation network according to claim 1, characterized in that, After updating the parameters using the contrast loss, the morphological standard vectors of reference pathological images from different slide preparation conditions are input into the domain discriminator. Adversarial training is then used to eliminate slide preparation differences, including: The intraoperative frozen pathology slide image block and the reference pathology image block are respectively input into the initial feature extraction network after the contrast loss update parameters to obtain their respective morphological representation vectors. The respective morphological representation vectors are input into the domain discriminator, which determines the source domain of each morphological representation vector and calculates the domain classification loss. The gradient of the domain classification loss is inverted by the gradient inversion layer and then passed to the initial feature extraction network, so that the initial feature extraction network updates its parameters in a way that confuses the domain discriminator in distinguishing the source of frozen pathological sections from reference pathological images. Optionally, generating multiple enhanced views for each frozen pathology image block includes performing at least two enhancement processes on the frozen pathology image block, such as color transformation, contrast adjustment, geometric transformation, and staining style transfer. Optionally, the reference pathological image is a paraffin-embedded pathological section image; Optionally, the initial feature extraction network is a neural network to be trained for extracting high-dimensional morphological features from pathological image patches.

3. The training method for the morphological representation network according to claim 1, characterized in that, The initial feature extraction network is a pathological baseline model pre-trained on a reference pathological image dataset; prior to the adversarial training, it also includes: Freeze the original weights of the pathological model; A low-rank adapter is inserted into the basic pathological model as a trainable model; The adversarial training is performed on the feature output of the low-rank adapter. By updating only the parameters of the low-rank adapter, the morphological feature capture capability learned by the pathological base model from the reference pathological image is losslessly generalized to the frozen pathological slide image.

4. A method for constructing interpretable semantic mapping relationships, characterized in that, include: Using the morphological representation learning network trained by any one of claims 1-3, high-dimensional morphological representation vectors of frozen pathological slide images are extracted. A graph structure is constructed using the high-dimensional morphological representation vectors as nodes and the similarity between representation vectors as edges. Unsupervised semantic clustering is performed on the graph structure to obtain multiple feature clusters; Obtain semantic annotations for the organizational morphological pattern descriptions of the multiple feature clusters, and establish a mapping relationship library between the feature decisions and the organizational morphological pattern descriptions; The mapping relation library is used in pathological analysis tasks to transform extracted high-dimensional features into clinically interpretable morphological descriptions. Optionally, the unsupervised semantic clustering of the graph structure is performed to obtain multiple feature clusters, including: Phase 1: Perform local node movement and calculate the change in modularity after each movement. Retain the movement results that improve the overall modularity to form an initial community of nodes. Refine the boundaries of the initial community and aggregate the nodes into multiple fine-grained communities. The second stage involves aggregating each fine-grained community into a super node, preserving the connections between super nodes, and retaining the node structure within the initial community to construct a coarse-grained community. Based on the coarse-grained community, node movement and community boundary refinement are performed again to obtain multi-level feature clusters.

5. A training method for an interpretable pathology report generation model, characterized in that, include: Obtain a training dataset containing frozen section images from thyroid nodule surgery and corresponding pathology report texts; Using the mapping database established by the method described in claim 4, semantically enhanced annotations are performed on image patches in the training dataset; Construct an initial generative model comprising a visual encoder and a text decoder, wherein the visual encoder is a morphological representation learning network trained by the method of claim 1, or a pathological basic model fine-tuned across domains by the method of claim 3. The intraoperative frozen section image is input into the visual encoder to extract feature sequences, and the pathology report text is subjected to text embedding processing to obtain text embedding features; The feature sequence and text embedding features are input into the text encoder to generate predicted text; Key image patches are selected based on their diagnostic contribution. These key image patches are then used to impose directional constraints on the attention distribution of the text decoder. The generation loss is calculated in conjunction with the pathology report text. A training objective containing supervised loss is constructed, and the parameters of the initial generation model are jointly optimized to obtain an interpretable pathology report generation model. Optionally, the interpretable pathology report generation model can output a pathology report containing histological morphology descriptions and analyze thyroid-related diseases.

6. The training method for the interpretable pathology report generation model according to claim 5, characterized in that, The training methods for the supervised loss include: Based on the selected key image blocks, the attention distribution of the text decoder during the generation process is supervised, guiding the text decoder to focus its attention weight on the image region corresponding to the key image block, so that the histomorphological description of the generated pathological report text is consistent with the pathological morphological features reflected by the key image block. Optionally, the supervised loss is also used to quantify the consistency of the contribution direction of each tissue morphological pattern to the prediction of lymph node metastasis. When the directions of multiple key morphological patterns are inconsistent, the loss value is increased to train the model to identify high-risk uncertain samples.

7. A method for learning and analyzing the morphological characterization of pathological images during thyroid nodule surgery, characterized in that, include: Obtain frozen section images of thyroid nodules during surgery for analysis; The intraoperative frozen section image is input into the morphological representation learning network trained by the method described in claim 1 to extract high-dimensional morphological representation vectors. Using the mapping relationship library established by the method described in claim 4, the high-dimensional morphological representation vector is mapped to the corresponding tissue morphological pattern description, and a pathological analysis report containing the tissue morphological pattern description is generated. Optionally, the morphological representation learning network can also be trained using the method of claim 3, i.e., by inserting a low-rank adapter into a pre-trained pathological baseline model and training only the low-rank adapter to adapt the pathological baseline model to frozen sections.

8. A computer device, characterized in that, The device includes: a memory and a processor; the memory is used to store a computer program; the processor executes the computer program to implement the steps of the method according to any one of claims 1-6, or to implement the steps of the method according to claim 7.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1-6, or implements the steps of the method as described in claim 7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-6, or the steps of the method according to claim 7.

Images