Pathological image analysis and processing methods, devices, equipment and storage media

Abstract

This invention discloses a method, apparatus, device, and storage medium for pathological image analysis and processing. The method includes: acquiring multiple instance features of a whole-slice image; constructing an instance feature set based on the instance features; initializing multiple learnable prior inductions based on prior knowledge to obtain initialized prior inductions; performing context modeling through the interaction between the instance feature set and the initialized prior inductions to obtain a context-aware instance feature set; generating a bag-level representation of the whole-slice image based on the context-aware instance feature set; and performing classification prediction on the whole-slice image based on the bag-level representation to obtain a classification prediction result. Because this invention initializes multiple learnable prior inductions based on prior knowledge and performs context modeling through the interaction between the instance feature set and the initialized prior inductions, compared to existing technologies, this invention not only reduces computational complexity but also improves classification prediction accuracy.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for pathological image analysis and processing. Background Technology

[0002] In computational pathology, whole-slide images (WSIs) are enormous, typically containing hundreds of thousands to millions of image patches (or instances). Due to the extremely high cost of fine-grained annotation, only slice-level diagnostic labels (such as disease categories) are usually available. Therefore, this problem is often modeled as a weakly supervised learning problem. Multiple-instance learning has become the mainstream paradigm for handling such tasks. It treats the entire WSI as a "bag" and the image patches as "instances," aiming to learn a model using only the bag-level labels to achieve accurate disease classification or prognostic prediction.

[0003] Existing multiple instance learning (MIL) methods can be mainly divided into two categories. The first category is non-interactive methods (non-interactive MIL), such as attention-based multiple instance learning and its variants. These methods typically first independently encode each image patch to obtain instance features, and then directly generate bag-level representations through aggregation mechanisms such as attention. Although such methods are relatively computationally efficient, the lack of explicit modeling of the dependencies between different instance features before aggregation leads to the loss of key contextual information, limiting the model's ability to capture complex pathological morphological patterns (such as the spatial distribution of tumors, the relationship between stroma reaction and cancer nests), and ultimately affecting classification performance.

[0004] The second category is interactive methods (interactive MIL), such as those based on the Transformer architecture or graph neural networks. These methods explicitly model the interactions between all instance features through global self-attention mechanisms or by constructing instance relationship graphs, thereby capturing long-range contextual information. However, these methods have significant drawbacks: First, their computational complexity is proportional to the square of the number of instances. When processing WSIs containing tens of thousands of instances, they incur enormous computational and memory overhead, making it difficult to scale to large-scale practical applications. Second, forcibly modeling the relationships between all instance pairs introduces a large number of redundant or even noisy connections, which not only reduces the model's inference efficiency but may also impair its robustness.

[0005] Therefore, there is an urgent need for a pathological image analysis and processing method that can reduce computational complexity and improve classification and prediction accuracy. Summary of the Invention

[0006] The main objective of this invention is to provide a method, apparatus, device, and storage medium for pathological image analysis and processing, aiming to solve the technical problems of high computational complexity and insufficient classification prediction accuracy in the prior art.

[0007] To achieve the above objectives, the present invention provides a method for pathological image analysis and processing, the method comprising the following steps:

[0008] Obtain multiple instance features of the full-slice image, and construct an instance feature set based on the instance features;

[0009] Multiple learnable prior inductions are initialized based on prior knowledge to obtain the initialized prior inductions;

[0010] Context modeling is performed through the interaction between the instance feature set and the initialized prior induction to obtain a context-aware instance feature set;

[0011] Based on the context-aware instance feature set, a bag-level representation of the full slice image is generated, and the full slice image is classified and predicted based on the bag-level representation to obtain the classification prediction result.

[0012] Optionally, the step of obtaining a context-aware instance feature set by performing context modeling through the interaction between the instance feature set and the initialized prior induction includes:

[0013] The initialized prior inducement is projected as a first query matrix, a first attention weight is determined based on the instance feature set, and the initialized prior inducement is updated using the first attention weight to obtain the first updated prior inducement.

[0014] The first updated prior induction is obtained by exchanging information between the first updated prior inductions through a multi-head self-attention mechanism.

[0015] The instance feature set is projected onto a second query matrix, a second attention weight is determined based on the second updated prior induction, and the instance feature set is updated using the second attention weight to obtain a context-aware instance feature set.

[0016] Optionally, the step of projecting the initialized prior inducement onto a first query matrix, determining a first attention weight based on the instance feature set, and updating the initialized prior inducement using the first attention weight to obtain a first updated prior inducement includes:

[0017] The initialized prior inducement is projected as a first query matrix, and the instance feature set is projected as a first key matrix and a first value matrix;

[0018] The first attention weight is determined based on the first query matrix and the first key matrix;

[0019] The first attention weight and the first value matrix are used to update the initialized prior induction to obtain the first updated prior induction.

[0020] Optionally, the step of projecting the instance feature set into a second query matrix, determining a second attention weight based on the second updated prior induction, and updating the instance feature set using the second attention weight to obtain a context-aware instance feature set includes:

[0021] The instance feature set is projected into a second query matrix, and the second updated prior inducement is projected into a second key matrix and a second value matrix;

[0022] The second attention weight is determined based on the second query matrix and the second key matrix;

[0023] The instance feature set is updated using the second attention weight and the second value matrix to obtain a context-aware instance feature set.

[0024] Optionally, the step of initializing multiple learnable prior inductions based on prior knowledge to obtain initialized prior inductions includes:

[0025] Acquire multimodal prior information and use the multimodal prior information as prior knowledge. The multimodal prior information includes at least one of visual feature information, text description information, and molecular expression profile information.

[0026] The prior knowledge is encoded into an embedding vector, and the embedding vector is used to initialize multiple learnable prior inductions to obtain initialized prior inductions.

[0027] Optionally, the method further includes:

[0028] Determine the first attention log number corresponding to the first attention weight, and normalize the first attention log number to obtain the normalized attention log number;

[0029] Based on the normalized attention logarithm, the similarity matrix between each of the initialized prior inductions is determined;

[0030] The diversity regularization loss is determined based on the values ​​of the off-diagonal elements in the similarity matrix.

[0031] The diversity regularization loss and classification prediction loss are combined to obtain the training objective function.

[0032] Optionally, the step of generating a bag-level representation of the full-slice image based on the context-aware instance feature set, and performing classification prediction on the full-slice image based on the bag-level representation to obtain a classification prediction result includes:

[0033] The attention pooling mechanism is used to aggregate each instance feature in the context-aware instance feature set to generate a bag-level representation of the full-slice image;

[0034] The whole slice image is classified and predicted based on the bag-level representation to obtain the classification prediction result.

[0035] Furthermore, to achieve the above objectives, the present invention also proposes a pathological image analysis and processing device, the device comprising:

[0036] The feature acquisition module is used to acquire multiple instance features of the whole slice image and construct an instance feature set based on the instance features;

[0037] The induction initialization module is used to initialize multiple learnable prior inductions based on prior knowledge to obtain the initialized prior inductions;

[0038] The hierarchical interaction module is used to perform context modeling through the interaction between the instance feature set and the initialized prior induction, and obtain a context-aware instance feature set.

[0039] The classification prediction module is used to generate a bag-level representation of the full slice image based on the context-aware instance feature set, and to perform classification prediction on the full slice image based on the bag-level representation to obtain the classification prediction result.

[0040] Furthermore, to achieve the above objectives, the present invention also proposes a pathological image analysis and processing device, the device comprising: a memory, a processor, and a pathological image analysis and processing program stored in the memory and executable on the processor, the pathological image analysis and processing program being configured to implement the steps of the pathological image analysis and processing method described above.

[0041] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing a pathological image analysis and processing program, wherein the pathological image analysis and processing program, when executed by a processor, implements the steps of the pathological image analysis and processing method described above.

[0042] This invention discloses a method for acquiring multiple instance features of a full-slice image, constructing an instance feature set based on these features, initializing multiple learnable prior inducements based on prior knowledge to obtain initialized prior inducements, performing context modeling through the interaction between the instance feature set and the initialized prior inducements to obtain a context-aware instance feature set, generating a bag-level representation of the full-slice image based on the context-aware instance feature set, and performing classification prediction on the full-slice image based on the bag-level representation to obtain a classification prediction result. Because this invention initializes multiple learnable prior inducements based on prior knowledge and performs context modeling through the interaction between the instance feature set and the initialized prior inducements to obtain a context-aware instance feature set, compared to existing technologies, this invention not only reduces computational complexity but also improves classification prediction accuracy. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating the first embodiment of the pathological image analysis and processing method of the present invention;

[0044] Figure 2 This is a schematic diagram of the overall architecture of HookMIL in the pathological image analysis and processing method of the present invention;

[0045] Figure 3 This is a flowchart illustrating the second embodiment of the pathological image analysis and processing method of the present invention;

[0046] Figure 4 Example diagram of text initialization based on prostate hierarchical architecture pattern;

[0047] Figure 5 This is a flowchart illustrating the third embodiment of the pathological image analysis and processing method of the present invention;

[0048] Figure 6 A comparative diagram of different prior-induced attention maps with and without diversity loss;

[0049] Figure 7 This is a schematic diagram showing the alignment of HookMIL's attention distribution with the tumor region on a whole-slice image.

[0050] Figure 8 This is a structural block diagram of the first embodiment of the pathological image analysis and processing device of the present invention;

[0051] Figure 9 This is a schematic diagram of the structure of a pathological image analysis and processing device in the hardware operating environment involved in the embodiments of the present invention.

[0052] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0053] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0054] This invention provides a method for pathological image analysis and processing, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the pathological image analysis and processing method of the present invention.

[0055] In this embodiment, the pathological image analysis and processing method includes steps S10 to S40:

[0056] Step S10: Obtain multiple instance features of the full slice image, and construct an instance feature set based on the instance features.

[0057] It should be noted that the executing entity in this embodiment can be a computer server device with data processing, network communication, and program execution functions applied in a pathological image analysis scenario, such as a server, tablet computer, or personal computer, or an electronic device capable of performing the above functions (such as a pathological image analysis and processing device). The following uses a system containing a pathological image analysis and processing device (hereinafter referred to as the system) as an example to illustrate this embodiment and the following embodiments.

[0058] It should be noted that this embodiment provides an implementation framework called HookMIL, which is a context-aware multi-instance learning framework based on learnable prior-induced context for whole-slice image analysis in computational pathology.

[0059] It should be understood that instance features can refer to fixed-dimensional numerical vectors (feature vectors) obtained after processing a single image patch (patch) from a whole slice image (WSI) using a pre-trained feature extraction model. An instance feature set can refer to a data matrix formed by sequentially arranging and combining the instance features corresponding to all image patches of a complete whole slice image.

[0060] In a practical implementation, the full-slice image can be divided into multiple image patches; then, a pre-trained feature extraction model is used to extract features from each image patch to obtain the corresponding feature vector (instance features); and finally, the feature vectors corresponding to all image patches are combined into an instance feature set.

[0061] For example, a whole slice image is segmented into Each image patch (instance) is processed by a pre-trained feature extraction model (such as UNI or CONCH) to extract features from each image patch, resulting in a corresponding feature vector. , constitute the instance feature set In this context, i is the instance index, and D is the feature dimension. For example, N can range from 5000 to 20000, and the feature dimension D can be 512 or 1024.

[0062] Step S20: Initialize multiple learnable prior inductions based on prior knowledge to obtain the initialized prior inductions.

[0063] It should be understood that prior knowledge can refer to knowledge related to pathological morphology and diagnosis that has been learned from other data sources or models before the current pathological image classification model is started.

[0064] It should be explained that prior induction can refer to a fixed set of learnable vectors (or tokens, anchors) (denoted as K, much smaller than the number of image patches N) that function as proxies or hubs for context modeling. They do not directly correspond to a specific image patch, but are designed to actively hook and summarize scattered, related pathological pattern information in the entire WSI.

[0065] For example, in a specific implementation, introduce A learnable prior inducement ,in much smaller The preferred range is Different initialization strategies can be provided to initialize multiple learnable prior inductions, incorporating different forms of prior knowledge. For example, 1. Default initialization (HookMIL-TN): The prior inductions are randomly initialized from a pre-defined truncated normal distribution, i.e. This strategy does not rely on external knowledge and serves as a baseline; 2. Visual feature initialization (HookMIL-Vis): Representative lesion regions are selected from the training set (e.g., determined through cluster centers or expert annotations), and their visual feature vectors are extracted as initial values ​​for prior induction. Specifically, the K-means algorithm can be used to cluster the features of all instances in the training set, taking the top... The visual feature vectors of the cluster centers are initialized.

[0066] It should be noted that initializing multiple learnable prior inducements based on prior knowledge is equivalent to directly encoding domain knowledge into the prior inducements, which significantly accelerates convergence and improves model performance.

[0067] Step S30: Context modeling is performed through the interaction between the instance feature set and the initialized prior induction to obtain a context-aware instance feature set.

[0068] It should be noted that the number of prior inducements after initialization is much smaller than the number of instance features in the instance feature set, making the computational complexity of the context modeling step linearly related to the number of instance features.

[0069] It should be understood that context modeling can be an indirect, hierarchical interaction process of instance features → prior induction → instance features. It avoids pairwise interactions between instance features, thereby reducing the computational complexity from O(N²) to O(NK+K²) (K is much smaller than N), thus achieving highly efficient global context awareness.

[0070] It should be noted that the step of obtaining a context-aware instance feature set by performing context modeling through the interaction between the instance feature set and the initialized prior induction includes: steps S301~S303:

[0071] Step S301: Project the initialized prior inducement onto a first query matrix, determine a first attention weight based on the instance feature set, and update the initialized prior inducement using the first attention weight to obtain the first updated prior inducement.

[0072] It should be explained that the initialized prior inducement can be projected as a first query matrix, and the instance feature set can be projected as a first key matrix and a first value matrix; a first attention weight is determined based on the first query matrix and the first key matrix; the initialized prior inducement is updated using the first attention weight and the first value matrix to obtain the first updated prior inducement.

[0073] It should be noted that the first query matrix, the first key matrix, and the first value matrix are respectively... .in, Let H be the learnable projection matrix, H be the initialized prior induction, and X be the set of instance features.

[0074] It should be added that the first attention weight The calculation formula is: First updated prior inducement The update formula is: .

[0075] It should be noted that the above step of updating the initialized prior inducement using the first attention weight and the first value matrix to obtain the first updated prior inducement enables each prior inducement to aggregate the instance information most relevant to it.

[0076] Step S302: Information is exchanged between the first updated prior inductions through a multi-head self-attention mechanism to obtain the second updated prior inductions.

[0077] It should be understood that multi-head self-attention is an extension of the attention mechanism. It breaks down an attention process into multiple parallel heads. Each head has an independent set of linear transformation parameters (projection matrices), allowing the model to focus on information at different positions in the input sequence from different angles and with different emphases in different representation subspaces. The prior inductions after the first update can exchange information through the multi-head self-attention mechanism to model higher-order contextual patterns.

[0078] Understandably, enabling information exchange between the first updated prior inducements through a multi-head self-attention mechanism can refer to using an attention computing unit with multiple parallel sub-modules to perform multi-perspective information fusion and feature enhancement on a set of learnable tokens (the first updated prior inducements) that are far fewer in number than image patches and have already initially aggregated image patch information, thereby forming a set of feature vectors that can represent complex, high-order pathological context patterns (i.e., the second updated prior inducements).

[0079] It should be noted that a multi-head self-attention mechanism is used to exchange information between the first updated prior inductions to obtain the formula corresponding to the second updated prior induction, as shown below:

[0080] ;

[0081] .

[0082] In the formula, MHSA stands for Multi-Head Self-Attention Mechanism. For the first updated prior inducement Intermediate results processed by a multi-head self-attention mechanism This is the second updated prior inducement. Representation layer normalization.

[0083] Step S303: Project the instance feature set into a second query matrix, determine the second attention weight based on the second updated prior induction, and update the instance feature set using the second attention weight to obtain a context-aware instance feature set.

[0084] It should be explained that the instance feature set can be projected into a second query matrix, and the second updated prior induction can be projected into a second key matrix and a second value matrix; a second attention weight is determined based on the second query matrix and the second key matrix; the instance feature set is updated using the second attention weight and the second value matrix to obtain a context-aware instance feature set.

[0085] It should be noted that the second query matrix, the second key matrix, and the second value matrix are respectively... .in, For another set of learnable projection matrices, This is the second updated prior inducement.

[0086] It should be added that the second attention weight The calculation formula is: Context-aware set of instance features The update formula is: .

[0087] It should be noted that the above step of updating the instance feature set using the second attention weights and the second value matrix to obtain a context-aware instance feature set makes the output... This is to integrate instance-level representations with rich context (i.e., context-aware set of instance features).

[0088] Step S40: Generate a bag-level representation of the full slice image based on the context-aware instance feature set, and perform classification prediction on the full slice image based on the bag-level representation to obtain the classification prediction result.

[0089] It should be noted that the bag-level representation can refer to a feature vector of a fixed dimension (such as a 512-dimensional or 1024-dimensional vector). It is the result of a global and generalized summary of all context-modeled instance features (i.e., the context-aware set of instance features) in the entire whole slice image (WSI).

[0090] Understandably, the classification prediction result can refer to the final disease category judgment made on the entire whole slice image (WSI), which can be a probability distribution or a specific category label.

[0091] In a specific implementation, attention pooling can be used to aggregate each instance feature in the context-aware instance feature set to generate a bag-level representation of the full slice image; the full slice image can then be classified and predicted based on the bag-level representation to obtain the classification prediction result.

[0092] It should be understood that, based on the bag-level representation, the whole-slice image can be classified and predicted using a large model (i.e., a large language model) to obtain the classification prediction result. This embodiment can provide a new approach for the pre-training of large models at the pathological slice level.

[0093] It should be noted that the formula for aggregating each instance feature in the context-aware instance feature set using the attention pooling mechanism to generate the bag-level representation of the full-slice image is as follows:

[0094] ;

[0095] .

[0096] In the formula, For instance features in a context-aware set of instance features, Let be the attention weight for the feature of the i-th instance, tanh be the hyperbolic tangent activation function, exp be the exponential function, and Z be the bag-level representation. , For learnable parameters, It is usually set to 256 or 512.

[0097] For example, refer to Figure 2 , Figure 2 This is a schematic diagram of the overall architecture of HookMIL in the pathological image analysis and processing method of the present invention. The diagram includes four modules: a feature extraction module, a priori induction initialization module, a priori induction projection module, and a priori induction aggregation module. The feature extraction module processes the input image through a visual encoder, extracting a large number of image block-level visual features (i.e., instance features). The priori induction initialization module uses various strategies such as template matching and semantic guidance to guide and initialize learnable induction parameters from multimodal prior information (such as visual feature information, textual description information, and molecular expression profile information) to initialize multiple learnable priori induction tokens, obtaining initialized priori induction tokens. Then, the initialized priori induction tokens interact with instance features through the priori induction projection module and the priori induction aggregation module to perform context modeling and obtain a context-aware set of instance features.

[0098] This embodiment discloses acquiring multiple instance features of a full-slice image, constructing an instance feature set based on these features, initializing multiple learnable prior inducements based on prior knowledge to obtain initialized prior inducements, performing context modeling through the interaction between the instance feature set and the initialized prior inducements to obtain a context-aware instance feature set, generating a bag-level representation of the full-slice image based on the context-aware instance feature set, and performing classification prediction on the full-slice image based on the bag-level representation to obtain a classification prediction result. Because this embodiment initializes multiple learnable prior inducements based on prior knowledge and performs context modeling through the interaction between the instance feature set and the initialized prior inducements to obtain a context-aware instance feature set, compared to existing technologies, this embodiment not only reduces computational complexity but also improves classification prediction accuracy.

[0099] refer to Figure 3 , Figure 3This is a flowchart illustrating the second embodiment of the pathological image analysis and processing method of the present invention.

[0100] Based on the first embodiment described above, in this embodiment, step S20 includes steps S201 to S202:

[0101] Step S201: Obtain multimodal prior information and use the multimodal prior information as prior knowledge. The multimodal prior information includes at least one of visual feature information, text description information, and molecular expression profile information.

[0102] Step S202: Encode the prior knowledge into an embedding vector, and use the embedding vector to initialize multiple learnable prior inductions to obtain initialized prior inductions.

[0103] It should be noted that, based on multimodal prior information including at least one of visual feature information, textual description information, and molecular expression profile information, this embodiment provides three additional initialization strategies besides the default initialization (HookMIL-TN), for a total of four initialization strategies, to incorporate different forms of prior knowledge: For example, 1. Default initialization (HookMIL-TN): Prior induction randomly initializes from a preset truncated normal distribution, i.e. This strategy does not rely on external knowledge and serves as a baseline; 2. Visual feature initialization (HookMIL-Vis): Representative lesion regions are selected from the training set (e.g., determined through cluster centers or expert annotations), and their visual feature vectors are extracted as initial values ​​for prior induction. Specifically, the K-means algorithm can be used to cluster the features of all instances in the training set, taking the top... 3. Initialization of visual feature vectors of cluster centers: Initialization of visual feature vectors of cluster centers; 4. Initialization of text embedding (HookMIL-Txt): Using a visual-language pathology model (such as CONCH), pathological diagnostic terms or tissue morphology descriptions (e.g., "invasive cancer nest", "glandular structure destruction", "lymphocyte infiltration") are encoded into text embedding vectors to initialize prior induction. For example, in the prostate cancer task, Gleason grading descriptors such as "Well-formed, round, separate glands" can be used; 5. Initialization of spatial transcriptomics (HookMIL-ST): In scenarios with spatial transcriptomics data, using a multimodal model (such as Loki), gene expression profiles or molecular descriptors related to tissue regions are mapped into embedding vectors to initialize prior induction, thereby guiding the learning of visual representations through molecular information.

[0104] Understandably, compared to default initialization, initialization based on multimodal prior information (such as visual feature information, textual description information, and molecular expression profile information) provides the model with a structured starting point that is highly relevant to the task. This enables the model to have the initial ability to identify key pathological morphologies or concepts in the early stages of training, significantly accelerating the model convergence process and ultimately achieving higher classification accuracy, AUC, and other performance metrics.

[0105] It should be understood that the default initialization (HookMIL-TN) provides a baseline scheme that does not rely on external data. Visual feature initialization, text embedding initialization, and spatial transcriptomics initialization, on the other hand, allow for flexible adaptation based on the availability of data in the actual application scenario (whether there are labeled regions, text descriptions, or molecular data).

[0106] refer to Figure 4 The diagram illustrates how the prostate cancer grading architecture is transformed into prior induction tokens (i.e., initialized prior inductions) through a specific process. The prior knowledge is textual descriptive information. The diagram explicitly lists specific morphological textual descriptions related to the Gleason grading of prostate cancer (e.g., "well-formed, round, separated glands," "infiltrative but still separated small glands," etc.). This descriptive text is first processed by a tokenizer, transforming it into discrete word sequences. These words are then input into a text Transformer model for encoding, ultimately outputting a series of dense text embedding vectors. As indicated by the flow arrows, these text embedding vectors are directly used to initialize the prior induction tokens.

[0107] This embodiment discloses the acquisition of multimodal prior information and the use of this multimodal prior information as prior knowledge. The multimodal prior information includes at least one of visual feature information, textual description information, and molecular expression profile information. The prior knowledge is encoded into embedding vectors, and these embedding vectors are used to initialize multiple learnable prior inductions to obtain initialized prior inductions. Because this embodiment uses multimodal prior information as prior knowledge and then encodes it into embedding vectors to initialize multiple learnable prior inductions, it accelerates the model convergence process and improves model performance compared to existing technologies.

[0108] refer to Figure 5 , Figure 5 This is a flowchart illustrating the third embodiment of the pathological image analysis and processing method of the present invention.

[0109] Based on the above embodiments, in this embodiment, the pathological image analysis and processing method further includes steps S01 to S04:

[0110] Step S01: Determine the first attention log number corresponding to the first attention weight, and normalize the first attention log number to obtain the normalized attention log number.

[0111] Step S02: Based on the normalized attention logarithm, determine the similarity matrix between each of the initialized prior inducements.

[0112] Step S03: Determine the diversity regularization loss based on the values ​​of the off-diagonal elements in the similarity matrix.

[0113] Step S04: Combine the diversity regularization loss and the classification prediction loss to obtain the training objective function.

[0114] In specific implementation, it can be made The formula for calculating the first attention weight is the attention logarithm (its) before softmax; then it is normalized: ,in Then calculate the similarity matrix between the initialized prior inductions: Finally, the diversity regularization loss is determined, which is the average of the sum of squares of the off-diagonal elements in the similarity matrix: This loss function encourages different priors to focus on different subsets of instances, thereby learning complementary morphological patterns.

[0115] It should be noted that the training objective function is in This is the cross-entropy loss, i.e., the classification prediction loss. The weighting coefficients for the diversity regularization loss are, for example, determined experimentally to have an optimal range of [value missing]. .

[0116] For example, refer to Figure 6 The figure visualizes the specific effects of diversity regularization loss on model training. The image compares the attention heatmaps of multiple prior inducements 1-8 on pathological slide images under different regularization strength coefficients (λ=0 and λ=0.1). When λ=0, i.e., without applying diversity regularization loss, the heatmap patterns generated by different prior inducements may be similar or overlapping, indicating that they have failed to effectively learn diverse patterns. When λ=0.1, i.e., with the diversity regularization loss applied, the attention regions of different prior inducements exhibit more obvious differentiation and complementarity (represented by color labels "high value" and "low value"), intuitively demonstrating that this method can effectively drive each learnable prior inducement token to focus on different, distinctive image regions or visual patterns, thereby enhancing the model's interpretability and feature representation capabilities.

[0117] refer to Figure 7This figure visually demonstrates the effectiveness of the method provided by this invention in multi-instance learning (MIL) of pathological images. Four whole-slice image (WSI) examples are shown in the figure, each containing the original whole-slice image on the left and the corresponding Attention Weights (ATTN) visualization image on the right. The whole-slice images present the pathological morphological features of human tissues (e.g., lung tissue). The attention weights map on the right reveals the degree of salience assigned to different instance features in the image by the core prior induction and hierarchical interaction of this invention. Red areas represent high attention values ​​(i.e., key areas that the model considers to contribute significantly to the final diagnostic prediction), while blue areas represent low attention values. This visualization result generates bag-level prediction results for the whole-slice images based on the context-aware instance features, as well as the intrinsic decision-making basis for attention pooling of the context-aware instance features. In particular, the context-aware instance feature set generated through hierarchical interaction allows attention to be more precisely focused on pathologically significant tissue regions, such as tumor cell-rich areas, rather than scattered across the background or non-critical tissues, which verifies the effectiveness of the interaction mechanism. This figure visually demonstrates the technical effect of this invention in achieving more accurate and interpretable pathological image classification by introducing learnable prior induction tokens and combining them with attention mechanisms for contextual modeling.

[0118] This embodiment discloses determining the first attention logarithm corresponding to the first attention weight, and normalizing the first attention logarithm to obtain a normalized attention logarithm; based on the normalized attention logarithm, determining the similarity matrix between each initialized prior inducement; determining the diversity regularization loss based on the values ​​of the off-diagonal elements in the similarity matrix; and combining the diversity regularization loss and the classification prediction loss to obtain the training objective function. Compared with the prior art, this embodiment, through the diversity regularization loss, can actively guide different prior inducements to mine diverse discriminative features in the data during model training, enabling each prior inducement to automatically focus on a unique pathological morphological pattern, improving the transparency and credibility of model decisions, and thus enhancing model interpretability.

[0119] Furthermore, this embodiment of the invention also proposes a storage medium storing a pathological image analysis and processing program, which, when executed by a processor, implements the steps of the pathological image analysis and processing method described above.

[0120] Reference Figure 8 , Figure 8 This is a structural block diagram of the first embodiment of the pathological image analysis and processing device of the present invention.

[0121] like Figure 8As shown, the pathological image analysis and processing device proposed in this embodiment of the invention includes: a feature acquisition module 501, an induction initialization module 502, a hierarchical interaction module 503, and a classification prediction module 504.

[0122] The feature acquisition module 501 is used to acquire multiple instance features of the full slice image and construct an instance feature set based on the instance features.

[0123] The induction initialization module 502 is used to initialize multiple learnable prior inductions based on prior knowledge to obtain initialized prior inductions.

[0124] The hierarchical interaction module 503 is used to perform context modeling through the interaction between the instance feature set and the initialized prior induction, and obtain a context-aware instance feature set.

[0125] The classification prediction module 504 is used to generate a bag-level representation of the full slice image based on the context-aware instance feature set, and to perform classification prediction on the full slice image based on the bag-level representation to obtain a classification prediction result.

[0126] The hierarchical interaction module 503 is further configured to project the initialized prior induction into a first query matrix, determine a first attention weight based on the instance feature set, and update the initialized prior induction using the first attention weight to obtain a first updated prior induction; exchange information between the first updated prior inductions through a multi-head self-attention mechanism to obtain a second updated prior induction; project the instance feature set into a second query matrix, determine a second attention weight based on the second updated prior induction, and update the instance feature set using the second attention weight to obtain a context-aware instance feature set.

[0127] The hierarchical interaction module 503 is further configured to project the initialized prior inducement into a first query matrix, and project the instance feature set into a first key matrix and a first value matrix; determine a first attention weight based on the first query matrix and the first key matrix; and update the initialized prior inducement using the first attention weight and the first value matrix to obtain a first updated prior inducement.

[0128] The hierarchical interaction module 503 is further configured to project the instance feature set into a second query matrix, and project the second updated prior induction into a second key matrix and a second value matrix; determine a second attention weight based on the second query matrix and the second key matrix; and update the instance feature set using the second attention weight and the second value matrix to obtain a context-aware instance feature set.

[0129] The classification prediction module 504 is further configured to aggregate each instance feature in the context-aware instance feature set using an attention pooling mechanism to generate a bag-level representation of the full slice image; and to perform classification prediction on the full slice image based on the bag-level representation to obtain a classification prediction result.

[0130] This device embodiment discloses acquiring multiple instance features of a full-slice image, constructing an instance feature set based on these features, initializing multiple learnable prior inducements based on prior knowledge to obtain initialized prior inducements, performing context modeling through the interaction between the instance feature set and the initialized prior inducements to obtain a context-aware instance feature set, generating a bag-level representation of the full-slice image based on the context-aware instance feature set, and performing classification prediction on the full-slice image based on the bag-level representation to obtain a classification prediction result. Because this device embodiment initializes multiple learnable prior inducements based on prior knowledge and performs context modeling through the interaction between the instance feature set and the initialized prior inducements to obtain a context-aware instance feature set, compared to existing technologies, this device embodiment not only reduces computational complexity but also improves classification prediction accuracy.

[0131] Based on the first embodiment of the pathological image analysis and processing device of the present invention, a second embodiment of the pathological image analysis and processing device of the present invention is proposed.

[0132] In this embodiment, the induction initialization module 502 is further configured to acquire multimodal prior information and use the multimodal prior information as prior knowledge. The multimodal prior information includes at least one of visual feature information, text description information, and molecular expression profile information. The prior knowledge is encoded into an embedding vector, and the embedding vector is used to initialize multiple learnable prior inductions to obtain initialized prior inductions.

[0133] Other embodiments or specific implementations of the pathological image analysis and processing device of the present invention can be referred to the above-described method embodiments, and will not be repeated here.

[0134] This application provides a pathological image analysis and processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the pathological image analysis and processing method in the first embodiment described above.

[0135] The following is for reference. Figure 9The diagram illustrates a structural schematic of a pathological image analysis and processing device suitable for implementing embodiments of this application. The pathological image analysis and processing device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The pathological image analysis and processing device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0136] like Figure 9 As shown, the pathological image analysis and processing device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the pathological image analysis and processing device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the pathological image analysis and processing equipment to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows pathological image analysis and processing equipment with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0137] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0138] The pathological image analysis and processing device provided in this application, employing the pathological image analysis and processing method in the above embodiments, can solve the technical problems of high computational complexity and insufficient classification prediction accuracy in the prior art. Compared with the prior art, the beneficial effects of the pathological image analysis and processing device provided in this application are the same as those of the pathological image analysis and processing method provided in the above embodiments, and other technical features in this pathological image analysis and processing device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0139] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0140] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0141] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0142] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0143] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0144] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for analyzing and processing pathological images, characterized in that, The method includes: Obtain multiple instance features of the full-slice image, and construct an instance feature set based on the instance features; Multiple learnable prior inductions are initialized based on prior knowledge to obtain the initialized prior inductions; Context modeling is performed through the interaction between the instance feature set and the initialized prior induction to obtain a context-aware instance feature set; Based on the context-aware instance feature set, a bag-level representation of the full slice image is generated, and the full slice image is classified and predicted based on the bag-level representation to obtain the classification prediction result; The step of obtaining a context-aware instance feature set by performing context modeling through the interaction between the instance feature set and the initialized prior induction includes: The initialized prior inducement is projected as a first query matrix, a first attention weight is determined based on the instance feature set, and the initialized prior inducement is updated using the first attention weight to obtain the first updated prior inducement. The first updated prior induction is obtained by exchanging information between the first updated prior inductions through a multi-head self-attention mechanism. The instance feature set is projected into a second query matrix, a second attention weight is determined based on the second updated prior induction, and the instance feature set is updated using the second attention weight to obtain a context-aware instance feature set. The step of initializing multiple learnable prior inductions based on prior knowledge to obtain initialized prior inductions includes: Acquire multimodal prior information and use the multimodal prior information as prior knowledge. The multimodal prior information includes at least one of visual feature information, text description information, and molecular expression profile information. The prior knowledge is encoded into an embedding vector, and the embedding vector is used to initialize multiple learnable prior inductions to obtain initialized prior inductions.

2. The pathological image analysis and processing method as described in claim 1, characterized in that, The steps of projecting the initialized prior inducement onto a first query matrix, determining a first attention weight based on the instance feature set, and updating the initialized prior inducement using the first attention weight to obtain the first updated prior inducement include: The initialized prior inducement is projected as a first query matrix, and the instance feature set is projected as a first key matrix and a first value matrix; The first attention weight is determined based on the first query matrix and the first key matrix; The first attention weight and the first value matrix are used to update the initialized prior induction to obtain the first updated prior induction.

3. The pathological image analysis and processing method as described in claim 1, characterized in that, The steps of projecting the instance feature set into a second query matrix, determining a second attention weight based on the second updated prior induction, and updating the instance feature set using the second attention weight to obtain a context-aware instance feature set include: The instance feature set is projected into a second query matrix, and the second updated prior inducement is projected into a second key matrix and a second value matrix; The second attention weight is determined based on the second query matrix and the second key matrix; The instance feature set is updated using the second attention weight and the second value matrix to obtain a context-aware instance feature set.

4. The pathological image analysis and processing method as described in claim 1, characterized in that, The method further includes: Determine the first attention log number corresponding to the first attention weight, and normalize the first attention log number to obtain the normalized attention log number; Based on the normalized attention logarithm, the similarity matrix between each of the initialized prior inductions is determined; The diversity regularization loss is determined based on the values ​​of the off-diagonal elements in the similarity matrix. The diversity regularization loss and classification prediction loss are combined to obtain the training objective function.

5. The pathological image analysis and processing method as described in claim 1, characterized in that, The step of generating a bag-level representation of the whole slice image based on the context-aware instance feature set, and performing classification prediction on the whole slice image based on the bag-level representation to obtain the classification prediction result includes: The attention pooling mechanism is used to aggregate each instance feature in the context-aware instance feature set to generate a bag-level representation of the full-slice image; The whole slice image is classified and predicted based on the bag-level representation to obtain the classification prediction result.

6. A pathological image analysis and processing device, characterized in that, The device includes: The feature acquisition module is used to acquire multiple instance features of the whole slice image and construct an instance feature set based on the instance features; The induction initialization module is used to initialize multiple learnable prior inductions based on prior knowledge to obtain the initialized prior inductions; The hierarchical interaction module is used to perform context modeling through the interaction between the instance feature set and the initialized prior induction, and obtain a context-aware instance feature set. The classification prediction module is used to generate a bag-level representation of the full slice image based on the context-aware instance feature set, and to perform classification prediction on the full slice image based on the bag-level representation to obtain the classification prediction result; The hierarchical interaction module is further configured to project the initialized prior induction into a first query matrix, determine a first attention weight based on the instance feature set, and update the initialized prior induction using the first attention weight to obtain a first updated prior induction; exchange information between the first updated prior inductions through a multi-head self-attention mechanism to obtain a second updated prior induction; project the instance feature set into a second query matrix, determine a second attention weight based on the second updated prior induction, and update the instance feature set using the second attention weight to obtain a context-aware instance feature set; The induction initialization module is further configured to acquire multimodal prior information and use the multimodal prior information as prior knowledge, wherein the multimodal prior information includes at least one of visual feature information, text description information and molecular expression profile information; encode the prior knowledge into an embedding vector, and use the embedding vector to initialize multiple learnable prior inductions to obtain initialized prior inductions.

7. A pathological image analysis and processing device, characterized in that, The device includes: a memory, a processor, and a pathological image analysis and processing program stored in the memory and executable on the processor, the pathological image analysis and processing program being configured to implement the steps of the pathological image analysis and processing method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores a pathological image analysis and processing program, which, when executed by a processor, implements the steps of the pathological image analysis and processing method as described in any one of claims 1 to 5.

Images