Pathological image analysis method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose a pathological image analysis method and device, the method comprising: acquiring a pathological image dataset, each pathological image having different magnifications; cutting each pathological image respectively to obtain a plurality of image blocks in each pathological image; performing feature extraction on each image block to obtain image features of each image block, and performing feature fusion on the image features of each image block according to the corresponding relationship between each image block to obtain multi-magnification features of each image block; constructing a panoramic image according to each image block of different magnifications, the spatial position relationship of each image block in the belonging pathological image, and the multi-magnification features of each image block; inputting the panoramic image into a constructed context information perception model to learn the graph structure information and the context information in each pathological image, and obtaining an analysis result of the pathological image dataset; and the analysis result is used for a target task. The present application solves the problem of insufficient feature information acquisition and low accuracy in the prior art.

Description

Technical Field

[0001] This invention relates to the field of medical image processing, and more particularly to a method, apparatus, electronic device, and storage medium for pathological image analysis. Background Technology

[0002] Histopathological diagnosis is the gold standard for cancer identification in modern medicine and is the most important and reliable method in cancer diagnosis. Currently, the process of diagnosing cancer through artificial pathological sections is lengthy, requiring experienced pathologists to examine tissue samples under microscopes at different magnifications after preparation. Based on various biological indicators, such as the number of mitotic cells, atypia of cancer cells, the proportion of cancer cells, and the ratio of tumor parenchyma to stroma, they determine whether the patient has cancer and its severity, completing the pathological diagnosis and prognostic assessment to implement effective treatment measures. Due to the existence of tumor heterogeneity, a more comprehensive utilization of the various information contained in pathological images is needed in tasks such as tumor subtype classification and prognostic prediction.

[0003] However, manually calculating various biological indicators by pathologists is not only time-consuming and inefficient, but may also lead to the omission of key characteristic indicators, resulting in misdiagnosis. Furthermore, cancer is a complex disease; its pathological manifestations vary significantly among different patients, at different stages of disease progression, and in different tissues and organs, exhibiting strong heterogeneity. This also presents a significant challenge to pathological diagnosis.

[0004] Therefore, there is an urgent need for a pathological image analysis method that can fully acquire the feature information of images and has high accuracy. Summary of the Invention

[0005] The embodiments of the present invention provide a pathological image analysis method, device, electronic device, and storage medium to solve the problem that pathological image analysis methods in related technologies cannot fully acquire the feature information of images, resulting in low image analysis accuracy.

[0006] The technical solution adopted in this invention is as follows:

[0007] According to one aspect of the present invention, a pathological image analysis method includes: acquiring a pathological image dataset, the pathological image dataset including multiple pathological images, each pathological image having a different magnification; segmenting each pathological image at different magnifications to obtain multiple image blocks in each pathological image; the image blocks belonging to different pathological images having different magnifications, and the image blocks belonging to the same pathological image having the same magnification, and the image blocks at different magnifications having a corresponding relationship based on a pyramid cascade arrangement between the pathological images; and performing feature extraction on each image block to obtain an image of each image block. The image features are analyzed, and feature fusion is performed on the image features of each image patch according to the correspondence between each image patch to obtain the multi-magnification features of each image patch; a panoramic image is constructed based on each image patch at different magnifications and its spatial position relationship in the corresponding pathological image, as well as the multi-magnification features of each image patch; the panoramic image is input into the constructed context information perception model to learn the graph structure information and context information in each pathological image to obtain the analysis results of the pathological image dataset; the analysis results are used for the target task, which includes at least one of disease-related classification tasks and regression tasks.

[0008] According to one aspect of the present invention, a pathological image analysis apparatus includes: a dataset acquisition module for acquiring a pathological image dataset, the pathological image dataset including multiple pathological images, each pathological image having a different magnification; an image segmentation module for segmenting each pathological image at different magnifications to obtain multiple image blocks in each pathological image; the magnification of image blocks belonging to different pathological images is different, and the magnification of image blocks belonging to the same pathological image is the same, and the image blocks at different magnifications have a corresponding relationship based on a pyramid cascading method between the pathological images; and a feature acquisition module for extracting features from each image block to obtain multiple image blocks in each pathological image. The system uses image features of image patches and performs feature fusion on the image features of each image patch according to the correspondence between the image patches to obtain the multi-magnification features of each image patch; a panoramic image construction module is used to construct panoramic images based on each image patch at different magnifications and its spatial position relationship in the corresponding pathological image, as well as the multi-magnification features of each image patch; an image analysis module is used to input the panoramic images into a pre-constructed context information perception model to learn the graph structure information and context information in each pathological image to obtain the analysis results of the pathological image dataset; the analysis results are used for a target task, which includes at least one of disease-related classification tasks and regression tasks.

[0009] According to one aspect of the present invention, an electronic device includes a processor and a memory, wherein the memory stores computer-readable instructions that, when executed by the processor, implement the pathological image analysis method as described above.

[0010] According to one aspect of the present invention, a storage medium having a computer program stored thereon, which, when executed by a processor, implements the pathological image analysis method as described above.

[0011] According to one aspect of the present invention, a computer program product includes a computer program stored in a storage medium, a processor of a computer device reads the computer program from the storage medium, and the processor executes the computer program, causing the computer device to implement the pathological image analysis method as described above when executed.

[0012] The above technical solution realizes a pathological image analysis method that fully acquires the feature information of the image.

[0013] Specifically, deep learning networks are used to automatically extract features from pathological images at different magnifications, and feature fusion is performed based on the correspondence between different magnifications. The panoramic pathological image is modeled as a graph, and a context information perception model is constructed to obtain structural and contextual information in the pathological image. This can maximize the acquisition of various types of information from the pathological image, not only avoiding the excessive reliance of diagnostic results on pathological image feature knowledge, but also effectively saving time. This solves the problem that existing pathological image analysis methods cannot fully acquire image feature information, resulting in low image analysis accuracy.

[0014] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram based on the implementation environment involved in this application;

[0017] Figure 2 This is a flowchart illustrating a pathological image analysis method according to an exemplary embodiment;

[0018] Figure 3 yes Figure 2A flowchart of step 150 in one embodiment corresponds to the following example;

[0019] Figure 4 yes Figure 3 A flowchart illustrating the feature splicing process in the corresponding embodiment;

[0020] Figure 5 yes Figure 2 A flowchart of step 170 in one embodiment corresponds to the following example;

[0021] Figure 6 yes Figure 2 The flowchart of the corresponding module in step 190 of the embodiment;

[0022] Figure 7 yes Figure 2 The flowchart of the Transformer module in step 190 of the corresponding embodiment;

[0023] Figure 8 This is a flowchart illustrating a pathological image analysis method according to an exemplary embodiment;

[0024] Figure 9 This is a block diagram illustrating a pathological image analysis device according to an exemplary embodiment;

[0025] Figure 10 This is a hardware structure diagram of an electronic device according to an exemplary embodiment;

[0026] Figure 11 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0027] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0028] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this disclosure means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0029] Computer-aided diagnosis offers a novel solution among existing technologies. With the continuous development of computer hardware and software, the use of artificial intelligence to assist diagnosis has gained increasing acceptance. Although the technology is advancing, analyzing pathological images using computers remains challenging. Traditional machine learning methods require manual feature extraction from pathological images, a process reliant on prior knowledge. Deep learning methods, however, can automatically extract features beneficial for pathological diagnosis without additional human intervention. While deep learning-based pathological image analysis has achieved considerable success, it typically uses images at a single magnification, such as 40x or 20x, rarely fusing features from different magnifications. Furthermore, current research largely relies on convolutional neural networks (CNNs). Due to the massive scale of digital pathological images, patch-based processing methods are commonly used. Specifically, the input image is tiled into fixed-size patches, each processed individually by a pre-trained CNN. Then, an aggregation function gathers information from each patch to construct an image-level embedding for downstream tasks (e.g., tumor grading). While CNNs are known for their strong local feature extraction capabilities, they often overlook structural and contextual information within the image. However, structural and contextual information often play a crucial role in pathological tasks.

[0030] Existing technologies mainly focus on utilizing features from pathological images at a single magnification, neglecting the use of features at different magnifications. Pathologists' diagnoses are time-consuming and require a high level of expertise. Traditional machine learning diagnoses of pathological images rely primarily on the effectiveness of feature extraction, which demands a high level of expertise from researchers. Currently, methods based on convolutional neural networks (CNNs) are limited by the size of the convolutional kernel. CNNs cannot capture complex neighborhood information, and the relationships between patches are difficult to correspond. Furthermore, this pixel-based processing is completely detached from biological meaning, making these methods poorly interpretable.

[0031] As can be seen from the above, there is still a deficiency in the related technologies that cannot fully obtain the feature information of the image.

[0032] Therefore, the pathological image analysis method provided in this application automatically extracts pathological image features at different magnifications through a pre-trained deep learning model, and performs feature fusion according to the correspondence between different magnifications, models the panoramic pathological image as a graph, and obtains structural information and contextual information in the pathological image by constructing a context information perception model. This method can obtain various types of information from the pathological image to the maximum extent, thereby achieving full acquisition of image feature information. This pathological image analysis method is applicable to pathological image analysis devices, which can be deployed on electronic devices configured with the von Neumann architecture, such as desktop computers, laptops, servers, etc.

[0033] Figure 1 This is a schematic diagram of an implementation environment involved in a pathological image analysis method. It should be noted that this implementation environment is merely an example adapted to the present invention and should not be considered as providing any limitation on the scope of the invention.

[0034] The implementation environment includes a data acquisition terminal 110 and a server terminal 130.

[0035] Specifically, the acquisition terminal 110 can also be considered an image acquisition device, including but not limited to electronic devices with shooting functions such as cameras, camcorders, and video recorders. For example, the acquisition terminal 110 is a camera.

[0036] Server 130 can be an electronic device such as a desktop computer, laptop computer, or server, or it can be a computer cluster consisting of multiple servers, or even a cloud computing center consisting of multiple servers. Server 130 is used to provide backend services, such as, but not limited to, pathological image analysis services.

[0037] The server 130 and the acquisition terminal 110 establish a network communication connection in advance via wired or wireless means, and data transmission between the server 130 and the acquisition terminal 110 is realized through this network communication connection. The transmitted data includes, but is not limited to, pathological images, etc.

[0038] In one application scenario, through the interaction between the acquisition terminal 110 and the server terminal 130, the acquisition terminal 110 acquires a pathological image and uploads the pathological image to the server terminal 130 to request the server terminal 130 to provide pathological image analysis services.

[0039] For server 130, after receiving the pathological image uploaded by acquisition terminal 110, it calls the pathological image analysis service to analyze the pathological image, fully obtain the feature information in the pathological image, and improve the accuracy of image analysis, thereby solving the problem of insufficient feature information and low accuracy in related technologies.

[0040] Please see Figure 2 This application provides a method for pathological image analysis, which is applicable to electronic devices, such as desktop computers, laptops, servers, etc.

[0041] In the following method embodiments, for ease of description, the execution subject of each step of the method is an electronic device, but this does not constitute a specific limitation.

[0042] like Figure 2 As shown, the method may include the following steps:

[0043] Step 110: Obtain the pathological image dataset, which includes multiple pathological images, each with a different magnification.

[0044] The pathological images in the pathological image dataset are a pyramid-structured dataset containing multiple magnifications. This can be understood as pathological images at different levels of the pyramid having different magnifications. Figure 4 As shown, pathological images at different magnifications focus on different aspects, i.e., they contain different image information. For example, pathological images at high magnification (e.g., 40x) contain detailed information such as cell morphology, while low magnification images (10x, 5x, etc.) contain macroscopic information such as structure and distribution. Therefore, fusing the features of images at different magnifications can yield richer image feature representations, which is beneficial for applications in various downstream target tasks related to diseases, such as classification and regression.

[0045] Specifically, the magnification of pathological images can be various, such as 5x, 10x, 20x, 40x, etc., without any limitation here.

[0046] Step 130: Cut the pathological images at different magnifications to obtain multiple image blocks in each pathological image.

[0047] Among them, the magnification of image blocks belonging to different pathological images is different, while the magnification of image blocks belonging to the same pathological image is the same. The image blocks with different magnifications have a corresponding relationship based on the pyramid cascading method between the pathological images.

[0048] Specifically, each pathological image is first preprocessed, then the foreground and background regions of each preprocessed pathological image are segmented, then the foreground region obtained from the segmentation of each pathological image is finally cut into image blocks of a set size.

[0049] In one possible implementation, taking pathological images with image patch magnifications of 40x and 20x as an example, a preliminary screening is first performed to remove pathological images of poor quality. Then, the remaining pathological images undergo preprocessing, using a threshold segmentation algorithm to segment the foreground and background regions. The segmented foreground region is then cut into image patches of a set size. It is important to note that the correspondence between image patches of different magnifications needs to be maintained during the slicing process. This correspondence is related to the pyramid cascading method between the pathological images to which the image patches of different magnifications belong. That is, one 20x image patch will correspond to four 40x image patches. The segmentation algorithm for the foreground and background regions can be any threshold segmentation algorithm, such as Otsu's method, adaptive threshold segmentation, maximum entropy threshold segmentation, iterative threshold segmentation, etc., and is not limited here.

[0050] Step 150: Extract features from each image block to obtain the image features of each image block, and perform feature fusion on the image features of each image block according to the correspondence between each image block to obtain the multi-magnification features of each image block.

[0051] In one possible implementation, the process of feature extraction from pathological images at different magnifications is achieved through deep learning networks, such as Convolutional Neural Networks (CNN), ResNet, VGG, BERT, etc., without any specific limitations.

[0052] In one possible implementation, for the cut image patch, a deep learning model pre-trained on ImageNet, such as ResNet34 or ResNet50, is used to automatically extract the deep feature information contained in the pathological image. The image patch of size will eventually become a column vector of length M. By concatenating the obtained low-magnification features (20x) with high-magnification features (40x), the final multi-magnification features of length 2M can be obtained.

[0053] Specifically, such as Figure 3As shown, step 150 may include the following steps:

[0054] Step 210: Based on the first number of image blocks in the pathological image at the next level of the pyramid, copy the image features of the second number of image blocks in the pathological image at the previous level of the pyramid to obtain the image features of the first number of image blocks at the previous level.

[0055] Step 230: The image features of the first number of image blocks in the previous level are concatenated with the image features of the first number of image blocks in the next level to obtain the magnification features of the first number of image blocks in the next level.

[0056] Step 250 continues until the image features of each image block in the pathological images of each level of the pyramid are stitched together to obtain the magnification features of each image block.

[0057] In one possible implementation Figure 4 The process of feature fusion is illustrated in the figure. Taking pathological images with image patch magnifications of 10x, 40x, and 20x as an example, the magnifications are arranged in a pyramid structure from top to bottom. Specifically, one 10x image patch corresponds to four 20x image patches and sixteen 40x image patches. One 20x image patch corresponds to four 40x image patches. Therefore, the feature representation of the 10x image patch is first extracted and then copied three times, resulting in four identical 10x features. These four 10x features are then fused with the corresponding four extracted 20x image patch features, resulting in four fused features that are twice the length of the original. These fused features are then copied to obtain four identical 20x fused features. Finally, these four 20x fused features are fused with the corresponding four extracted 40x image patch features to obtain the final multi-magnification features. In this way, feature fusion at more magnifications can be achieved.

[0058] In the above process, the embodiments of the present invention integrate pathological image features at high magnification (e.g., 40x, 20x, etc.) and pathological image features at low magnification (e.g., 10x, 5x, etc.), and achieve the purpose of multi-magnification feature fusion through pyramid cascading. The resulting multi-magnification features include not only detailed information but also macroscopic information, resulting in rich image feature representation and fully acquiring the feature information of the image.

[0059] Step 170: Construct a panoramic image based on each image patch at different magnifications, their spatial relationship in the corresponding pathological image, and the multi-magnification features of each image patch.

[0060] Specifically, such as Figure 5 As shown, step 170 may include the following steps:

[0061] Step 410: Use each image patch as a node of the panoramic image and use the magnification features of each image patch as node features of each node in the panoramic image.

[0062] Step 430: Construct an adjacency matrix based on the spatial positional relationship of each image block in its respective pathological image, and construct paths between nodes based on the adjacency matrix.

[0063] Step 450: Obtain the panoramic image from each node and its characteristics, and the paths between each node.

[0064] One possible implementation involves using a classification algorithm to construct an adjacency matrix based on the spatial relationships between nodes in the panoramic image, such as the K-nearest neighbors algorithm, support vector machine, Naive Bayes, etc., without any specific limitations.

[0065] In one possible implementation, an adjacency matrix is ​​constructed using the K-nearest neighbor algorithm based on the true spatial relationships of each image patch within its panoramic pathological image.

[0066] Specifically, the adjacency matrix can be defined as:

[0067]

[0068] Where i and j represent different nodes, KNN(i) is the set of nodes that are classified as nearest neighbors according to the k-nearest neighbor algorithm, and A ij It is an adjacency matrix.

[0069] Step 190: Input the panoramic image into the constructed context information perception model to learn the graph structure information and context information in each pathological image to obtain the analysis results of the pathological image dataset; the analysis results are used for the target task.

[0070] It is worth noting that in the target task, if a patient has multiple panoramic pathological images, the image constructed from each panoramic pathological image is regarded as a sub-image, and all the sub-images of the patient are combined to form the final pathological image data.

[0071] In one possible implementation, the context information-aware model includes a graph module and a Transformer module. Based on the graph module, the adjacency relationships between different nodes are learned through the adjacency matrix of the panoramic image to obtain the graph structure information of the panoramic image. Based on the Transformer module, the global spatial dependencies and context information in the pathological image are learned through a self-attention mechanism based on the node features of the panoramic image.

[0072] Specifically, such as Figure 5As shown, based on the graph module, learning the adjacency relationships between different nodes through the adjacency matrix of the panoramic image to obtain the graph structure information of the panoramic image may include the following steps:

[0073] Step 510: Through the aggregation layer in the graph module, aggregate each node of the panoramic image and each neighbor node of the node in the adjacency matrix to obtain intermediate data.

[0074] Step 530: The intermediate data is nonlinearly transformed using the multilayer perceptron and activation function in the graph module to obtain the graph structure information in the panoramic image.

[0075] In one possible implementation, the graph module uses an L-layer graph isomorphic network (GIN) structure to learn the adjacency relationships between different graph nodes. It learns graph structure information by using a summation aggregation operation, and then adds a multilayer perceptron (MLP) and a linear rectified activation function (ReLU) after the aggregation operation to complete the nonlinear transformation.

[0076] The specific aggregation operation formula is as follows:

[0077]

[0078]

[0079] in, and Let v represent a graph node v in layer l and its neighbor nodes u in layer l, where ∈ are learnable parameters or setpoints, MLP is a multilayer perceptron, and ReLU is the activation function. It is the aggregation result of each node and its neighboring nodes in each layer. Concat is to aggregate the aggregation results of each layer again to obtain the graph structure information h(v).

[0080] Specifically, such as Figure 7 As shown, learning global spatial dependencies and contextual information in pathological images based on the node features of the panoramic image using a self-attention mechanism, based on the Transformer module, may include the following steps:

[0081] Step 610: By using the MHA layer and Norm layer in the repeating block, the global spatial dependency and contextual information in the pathological image are learned based on the influence of each node in the panoramic image on the target task, and the feature map of the repeating block is obtained.

[0082] Step 630: The attention value of each repeated block is calculated based on the feature map of each repeated block through the MLP layer and Norm layer in the repeated block.

[0083] Step 650: The attention values ​​of each repeated block are weighted and calculated to obtain the context information of the panoramic image.

[0084] In one possible implementation, the transformer module consists of N repeating blocks. Specifically, each repeating block consists of a multi-head self-attention MHA layer, an MLP layer with residual connections, and two normalized Norm layers.

[0085] The specific calculation process is as follows:

[0086] X i =H+MLP(Norm(H));

[0087] H = X i-1 +MHA(Norm(X i-1 )).

[0088] Among them, X i H is the feature map output of the i-th repeating block, and H represents the output of the multi-head self-attention layer in each repeating block.

[0089] The calculation process for MHA is as follows:

[0090] MHA(Q,K,V)=[head1,...,head c ]×W0.

[0091] Where W0 represents the weighted matrix used to aggregate multiple attention heads, head represents the attention value, c represents the number of attention heads, and Q, K, and V represent the query vector, key vector, and value vector, respectively.

[0092] The calculation process for attention is as follows:

[0093]

[0094]

[0095] in, This represents the weighted matrix of the query vector, key vector, and value vector, and softmax represents the normalized exponential function.

[0096] Through the above process, this embodiment of the invention learns the global spatial dependence and contextual information in pathological images based on the node features of the panoramic image through a self-attention mechanism. After learning by the graph module and transformer module, the multi-magnification panoramic image data can obtain further and deeper feature representations, which can then be used for disease-related target tasks such as classification and regression.

[0097] In one possible implementation, the context-aware model is a trained machine learning model capable of image analysis of pathological images. The specific training process includes:

[0098] Step S1: Obtain the training set and the test set. The training set includes pathological images of multiple samples.

[0099] Step S2: Input the current sample in the training set into the machine learning model for training and obtain the loss value;

[0100] Step S3: If the loss value satisfies the convergence condition, the context-aware model is obtained.

[0101] Step S4: Otherwise, update the model parameters of the machine learning model, and obtain the next sample input to update the model parameters and continue training the machine learning model until the loss value meets the convergence condition.

[0102] In one possible implementation, when performing classification tasks such as determining the benignity or malignancy of tumors, the loss function can be defined as:

[0103]

[0104] Where N represents the total number of samples, y n Indicates sample x n The true label, Indicates sample x n The predicted probability, x n This represents the feature vector obtained from the first layer of fusion.

[0105] In one possible implementation, when performing regression tasks such as predicting sample survival, the loss function can be defined as:

[0106]

[0107] Among them, t i and t j Representing the i-th and j-th time nodes, R(t) i ) represents time node t i The risk set at time, δ i For data censoring markers, δ i =0,1.

[0108] The context information perception model is trained until convergence using the loss function corresponding to the target task, and the weights of each layer of the neural network are obtained. The model is then applied to new data to achieve diagnosis or prognosis prediction, thereby realizing accurate diagnosis and analysis of diseases.

[0109] Through the above process, this embodiment of the invention uses a deep learning network to automatically extract pathological image features at different magnifications, and performs feature fusion based on the correspondence between different magnifications. The panoramic pathological image is modeled as a graph, and a context information perception model is constructed to obtain structural and contextual information in the pathological image. This can maximize the acquisition of various types of information from the pathological image, not only avoiding the excessive reliance of diagnostic results on pathological image feature knowledge, but also effectively saving time. This solves the problem that existing pathological image analysis methods cannot fully acquire the feature information of the image.

[0110] Please see Figure 8 , Figure 8 This is a flowchart illustrating a pathological image analysis method in an application scenario. In this scenario, the pathological image analysis method consists of three parts: multi-magnification feature extraction, multi-magnification pathological panoramic image construction, and context information perception model construction.

[0111] In the multi-magnification feature extraction part, deep learning features are extracted from pathological image patches at different magnifications using a pre-trained CNN model, and multi-magnification feature fusion is achieved using a pyramid cascade approach. Then, a multi-magnification pathological panorama is constructed based on the spatial distribution relationship of pathological image patches in the panoramic pathological image and the extracted multi-magnification features. The obtained graph structure data is then sequentially fed into the GNN module and the Transformer module to learn the rich contextual information in the pathological image, and finally serves the target task, which can be at least one of the disease-related classification task and regression task.

[0112] Through the above process, this invention uses a deep learning network to automatically extract pathological image features at different magnifications, and performs feature fusion based on the correspondence between different magnifications. The panoramic pathological image is modeled as a graph, and a context information perception model is constructed to obtain structural and contextual information in the pathological image. This can maximize the acquisition of various types of information from the pathological image, not only avoiding the excessive reliance of diagnostic results on pathological image feature knowledge, but also effectively saving time. This solves the problem that existing pathological image analysis methods cannot fully acquire the feature information of the image.

[0113] The following are embodiments of the apparatus described in this application, which can be used to execute the pathological image analysis method involved in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments of the pathological image analysis method involved in this application.

[0114] Please see Figure 9 This application provides a pathological image analysis device 800.

[0115] The device 800 includes, but is not limited to: a dataset acquisition module 810, an image segmentation module 830, a feature acquisition module 850, a panoramic image construction module 870, and an image analysis module 890.

[0116] Among them, the dataset acquisition module 810 is used to acquire a pathological image dataset, which includes multiple pathological images, each with a different magnification.

[0117] The image segmentation module 830 is used to segment each pathological image at different magnifications to obtain multiple image blocks in each pathological image; the magnifications of image blocks belonging to different pathological images are different, and the magnifications of image blocks belonging to the same pathological image are the same. The image blocks at different magnifications have a corresponding relationship based on the pyramid cascade method between the pathological images.

[0118] The feature acquisition module 850 is used to extract features from each image block to obtain the image features of each image block, and to fuse the image features of each image block according to the correspondence between each image block to obtain the magnification features of each image block.

[0119] The panoramic image construction module 870 is used to construct panoramic images based on image blocks at different magnifications and their spatial position relationships in the corresponding pathological images, as well as the multimagnification characteristics of each image block.

[0120] The image analysis module 890 is used to input panoramic images into the constructed context information perception model, learn the graph structure information and context information in each pathological image, and obtain the analysis results of the pathological image dataset. The analysis results are used for the target task, which includes at least one of the disease-related classification task and regression task.

[0121] It should be noted that the pathological image analysis provided in the above embodiments is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed. That is, the internal structure of the pathological image analysis device will be divided into different functional modules to complete all or part of the functions described above.

[0122] Furthermore, the embodiments of the pathological image analysis device and the pathological image analysis method provided in the above embodiments belong to the same concept, and the specific way in which each module performs its operation has been described in detail in the method embodiments, and will not be repeated here.

[0123] Figure 10 A schematic diagram of the structure of an electronic device according to an exemplary embodiment is shown.

[0124] It should be noted that this electronic device is merely an example adapted to this application and should not be construed as providing any limitation on the scope of use of this application. Furthermore, this electronic device should not be interpreted as requiring or depending on any specific feature. Figure 10 One or more components of the exemplary electronic device 2000 shown.

[0125] The hardware structure of electronic devices 2000 can vary significantly due to differences in configuration or performance, such as... Figure 10 As shown, the electronic device 2000 includes: a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.

[0126] Specifically, power supply 210 is used to provide operating voltage for various hardware devices on electronic device 2000.

[0127] Interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices.

[0128] Of course, in other examples adapted in this application, interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input / output interface 235, and at least one USB interface 237, etc. Figure 10 As shown, this does not constitute a specific limitation.

[0129] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it include the operating system 251, application programs 253, and data 255, etc., and the storage method can be temporary storage or permanent storage.

[0130] The operating system 251 is used to manage and control the various hardware devices and application programs 253 on the electronic device 2000, so as to enable the central processing unit 270 to perform calculations and processing on the massive data 255 in the memory 250. It can be Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0131] Application 253 is a computer program that performs at least one specific task based on operating system 251, and may include at least one module ( Figure 10 (Not shown), each module may contain a computer program for the electronic device 2000. For example, the information recommendation device may be considered as an application program 253 deployed on the electronic device 2000.

[0132] Data 255 can be photos, pictures, etc. stored on a disk, or pathological image data, etc., stored in memory 250.

[0133] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read computer programs stored in the memory 250, thereby performing operations and processing on massive amounts of data 255 stored in the memory 250. For example, a pathological image analysis method may be implemented by the central processing unit 270 reading a series of computer programs stored in the memory 250.

[0134] Furthermore, this application can also be implemented through hardware circuits or a combination of hardware circuits and software. Therefore, the implementation of this application is not limited to any specific hardware circuit, software, or combination thereof.

[0135] Please see Figure 11 This application provides an electronic device 4000, which may include: a desktop computer, a laptop computer, a server, etc.

[0136] exist Figure 11 The electronic device 4000 includes at least one processor 4001, at least one communication bus 4002, and at least one memory 4003.

[0137] The processor 4001 and memory 4003 are connected, for example, via a communication bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0138] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0139] The communication bus 4002 may include a path for transmitting information between the aforementioned components. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 11 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0140] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0141] The memory 4003 stores a computer program, and the processor 4001 reads the computer program stored in the memory 4003 through the communication bus 4002.

[0142] When the computer program is executed by the processor 4001, it implements the pathological image analysis methods in the above embodiments.

[0143] In addition, this application provides a storage medium storing a computer program, which, when executed by a processor, implements the pathological image analysis methods in the above embodiments.

[0144] This application provides a computer program product comprising a computer program stored in a storage medium. A processor of a computer device reads the computer program from the storage medium and executes the computer program, causing the computer device to perform the pathological image analysis methods described in the above embodiments.

[0145] Compared with related technologies, the beneficial effects of the present invention are:

[0146] 1. This invention uses a deep learning network to automatically extract features from pathological images at different magnifications and performs feature fusion based on the correspondence between different magnifications. It models the panoramic pathological image as a panoramic view and acquires structural and contextual information from the pathological image by constructing a context information perception model. This maximizes the acquisition of various types of information from the pathological image, avoiding excessive reliance on pathological image feature knowledge for diagnostic results and effectively saving time. This solves the problem that existing pathological image analysis methods cannot fully acquire the feature information of the image.

[0147] 2. This invention combines the features of pathological images at different magnifications to learn multi-scale feature representations of tumors, and combines the powerful context information acquisition capabilities of the context information perception model to model panoramic pathological images, fully acquire feature information in pathological images, and thus achieve accurate diagnosis and analysis of diseases.

[0148] 3. This invention uses panoramic pathological images, rather than manually selected regions of interest, to preserve the information in the pathological images to the maximum extent. It proposes a feature acquisition method based on multi-magnification panoramic pathological images, which can effectively utilize information at different scales in the pathological images to complete tasks such as diagnosis and prognosis.

[0149] 4. This invention designs a context information perception model based on graph transformer. It can learn the structural information in pathological images by obtaining the adjacency relationship of graph nodes through the adjacency matrix. At the same time, it implicitly captures the global contextual dependencies in pathological images through the multi-head self-attention mechanism in the transformer module. This allows for better learning of the structural features and global contextual information contained in pathological image data, providing richer feature representations for subsequent diagnosis and prognosis tasks.

[0150] 5. This invention quantifies and streamlines the complex clinical pathological diagnosis process, providing auxiliary reference for clinical diagnosis and prognosis. It also provides an interpretation function for model prediction results, intuitively demonstrating the reasons and results of how data in pathological images affect the model's outcome. This reduces the obstacles to the model's application in real-world scenarios and provides an interpretable method for pathological image analysis.

[0151] 6. This invention can effectively save the cost of manual data analysis and classification, while avoiding excessive reliance on doctors' technical skills, thus saving doctors a lot of human and material resources. It integrates pathological image features at different magnifications, making the information contained in the data richer.

[0152] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0153] The above description is merely a preferred exemplary embodiment of the present invention and is not intended to limit the implementation of the present invention. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of protection claimed in the claims.

Claims

1. A method of pathological image analysis, characterized in that, The method includes: Obtain a pathological image dataset, which includes multiple pathological images, each with a different magnification. Each of the pathological images at different magnifications is cut into multiple image blocks in each pathological image; The magnification of image blocks belonging to different pathological images is different, while the magnification of image blocks belonging to the same pathological image is the same. The image blocks with different magnifications have a corresponding relationship based on the pyramid cascading method between the pathological images. Feature extraction is performed on each of the image blocks to obtain the image features of each image block, and feature fusion is performed on the image features of each image block according to the correspondence between the image blocks to obtain the multi-magnification features of each image block; A panoramic image is constructed based on the image blocks at different magnifications, their spatial relationships in the corresponding pathological images, and the multi-magnification features of each image block. The panoramic image is input into a pre-constructed context information awareness model, which includes a graph module and a Transformer module. Based on the graph module, the adjacency relationships between different nodes are learned through an adjacency matrix used to construct the panoramic image, thereby obtaining the graph structure information of the panoramic image. Based on the self-attention mechanism in the Transformer module, the influence of each node in the panoramic image on the target task is learned, thereby obtaining the context information of the panoramic image. The graph structure information and context information in each pathological image are then learned to obtain the analysis results of the pathological image dataset. The analysis results are used for the target task, which includes at least one of disease-related classification and regression tasks.

2. The method of claim 1, wherein, The construction of a panoramic image based on the image patches at different magnifications, their spatial relationships within their respective pathological images, and the multi-magnification features of each image patch includes: Each of the aforementioned image patches is used as a node in the panoramic image, and the magnification features of each of the aforementioned image patches are used as node features of each node in the panoramic image. An adjacency matrix is ​​constructed based on the spatial positional relationship of each image patch in its respective pathological image, and a path is constructed between each node based on the adjacency matrix; The panoramic image is obtained from each node and its characteristics, as well as the paths between the nodes.

3. The method of claim 1, wherein, The graph module includes an aggregation layer, a multilayer perceptron, and an activation function; Based on the graph module, the adjacency relationships between different nodes are learned through the adjacency matrix used to construct the panoramic image, thereby obtaining the graph structure information of the panoramic image, including: Through the aggregation layer in the graph module, the nodes of the panoramic image and the neighboring nodes of the nodes in the adjacency matrix are aggregated to obtain intermediate data. The intermediate data is nonlinearly transformed using the multilayer perceptron and activation function in the graph module to obtain the graph structure information in the panoramic image.

4. The method of claim 1, wherein, The Transformer module includes multiple repeating blocks, which include a multi-head self-attention MHA layer, a multilayer perceptron (MLP) layer with residual connections, and a normalized Norm layer. The influence of each node in the panoramic image on the target task is learned based on the self-attention mechanism in the Transformer module to obtain the context information of the panoramic image, including: By using the MHA layer and Norm layer in the repeating block, the global spatial dependency and contextual information in the pathological image are learned based on the influence of each node in the panoramic image on the target task, and the feature map of the repeating block is obtained. The attention value of each repeated block is calculated based on the feature map of each repeated block through the MLP layer and Norm layer in the repeated block; The attention values ​​of each repeating block are weighted and calculated to obtain the context information of the panoramic image.

5. The method of claim 1, wherein, The step of segmenting each of the pathological images at different magnifications to obtain multiple image blocks in each pathological image includes: Preprocess each of the aforementioned pathological images; The foreground and background regions of each of the preprocessed pathological images are segmented. For each foreground region obtained from the segmentation of the pathological image, the foreground region is cut into image blocks of a set size.

6. The method of claim 1, wherein, The step of fusing image features of each image block according to the correspondence between the image blocks to obtain the magnification features of each image block includes: The pyramid cascading method between the pathological images to which each image block belongs is determined based on the correspondence between the image blocks; Based on the first number of image blocks in the pathological image at the next level of the pyramid, the image features of the second number of image blocks in the pathological image at the previous level of the pyramid are copied to obtain the image features of the first number of image blocks at the previous level. The image features of each image block of the first quantity in the previous level are concatenated with the image features of each image block of the first quantity in the next level to obtain the magnification features of each image block of the first quantity in the next level. The image features of each image block in the pathological images of each level of the pyramid are stitched together to obtain the magnification features of each image block.

7. A pathological image analysis apparatus characterized by comprising: The device includes: The dataset acquisition module is used to acquire a pathological image dataset, which includes multiple pathological images, each with a different magnification. The image segmentation module is used to segment each of the pathological images at different magnifications to obtain multiple image blocks in each pathological image; the magnifications of the image blocks belonging to different pathological images are different, and the magnifications of the image blocks belonging to the same pathological image are the same; the image blocks at different magnifications have a corresponding relationship based on the pyramid cascade method between the pathological images. The feature acquisition module is used to extract features from each of the image blocks to obtain the image features of each of the image blocks, and to perform feature fusion on the image features of each of the image blocks according to the correspondence between the image blocks to obtain the multi-magnification features of each of the image blocks; A panoramic image construction module is used to construct a panoramic image based on each image patch at different magnifications and its spatial position relationship in the corresponding pathological image, as well as the multi-magnification features of each image patch. An image analysis module is used to input the panoramic image into a pre-constructed context information awareness model. The context information awareness model includes a graph module and a Transformer module. Based on the graph module, adjacency relationships between different nodes are learned through an adjacency matrix used to construct the panoramic image, obtaining the graph structure information of the panoramic image. Based on the self-attention mechanism in the Transformer module, the influence of each node in the panoramic image on the target task is learned, obtaining the context information of the panoramic image. The graph structure information and context information in each pathological image are learned to obtain the analysis results of the pathological image dataset. The analysis results are used for the target task, which includes at least one of disease-related classification and regression tasks.

8. An electronic device, comprising: include: At least one processor and at least one memory, wherein program instructions or code are stored in the memory; The program instructions or code are loaded and executed by the processor, causing the electronic device to implement the pathological image analysis method as described in any one of claims 1 to 6.

9. A storage medium having stored thereon program instructions or code, characterized in that, The program instructions or code are loaded and executed by the processor to implement the pathological image analysis method as described in any one of claims 1 to 6.

Images