Pathological image analysis method and apparatus, electronic device, and storage medium

Abstract

Disclosed in embodiments of the present invention are a pathological image analysis method and apparatus. The method comprises: acquiring a pathological image data set, pathological images having different magnifications; respectively cutting the pathological images to obtain a plurality of image blocks in the pathological images; performing feature extraction on the image blocks to obtain image features of the image blocks, and performing feature fusion on the image features of the image blocks on the basis of the correspondence between the image blocks to obtain multi-magnification features of the image blocks; constructing a panoramic image on the basis of the image blocks having different magnifications, a spatial positional relationship of the image blocks in the pathological images to which the image blocks belong, and the multi-magnification features of the image blocks; inputting the panoramic image into a constructed context information perception model to learn graph structure information and context information in the pathological images, so as to obtain an analysis result of the pathological image data set; and applying the analysis result to a target task. The present invention solves the problem in the prior art of low accuracy caused by failure in full acquisition of feature information.

Description

Pathological image analysis method, device, electronic device and storage medium Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a pathological image analysis method, device, electronic equipment and storage medium. Background Art

[0002] Histopathological diagnosis is the gold standard for identifying cancer in modern medicine and is the most important and reliable method for cancer diagnosis. The current manual pathology section diagnostic process is lengthy, requiring experienced pathologists to prepare tissue sections and then examine them under microscopes at varying magnifications to identify cancerous tissue. Multiple biological indicators, such as the number of cancer cell mitoses, cancer cell atypia, the proportion of cancer cells, and the ratio of tumor parenchyma to stroma, are used to determine whether a patient has cancer and its severity. This allows for pathological diagnosis and prognostic assessment, enabling effective treatment. Due to tumor heterogeneity, the use of pathology images for tasks such as tumor subtype classification and prognosis prediction requires more comprehensive utilization of the various information contained in them.

[0003] However, manually calculating various biological indicators is not only time-consuming and inefficient, but can also lead to missed key features, resulting in misdiagnosis. Furthermore, cancer is a complex disease, with pathological manifestations varying significantly between patients, at different stages of disease progression, and in different tissues and organs. This creates significant challenges for pathological diagnosis.

[0004] Therefore, there is an urgent need for a pathological image analysis method that can fully obtain the characteristic information of the image and has high accuracy.

[0005] Summary of the Invention

[0006] The embodiments of the present invention provide a pathological image analysis method, device, electronic device and storage medium to solve the problem in the related art that the pathological image analysis method cannot fully obtain the characteristic information of the image, resulting in low image analysis accuracy.

[0007] Wherein, the technical solution adopted by the present invention is:

[0008] According to one aspect of the present invention, a pathological image analysis method comprises: obtaining a pathological image data set, wherein the pathological image data set comprises a plurality of pathological images, each of the pathological images having a different magnification; cutting each of the pathological images having different magnifications respectively to obtain a plurality of image blocks in each of the pathological images; the image blocks belonging to different pathological images have different magnifications, the image blocks belonging to the same pathological image have the same magnifications, and the image blocks of different magnifications have a corresponding relationship based on a pyramid cascade method between the pathological images; performing feature extraction on each of the image blocks to obtain an image block of each of the image blocks. The method comprises the following steps: obtaining image features of each image block, performing feature fusion on the image features of each image block according to the correspondence between the image blocks, and obtaining multi-magnification features of each image block; constructing a panoramic image according to the spatial position relationship between the image blocks of different magnifications in the pathological image to which they belong, and the multi-magnification features of each image block; inputting the panoramic image into the constructed context information perception model, learning the graph structure information and context information in each pathological image, and obtaining an analysis result of the pathological image dataset; and using the analysis result for a target task, which includes at least one of a classification task and a regression task related to the disease.

[0009] According to one aspect of the present invention, a pathological image analysis device comprises: a data set acquisition module for acquiring a pathological image data set, wherein the pathological image data set comprises a plurality of pathological images, each of the pathological images having a different magnification; an image cutting module for cutting each of the pathological images of different magnifications respectively to obtain a plurality of image blocks in each of the pathological images; the image blocks belonging to different pathological images have different magnifications, and the image blocks belonging to the same pathological image have the same magnifications, and the image blocks of different magnifications have a corresponding relationship based on a pyramid cascade method between the pathological images; a feature acquisition module for extracting features from each of the image blocks to obtain each of the pathological images. The method comprises the following steps: obtaining image features of the image blocks, performing feature fusion on the image features of each image block according to the correspondence between the image blocks, and obtaining multi-magnification features of each image block; constructing a panoramic image according to the image blocks of different magnifications and their spatial positional relationships in the corresponding pathological images, and constructing a panoramic image according to the multi-magnification features of each image block; inputting the panoramic image into the constructed context information perception model, learning the graph structure information and context information in each pathological image, and obtaining the analysis results of the pathological image data set; and using the analysis results for the target task, which includes at least one of a classification task and a regression task related to the disease.

[0010] According to one aspect of the present invention, an electronic device includes a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the pathological image analysis method described above is implemented.

[0011] According to one aspect of the present invention, a storage medium stores a computer program thereon, and when the computer program is executed by a processor, the pathological image analysis method described above is implemented.

[0012] According to one aspect of the present invention, a computer program product includes a computer program, the computer program is stored in a storage medium, a processor of a computer device reads the computer program from the storage medium, and the processor executes the computer program, so that the pathological image analysis method described above is implemented when the computer device executes the computer program.

[0013] In the above technical solution, a pathological image analysis method for fully acquiring characteristic information of an image is implemented.

[0014] Specifically, a deep learning network is used to automatically extract pathological image features at different magnifications, and feature fusion is performed based on the correspondence between different magnifications. The panoramic pathological image is modeled as a graph, and a contextual information perception model is constructed to obtain structural information and contextual information in the pathological image. This can maximize the acquisition of various types of information from the pathological image, which not only avoids the excessive dependence of the diagnostic results on the knowledge of pathological image features, but also effectively saves time, thereby solving the problem that the pathological image analysis method in the existing technology cannot fully obtain the feature information of the image, resulting in low image analysis accuracy.

[0015] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts.

[0017] FIG1 is a schematic diagram of an implementation environment according to the present application;

[0018] FIG2 is a flow chart showing a pathological image analysis method according to an exemplary embodiment;

[0019] FIG3 is a flow chart of step 150 in one embodiment of the embodiment corresponding to FIG2 ;

[0020] FIG4 is a schematic diagram of the process of feature splicing in the embodiment corresponding to FIG3 ;

[0021] FIG5 is a flow chart of step 170 in one embodiment of the embodiment corresponding to FIG2 ;

[0022] FIG6 is a flowchart of the diagram module in step 190 of the embodiment corresponding to FIG2;

[0023] FIG7 is a flowchart of the Transformer module in step 190 of the embodiment corresponding to FIG2 ;

[0024] FIG8 is a schematic flow chart showing a pathological image analysis method according to an exemplary embodiment;

[0025] FIG9 is a block diagram of a pathological image analysis device according to an exemplary embodiment;

[0026] FIG10 is a hardware structure diagram of an electronic device according to an exemplary embodiment;

[0027] Fig. 11 is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0028] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.

[0029] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present disclosure refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0030] Among existing technologies, computer-aided diagnosis (CAD) offers a new solution. With the continuous advancement of computer hardware and software, AI-assisted diagnosis is gaining increasing recognition. Despite technological advancements, computer-assisted pathology image analysis remains challenging and challenging. Traditional machine learning methods require manual feature extraction from pathology images, a process that relies on prior knowledge. Deep learning methods, however, can automatically extract features from images that are beneficial for pathological diagnosis without requiring additional human intervention. Deep learning-based pathology image analysis has achieved considerable success, but these methods typically utilize images at a single magnification, such as 40x or 20x. Few studies integrate pathology features from different magnifications. Furthermore, much current work relies on convolutional neural networks (CNNs). Due to the large scale of digital pathology images, most approaches employ patch-based processing. Specifically, the input image is tiled into fixed-size patches, which are then processed individually by a pre-trained CNN. An aggregation function then aggregates information from each patch to construct an image-level embedding for downstream tasks, such as tumor grading. While convolutional neural networks are known for their strong local feature extraction capabilities, they often overlook structural and contextual information within the image. However, for pathological tasks, structural information and contextual information often play important roles.

[0031] Existing technologies mainly focus on utilizing the features of pathological images at a single magnification, ignoring the use of features at different magnifications. Pathologists' diagnosis takes a long time and requires a high level of professional ability. Traditional machine learning diagnosis of pathological images mainly relies on the effect of feature extraction, which requires a high level of professional knowledge from researchers. Current methods based on convolutional neural networks (CNNs) are limited by the size of the convolution kernel. CNNs cannot capture complex neighborhood information, and the relationship between patches is difficult to correspond. In addition, this pixel-based processing is completely detached from biological meaning, making these methods poorly interpretable.

[0032] As can be seen from the above, the related art still has the defect of being unable to fully obtain the feature information of the image.

[0033] To this end, the pathological image analysis method provided in this application automatically extracts pathological image features at different magnifications through a pre-trained deep learning model, and performs feature fusion based on the correspondence between different magnifications, models the panoramic pathological image as a graph, and acquires structural information and contextual information in the pathological image by constructing a context information perception model. It can obtain various types of information from the pathological image to the maximum extent, thereby fully acquiring the characteristic information of the image. The pathological image analysis method is suitable for a pathological image analysis device, and the pathological image analysis device can be deployed on an electronic device configured with a von Neumann architecture. For example, the electronic device can be a desktop computer, a laptop computer, a server, etc.

[0034] Figure 1 is a schematic diagram of an implementation environment involved in a pathological image analysis method. It should be noted that this implementation environment is only an example adapted to the present invention and should not be considered as providing any limitation on the scope of application of the present invention.

[0035] The implementation environment includes a collection end 110 and a service end 130 .

[0036] Specifically, the acquisition end 110 can also be considered as an image acquisition device, including but not limited to electronic devices with a shooting function such as a camera, a still camera, and a video camera. For example, the acquisition end 110 is a camera.

[0037] Server 130 can be an electronic device such as a desktop computer, laptop computer, or server. It can also be a computer cluster consisting of multiple servers, or even a cloud computing center consisting of multiple servers. Server 130 is used to provide background services, such as, but not limited to, pathology image analysis services.

[0038] A network communication connection is pre-established between the server 130 and the acquisition terminal 110 via a wired or wireless method, and data transmission between the server 130 and the acquisition terminal 110 is achieved via the network communication connection. The transmitted data includes but is not limited to pathological images and the like.

[0039] In one application scenario, through the interaction between the acquisition terminal 110 and the server 130 , the acquisition terminal 110 acquires a pathological image and uploads the pathological image to the server 130 to request the server 130 to provide a pathological image analysis service.

[0040] For the server 130, after receiving the pathological image uploaded by the acquisition terminal 110, it calls the pathological image analysis service to analyze the pathological image, fully obtain the characteristic information in the pathological image, and improve the accuracy of image analysis, so as to solve the problem of insufficient characteristic information acquisition and low accuracy in related technologies.

[0041] Please refer to FIG2 . An embodiment of the present application provides a pathological image analysis method, which is applicable to electronic devices, for example, the electronic devices may be desktop computers, laptop computers, servers, and the like.

[0042] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.

[0043] As shown in FIG2 , the method may include the following steps:

[0044] Step 110 : Acquire a pathological image dataset, where the pathological image dataset includes multiple pathological images, each of which has a different magnification.

[0045] The pathological images in the pathological image dataset are a type of data with a pyramid structure and contain multiple magnifications. It can also be understood that pathological images at different levels of the pyramid have different magnifications, as shown in Figure 4. Pathological images at different magnifications focus on different aspects, that is, they contain different image information. For example, pathological images at high magnifications (such as 40x) contain detailed information such as cell morphology, while low magnification images (10x, 5x, etc.) contain macroscopic information such as structure and distribution. Therefore, fusing image features at different magnifications can obtain richer image feature representations, which is beneficial for application in various downstream disease-related target tasks such as classification and regression.

[0046] Specifically, the magnification of the pathological image can be various, such as 5x, 10x, 20x, 40x, etc., which is not limited here.

[0047] Step 130 : Cut each pathology image of different magnifications to obtain a plurality of image blocks in each pathology image.

[0048] The image blocks belonging to different pathological images have different magnifications, the image blocks belonging to the same pathological image have the same magnification, and the image blocks with different magnifications have a corresponding relationship based on the pyramid cascade mode between the pathological images.

[0049] Specifically, each pathological image is first preprocessed, and then each preprocessed pathological image is segmented into a foreground area and a background area. Then, for each pathological image, the foreground area obtained by segmentation is finally cut into image blocks of a set size.

[0050] In one possible implementation, taking the pathological image with image block magnifications of 40x and 20x as an example, a preliminary screening is first performed to eliminate pathological images of poor quality, and then the retained pathological images are preprocessed, and a threshold segmentation algorithm is used to segment the foreground area and the background area, and the segmented foreground area is cut into image blocks of a set size. It should be noted that the correspondence between image blocks of different magnifications needs to be maintained when cutting. This correspondence is related to the pyramid cascade method between the pathological images to which the image blocks of different magnifications belong, that is, an image block at 20x will correspond to 4 image blocks at 40x. The segmentation algorithm for segmenting the foreground area and the background area can be any threshold segmentation algorithm, such as Otsu's method, adaptive threshold segmentation method, maximum entropy threshold segmentation method, iterative threshold segmentation method, etc., which is not limited here.

[0051] Step 150 , extracting features from each image block to obtain image features of each image block, and performing feature fusion on the image features of each image block according to the corresponding relationship between the image blocks to obtain multi-rate features of each image block.

[0052] In one possible implementation, the process of extracting features from pathological images at different magnifications is implemented through a deep learning network, such as a convolutional neural network CNN, a ResNet network, a VGG network, a BERT network, etc., which is not limited here.

[0053] In one possible implementation, for the cut image blocks, a deep learning model pre-trained on ImageNet, such as Resnet34, Resnet50, etc., is used to automatically extract the deep feature information contained in the pathological image. The image blocks of different sizes will eventually become a column vector of length M. By concatenating the obtained low-magnification features (20x) and high-magnification features (40x), the final multi-magnification features with a length of 2M can be obtained.

[0054] Specifically, as shown in FIG3 , step 150 may include the following steps:

[0055] Step 210 : according to the first number of image blocks in the pathological image in the next level pyramid, the image features of the second number of image blocks in the pathological image in the previous level pyramid are copied respectively to obtain the image features of the first number of image blocks in the previous level.

[0056] Step 230 : splicing the image features of the first number of image blocks at the previous level with the image features of the first number of image blocks at the next level to obtain the multi-magnification features of the first number of image blocks at the next level.

[0057] Step 250 , until the image features of each image block in the pathological images at each level of the pyramid are completely spliced, a multi-magnification feature of each image block is obtained.

[0058] In one possible implementation, Figure 4 shows the process of feature fusion. As shown in the figure, taking the image block magnifications of pathological images with 10x, 40x and 20x as an example, a pyramid structure is formed from top to bottom in the order of low magnification to high magnification. Specifically, one image block at 10x will correspond to 4 image blocks at 20x, 16 image blocks at 40x, and one image block at 20x will correspond to 4 image blocks at 40x. Therefore, the feature representation of the image block at 10x is first extracted, and then it is copied 3 times to obtain a total of 4 identical features at 10x. These 4 10x features are then fused with the corresponding 4 20x image block features extracted to obtain 4 fused features with a length twice that of the original. The spliced ​​fused features are copied to obtain 4 identical 20x fused features. These 4 20x fused features are then fused with the corresponding 4 40x image block features extracted to obtain the final multi-magnification features. In this way, feature fusion at more magnifications can be achieved.

[0059] In the above process, the embodiment of the present invention integrates the pathological image features at high magnification (such as 40x, 20x, etc.) and the pathological image features at low magnification (such as 10x, 5x, etc.), and achieves the purpose of multi-magnification feature fusion through pyramid cascade. The obtained multi-magnification features include not only detail information but also macro information, and obtain rich image feature representation, which fully obtains the feature information of the image.

[0060] Step 170 : constructing a panoramic image based on the image blocks of different magnifications and their spatial positional relationships in the corresponding pathological images, as well as the multi-magnification features of the image blocks.

[0061] Specifically, as shown in FIG5 , step 170 may include the following steps:

[0062] Step 410 : Use each image block as a node of the panoramic image and use the multi-magnification feature of each image block as the node feature of each node in the panoramic image.

[0063] Step 430 : constructing an adjacency matrix based on the spatial position relationship of each image block in the pathological image to which it belongs, and constructing a path between each node based on the adjacency matrix.

[0064] Step 450 , obtaining a panoramic view from each node and its node features, and the paths between each node.

[0065] In one possible implementation, an adjacency matrix is ​​constructed according to the spatial position relationship of each node in the panoramic image through a classification algorithm, such as a K-nearest neighbor algorithm, a support vector machine, a naive Bayes algorithm, etc., which is not limited here.

[0066] In a possible implementation, an adjacency matrix is ​​constructed by using a K-nearest neighbor algorithm according to the real spatial position relationship of each image block in its panoramic pathology image.

[0067] Specifically, the adjacency matrix can be defined as:

[0068] Among them, i and j represent different nodes, KNN(i) is the node set divided into neighbor nodes according to the k value set by the K-nearest neighbor algorithm, A ij is the adjacency matrix.

[0069] In step 190 , the panoramic image is input into the constructed context information perception model to learn the image structure information and context information in each pathological image to obtain the analysis results of the pathological image dataset; the analysis results are used for the target task.

[0070] It is worth noting that in the target task, if a patient has multiple panoramic pathology images, the image constructed from each panoramic pathology image is regarded as a sub-image, and all the sub-images of the patient are combined into the final pathology image data.

[0071] In one possible implementation, the context information perception model includes a graph module and a Transformer module. Based on the graph module, the adjacency relationship between different nodes is learned through the adjacency matrix of the panoramic image to obtain the graph structure information of the panoramic image. Based on the Transformer module, the global spatial dependency and context information in the pathological image are learned according to the node features of the panoramic image through the self-attention mechanism.

[0072] Specifically, as shown in FIG5 , based on the graph module, learning the adjacency relationship between different nodes through the adjacency matrix of the panoramic graph and obtaining the graph structure information of the panoramic graph may include the following steps:

[0073] Step 510 : Aggregate each node of the panoramic graph and each neighboring node of the node in the adjacency matrix through the aggregation layer in the graph module to obtain intermediate data.

[0074] Step 530 : Perform nonlinear transformation on the intermediate data through the multi-layer perceptron and activation function in the graph module to obtain graph structure information in the panoramic image.

[0075] In one possible implementation, the graph module uses an L-layer graph isomorphism network (GIN) structure to learn the adjacency relationship between different graph nodes, learns the graph structure information by using a summation aggregation operation, and adds a multi-layer perceptron (MLP) and a linear rectifier activation function (ReLU) after the aggregation operation to complete the nonlinear transformation.

[0076] The specific aggregation operation formula is as follows:

[0077] in, and Represents a graph node v at layer l and its neighbor node u at layer l, ∈ is a learnable parameter or set value, MLP is a multi-layer perceptron, ReLU is an activation function, It is the aggregation result of each node in each layer and its neighboring nodes. Concat is to aggregate the aggregation results of each layer again to obtain the graph structure information h(v).

[0078] Specifically, as shown in FIG7 , based on the Transformer module, learning the global spatial dependency and context information in the pathological image according to the node features of the panoramic image through the self-attention mechanism can include the following steps:

[0079] Step 610 , using the MHA layer and the Norm layer in the repeated block, the global spatial dependency and context information in the pathological image are learned according to the influence of each node in the panoramic image on the target task, and a feature map of the repeated block is obtained.

[0080] In step 630 , the attention value of each repeated block is calculated based on the feature map of each repeated block through the MLP layer and the Norm layer in the repeated block.

[0081] Step 650 : Perform weighted calculation on the attention values ​​of each repeated block to obtain context information of the panoramic image.

[0082] In one possible implementation, the transformer module consists of N repeated blocks. Specifically, each repeated block consists of a multi-head self-attention (MHA) layer, an MLP layer with residual connections, and two normalization (Norm) layers.

[0083] The specific calculation process is as follows: X i =H+MLP(Norm(H)); H=X i-1 +MHA(Norm(X i-1 )).

[0084] Among them, X i is the feature map output by the i-th repeated block, and H represents the output of the multi-head self-attention layer in each repeated block.

[0085] The calculation process of MHA is as follows: MHA(Q,K,V)=[head1,...,head c ]×W0.

[0086] Among them, W0 represents the weight matrix used to aggregate multiple attention heads, head represents the attention value, c represents the number of attention heads, Q, K, and V represent the query vector, key-value vector, and value vector, respectively.

[0087] The attention calculation process is as follows:

[0088] in, Represents the weighted matrix of query vector, key-value vector and value vector, and softmax represents the normalized exponential function.

[0089] Through the above process, the embodiment of the present invention learns the global spatial dependencies and contextual information in the pathological image based on the node features of the panoramic image through the self-attention mechanism. After learning the multi-magnification panoramic image data through the graph module and the transformer module, a further and deeper feature representation can be obtained, which can then be used for classification, regression and other disease-related target tasks.

[0090] In one possible implementation, the context-aware model is a trained machine learning model capable of performing image analysis on pathological images. The specific training process includes:

[0091] Step S1, obtaining a training set and a test set, where the training set includes pathological images of multiple samples;

[0092] Step S2: Input the current sample in the training set into the machine learning model for training to obtain the loss value;

[0093] Step S3: If the loss value meets the convergence condition, a context-aware model is obtained;

[0094] Step S4: Otherwise, update the model parameters of the machine learning model, obtain the next sample input and continue training the machine learning model after updating the model parameters until the loss value meets the convergence condition.

[0095] In one possible implementation, when performing classification tasks such as determining whether a tumor is benign or malignant, the loss function can be defined as:

[0096] Where N represents the total number of samples, y n Represents sample x n The true label, Represents sample x nThe predicted probability, x n Represents the feature vector obtained by the first layer fusion.

[0097] In one possible implementation, when performing regression tasks such as predicting sample survival, the loss function can be defined as:

[0098] Among them, t i and t j are the i-th and j-th time nodes respectively, R(t i ) is the time node t i The risk set at time δ i is the data missing mark, δ i =0,1.

[0099] The contextual information perception model is trained using the loss function corresponding to the target task until convergence, and the weights of each layer of the neural network are obtained. The model is then applied to new data to achieve diagnosis or prognosis prediction, thereby achieving accurate diagnosis and analysis of the disease.

[0100] Through the above process, the embodiment of the present invention automatically extracts pathological image features at different magnifications through a deep learning network, and performs feature fusion according to the correspondence between different magnifications, models the panoramic pathological image as a graph, and acquires structural information and contextual information in the pathological image by constructing a context information perception model. It can obtain various types of information from the pathological image to the maximum extent, which not only avoids the excessive dependence of the diagnostic results on the knowledge of pathological image features, but also effectively saves time, thereby solving the problem that the pathological image analysis method in the prior art cannot fully acquire the feature information of the image.

[0101] Please refer to Figure 8, which is a flow chart of a pathological image analysis method in an application scenario. In this application scenario, the pathological image analysis method consists of three parts: multi-magnification feature extraction, multi-magnification pathological panoramic image construction, and context information perception model construction.

[0102] In the multi-magnification feature extraction part, deep learning features are extracted from pathological image blocks of different magnifications through a pre-trained CNN model, and multi-magnification feature fusion is achieved using a pyramid cascade method. Then, a multi-magnification pathological panorama is constructed based on the spatial distribution relationship of pathological image blocks in the panoramic pathological image and the extracted multi-magnification features. The obtained graph structure data is then sent to the GNN module and the Transformer module in sequence to learn the rich contextual information in the pathological image, and finally serve the target task, which can be at least one of the disease-related classification tasks and regression tasks.

[0103] Through the above process, the present invention automatically extracts pathological image features at different magnifications through a deep learning network, and performs feature fusion according to the correspondence between different magnifications, models the panoramic pathological image as a graph, and acquires structural information and contextual information in the pathological image by constructing a contextual information perception model. It can obtain various types of information from the pathological image to the maximum extent, which not only avoids the excessive dependence of the diagnostic results on the knowledge of pathological image features, but also effectively saves time, thereby solving the problem that the pathological image analysis method in the prior art cannot fully acquire the feature information of the image.

[0104] The following are embodiments of the device of the present application, which can be used to perform the pathological image analysis method involved in the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the pathological image analysis method involved in the present application.

[0105] Please refer to FIG9 , a pathological image analysis device 800 is provided in an embodiment of the present application.

[0106] The apparatus 800 includes but is not limited to: a data set acquisition module 810 , an image segmentation module 830 , a feature acquisition module 850 , a panoramic image construction module 870 and an image analysis module 890 .

[0107] The data set acquisition module 810 is used to acquire a pathological image data set, which includes multiple pathological images, each of which has a different magnification;

[0108] An image segmentation module 830 is configured to segment pathology images of different magnifications to obtain multiple image blocks in each pathology image. Image blocks belonging to different pathology images have different magnifications, while image blocks belonging to the same pathology image have the same magnification. Image blocks of different magnifications have a corresponding relationship based on a pyramid cascade method between the pathology images.

[0109] The feature acquisition module 850 is used to extract features from each image block to obtain image features of each image block, and to fuse the image features of each image block according to the corresponding relationship between the image blocks to obtain multi-rate features of each image block;

[0110] A panoramic image construction module 870 is configured to construct a panoramic image based on the image blocks of different magnifications and their spatial positional relationships in the corresponding pathological images, as well as the multi-magnification characteristics of each image block;

[0111] The image analysis module 890 is used to input the panoramic image into the constructed context information perception model, learn the image structure information and context information in each pathological image, and obtain the analysis results of the pathological image data set; the analysis results are used for the target task, which includes at least one of the classification task and regression task related to the disease.

[0112] It should be noted that the pathological image analysis provided in the above embodiment is only illustrated by the division of the above-mentioned functional modules when performing pathological image analysis. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the pathological image analysis device will be divided into different functional modules to complete all or part of the functions described above.

[0113] In addition, the pathological image analysis device and the pathological image analysis method provided in the above embodiments belong to the same concept, and the specific manner in which each module performs operations has been described in detail in the method embodiment and will not be repeated here.

[0114] Fig. 10 shows a schematic structural diagram of an electronic device according to an exemplary embodiment.

[0115] It should be noted that the electronic device is only an example adapted for the present application and should not be considered to provide any limitation on the scope of use of the present application. The electronic device should not be interpreted as needing to rely on or necessarily having one or more components in the exemplary electronic device 2000 shown in FIG10.

[0116] The hardware structure of the electronic device 2000 may vary greatly due to different configurations or performances. As shown in FIG. 10 , the electronic device 2000 includes a power supply 210 , an interface 230 , at least one memory 250 , and at least one central processing unit (CPU) 270 .

[0117] Specifically, the power supply 210 is used to provide operating voltage for various hardware devices on the electronic device 2000 .

[0118] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices.

[0119] Of course, in other examples adapted by this application, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, as shown in FIG10 , which is not specifically limited here.

[0120] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon include an operating system 251, application 253 and data 255, etc. The storage method can be temporary storage or permanent storage.

[0121] Among them, the operating system 251 is used to manage and control the various hardware devices and application programs 253 on the electronic device 2000 to enable the central processing unit 270 to calculate and process the massive data 255 in the memory 250. It can be Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0122] The application 253 is a computer program that performs at least one specific task based on the operating system 251. It may include at least one module (not shown in FIG10 ), each of which may include a computer program for the electronic device 2000. For example, the information recommendation device may be considered as the application 253 deployed on the electronic device 2000.

[0123] The data 255 may be photos, pictures, etc. stored in a disk, or may be pathological image data, etc. stored in the memory 250 .

[0124] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read computer programs stored in the memory 250, thereby performing operations and processing on the massive amount of data 255 in the memory 250. For example, the pathological image analysis method may be implemented by the central processing unit 270 reading a series of computer programs stored in the memory 250.

[0125] In addition, the present application can also be implemented through hardware circuits or hardware circuits combined with software. Therefore, the implementation of the present application is not limited to any specific hardware circuits, software, or a combination of the two.

[0126] Please refer to FIG. 11 . An electronic device 4000 is provided in an embodiment of the present application. The electronic device 4000 may include a desktop computer, a laptop computer, a server, etc.

[0127] In FIG. 11 , the electronic device 4000 includes at least one processor 4001 , at least one communication bus 4002 , and at least one memory 4003 .

[0128] The processor 4001 and the memory 4003 are connected, for example, via a communication bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0129] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0130] Communication bus 4002 may include a path for transmitting information between the aforementioned components. Communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, for example. Communication bus 4002 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, FIG11 shows only one thick line, but this does not indicate that there is only one bus or only one type of bus.

[0131] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0132] The memory 4003 stores a computer program, and the processor 4001 reads the computer program stored in the memory 4003 through the communication bus 4002 .

[0133] When the computer program is executed by the processor 4001 , the pathological image analysis method in the above-mentioned embodiments is implemented.

[0134] In addition, an embodiment of the present application provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the pathological image analysis method in the above-mentioned embodiments is implemented.

[0135] In an embodiment of the present application, a computer program product is provided, which includes a computer program stored in a storage medium. A processor of a computer device reads the computer program from the storage medium and executes the computer program, causing the computer device to perform the pathological image analysis method of each of the above embodiments.

[0136] Compared with the related art, the present invention has the following beneficial effects:

[0137] 1. The present invention uses a deep learning network to automatically extract the features of pathological images at different magnifications, and performs feature fusion based on the correspondence between different magnifications. The panoramic pathological image is modeled as a panoramic map, and the structural information and contextual information in the pathological image are acquired by constructing a contextual information perception model. It can obtain various types of information from the pathological image to the maximum extent, which not only avoids the excessive dependence of the diagnostic results on the knowledge of pathological image features, but also effectively saves time, thereby solving the problem that the pathological image analysis method in the prior art cannot fully obtain the feature information of the image.

[0138] 2. The present invention combines the features of pathological images at different magnifications to learn the multi-scale feature representation of tumors. Combined with the powerful contextual information acquisition capability of the contextual information perception model, it models the panoramic pathological images, fully acquires the feature information in the pathological images, and thus realizes the accurate diagnosis and analysis of the disease.

[0139] 3. The present invention uses panoramic pathological images rather than manually selected regions of interest, which maximizes the retention of information in the pathological images. It proposes a feature acquisition method based on multi-magnification panoramic pathological images, which can effectively utilize information at different scales in pathological images to complete diagnosis, prognosis and other tasks.

[0140] 4. The present invention designs a contextual information perception model based on graph transformer, which can obtain the adjacency relationship of graph nodes according to the adjacency matrix display and learn the structural information in the pathological image. At the same time, the multi-head self-attention mechanism in the transformer module is used to implicitly capture the global contextual dependencies in the pathological image, which can better learn the structural features and global contextual information contained in the pathological image data, and provide richer feature representations for subsequent diagnosis and prognosis tasks.

[0141] 5. The present invention quantifies and streamlines the complex clinical pathological diagnostic process, which can provide auxiliary reference for clinical diagnosis and prognosis, and provides an explanation function for the model prediction results. It intuitively shows the reasons and results of the data in the pathological image affecting the results in the model, reduces the obstacles for the model to enter practical application scenarios, and provides an explainable pathological image analysis method.

[0142] 6. The present invention can effectively save the cost of manual analysis and classification of data, while avoiding excessive dependence on the doctor's technical level, saving a lot of manpower and material costs of doctors, integrating the pathological image features of different magnifications, and the information contained in the data is richer.

[0143] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0144] The above content is only a preferred exemplary embodiment of the present invention and is not intended to limit the implementation scheme of the present invention. Ordinary technicians in this field can easily make corresponding changes or modifications based on the main concept and spirit of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection required by the claims.

Claims

1. A pathological image analysis method, characterized in that: The method comprises: Acquire a pathological image data set, wherein the pathological image data set includes a plurality of pathological images, each of the pathological images having a different magnification; The pathological images of different magnifications are cut respectively to obtain a plurality of image blocks in each pathological image; the image blocks belonging to different pathological images have different magnifications, the image blocks belonging to the same pathological image have the same magnifications, and the image blocks of different magnifications have a corresponding relationship based on the pyramid cascade mode between the pathological images; Extracting features from each of the image blocks to obtain image features of each of the image blocks, and fusing the image features of each of the image blocks according to the corresponding relationship between the image blocks to obtain multi-rate features of each of the image blocks; Constructing a panoramic image according to the image blocks of different magnifications and their spatial positional relationships in the corresponding pathological images, and the multi-magnification characteristics of the image blocks; The panoramic image is input into the constructed context information perception model, the graph structure information and context information in each of the pathological images are learned, and the analysis results of the pathological image data set are obtained; the analysis results are used for the target task, and the target task includes at least one of the classification task and regression task related to the disease.

2. The method according to claim 1, characterized in that The method of constructing a panoramic image according to the image blocks of different magnifications and their spatial positional relationship in the corresponding pathological image, and the multi-magnification features of the image blocks, comprises: Using each of the image blocks as a node of the panoramic image, and using the multi-rate feature of each of the image blocks as a node feature of each node in the panoramic image; Constructing an adjacency matrix according to the spatial position relationship of each image block in the pathological image to which it belongs, and constructing a path between each node according to the adjacency matrix; The panoramic image is obtained from each node and its node features, and the paths between the nodes.

3. The method according to claim 1, characterized in that The context information perception model includes a graph module and a Transformer module; The panoramic image is input into the constructed context information perception model, and the image structure information and context information in each pathological image are learned to obtain the analysis of the pathological image data set. Results, including: Based on the graph module, the adjacency relationship between different nodes is learned by using an adjacency matrix for constructing the panoramic graph to obtain graph structure information of the panoramic graph; Based on the self-attention mechanism in the Transformer module, the influence of each node in the panoramic image on the target task is learned to obtain the context information of the panoramic image; The analysis result is generated based on the graph structure information and context information of the panoramic image.

4. The method according to claim 3, characterized in that The graph module includes an aggregation layer, a multi-layer perceptron, and an activation function; The method of learning the adjacency relationship between different nodes based on the graph module by using the adjacency matrix for constructing the panoramic graph to obtain the graph structure information of the panoramic graph includes: Aggregating each node of the panoramic graph and each neighboring node of the node in the adjacency matrix through the aggregation layer in the graph module to obtain intermediate data; The intermediate data is nonlinearly transformed through the multi-layer perceptron and activation function in the graph module to obtain the graph structure information in the panoramic image.

5. The method according to claim 3, characterized in that The Transformer module includes a plurality of repeated blocks, each of which includes a multi-head self-attention MHA layer, a multi-layer perceptron MLP layer with residual connections, and a normalization Norm layer; The self-attention mechanism in the Transformer module is used to learn the influence of each node in the panoramic image on the target task to obtain context information of the panoramic image, including: By using the MHA layer and the Norm layer in the repeated block, the global spatial dependency and context information in the pathological image are learned according to the influence of each node in the panoramic image on the target task, thereby obtaining a feature map of the repeated block; By using the MLP layer and the Norm layer in the repeated block, the attention value of each repeated block is calculated according to the feature map of each repeated block; The attention values ​​of the repeated blocks are weightedly calculated to obtain context information of the panoramic image.

6. The method according to claim 1, characterized in that The pathological images of different magnifications are cut respectively to obtain a plurality of image blocks in each pathological image, including: Preprocessing each of the pathological images; Segmenting the foreground area and the background area of ​​each of the preprocessed pathological images; For each foreground area obtained by segmenting the pathological image, the foreground area is cut into image blocks of a set size.

7. The method according to claim 1, characterized in that The step of fusing the image features of each image block according to the correspondence between the image blocks to obtain the multi-rate features of each image block includes: Determining a pyramid cascade mode between the pathological images to which each of the image blocks belongs based on the corresponding relationship between the image blocks; According to the first number of image blocks in the pathological image in the next level pyramid, the image features of the second number of image blocks in the pathological image in the previous level pyramid are respectively copied to obtain the image features of the first number of image blocks in the previous level; splicing the image features of the first number of image blocks at the previous level with the image features of the first number of image blocks at the next level to obtain the multi-rate features of the first number of image blocks at the next level; Until the image features of each image block in the pathological images at each level of the pyramid are completed, the multi-magnification features of each image block are obtained.

8. A pathological image analysis device, characterized in that: The device comprises: A data set acquisition module, used to acquire a pathological image data set, wherein the pathological image data set includes a plurality of pathological images, each of which has a different magnification; An image cutting module, used for cutting the pathological images of different magnifications respectively to obtain a plurality of image blocks in each pathological image; the image blocks belonging to different pathological images have different magnifications, the image blocks belonging to the same pathological image have the same magnifications, and the image blocks of different magnifications have a corresponding relationship based on the pyramid cascade mode between the pathological images; A feature acquisition module is used to extract features from each of the image blocks to obtain image features of each of the image blocks, and to fuse the image features of each of the image blocks according to the corresponding relationship between the image blocks to obtain multi-rate features of each of the image blocks; A panoramic image construction module, used to construct a panoramic image according to the image blocks of different magnifications and their spatial positional relationships in the corresponding pathological images, as well as the multi-magnification characteristics of the image blocks; The image analysis module is used to input the panoramic image into the constructed context information perception model, learn the image structure information and context information in each pathological image, and obtain the analysis result of the pathological image data set; the analysis result is used for the target task, and the target task includes At least one of the disease-related classification tasks and regression tasks.

9. An electronic device, characterized in that: include: at least one processor and at least one memory, wherein: The memory stores program instructions or codes; The program instructions or codes are loaded and executed by the processor, so that the electronic device implements the pathological image analysis method according to any one of claims 1 to 7.

10. A storage medium having program instructions or codes stored thereon, characterized in that: The program instructions or codes are loaded and executed by a processor to implement the pathological image analysis method according to any one of claims 1 to 7.

Images