Cancer molecular subtype classification method, model training method, equipment and medium
By comparing the combination of learning model and classification model, feature vectors are generated and sparse attention calculation is performed, the problem of limited classification ability of deep learning models when identifying morphological differences in tumors is solved, and the accuracy of recognition of cancer molecular subtypes is improved and the effectiveness of targeted therapies.
Patent Information
- Application Number
- CN202311762277.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-20
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-12-20
AI Technical Summary
In the prior art, deep learning models usually divide the full slide image into thousands of patches due to GPU memory limitations when identifying morphological differences in tumors, resulting in limited classification capabilities and affecting the accuracy of targeted treatment in cancer patients.
The pre-trained contrast learning model is used to segment the full slide image, generate the first feature vector, the second feature vector and the anchor vector, and use the classification model to perform sparse attention calculation and convolution operations to generate position information and Predictive results of key feature information for classification of cancer molecular subtypes.
The classification ability of deep learning models for digital pathological images is improved, the effectiveness of targeted treatment for cancer patients is enhanced, and the generation of richer feature representations and consideration of spatial structure information is enhanced.
Smart Images

Figure CN117953270B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a cancer molecular subtype classification method, model training method, device and medium. Background Art
[0002] Currently, the identification of cancer molecular subtypes relies primarily on genomic or transcriptomic analysis, which requires complex data acquisition and analysis expertise, leading to extended turnaround times and higher costs.
[0003] To address the above technical issues, in related technologies, deep learning models are used to identify morphological differences within tumors, especially whole slide images (WSIs) stained with hematoxylin and eosin (H&E), which have rich clinical applications and contain sufficient tumor information to directly infer molecular subtypes. Therefore, there is a growing demand to identify molecular subtypes through whole slide images in an inexpensive and scalable manner and to accurately divide patients into subgroups to assist personalized treatment.
[0004] However, a digital pathology image has approximately 1 billion pixels. Due to the limitation of GPU memory, the method of deep learning models to identify morphological differences within tumors usually divides the whole slide image into thousands of patches and then uses hand-crafted features for downstream analysis. This limits the classification ability of deep learning models for digital pathology images and affects the accuracy of guiding targeted treatment for cancer patients in clinical practice. Summary of the invention
[0005] The purpose of this application is to provide a cancer molecular subtype classification method, model training method, equipment and medium, which can improve the classification ability of deep learning models for digital pathology images and improve the effectiveness of guiding targeted treatment of cancer patients in clinical practice.
[0006] The present application embodiment provides a method for classifying cancer molecular subtypes, comprising:
[0007] Acquire whole slide images;
[0008] Segmenting the whole slide image to obtain a plurality of image blocks;
[0009] Using a pre-trained contrast learning model, feature contrast learning is performed on the image block to generate a first feature vector, a second feature vector, a third feature vector and an anchor vector; the first feature vector, the second feature vector and the anchor vector are feature vectors obtained by performing feature extraction processing on the image block along different view directions, and the third feature vector is a feature vector selected from a preset feature vector set and most similar to the anchor vector;
[0010] Using a pre-trained classification model, performing a sparse attention calculation operation and a convolution operation on the first feature vector, the second feature vector, and the third feature vector to generate a prediction result; the prediction result includes the position information and key feature information of the image block;
[0011] Based on the prediction results, the whole slide image is classified into cancer molecular subtypes.
[0012] In some embodiments, the whole slide image is segmented to obtain a plurality of image blocks, including:
[0013] Downsampling the whole slide image to obtain a downsampled image;
[0014] Segmenting the downsampled image to obtain a plurality of image blocks;
[0015] The image blocks are screened to eliminate the image blocks lacking tissue or showing minimal tissue.
[0016] In some embodiments, the contrastive learning model includes a first branch network, a second branch network, and a third branch network;
[0017] The method of using a pre-trained contrast learning model to perform feature contrast learning on the image block to generate a first feature vector, a second feature vector, a third feature vector and an anchor vector includes:
[0018] Using the first branch network, the second branch network and the third branch network, feature extraction processing is performed on the image block along different view directions to generate the first feature vector, the second feature vector and the anchor vector; the network weight parameter of the second branch network is updated according to the network weight parameter of the first branch network, and the second branch network and the third branch network share the same network weight parameter;
[0019] A similarity calculation is performed between the anchor point vector and a preset vector in the preset feature vector set, and a plurality of feature vectors most similar to the anchor point vector are selected from the preset feature vector set to obtain a plurality of the third feature vectors.
[0020] In some embodiments, the first branch network, the second branch network and the third branch network respectively include a data enhancement layer, a plurality of extraction networks and a projection layer, and the extraction network is a Swin Transformer structure;
[0021] The performing feature extraction processing on the image block comprises:
[0022] Using the data enhancement layer, upsampling the image block to obtain an enhanced image block;
[0023] Using each of the extraction networks, performing feature extraction processing on the enhanced image block layer by layer, and outputting a corresponding feature extraction vector in the last feature extraction network;
[0024] The projection layer is used to perform spatial projection transformation processing on the feature extraction vector to obtain the first feature vector, the second feature vector or the anchor point vector.
[0025] In some embodiments, the classification model includes a sparse attention network and a convolutional neural network;
[0026] The using of the pre-trained classification model to perform a sparse attention calculation operation and a convolution operation on the first feature vector pair and the second feature vector pair to generate a prediction result includes:
[0027] Using the sparse attention network, a sparse attention calculation operation is performed on the first eigenvector, the second eigenvector and the third eigenvector to generate a first position feature matrix; the first position feature matrix includes global position information between the first eigenvector, the second eigenvector and each of the third eigenvectors;
[0028] Using the convolutional neural network, performing a layer-by-layer convolution operation on the first eigenvector, the second eigenvector and the third eigenvector to generate a second position feature matrix; the second position feature matrix includes local position information of the first eigenvector, the second eigenvector and each of the third eigenvectors;
[0029] The first position feature matrix and the second position feature matrix are aggregated to obtain a prediction result.
[0030] The present application also provides a model training method, including:
[0031] Constructing a training set and a validation set; the training set and the validation set are both composed of a plurality of sample image blocks, and the sample image blocks contain characteristic information of at least one cancer molecular subtype;
[0032] Using the contrast learning model to be trained, performing feature contrast learning on the sample image blocks of the training set to generate a first sample vector, a second sample vector, a third sample vector and a sample anchor vector; the first sample vector, the second sample vector and the sample anchor vector are feature vectors obtained by performing feature extraction processing on the sample image blocks along different view directions, and the third sample vector is a feature vector selected from a preset feature vector set and most similar to the sample anchor vector;
[0033] Construct a first vector pair using the first sample vector and the second sample vector, and construct a second vector pair using the second sample vector and the third sample vector;
[0034] Using the classification model to be trained, performing a sparse attention calculation operation and a convolution operation on the first vector pair and the second vector pair to generate a sample prediction result; the sample prediction result includes position information and key feature information of the sample image block;
[0035] According to the model loss information, the weight parameters of the contrastive learning model to be trained, the weight parameters of the classification model to be trained and the preset feature vector are updated, and when the model loss information meets the termination condition, the pre-trained contrastive learning model and the pre-trained classification model are obtained; the model loss information characterizes the degree of matching between the feature information of the sample image blocks of the verification set and the sample prediction results.
[0036] In some embodiments, the model loss information includes contrastive learning loss information;
[0037] The expression of the contrastive learning loss information is:
[0038]
[0039]
[0040]
[0041] Among them, L CTrains To compare the learning loss information, is the contrast loss information of positive feature samples, is the contrast loss information of negative feature samples, z1 is the first vector pair, is a positive feature sample pair set, which contains the first vector pair and the second vector pair of the current sample image block, z - is a negative feature sample pair set, which includes the first vector pair and the second vector pair of other sample image blocks. is the i-th element in the positive feature sample pair set, i = 1, 2, ..., S, S + 1, S is the number of second vector pairs of the current sample image block, S is the number of elements in the positive feature sample pair set, N is the number of elements in the negative feature sample pair set, and τ is the temperature hyperparameter.
[0042] In some embodiments, the model loss information includes classification loss information;
[0043] The expression of the classification loss information is:
[0044] L total=L classification +L sparse ,
[0045]
[0046] p(E mn ):Bern(E mn |p mn ),
[0047] p mn =σ(f q (x m ) T f k (x n )+b),
[0048]
[0049] Among them, L total is the total classification loss information, L classification is the classification loss information, L sparse is the sparse loss information, is the Bernoulli probability distribution of the image patch feature connection, The Bernoulli probability of connecting the features of the mth and nth image patches, p(E) is the prior probability distribution of the image block feature connection, p(E mn ) is the prior probability of the connection between the mth and nth image block features, p(E mn )∈p(E),p mn is the pre-probability of the connection between the mth and nth image patch features, f is the convolution operation, σ is the sigmoid operation, U is the sample extracted from the continuous uniform distribution, τ′ is the approximation hyperparameter, and x m is the feature vector of the mth image block, x n is the feature vector of the nth image block, f q is the convolution operation of the qth convolutional layer, f k is the convolution operation of the kth convolutional layer, b is the bias value, E mn Represents the connection strength from the mth image block to the nth image block.
[0050] An embodiment of the present application further provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the above method when executing the computer program.
[0051] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.
[0052] The beneficial effects of the present application are as follows: using a contrastive learning model, feature contrastive learning is performed on the image blocks obtained by segmenting the whole slide image to generate a first feature vector, a second feature vector and a third feature vector, and then using a classification model, sparse attention calculation operations and convolution operations are performed on the first feature vector, the second feature vector and the third feature vector to generate a prediction result containing the position information and key feature information of the image block, and then based on the prediction result, the whole slide image is classified into cancer molecular subtypes. Since the first feature vector and the second feature vector are directly generated by the contrastive learning model, the third feature vector is a feature vector selected from a preset feature vector set that is most similar to the anchor vector generated by the contrastive learning model, so as to achieve the technical effect of augmenting positive feature samples to generate richer feature representations, improve the ability to capture domain-specific features related to downstream tasks, perform sparse attention calculation operations and convolution operations on the first feature vector, the second feature vector and the third feature vector, take into account spatial structure information and related key feature information, improve the classification ability of the deep learning model for digital pathology images, and improve the effectiveness of guiding targeted treatment of cancer patients in clinical practice. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 This is an optional flow chart of the cancer molecular subtype classification method provided in the embodiments of the present application.
[0054] Figure 2 It is a flowchart of the specific method of step S103 provided in an embodiment of the present application.
[0055] Figure 3 It is a flowchart of the specific method of step S104 provided in an embodiment of the present application.
[0056] Figure 4 It is an optional flowchart of the model training method provided in the embodiment of the present application.
[0057] Figure 5 This is an optional structural schematic diagram of the cancer molecular subtype classification device provided in the embodiment of the present application.
[0058] Figure 6 It is an optional structural diagram of the model training device provided in an embodiment of the present application.
[0059] Figure 7 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application.
[0060] Figure 8 It is an optional structural diagram of the contrastive learning model provided in the embodiment of the present application. DETAILED DESCRIPTION
[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0062] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0064] First, some nouns involved in this application are analyzed:
[0065] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0066] Contrastive Learning: Contrastive learning is a type of self-supervised learning. Contrastive learning does not need to rely on manually labeled data, but directly uses the data itself as supervision information. Using contrastive learning methods, machine learning models can be trained to distinguish between similar and different images. Self-supervised learning in the image field is divided into two types: generative self-supervised learning and discriminative self-supervised learning. Contrastive learning uses typical discriminative self-supervised learning. The general principle of contrastive learning is: by automatically constructing similar instances (positive samples) and dissimilar instances (negative samples), a learning model is obtained through learning, and the learning compares positive samples and negative samples in the projection space, so that similar instances are closer in the projection space, while dissimilar instances are farther apart in the projection space, and the difference becomes larger.
[0067] Transformer model: A neural network model based on full self-attention, which can calculate the representation of input and output by relying on the self-attention mechanism. The Transformer model is a model framework built on the attention mechanism, and its overall architecture can be divided into input layer, encoding layer, decoding layer and output layer. If the Transformer model inputs the image feature matrix into the encoding layer, the encoding layer will send the image feature matrix after self-attention calculation to the feedforward neural network of this layer after performing self-attention calculation on the image feature matrix, and then send the image feature matrix to the next encoding layer through the feedforward neural network of this layer, and repeat the corresponding processing until all the multiple encoding layers are executed, and then use the residual block to decode the features, and finally use a convolutional layer to obtain the result of feature extraction.
[0068] Convolutional Neural Networks (CNN): A type of feedforward neural network that includes convolutional calculations and has a deep structure. It is one of the representative algorithms of deep learning. Convolutional neural networks have the ability of representation learning and can perform shift-invariant classification on input information according to its hierarchical structure. Therefore, they are also called "Shift-Invariant Artificial Neural Networks (SIANN)". Convolutional neural networks are constructed by imitating the visual perception mechanism of organisms. They can perform supervised learning and unsupervised learning. The convolution kernel parameter sharing in the hidden layer and the sparsity of the inter-layer connection enable convolutional neural networks to learn grid-like topology features such as pixels and audio with a small amount of computation, with stable effects and no additional feature engineering requirements for data.
[0069] In the related art, deep learning models are used to identify morphological differences within tumors, especially whole slide images (WSIs) stained with hematoxylin and eosin (H&E), which are rich in clinical applications and contain sufficient tumor information to directly infer molecular subtypes. However, a digital pathology image is about 1 billion pixels. Due to the limitation of GPU memory, the method of deep learning models to identify morphological differences within tumors usually divides the whole slide image into thousands of patches and then uses hand-crafted features for downstream analysis, which limits the classification ability of deep learning models for digital pathology images. Some people have also proposed using a pre-trained deep learning network to extract instance-level patch embeddings from non-overlapping tissue patches in WSIs and apply global pooling to generate WSI-level embeddings. However, in the instance-level feature extraction step, a ResNet encoder based on the ImageNet dataset is usually used, which is an out-of-domain dataset for natural images and cannot capture all domain-specific features related to downstream tasks, such as cancer phenotypes.
[0070] Based on this, the embodiments of the present application provide a cancer molecular subtype classification method, model training method, device and medium, aiming to improve the classification ability of deep learning models for digital pathology images and improve the effectiveness of guiding targeted treatment of cancer patients in clinical practice.
[0071] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0072] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0073] The cancer molecular subtype classification method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The cancer molecular subtype classification method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN) and big data and artificial intelligence platforms and other basic cloud computing services Cloud server; the software can be an application that implements a text classification method, etc., but is not limited to the above forms.
[0074] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network personal computers (Personal Computer, PC), minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0075] See also Figure 1 , Figure 1 is an optional flow chart of the method for classifying cancer molecular subtypes provided in the embodiments of the present application. In some embodiments of the present application, Figure 1 The method may specifically include but is not limited to steps S101 to S105. Figure 1 These five steps are introduced in detail.
[0076] Step S101, acquiring a whole slide image.
[0077] Step S102, segmenting the whole slide image to obtain a plurality of image blocks.
[0078] In some embodiments, step S102 specifically includes the following steps:
[0079] Downsampling the whole slide image to obtain a downsampled image;
[0080] Segment the downsampled image to obtain multiple image blocks;
[0081] The image patches are screened to remove those that lack tissue or exhibit minimal tissue.
[0082] Specifically, in the segmentation processing stage, the tissue area of each whole slide image is automatically segmented using the traditional Otsu threshold. First, the whole slide image is read into memory at a downsampled resolution (default 20× magnification) to obtain a downsampled image, and then the tissue area of each downsampled image is segmented into square image blocks with non-overlapping edge lengths of 360 μm, and the pixel size of the image block is adjusted to 256×256. Finally, the image blocks are screened by applying a minimum threshold of 50 to the median of each 8-bit RGB channel, and image blocks lacking tissue or showing minimal tissue are eliminated, and image blocks that meet the screening criteria are output.
[0083] Step S103, using the pre-trained contrast learning model, perform feature contrast learning on the image block to generate a first feature vector, a second feature vector, a third feature vector and an anchor vector.
[0084] The first feature vector, the second feature vector and the anchor vector are feature vectors obtained by performing feature extraction processing on the image block along different viewing directions, and the third feature vector is a feature vector selected from a preset feature vector set that is most similar to the anchor vector.
[0085] It can be understood that, for an image block, the first feature vector, the second feature vector and the third feature vector generated by the feature extraction process of the image block are positive feature samples of the image block, and the first feature vector, the second feature vector and the third feature vector generated by the feature extraction process of other image blocks are negative feature samples of the image block. When generating the first feature vector, the second feature vector and the third feature vector, the positive feature samples and the negative feature samples are compared in the projection space, so that similar instances are closer in the projection space, while dissimilar instances are farther away in the projection space, and the difference becomes larger.
[0086] It can be understood that the anchor point vector is the feature vector corresponding to the anchor point, the anchor point is a data sample in the image block, a feature vector of a data point belonging to the same distribution as the anchor point is a positive feature sample, and another feature vector of a data point belonging to a different distribution is a negative feature sample.
[0087] Specifically, each image block is input into a pre-trained contrastive learning model, feature contrast learning is performed in the contrastive learning model, each image block is input into the first branch network, the second branch network and the third branch network respectively, feature extraction processing is performed along different view directions in the first branch network, the second branch network and the third branch network to generate a first feature vector, a second feature vector and an anchor vector, and then the anchor vector is compared with all feature vectors in a preset feature vector set to find several feature vectors that are most similar to the anchor vector, and several third feature vectors are obtained, so as to achieve the technical effect of augmenting the positive feature samples to produce a richer feature representation.
[0088] Step S104, using the pre-trained classification model, sparse attention calculation operations and convolution operations are performed on the first eigenvector, the second eigenvector, and the third eigenvector to generate a prediction result.
[0089] The prediction result includes the location information and key feature information of the image block.
[0090] Specifically, the first eigenvector, the second eigenvector and the third eigenvector are input into a pre-trained classification model. On the one hand, a sparse attention calculation operation is performed on the first eigenvector, the second eigenvector and the third eigenvector to predict the key feature information of the image block. On the other hand, a convolution operation is performed on the first eigenvector, the second eigenvector and the third eigenvector. The first eigenvector, the second eigenvector and the third eigenvector are rearranged into a two-dimensional feature map, and different levels of spatial information are obtained through convolution kernels of different sizes, so as to predict the position information of the image block. The prediction information output by the sparse attention calculation operation and the convolution operation is combined to generate a prediction result.
[0091] Step S105 , classifying the whole slide image into cancer molecular subtypes according to the prediction results.
[0092] Specifically, after the prediction processing of the pre-trained contrast learning model and the pre-trained classification model, the generated prediction results include the prediction probability of each type of cancer molecular subtype in the training data used. According to the prediction probability of the prediction results, the whole slide image is classified into cancer molecular subtypes, thereby realizing the identification of cancer molecular subtypes.
[0093] In the steps S101 to S105 shown in the embodiment of the present application, a contrastive learning model is used to perform feature contrastive learning on the image block obtained by segmenting the whole slide image to generate a first feature vector, a second feature vector and a third feature vector, and then a classification model is used to perform sparse attention calculation operations and convolution operations on the first feature vector, the second feature vector and the third feature vector to generate a prediction result containing the position information and key feature information of the image block, and then the whole slide image is classified into cancer molecular subtypes based on the prediction result. Since the first feature vector and the second feature vector are directly generated by the contrastive learning model, the third feature vector is a feature vector selected from a preset feature vector set that is most similar to the anchor vector generated by the contrastive learning model, so as to achieve the technical effect of augmenting positive feature samples to generate richer feature representations, improve the ability to capture domain-specific features related to downstream tasks, perform sparse attention calculation operations and convolution operations on the first feature vector, the second feature vector and the third feature vector, take into account spatial structure information and related key feature information, improve the classification ability of the deep learning model for digital pathology images, and improve the effectiveness of guiding targeted treatment of cancer patients in clinical practice.
[0094] See also Figure 2 , Figure 2 is a flowchart of a specific method of step S103 provided in an embodiment of the present application. In some embodiments of the present application, step S103 may specifically include but is not limited to steps S201 to S202. Figure 2 These two steps are introduced in detail.
[0095] See also Figure 8 In this embodiment, the contrastive learning model uses the CTranspath model structure as the backbone architecture, and the contrastive learning model includes a first branch network, a second branch network, and a third branch network.
[0096] Step S201, using the first branch network, the second branch network and the third branch network, respectively perform feature extraction processing on the image block along different view directions to generate a first feature vector, a second feature vector and an anchor vector.
[0097] The network weight parameters of the second branch network are updated according to the network weight parameters of the first branch network, and the second branch network and the third branch network share the same network weight parameters.
[0098] Specifically, the image blocks are respectively input into the first branch network, the second branch network and the third branch network, and feature extraction processing is performed on the image blocks along different viewing directions. The first branch network generates a first feature vector, the second branch network generates a second feature vector, and the third branch network generates an anchor vector. Since the network weight parameters of the second branch network are updated according to the network weight parameters of the first branch network, the second branch network and the third branch network share the same network structure and network weight parameters, so that the first feature vector, the second feature vector and the anchor vector have the same semantic information, which can better match the feature vectors in the preset feature vector set.
[0099] More specifically, the first branch network uses parameter θ as the network weight parameter for generating the first feature vector, the second branch network uses parameter ξ as the network weight parameter for generating the second feature vector, and the third branch network uses the same network structure and network weight parameter as the second branch network, and the feature vector in the preset feature vector set stores the feature information of the second branch network. During the training process, the parameter θ is updated according to the loss information of the iterative training result, and the parameter ξ is updated following the update of the parameter θ, and the update method is ξ←mξ+(1-m)θ, where m is the update weight parameter, and the feature vector in the preset feature vector is updated following the update of the network structure and network weight parameter of the second branch network at the end of each iteration.
[0100] See again Figure 8 The first branch network, the second branch network and the third branch network respectively include a data enhancement layer, multiple extraction networks and a projection layer, and the extraction network is a Swin Transformer structure.
[0101] In some embodiments, the feature extraction process is performed on the image block, specifically including:
[0102] Using the data enhancement layer, the image block is upsampled to obtain an enhanced block;
[0103] Using each extraction network, the enhanced image block is subjected to layer-by-layer feature extraction processing, and the last feature extraction network outputs the corresponding feature extraction vector;
[0104] The projection layer is used to perform spatial projection transformation processing on the feature extraction vector to obtain the first feature vector, the second feature vector or the anchor point vector.
[0105] Specifically, the image block is first input into the data enhancement layer, and the image block is upsampled to obtain an enhanced block to increase the data volume of the image block. The enhanced block is then input into the first extraction network, and after feature extraction processing layer by layer in each extraction network, the corresponding feature extraction vector is output in the last feature extraction network. In the extraction network, a convolutional neural network structure is used to replace the original patch partition. The convolutional neural network has three convolutional layers with kernel sizes of 3×3, 3×3 and 1×1, respectively, which is similar to the current ResNet structure. The first branch network, the second branch network and the third branch network are designed to use four extraction networks to generate hierarchical feature extraction vectors in sequential stages. Finally, the feature extraction vector output by the last feature extraction network is input into the projection layer and projected to the corresponding projection space to generate a first feature vector, a second feature vector or an anchor vector.
[0106] Step S202 , calculating similarity between the anchor point vector and a preset vector in a preset feature vector set, selecting a number of feature vectors that are most similar to the anchor point vector from the preset feature vector set, and obtaining a number of third feature vectors.
[0107] Specifically, cosine similarity calculation is performed on the anchor vector and a preset vector in the preset feature vector set to obtain a similarity calculation result. Based on the similarity calculation result, several feature vectors with the highest similarity to the anchor vector are selected from the preset feature vector set as the third feature vector.
[0108] See also Figure 3 , Figure 3 is a flowchart of a specific method of step S104 provided in an embodiment of the present application. In some embodiments of the present application, step S104 may specifically include but is not limited to steps S301 to S303. Figure 3 These three steps are introduced in detail.
[0109] In this embodiment, the classification model includes a sparse attention network and a convolutional neural network.
[0110] Step S301, using a sparse attention network, performing a sparse attention calculation operation on the first eigenvector, the second eigenvector, and the third eigenvector to obtain a first position feature matrix;
[0111] The first position feature matrix includes the global position information between the first eigenvector, the second eigenvector and each third eigenvector.
[0112] Specifically, the first eigenvector, the second eigenvector and the third eigenvector of the input are linearly transformed to obtain the similarity matrix between the query matrix and the key matrix. In the similarity matrix, each row represents the similarity between a query vector and all key vectors, where the similarity can be calculated by dot product, cosine similarity, etc. In sparse attention, only the first N largest elements in each row of the similarity matrix will be retained, and the remaining elements are set to 0. Then, these N elements are softmax normalized to obtain the attention distribution of each query matrix to all key matrices.
[0113] More specifically, in a similarity matrix with N elements, the estimated probability of a connection between elements is The attention value between image blocks can be calculated according to the following formula,
[0114]
[0115] Among them, A is the attention value, s() represents probability-based sampling, ⊙ represents element-wise multiplication, Q and K are the query matrix and key matrix, respectively.
[0116] In this way, the amount of calculation can be reduced, and the attention can be more focused on key vectors with higher similarity, thereby improving the performance of the model.
[0117] Step S302: Using a convolutional neural network, perform layer-by-layer convolution operations on the first eigenvector, the second eigenvector, and the third eigenvector to obtain a second position feature matrix.
[0118] The second position feature matrix includes the local position information of the first eigenvector, the second eigenvector and each third eigenvector.
[0119] Specifically, the first eigenvector, the second eigenvector and each third eigenvector are input into the convolutional neural network, and convolution calculation is performed in the convolutional neural network in a sliding window manner to obtain the spatial information of the first eigenvector, the second eigenvector and each third eigenvector at different levels, and finally generate a second position feature matrix. More specifically, the convolutional neural network is composed of three convolutional blocks, the first convolutional block includes a convolutional layer with a kernel size of 7 and a stride of 1, the second convolutional block includes a convolutional layer with a kernel size of 5 and a stride of 2, and the third convolutional block includes a convolutional layer with a kernel size of 3 and a stride of 1. The convolutional neural network can be defined as:
[0120]
[0121] Where k = 1, ..., P, P is the number of convolutional blocks, f krepresents the convolution operation of the kth convolution block, C represents the dimension of the feature matrix at the second position, is the output of the kth convolutional block in the convolutional neural network.
[0122] Step S303: Aggregate the first position feature matrix and the second position feature matrix to obtain a prediction result.
[0123] Specifically, aggregating the first position feature matrix and the second position feature matrix can be to add the first position feature matrix and the second position feature matrix, fuse them into a vector through a cat operation, and then input them into a fully connected layer to generate corresponding prediction results, and the prediction results include the predicted probability of each type of cancer molecular subtype in the training data used.
[0124] See also Figure 4 , Figure 4 is an optional flow chart of the model training method provided in the embodiments of the present application. In some embodiments of the present application, Figure 4 The method may specifically include but is not limited to steps S401 to S405. Figure 4 These five steps are introduced in detail.
[0125] Step S401, constructing a training set and a validation set.
[0126] The training set and the validation set are both composed of multiple sample image blocks, and the sample image blocks contain characteristic information of at least one cancer molecular subtype.
[0127] Specifically, the sample image blocks included two cancer molecular subtypes, TCGA-BRCA and / or TCGA-CRC. In the TCGA-BRCA cohort, a total of 875,526 sample image blocks were carefully sorted from 674 whole slide images of 633 patients. In the TCGA-CRC cohort, 418,592 sample image blocks were carefully selected from 412 whole slide images representing 393 patients in the TCGA-CRC cohort. Then, the training set and validation set were constructed based on the training data division experience of the deep learning model.
[0128] The 633 TCGA-BRCA and 393 TCGA-CRC datasets were divided, and the preprocessed datasets were divided into 539 and 330 training sets, and 135 and 82 validation sets, respectively, in a ratio of 8:2 (the constructed training and validation sets were mainly used to generate WSI-level embeddings by global pooling later, and the four molecular subtypes of breast cancer (LumA, LumB, Her2, TNBC) and the four molecular subtypes of colorectal cancer (CMS1, CMS2, CMS3, CMS4) were trained and validated respectively).
[0129] Step S402, using the contrast learning model to be trained, performing feature contrast learning on the sample image blocks of the training set to generate a first sample vector, a second sample vector, a third sample vector and a sample anchor point vector.
[0130] The first sample vector, the second sample vector and the sample anchor vector are feature vectors obtained by performing feature extraction processing on the sample image block along different viewing directions, and the third sample vector is a feature vector selected from a preset feature vector set that is most similar to the sample anchor vector.
[0131] Step S403: construct a first vector pair using the first sample vector and the second sample vector, and construct a second vector pair using the second sample vector and the third sample vector.
[0132] Step S404, using the classification model to be trained, performing sparse attention calculation operations and convolution operations on the first vector pair and the second vector pair to generate a sample prediction result; the sample prediction result includes position information and key feature information of the sample image block.
[0133] Step S405, based on the model loss information, update the weight parameters of the contrastive learning model to be trained, the weight parameters of the classification model to be trained and the preset feature vector, and when the model loss information meets the termination conditions, obtain the pre-trained contrastive learning model and the pre-trained classification model; the model loss information represents the degree of matching between the feature information of the sample image blocks of the verification set and the sample prediction results.
[0134] In some embodiments, the model loss information includes contrastive learning loss information;
[0135] The expression of the contrastive learning loss information is:
[0136]
[0137]
[0138]
[0139] Among them, L CTrains To compare the learning loss information, is the contrast loss information of positive feature samples, is the contrast loss information of negative feature samples, z1 is the first vector pair, is a positive feature sample pair set, which contains the first vector pair and the second vector pair of the current sample image block, z - is a negative feature sample pair set, which includes the first vector pair and the second vector pair of other sample image blocks. is the i-th element in the positive feature sample pair set, i = 1, 2, ..., S, S + 1, S is the number of second vector pairs of the current sample image block, S is the number of elements in the positive feature sample pair set, N is the number of elements in the negative feature sample pair set, and τ is the temperature hyperparameter.
[0140] In some embodiments, the model loss information includes classification loss information;
[0141] The expression of the classification loss information is:
[0142] L total =L classification +L sparse ,
[0143]
[0144] p(E mn ):Bern(E mn |p mn ),
[0145] p mn =σ(f q (x m ) T f k (x n )+b),
[0146]
[0147] Among them, L total is the total classification loss information, L classification is the classification loss information, L sparse is the sparse loss information, is the Bernoulli probability distribution of the image patch feature connection, The Bernoulli probability of connecting the features of the mth and nth image patches, p(E) is the prior probability distribution of the image block feature connection, p(E mn ) is the prior probability of the connection between the mth and nth image block features, p(E mn )∈p(E),p mn is the pre-probability of the connection between the mth and nth image patch features, f is the convolution operation, σ is the sigmoid operation, U is the sample extracted from the continuous uniform distribution, τ′ is the approximation hyperparameter, and x m is the feature vector of the mth image block, x n is the feature vector of the nth image block, f q is the convolution operation of the qth convolutional layer, f k is the convolution operation of the kth convolutional layer, b is the bias value, E mnRepresents the connection strength from the mth image block to the nth image block.
[0148] Specifically, E mn is a real value used to calculate the attention score between the mth image patch and the nth image patch. In the self-attention mechanism, the attention score is obtained by multiplying the transpose of the query vector (Q) and the key vector (K), and then performing softmax normalization. Here, E mn As part of the prior probability p(E), it is used to control the connection sparsity between image patches. If E mn If the value of is small, the connection probability between the mth image block and the nth image block will be reduced, thereby achieving connection sparsity. On the contrary, if E mn If the value of is large, the connection probability between the m-th image block and the n-th image block will increase, thereby achieving dense connection.
[0149] Specifically, sparse attention sparses the self-attention value by defining the probability of connection between features of image patches in existing data and only calculating the self-attention of connections extracted from this probability.
[0150] For the connection E∈{0,1} between the features of N image patches of length N N×N , the estimated Bernoulli probability The definition is as follows:
[0151]
[0152] ρ mn =σ(f q (x m ) T f k (x n )+b)
[0153] Among them, Bern represents Bernoulli probability distribution, f and σ represent convolutional neural network and sigmoid operation respectively, and m and n are the feature vectors of the mth and nth image blocks respectively.
[0154] Bernoulli probability based on estimates For the connection between image blocks, the attention value between image blocks can be calculated according to the following formula
[0155]
[0156] Where S() represents probability-based extraction, ⊙ represents factor-based multiplication, Q and K are the query matrix and key matrix respectively. Use the calculated attention concentration value to update a feature matrix for sequential data Such as the traditional self-attention concentration mechanism:
[0157] H=AV
[0158] in, Represents a matrix of values.
[0159] Since sampling from discrete probability distributions is not differentiable, we use the following continuous probability distribution approximation and use the reparameterization trick for sampling:
[0160]
[0161] Among them, U is a sample drawn from a continuous uniform distribution, and τ′ is a hyperparameter of the approximation degree.
[0162] In order to give the image block the sparseness of the connection between features, the prior probability p(E) about E is defined, then the estimated probability q p hi can be defined as an additional loss value for approximating the prior probability by the following KL divergence:
[0163]
[0164] p(E mn )~Bern(E mn |p mn ),
[0165] Among them, p mn It is the pre-probability of whether there is a connection between the m-th and n-th images, which can be determined according to the degree of sparsity.
[0166] It is understandable that the probability q p hi is obtained by calculating the KL divergence between the prior probability p(E) and the attention probability distribution. KL divergence is a measure of the difference between two probability distributions. The smaller its value, the more similar the two distributions are. By making the attention probability distribution as close as possible to the prior probability distribution, the sparsity of the connection between the image block features can be achieved. KL divergence can be added to the model training as an additional loss function to help the model learn a better attention probability distribution. The estimated probability q p The shape of hi is the same as the attention probability distribution, where each element represents the connection probability between image patches.
[0167] Therefore, the expression of classification loss information is as follows:
[0168] L total =L classification +L sparse
[0169] Among them, L total is the total classification loss information, L classification is the classification loss information, L sparse is the sparse loss information.
[0170] See also Figure 5 The present application also provides a cancer molecular subtype classification device, which can implement the above-mentioned cancer molecular subtype classification method, and the device includes:
[0171] The first module 501 is used to obtain a whole slide image;
[0172] The second module 502 is used to segment the whole slide image to obtain multiple image blocks;
[0173] The third module 503 is used to perform feature contrast learning on the image block using the pre-trained contrast learning model to generate a first feature vector, a second feature vector, a third feature vector and an anchor vector; the first feature vector, the second feature vector and the anchor vector are feature vectors obtained by performing feature extraction processing on the image block along different view directions, and the third feature vector is a feature vector selected from a preset feature vector set that is most similar to the anchor vector;
[0174] The fourth module 504 is used to perform a sparse attention calculation operation and a convolution operation on the first feature vector, the second feature vector and the third feature vector using a pre-trained classification model to generate a prediction result; the prediction result includes position information and key feature information of the image block;
[0175] The fifth module 505 is used to classify the whole slide image into cancer molecular subtypes based on the prediction results.
[0176] The specific implementation of the cancer molecular subtype classification device is basically the same as the specific embodiment of the above-mentioned cancer molecular subtype classification method, and will not be repeated here.
[0177] See also Figure 6 The embodiment of the present application also provides a model training device, which can implement the above-mentioned model training method, and the device includes:
[0178] The first training module 601 is used to construct a training set and a validation set; both the training set and the validation set are composed of a plurality of sample image blocks, and the sample image blocks contain characteristic information of at least one cancer molecular subtype;
[0179] The second training module 602 is used to perform feature contrast learning on the sample image blocks of the training set using the contrast learning model to be trained, and generate a first sample vector, a second sample vector, a third sample vector and a sample anchor vector; the first sample vector, the second sample vector and the sample anchor vector are feature vectors obtained by performing feature extraction processing on the sample image blocks along different view directions, and the third sample vector is a feature vector selected from a preset feature vector set and most similar to the sample anchor vector;
[0180] A third training module 603, configured to construct a first vector pair using the first sample vector and the second sample vector, and to construct a second vector pair using the second sample vector and the third sample vector;
[0181] The fourth training module 604 is used to perform a sparse attention calculation operation and a convolution operation on the first vector pair and the second vector pair using the classification model to be trained to generate a sample prediction result; the sample prediction result includes position information and key feature information of the sample image block;
[0182] The fifth training module 605 is used to update the weight parameters of the contrastive learning model to be trained, the weight parameters of the classification model to be trained and the preset feature vector according to the model loss information, and obtain the pre-trained contrastive learning model and the pre-trained classification model when the model loss information meets the termination conditions; the model loss information represents the degree of matching between the feature information of the sample image blocks of the verification set and the sample prediction results.
[0183] The specific implementation of the model training device is basically the same as the specific implementation of the above-mentioned model training method, and will not be repeated here.
[0184] Figure 7 It is a block diagram of an electronic device according to an exemplary embodiment.
[0185] Refer to the following Figure 7 An electronic device 700 according to this embodiment of the present disclosure is described. Figure 7 The electronic device 700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0186] like Figure 7 As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one storage unit 720, a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710), a display unit 740, etc.
[0187] The storage unit stores program codes, which can be executed by the processing unit 710, so that the processing unit 710 performs the steps according to various exemplary embodiments of the present disclosure described in the above method section of this specification. For example, the processing unit 710 can perform the following steps: Figure 1 , Figure 2 , Figure 3 and Figure 4 Follow the steps shown in .
[0188] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 7201 and / or a cache storage unit 7202 , and may further include a read-only storage unit (ROM) 7203 .
[0189] The storage unit 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0190] Bus 730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
[0191] The electronic device 700 may also communicate with one or more external devices 700' (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 760. The network adapter 760 may communicate with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0192] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the above method when executed by a processor.
[0193] The cancer molecular subtype classification method, model training method, device and medium provided in the embodiment of the present application use a contrast learning model to perform feature contrast learning on the image block obtained by segmenting the whole slide image to generate a first feature vector, a second feature vector and a third feature vector, and then use a classification model to perform sparse attention calculation operations and convolution operations on the first feature vector, the second feature vector and the third feature vector to generate a prediction result containing the position information and key feature information of the image block, and then classify the whole slide image into cancer molecular subtypes based on the prediction result. Since the first feature vector and the second feature vector are directly generated by the contrast learning model, the third feature vector is a feature vector selected from a preset feature vector set that is most similar to the anchor vector generated by the contrast learning model, so as to achieve the technical effect of augmenting positive feature samples to generate richer feature representations, improve the ability to capture domain-specific features related to downstream tasks, perform sparse attention calculation operations and convolution operations on the first feature vector, the second feature vector and the third feature vector, take into account spatial structure information and related key feature information, improve the classification ability of the deep learning model for digital pathology images, and improve the effectiveness of guiding targeted treatment of cancer patients in clinical practice.
[0194] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the implementation of the present disclosure.
[0195] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0196] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.
[0197] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).
[0198] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.
[0199] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.
[0200] The exemplary embodiments of the present disclosure are specifically shown and described above. It should be understood that the present disclosure is not limited to the detailed structures, configurations or implementations described herein; on the contrary, the present disclosure is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.
Claims
1. A method for classifying cancer molecular subtypes, It is characterized in that include: Acquire whole slide images; Segmenting the whole slide image to obtain a plurality of image blocks; Using a pre-trained contrast learning model, feature contrast learning is performed on the image block to generate a first feature vector, a second feature vector, a third feature vector and an anchor vector; the first feature vector, the second feature vector and the anchor vector are feature vectors obtained by performing feature extraction processing on the image block along different view directions, and the third feature vector is a feature vector selected from a preset feature vector set and most similar to the anchor vector; Using a pre-trained classification model, performing a sparse attention calculation operation and a convolution operation on the first feature vector, the second feature vector, and the third feature vector to generate a prediction result; the prediction result includes the position information and key feature information of the image block; According to the prediction result, classifying the whole slide image into cancer molecular subtypes; The contrastive learning model includes a first branch network, a second branch network and a third branch network; The method of using a pre-trained contrast learning model to perform feature contrast learning on the image block to generate a first feature vector, a second feature vector, a third feature vector and an anchor vector includes: Using the first branch network, the second branch network and the third branch network, feature extraction processing is performed on the image block along different view directions to generate the first feature vector, the second feature vector and the anchor vector; the network weight parameter of the second branch network is updated according to the network weight parameter of the first branch network, and the second branch network and the third branch network share the same network weight parameter; Performing similarity calculation on the anchor point vector and a preset vector in the preset feature vector set, selecting a number of feature vectors that are most similar to the anchor point vector from the preset feature vector set, and obtaining a number of the third feature vectors; The first branch network, the second branch network and the third branch network respectively include a data enhancement layer, a plurality of extraction networks and a projection layer, and the extraction network is a Swin Transformer structure; The performing feature extraction processing on the image block comprises: Using the data enhancement layer, upsampling the image block to obtain an enhanced image block; Using each of the extraction networks, performing feature extraction processing on the enhanced image block layer by layer, and outputting a corresponding feature extraction vector in the last feature extraction network; The projection layer is used to perform spatial projection transformation processing on the feature extraction vector to obtain the first feature vector, the second feature vector or the anchor point vector.
2. The method for classifying cancer molecular subtypes according to claim 1, It is characterized in that The whole slide image is segmented to obtain a plurality of image blocks, including: Downsampling the whole slide image to obtain a downsampled image; Segmenting the downsampled image to obtain a plurality of image blocks; The image blocks are screened to eliminate the image blocks lacking tissue or showing minimal tissue.
3. The method for classifying cancer molecular subtypes according to claim 1, It is characterized in that The classification model includes a sparse attention network and a convolutional neural network; The using of the pre-trained classification model to perform a sparse attention calculation operation and a convolution operation on the first feature vector pair and the second feature vector pair to generate a prediction result includes: Using the sparse attention network, a sparse attention calculation operation is performed on the first eigenvector, the second eigenvector and the third eigenvector to generate a first position feature matrix; the first position feature matrix includes global position information between the first eigenvector, the second eigenvector and each of the third eigenvectors; Using the convolutional neural network, performing a layer-by-layer convolution operation on the first eigenvector, the second eigenvector and the third eigenvector to generate a second position feature matrix; the second position feature matrix includes local position information of the first eigenvector, the second eigenvector and each of the third eigenvectors; The first position feature matrix and the second position feature matrix are aggregated to obtain a prediction result.
4. A model training method, It is characterized in that include: Constructing a training set and a validation set; the training set and the validation set are both composed of a plurality of sample image blocks, and the sample image blocks contain characteristic information of at least one cancer molecular subtype; Using the contrast learning model to be trained, performing feature contrast learning on the sample image blocks of the training set to generate a first sample vector, a second sample vector, a third sample vector and a sample anchor vector; the first sample vector, the second sample vector and the sample anchor vector are feature vectors obtained by performing feature extraction processing on the sample image blocks along different view directions, and the third sample vector is a feature vector selected from a preset feature vector set and most similar to the sample anchor vector; Construct a first vector pair using the first sample vector and the second sample vector, and construct a second vector pair using the second sample vector and the third sample vector; Using the classification model to be trained, performing a sparse attention calculation operation and a convolution operation on the first vector pair and the second vector pair to generate a sample prediction result; the sample prediction result includes position information and key feature information of the sample image block; According to the model loss information, the weight parameters of the contrastive learning model to be trained, the weight parameters of the classification model to be trained and the preset feature vector are updated, and when the model loss information meets the end condition, a pre-trained contrastive learning model and a pre-trained classification model are obtained; the model loss information represents the matching degree between the feature information of the sample image block of the verification set and the sample prediction result; The contrastive learning model includes a first branch network, a second branch network and a third branch network; The method of using the contrastive learning model to be trained to perform feature contrastive learning on the sample image blocks of the training set to generate a first sample vector, a second sample vector, a third sample vector and a sample anchor vector includes: Using the first branch network, the second branch network and the third branch network, feature extraction processing is performed on the sample image block along different view directions to generate the first sample vector, the second sample vector and the sample anchor vector; the network weight parameter of the second branch network is updated according to the network weight parameter of the first branch network, and the second branch network and the third branch network share the same network weight parameter; Calculating the similarity between the sample anchor point vector and a preset vector in the preset feature vector set, selecting a number of feature vectors that are most similar to the sample anchor point vector from the preset feature vector set, and obtaining a number of the third sample vectors; The first branch network, the second branch network and the third branch network respectively include a data enhancement layer, a plurality of extraction networks and a projection layer, and the extraction network is a Swin Transformer structure; The performing feature extraction processing on the sample image block comprises: Using the data enhancement layer, upsampling the sample image block to obtain an enhanced image block; Using each of the extraction networks, performing feature extraction processing on the enhanced image block layer by layer, and outputting a corresponding feature extraction vector in the last feature extraction network; The projection layer is used to perform spatial projection transformation processing on the feature extraction vector to obtain the first sample vector, the second sample vector or the sample anchor point vector.
5. The model training method according to claim 4, It is characterized in that The model loss information includes contrastive learning loss information; The expression of the contrastive learning loss information is: Among them, L CTrains To compare the learning loss information, is the contrast loss information of positive feature samples, is the contrast loss information of negative feature samples, z 1 is the first vector pair, is a positive feature sample pair set, which contains the first vector pair and the second vector pair of the current sample image block, z - is a negative feature sample pair set, which includes the first vector pair and the second vector pair of other sample image blocks. is the i-th element in the positive feature sample pair set, i = 1, 2, ..., S, S + 1, S is the number of second vector pairs of the current sample image block, S is the number of elements in the positive feature sample pair set, N is the number of elements in the negative feature sample pair set, and τ is the temperature hyperparameter.
6. The model training method according to claim 4, It is characterized in that The model loss information includes classification loss information; The expression of the classification loss information is: L total =L classification +L sparse , p mn =σ(f q (x m ) T f k (x n )+b), Among them, L total is the total classification loss information, L classification is the classification loss information, L sparse is the sparse loss information, is the Bernoulli probability distribution of the image patch feature connection, The Bernoulli probability of connecting the features of the mth and nth image patches, p(E) is the prior probability distribution of the image block feature connection, p(E mn ) is the prior probability of the connection between the mth and nth image block features, p(E mn )∈p(E),p mn is the pre-probability of the connection between the mth and nth image patch features, f is the convolution operation, σ is the sigmoid operation, U is the sample extracted from the continuous uniform distribution, τ′ is the approximation hyperparameter, and x m is the feature vector of the mth image block, x n is the feature vector of the nth image block, f q is the convolution operation of the qth convolutional layer, f k is the convolution operation of the kth convolutional layer, b is the bias value, E mn Represents the connection strength from the mth image block to the nth image block.
7. An electronic device, It is characterized in that The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 6 when executing the computer program.
8. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
