Pathological image semi-supervised automatic classification method based on local augmentation of images
By combining local augmentation techniques and a consistency loss function, the problems of time-consuming annotation and insufficient diversity in pathological image classification are solved, and efficient semi-supervised classification of pathological images is achieved.
Patent Information
- Application Number
- CN202310071136.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-07
- Publication Date
- 2025-12-05
- Estimated Expiration
- 2043-02-07
AI Technical Summary
Existing pathological image classification methods rely on labeled data, and the annotation process is time-consuming and labor-intensive. Common image augmentation methods are insufficient for the diversity of pathological images, and consistency regularization methods are difficult to fully represent the features of pathological images, resulting in poor semi-supervised classification performance.
Local augmentation techniques are employed to augment unlabeled pathological images. Combined with prediction consistency and local feature consistency loss functions, semi-supervised learning is performed through local representation networks and representation aggregation networks to improve the classification accuracy of pathological images.
It effectively increases the augmented diversity of pathological images, enhances the expressive power of deep neural networks for unlabeled pathological image features, improves the accuracy of semi-supervised classification, and avoids the performance degradation caused by mixed backgrounds.
Smart Images
Figure CN116563592B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of pathological image semi-supervised pathological image classification, and particularly relates to a pathological image semi-supervised automatic classification method based on image local augmentation. BACKGROUND
[0002] A pathological image is a gold standard for current most cancer clinical diagnosis, and provides an important basis for formulating a subsequent treatment plan. With the emergence of whole slide scanning technology and convolutional neural networks, pathological image classification work has made great progress and achieved remarkable research results. Existing pathological image classification networks mostly rely heavily on labeled data to obtain satisfactory results. In fact, the acquisition of digital pathological image data labels requires a pathologist with professional knowledge to perform manual annotation, and the annotation process is extremely time-consuming, so that the digital pathological image analysis method based on supervised learning has obvious limitations. In recent years, semi-supervised learning has gradually been valued in the field of pathological image classification, which uses a small amount of labeled data as guidance to achieve more accurate prediction by making full use of the distribution information in unlabeled data. At present, the semi-supervised learning theory based on consistency regularization has achieved great success in semi-supervised learning, and its core idea is to train a deep neural network using labeled data while designing a corresponding consistency regularization loss function to ensure that the neural network has consistent output for the perturbed unlabeled data, thereby effectively improving the feature expression ability of the unlabeled data.
[0003] However, when the existing consistency regularization method is applied to the pathological image classification task, the following problems will be encountered. First, image augmentation is currently the main means of perturbing natural images. However, for a relatively small number of pathological images, common image augmentation methods such as image rotation, flipping and other geometric transformations have limited perturbation effects, making it difficult to ensure the diversity of augmented unlabeled pathological images. Second, the recently proposed data augmentation method based on image fusion, such as CutMix and Mixup, can generate more diverse pathological images, but may have performance degradation problems due to the mixing of non-important image regions such as background. Finally, the current common semi-supervised classification method based on consistency regularization mostly focuses on the consistency of the prediction results when designing the consistency loss function. However, considering the complexity of pathological images, only the consistency constraint on the prediction results will make it difficult for the network to fully represent the pathological images, and thus the semi-supervised classification cannot achieve the optimal effect. SUMMARY
[0004] The present application is to solve the above-mentioned prior art, and proposes a pathological image semi-supervised automatic classification method based on image local augmentation, so as to effectively increase the diversity of augmented images, and use the pointing prediction consistency and pointing local feature consistency loss function to promote the deep neural network to learn more expressive unlabelled pathological image features, thereby effectively improving the semi-supervised classification accuracy of pathological images.
[0005] In order to achieve the above-mentioned application purposes, the present application adopts the following technical solutions:
[0006] The pathological image semi-supervised automatic classification method based on image local augmentation has the following steps:
[0007] Step 1, obtaining L labeled pathological images and their corresponding class labels to form a labeled dataset, and the lth labeled pathological image is denoted as x l , and the class label corresponding to x l is denoted as y l , l=1,2,…,L;
[0008] Obtaining U unlabelled pathological images to form an unlabelled dataset, and the uth unlabelled pathological image is denoted as z u , u=1,2,…,U;
[0009] Step 2, using local augmentation method to augment the data of unlabelled pathological images;
[0010] Step 2.1, copying the uth unlabelled pathological image z u to obtain R same images, wherein the rth image is denoted as r=1,2,…,R;
[0011] Step 2.2, defining a set T containing K t geometric transformation-based data augmentation methods, and calculating the augmentation probability θ t =t / (K t +1) of using any tth geometric transformation-based data augmentation method to augment the local region of the image, t=1,2,…,K t ;
[0012] Step 2.3, using a sliding window with a size of dxd and a stride of s to process the rth image in turn to obtain several local regions of the rth image , and selecting one local region in the image as Wherein, M represents the number of horizontal movement of the sliding window, N represents the number of vertical movement of the sliding window, and M=(W-(d-s))s, N=(H-(d-s))s, W and H represent the length and width of the pathological image respectively, m=1, 2, …, M, n=1, 2, …, N;
[0013] Randomly generate a number between 0 and 1 m*n , when 0≤α m*n ≤θ1, do not augment the local region ; when θ t ≤α m*n ≤θ t+1 , select the tth geometric transformation-based data augmentation method to augment the local region , and obtain the augmented local region , so as to form the rth augmented image from all the augmented local regions , and further obtain the augmented images of all the unlabeled pathological images Wherein, θ1 represents the augmentation probability of using any first geometric transformation-based data augmentation method to augment the local region of the image;
[0014] Step 3, construct a local feature network and a feature aggregation network;
[0015] Step 3.1, the local feature network is stacked by Q1 convolution modules, W1 cascaded Inception modules and L1 fully connected layers in turn, wherein each convolution module is composed of C1 convolution layers and D1 pooling layers in turn; each cascaded Inception module is composed of I1 Inception B modules and D2 pooling layers in turn;
[0016] Set the step length of all convolution layers to c, and set the activation functions of all convolution layers and the first L1-1 fully connected layers to be Relu functions; the activation function of the L1th fully connected layer is a Softmax function;
[0017] Step 3.2, the feature aggregation network is stacked by Q2 convolution modules, W2 cascaded Inception modules and L2 fully connected layers in turn;
[0018] Set the step length of all convolution layers to c, and set the activation functions of all convolution layers and the first L2-1 fully connected layers to be Relu functions; the activation function of the L2th fully connected layer is a Softmax function;
[0019] Step 4, use the labeled pathological images and the cross-entropy loss function to perform supervised learning on the local feature network and the feature aggregation network;
[0020] Step 4.1, the first labeled pathology image x l is divided into MxN labeled pathology image blocks with the same size and respectively represent the ith labeled image block of the first labeled pathology image x l and its corresponding class label, and are the same as y l , i = 1, 2, …, MxN;
[0021] Step 4.2, the labeled pathology image block is input into a local representation network for processing to obtain the classification prediction score of the labeled pathology image block wherein, represents the classification prediction score of the ith labeled image block .
[0022] The features output by the W1 cascaded Inception modules in the local representation network are taken as the high-dimensional features of the labeled pathology image block wherein f i l represents the features output by the Q1 convolution modules and the W1 cascaded Inception modules after the ith labeled image block .
[0023] Step 4.3, the image block classification loss L P_cls is established by using formula (1) to supervise the learning of the local representation network;
[0024]
[0025] In formula (1), H(·) represents the cross-entropy loss;
[0026] Step 4.4, the high-dimensional features of the labeled pathology image block are arranged according to the spatial positions of the respective labeled pathology image blocks in the pathology image x l to obtain the feature matrix F l of the labeled pathology image x l .
[0027] Step 4.5, the feature matrix F l of the labeled pathology image x l is input into a representation aggregation network for processing to generate the prediction score p l of the labeled pathology image x l And use equation (2) to establish the image classification loss L I_cls It is used for supervised learning of local representation networks and local representation networks in combination;
[0028]
[0029] Step 5: Use unlabeled pathological images and the pointing consistency loss function to perform unsupervised learning on the local representation network and the representation aggregation network;
[0030] Step 5.1: Transfer the u-th unlabeled pathological image z u and its rth augmented image Each image is divided into M×N unlabeled pathological image blocks of the same size. and augmented pathological image blocks in, This represents the u-th unlabeled pathological image z. u The i-th unlabeled image patch is divided into Represents the r-th augmented image The i-th augmented image patch is divided into sections;
[0031] Step 5.2, and The features are input into the local representation network for processing, and the output features of W1 cascaded Inception modules in the local representation network are extracted and used accordingly. High-dimensional features and High-dimensional features in, Represents the i-th unlabeled image patch The features output after passing through Q1 convolutional modules and W1 cascaded Inception modules; express The features output after passing through Q1 convolutional modules and W1 cascaded Inception modules;
[0032] Step 5.3, According to their respective unlabeled image patches, in the u-th unlabeled pathological image z u Arrange the images according to their spatial positions to obtain the u-th unlabeled pathological image z. u The characteristic matrix G u ;Will According to their respective augmented image blocks, in the r-th augmented image Arrange the images in spatial order to obtain the r-th augmented image. Feature matrix
[0033] Step 5.4, establish local feature direction consistency loss L using formula (3) DFC , for unsupervised learning of local representation network;
[0034]
[0035] In formula (3), J DPC (·, ·) represents KL loss calculation between high-dimensional images, and has:
[0036]
[0037]
[0038] In formula (4) and (5), I(g) is an indicator function, S DFC represents the feature direction matrix, and has:
[0039]
[0040] In formula (6), E(g) represents entropy calculation;
[0041] Step 5.5, the feature matrix G u and is input into the representation aggregation network for processing, and generates the predicted score e u of the u-th unlabeled pathological image z u and the predicted score of the r-th augmented image and establishes the prediction direction consistency loss L DPC using formula (7), for joint unsupervised learning of local representation network and local representation network;
[0042]
[0043] In formula (7), J DPC (·, ·) represents KL loss calculation between predicted scores of images, and has:
[0044]
[0045]
[0046] In formula (8) and (9), S DPC represents the predicted score direction matrix, and has:
[0047]
[0048] Step 5.6, establish the final direction consistency loss L DCA using formula (11):
[0049] L DCA =L DPC +L DFC (11)
[0050] Step 6, constructing a total loss function, using an alternating iterative optimization method to perform semi-supervised pathological image classification training, and classifying pathological images to obtain corresponding classification results.
[0051] The pathological image semi-supervised automatic classification method based on image local augmentation has the characteristics that the step 6 comprises the following steps:
[0052] Step 6.1, constructing a total target function L using formula (12) total :
[0053] L total =α(L P_cls +L I_cls )+βL DCA (12)
[0054] In formula (12), alpha and beta represent two weight coefficients respectively;
[0055] Step 6.2, setting the total training step number as V, setting the initial learning rate as l1, setting the number of labeled pathological images sampled from the labeled data set as B1 and the number of unlabeled pathological images sampled from the unlabeled data set as B2 each time;
[0056] Step 6.3, after using the sampled labeled pathological images to perform supervised learning on the local representation network and the representation aggregation network for t times, increasing the alternating training of supervised learning and unsupervised learning using the sampled unlabeled pathological images;
[0057] Step 6.4, using the trained local representation network and the representation aggregation network to process the pathological image data set to be predicted to obtain corresponding classification results.
[0058] The electronic device comprises a memory and a processor, and the memory is used to store a program supporting the processor to execute the pathological image semi-supervised automatic classification method, and the processor is configured to execute the program stored in the memory.
[0059] The computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the pathological image semi-supervised automatic classification method.
[0060] Compared with the prior art, the pathological image semi-supervised automatic classification method has the beneficial effects that:
[0061] 1. The present application designs a novel pathological image local augmentation technology, which realizes effective disturbance of unlabelled pathological images by image block sampling on pathological images and combining multiple types of image geometric transformation for local image augmentation. This image local augmentation technology not only avoids the relatively single disturbance result caused by the constraint of common image augmentation methods, but also avoids the problem of decreased pathological image classification accuracy caused by mixing irrelevant tissue regions.
[0062] 2. The present application further proposes two local augmentation consistency loss functions, prediction consistency and local feature consistency, for constraining the consistency relationship of unlabelled images before and after disturbance from the prediction result and image feature level. Meanwhile, the pointing alignment design idea is introduced in the process of constructing the above loss function, that is, the prediction score (or image high-dimensional feature) with lower confidence is aligned to the prediction score (or image high-dimensional feature) with higher confidence, so as to encourage the deep neural network to output unlabelled image features with stronger expression, and realize accurate and efficient pathological image semi-supervised classification. BRIEF DESCRIPTION OF DRAWINGS
[0063] Figure 1 is a flowchart of the present application;
[0064] Figure 2 is a local augmentation result schematic diagram of the present application;
[0065] Figure 3 is a local representation network and representation aggregation network structure diagram of the present application;
[0066] Figure 4 is a pathological image classification technology schematic diagram based on local augmentation and pointing consistency loss of the present application. DETAILED DESCRIPTION
[0067] In this embodiment, a pathological image semi-supervised automatic classification method based on image local augmentation, as shown in Figure 1 , is performed according to the following steps:
[0068] Step 1, obtaining L labeled pathological images and their corresponding class labels to form a labeled dataset, denoted as x l , and the class label corresponding to x l is denoted as y l , l = 1, 2, …, L;
[0069] Obtaining U unlabelled pathological images to form an unlabelled dataset, denoted as z u , u = 1, 2, …, U;
[0070] In this embodiment, 140 and 96 pathological images in Bioimaging2015 and BACH datasets are selected as labeled samples, and 140 and 97 pathological images are selected as unlabeled samples;
[0071] Step 2, using local augmentation method to augment the unlabeled pathological image data;
[0072] Step 2.1, in order to alleviate the problem that the common image augmentation method is difficult to ensure the diversity of the augmented unlabeled pathological image, and the data augmentation method based on image fusion has the problem of performance decline caused by mixing non-important image regions such as background, the application considers that the pathological image is augmented by using the local augmentation method. First, the u-th unlabeled pathological image z u R identical images are copied, where any r-th image is denoted as r=1,2,…,R;in specific implementation, R is 8;
[0073] Step 2.2, a set of K t geometric transformation-based data augmentation methods T is defined, and the augmentation probability θ t of using any t-th geometric transformation-based data augmentation method to augment the local region of the image is calculated t t=1,2,…,K t ; in specific implementation, the data augmentation method set T includes horizontal flip, rotation by 90°, 180° and 270°, and the combination of rotation and flip, that is: T=[Rotate90,Rotate180,Rotate270,LRMirror,LRMirror_Rotate90,LRMirror_Rotate180,LRMirror_Rotate270]; K t is 7;
[0074] Step 2.3, using a sliding window with a size of d×d and a stride of s to process the r-th image in turn, to obtain several local regions of the r-th image , and select one local region in the image as where M represents the number of horizontal movements of the sliding window, N represents the number of vertical movements of the sliding window, and M=(W-(d-s)) / s, N=(H-(d-s)) / s, W and H represent the length and width of the pathological image respectively, m=1,2,…,M, n=1,2,…,N; in specific embodiment, the size of each pathological image is 2048×1536; d and s are both 512;
[0075] a number α between 0 and 1 is randomly generatedm*n When 0≤α m*n When ≤θ1, no local region is affected. To augment; when θ t ≤α m*n ≤θ t+1 When, the t-th geometric transformation-based data augmentation method is selected for the local region. Augmentation is performed to obtain the augmented local region. Thus, the r-th augmented image is composed of all the augmented local regions. Pathological images before and after using local augmentation techniques, such as Figure 2 As shown; thus, augmented images of all unlabeled pathological images are obtained. Where θ1 represents the augmentation probability of local region augmentation of the image using any first geometric transformation-based data augmentation method;
[0076] Step 3: Construct local representation networks and representation aggregation networks;
[0077] Step 3.1, Local representation network such as Figure 3 As shown in Figure (a), it is composed of Q1 convolutional modules, W1 cascaded Inception modules, and L1 fully connected layers stacked sequentially. Each convolutional module consists of C1 convolutional layers and D1 pooling layers. Each cascaded Inception module consists of I1 Inception B modules and D2 pooling layers. In the specific implementation, Q is 3, W is 2, L1, C1, and I1 are 2, and D1 and D2 are 2.
[0078] Set the stride of all convolutional layers to c, and set the activation function of all convolutional layers and the first L1-1 fully connected layers to the ReLU function; the activation function of the L1 fully connected layer is the Softmax function; in the specific implementation, c is set to 3.
[0079] Step 3.2, Characterize the aggregation network as follows Figure 3 As shown in (b), it is composed of Q2 convolutional modules, W2 cascaded Inception modules, and L2 fully connected layers stacked in sequence; in the specific implementation, Q2 and W2 are 2, and L2 is 4.
[0080] Set the stride of all convolutional layers to c, and set the activation function of all convolutional layers and the first L2-1 fully connected layers to ReLU; set the activation function of the L2 fully connected layer to Softmax.
[0081] Step 4: Supervised learning of the local representation network and the representation aggregation network is performed using labeled pathological images and the cross-entropy loss function;
[0082] Step 4.1, the process of supervised learning of the local representation network and the representation aggregation network using the labeled pathological image and the cross-entropy loss function is as follows Figure 4 As shown in the upper half, the ith labeled pathological image x l is divided into MxN labeled pathological image blocks with the same size and respectively represent the ith labeled image block in the labeled pathological image x l and its corresponding class label, and y l is the same as y i , i = 1, 2, …, MxN;
[0083] Step 4.2, the labeled pathological image block x is input into the local representation network for processing to obtain the classification prediction score of the labeled pathological image block x wherein, represents the classification prediction score of the ith labeled image block x .
[0084] The features output by the W1 cascaded Inception modules in the local representation network are taken as the high-dimensional features of the labeled pathological image block x wherein f i l represents the feature output by the Q1 convolution modules and the W1 cascaded Inception modules after the ith labeled image block x .
[0085] Step 4.3, the image block classification loss L P_cls is established by formula (1) to supervise the learning of the local representation network;
[0086]
[0087] In formula (1), H(·) represents the cross-entropy loss;
[0088] Step 4.4, the high-dimensional features f of the labeled pathological image blocks x are arranged according to the spatial positions of the respective labeled pathological image blocks in the pathological image x l to obtain the feature matrix F l of the labeled pathological image x l .
[0089] Step 4.5, the feature matrix F l of the labeled pathological image x lThe input is processed in a representation aggregation network to generate labeled pathological images x. l Predicted score p l And use equation (2) to establish the image classification loss L I_cls It is used for supervised learning of local representation networks and local representation networks in combination;
[0090]
[0091] Step 5: Use unlabeled pathological images and the pointing consistency loss function to perform unsupervised learning on the local representation network and the representation aggregation network;
[0092] Step 5.1: Unsupervised learning of the local representation network and the representation aggregation network is performed using unlabeled pathological images and a pointing consistency loss function. Figure 4 The lower half shows the u-th unlabeled pathological image z. u and its rth augmented image Each image is divided into M×N unlabeled pathological image blocks of the same size. and augmented pathological image blocks in, This represents the u-th unlabeled pathological image z. u The i-th unlabeled image patch is divided into Represents the r-th augmented image The i-th augmented image patch is divided into sections;
[0093] Step 5.2, and The features are input into the local representation network for processing, and the output features of W1 cascaded Inception modules in the local representation network are extracted and used accordingly. High-dimensional features and High-dimensional features in, Represents the i-th unlabeled image patch The features output after passing through Q1 convolutional modules and W1 cascaded Inception modules; express The features output after passing through Q1 convolutional modules and W1 cascaded Inception modules;
[0094] Step 5.3, According to their respective unlabeled image patches, in the u-th unlabeled pathological image z u Arrange the images according to their spatial positions to obtain the u-th unlabeled pathological image z. u The characteristic matrix G u ;Will According to the spatial position of each augmented image block in the rth augmented image , a feature matrix of the rth augmented image is obtained
[0095] Step 5.4, considering that the pathological image itself has certain complexity, the present application proposes to introduce a consistency regularization loss function at the feature level to promote G u and as similar as possible, thereby promoting the expression ability of the local representation network to the pathological image block, and further improving the classification performance of the semi-supervised pathological image. In addition, the present application also considers that when using the existing consistency loss function to train the local representation network, the confidence between the original image and the augmented image prediction feature is ignored, which may cause the local feature of the pathological image with high confidence to deviate to the local feature with low confidence, resulting in the problem of classification accuracy decline. Therefore, the present application proposes to use formula (3) to establish the local feature pointing consistency loss L DFC , so that the local feature with low confidence is aligned to the local feature with high confidence, thereby promoting the local representation network to fully learn the unlabeled data;
[0096]
[0097] In formula (3), J DPC (·, ·) represents the KL loss calculation of the high-dimensional interval of the image, and has:
[0098]
[0099]
[0100] In formula (4) and (5), I(g) is an indicator function, S DFC represents a feature pointing matrix, and has:
[0101]
[0102] In formula (6), E(g) represents entropy calculation;
[0103] Step 5.5, input the feature matrix G u and into the representation aggregation network for processing, and generate the prediction score e u of the uth unlabeled pathological image z u and the prediction score of the rth augmented image The present application further uses formula (7) to establish the prediction pointing consistency loss L DPCThe prediction scores with lower confidence are aligned with the prediction scores with higher confidence, so as to improve the local representation network and the prediction accuracy of the local representation network on the unlabeled data.
[0104]
[0105] In formula (7), J DPC (·,·) represents KL loss calculation between prediction scores of images, and has:
[0106]
[0107]
[0108] In formula (8) and (9), S DPC represents a prediction score orientation matrix, and has:
[0109]
[0110] Step 5.6, establish the final orientation consistency loss L DCA using formula (11):
[0111] L DCA = L DPC + L DFC (11)
[0112] Step 6, construct a total loss function, use an alternating iterative optimization method for semi-supervised pathological image classification training, and classify the pathological images to obtain the corresponding classification results.
[0113] Step 6.1, construct the total target function L total using formula (12):
[0114] L total = α(L P_cls + L I_cls )+βL DCA (12)
[0115] In formula (13), α and β represent two weight coefficients respectively; in the specific implementation, α and β are both 1.0;
[0116] Step 6.2, set the total number of training steps as V; set the initial learning rate as l1; set the number of labeled pathological images sampled from the labeled data set each time as B1, and the number of unlabeled pathological images sampled from the unlabeled data set as B2; in the specific implementation, V is 100; l1 is 10 -3 ; B1 and B2 are both 4;
[0117] Step 6.3, after v times of supervised learning of the local feature network and the feature aggregation network using the sampled labeled pathological images, alternating training of supervised learning and unsupervised learning using sampled unlabeled pathological images is added; in specific implementation, v is all 10;
[0118] Step 6.4, the trained local feature network and the feature aggregation network are used to process the pathological image data set to be predicted, so as to obtain the corresponding classification result.
[0119] In this embodiment, an electronic device includes a memory and a processor, the memory is used to store a program supporting the processor to execute the pathological image semi-supervised automatic classification method described above, and the processor is configured to execute the program stored in the memory.
[0120] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to perform the steps of the pathological image semi-supervised automatic classification method described above.
[0121] In order to quantitatively evaluate the performance of the proposed method, this embodiment shows the performance comparison results of the proposed method and the supervised pathological image classification methods proposed by Araújo, Kamyar, Alexander, Wang and Shaban, and the semi-supervised cell nucleus detection methods such as Π-model, Mean Teacher, Self-training and Semi-HIC on Bioimaging2015 and BACH data sets, as shown in Table 1.
[0122] Table 1 Comparison of classification error rate (%) with other methods
[0123]
[0124] In the pathological image classification of the present application, the local augmentation technology is used to augment the unlabeled data, so that the augmented images can ensure diversity while avoiding the introduction of irrelevant tissue areas in other images to cause performance degradation problems; on this basis, the present application further utilizes the consistency loss function to prompt the neural network to learn more expressive features from the pathological images, effectively improving the accuracy of the semi-supervised classification of pathological images.
Claims
1. A semi-supervised automatic classification method for pathological images based on local image augmentation, characterized in that, Includes the following steps: Step 1: Obtain L labeled pathological images and their corresponding category labels to form a labeled dataset. Let the l-th labeled pathological image be denoted as x. l And x l The corresponding category label is denoted as y. l l = 1, 2, ..., L; We obtain U unlabeled pathological images to form an unlabeled dataset, and let the u-th unlabeled pathological image be denoted as z. u u = 1, 2, ..., U; Step 2: Use local augmentation to augment the data of unlabeled pathological images; Step 2.1: Transfer the u-th unlabeled pathological image z u R identical images are copied, where any r-th image is denoted as . Step 2.2: Define a set containing K t Given a set T of geometric transformation-based data augmentation methods, calculate the augmentation probability θ of using any t-th geometric transformation-based data augmentation method to augment a local region of an image. t =t / (K) t +1), t=1,2,…,K t ; Step 2.3: Use a sliding window of size d×d and stride s to view the r-th image. The sliding window process is performed sequentially to obtain the r-th image. Select several local regions of the image A local region is denoted as Where M represents the number of horizontal movements of the sliding window, N represents the number of vertical movements of the sliding window, and M = (W - (ds)) / s, N = (H - (ds)) / s, W and H represent the length and width of the pathological image, respectively, m = 1, 2, ..., M, n = 1, 2, ..., N; Randomly generate a number α between 0 and 1. m*n When 0≤α m*n When ≤θ1, no local region is affected. To augment; when θ t ≤α m*n ≤θ t+1 When, the t-th geometric transformation-based data augmentation method is selected for the local region. Augmentation is performed to obtain the augmented local region. Thus, the r-th augmented image is composed of all the augmented local regions. This leads to augmented images of all unlabeled pathological images. Where θ1 represents the augmentation probability of local region augmentation of the image using any first geometric transformation-based data augmentation method; Step 3: Construct local representation networks and representation aggregation networks; Step 3.1: The local representation network is composed of Q1 convolutional modules, W1 cascaded Inception modules, and L1 fully connected layers stacked sequentially. Each convolutional module consists of C1 convolutional layers and D1 pooling layers sequentially; each cascaded Inception module consists of I1 InceptionB modules and D2 pooling layers sequentially. Set the stride of all convolutional layers to c, and set the activation function of all convolutional layers and the first L1-1 fully connected layers to the ReLU function; the activation function of the L1 fully connected layer is the Softmax function. Step 3.2: The representation aggregation network is composed of Q2 convolutional modules, W2 cascaded Inception modules, and L2 fully connected layers stacked sequentially. Set the stride of all convolutional layers to c, and set the activation function of all convolutional layers and the first L2-1 fully connected layers to ReLU; set the activation function of the L2 fully connected layer to Softmax. Step 4: Supervised learning of the local representation network and the representation aggregation network is performed using labeled pathological images and the cross-entropy loss function; Step 4.1: Transfer the first labeled pathological image x l Divide into M×N labeled pathological image blocks of the same size. and Let x represent the l-th labeled pathological image. l The i-th labeled image patch and its corresponding category label, and With y l Same, i = 1, 2, ..., M × N; Step 4.2: Transfer the labeled pathological image blocks The input is processed in a local representation network to obtain labeled pathological image patches. Classification prediction score in, This represents the i-th labeled image patch. Classification prediction score; The features output by W1 cascaded Inception modules in the local representation network are used as labeled pathological image blocks. High-dimensional features Among them, f i l This represents the i-th labeled image patch. The features output after passing through Q1 convolutional modules and W1 cascaded Inception modules; Step 4.3: Use equation (1) to establish the image patch classification loss L. P_cls It is used for supervised learning of local representation networks; In equation (1), H(·) represents the cross-entropy loss; Step 4.4: Transfer the labeled pathological image blocks High-dimensional features According to their respective labeled pathological image blocks in pathological image x l The spatial positions in the image are arranged to obtain labeled pathological images x. l The characteristic matrix F l ; Step 4.5: Transfer the labeled pathological image x l The characteristic matrix F l The input is processed in a characterization aggregation network to generate the labeled pathological image x. l Predicted score p l And use equation (2) to establish the image classification loss L I_cls It is used for supervised learning of local representation networks and local representation networks in combination; Step 5: Use unlabeled pathological images and the pointing consistency loss function to perform unsupervised learning on the local representation network and the representation aggregation network; Step 5.1: Transfer the u-th unlabeled pathological image z u and its rth augmented image Each image is divided into M×N unlabeled pathological image blocks of the same size. and augmented pathological image blocks in, This represents the u-th unlabeled pathological image z. u The i-th unlabeled image patch is divided into Represents the r-th augmented image The i-th augmented image patch is divided into sections; Step 5.2, and The features are input into the local representation network for processing, and the output features of W1 cascaded Inception modules in the local representation network are extracted and used accordingly. High-dimensional features and High-dimensional features in, Represents the i-th unlabeled image patch The features output after passing through Q1 convolutional modules and W1 cascaded Inception modules; express The features output after passing through Q1 convolutional modules and W1 cascaded Inception modules; Step 5.3, According to their respective unlabeled image patches, in the u-th unlabeled pathological image z u Arrange the images according to their spatial positions to obtain the u-th unlabeled pathological image z. u The characteristic matrix G u ;Will According to their respective augmented image blocks, in the r-th augmented image Arrange the images in spatial order to obtain the r-th augmented image. Feature matrix Step 5.4: Use equation (3) to establish the local feature orientation consistency loss L DFC It is used for unsupervised learning of local representation networks; In equation (3), J DFC (·,·) represents the calculation of KL loss over the high dimensions of the image, and we have: In equations (4) and (5), I(g) is the indicator function, and S DFC Let represent the eigendirection matrix, and we have: In equation (6), E(g) represents the entropy calculation; Step 5.5: Transfer the feature matrix G u and The input is processed in a characterization aggregation network to generate the u-th unlabeled pathological image z. u Predicted score e u and the rth augmented image Predicted scores And use equation (7) to establish the prediction-oriented consistency loss L DPC It is used to perform unsupervised learning on local representation networks and local representation networks jointly; In equation (7), J DPC (·,·) represents the calculation of KL loss between the predicted scores of the image, and we have: In equations (8) and (9), S DPC Let represent the predictive scoring orientation matrix, and we have: Step 5.6: Use equation (11) to establish the eventual consistency loss L. DCA : L DCA =L DPC +L DFC (11) Step 6: Construct the overall loss function, use alternating iterative optimization to train semi-supervised pathological image classification, and classify the pathological images to obtain the corresponding classification results.
2. The semi-supervised automatic classification method for pathological images based on local image augmentation according to claim 1, characterized in that, Step 6 includes the following steps: Step 6.1: Construct the overall objective function L using equation (12). total : L total =α(L P_cls +L I_cls )+βL DCA (12) In equation (12), α and β represent two weighting coefficients, respectively; Step 6.2: Set the total number of training steps to V; The initial learning rate is set to l1; the number of labeled pathological images sampled from the labeled dataset in each training session is set to B1, and the number of unlabeled pathological images sampled from the unlabeled dataset is set to B2. Step 6.3: After performing t supervised learning on the local representation network and the representation aggregation network using sampled labeled pathological images, alternate between supervised and unsupervised training using sampled unlabeled pathological images. Step 6.4: Use the trained local representation network and representation aggregation network to process the pathological image dataset to be predicted, thereby obtaining the corresponding classification results.
3. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the semi-supervised automatic classification method for pathological images according to claim 1 or 2, and the processor is configured to execute the program stored in the memory.
4. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program, when run by a processor, executes the steps of the semi-supervised automatic classification method for pathological images as described in claim 1 or 2.