Pathology image classification method and system combining stable learning and hybrid augmentation

Abstract

This invention belongs to the fields of medical image processing and deep learning technology. It discloses a pathological image classification method and system combining stable learning and hybrid enhancement. The method involves acquiring a pathological image dataset, dividing it into a training set, a validation set, a test set, and an external validation set, and preprocessing the dataset. A deep learning network combining stable learning and hybrid enhancement is constructed and trained using the training set. The optimal deep learning network model is obtained using the validation set. The test set and the external validation set are input into the optimal deep learning network model to output the pathological image classification results. This invention utilizes a well-fitting pathological image classification model to effectively improve the overfitting problem and weak recognition ability of traditional models for domain-biased data, enhancing the recognition accuracy of independent and identically distributed data, improving the robustness and generalization ability of the pathological image classification model, and increasing the diagnostic accuracy of pathological images.

Description

Technical Field

[0001] This invention belongs to the fields of medical image processing and deep learning technology, and particularly relates to a method and system for classifying pathological images that combines stable learning and hybrid enhancement. Background Technology

[0002] Currently, pathological image classification plays a crucial role in improving medical quality, accelerating disease diagnosis and treatment, and promoting medical research. However, with the explosive growth of pathology applications, the global scarcity of pathologists poses a severe challenge to efficient, accurate, and convenient endometrial care. Computer-aided diagnosis offers an automated alternative for pathological images, improving the efficiency of pathologists. It primarily utilizes feature-engineered machine learning methods or end-to-end deep learning methods. However, these methods are typically based on the assumption that training and test data are identical and independently distributed. This assumption is not always valid in reality. When the probability distributions of training and test data differ, i.e., when domain shift occurs, the model's performance often degrades, leading to inaccurate disease diagnosis. Therefore, enhancing the model's generalization ability and robustness, and mitigating the impact of domain-shifted data, is of great significance for disease diagnosis and has become a key focus for researchers. Feature-engineered machine learning performs well on small sample datasets; however, these algorithms heavily rely on hand-crafted features and domain-specific knowledge. End-to-end deep learning methods can automatically handle more complex data and patterns without requiring manual feature extraction. While machine learning methods that utilize feature engineering or end-to-end deep learning methods offer satisfactory performance, they are typically based on the iid assumption.

[0003] Due to variations in slice thickness, storage time, demographic characteristics, and data acquisition methods, pathological images may differ in quality, color, style, and resolution. This can lead to domain shifts, causing the original model to perform poorly in recognizing domain-shifted data caused by image corruption or adversarial noise, resulting in inaccurate disease diagnoses.

[0004] Single-domain generalization methods use only one source domain to learn a model that can recognize common patterns and features distributed across different target domains. Single-domain generalization methods are often used as general solutions to domain offset problems, such as self-challenging representation learning, hybrid augmentation, and deep stable learning networks. Specifically, self-challenging representation learning improves the generalization of convolutional neural networks to domain-offset data by repeatedly eliminating primacy features during training and forcing the network to activate residual features relevant to the label. Hybrid augmentation improves the robustness and uncertainty measure of the model by combining data augmentation operations with consistency loss. Deep stable learning networks improve the generalization ability of deep models by eliminating statistical correlations between relevant and irrelevant features through sample weighting. They have achieved satisfactory performance on computer vision images containing relatively easily distinguishable objects (e.g., houses, people, or animals) with contrasting backgrounds. However, in pathological images, the distinction between different tissues depends on more complex features, ranging from global (gland-to-stromal ratio) to local (cellular structural atypia). When these methods are directly applied to pathological images, the significant differences between pathological and computer vision images can lead to suboptimal performance.

[0005] Based on the above analysis, the problems and shortcomings of existing technologies are as follows: Existing machine learning methods or end-to-end deep learning methods utilizing feature engineering are usually based on the assumption that training and test data are identical and independently distributed, an assumption that does not always hold true in reality; when the probability distributions of training and test data differ, i.e., when domain shift occurs, the model's performance often deteriorates, leading to inaccurate disease diagnosis. Currently, advanced single-domain generalization methods are still in their early stages in pathological image analysis. When existing single-domain generalization methods are directly applied to pathological images, the significant differences between pathological images and computer vision images may limit their application. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides a pathological image classification method and system that combines stable learning and hybrid enhancement.

[0007] This invention is implemented as follows: a pathological image classification method combining stable learning and hybrid enhancement. The method includes: acquiring a pathological image dataset; dividing the pathological image dataset into a training set, a validation set, a test set, and an external validation set; and preprocessing the pathological image dataset; constructing a deep learning network combining stable learning and hybrid enhancement and training it using the training set; obtaining the optimal deep learning network model using the validation set; inputting the test set and the external validation set into the optimal deep learning network model; and outputting the pathological image classification result.

[0008] Furthermore, the pathological image classification method combining stable learning and hybrid enhancement includes the following steps:

[0009] Step 1: Obtain two pathological image datasets containing the same category but prepared at different times. Divide the first dataset into a training set, a validation set, and a test set according to a certain ratio. Use the second dataset as an external validation set and preprocess the datasets.

[0010] Step 2: Construct a stable learning network and a hybrid reinforcement module, and then construct a deep learning network that combines stable learning and hybrid reinforcement.

[0011] Step 3: Train the network using the training set, and validate the network using the validation set during the training process to select the network model.

[0012] Step four: Input the test set and external validation set into the selected network model to obtain the pathological image classification results.

[0013] Furthermore, the preprocessing of the pathological image dataset in step one includes:

[0014] Each image in the pathology image dataset was resized to 224 pixels by 224 pixels to match the input size of the deep learning network.

[0015] Furthermore, step two involves constructing a stable learning network and a hybrid reinforcement module, and building a deep learning network that combines stable learning and hybrid reinforcement, including:

[0016] (1) Establish a stable learning network;

[0017] The stable learning network includes a feature extractor f, a classifier g, and a sample weighting module. The network eliminates the dependencies between features in the representation space through the sample weighting module and measures the general independence between features through random Fourier features. During the training phase, the stable learning network uses the sample weighting module to assign weights to each batch of samples. The sample weighting module calculates the sample weights while saving global information.

[0018] Sample X L The feature map Z is obtained after feature extractor f. L =f(X) L Then the sample weighting module will process the feature map Z... L With the pre-saved global feature matrix Connect, forming the result Z after the connection. O ,but:

[0019]

[0020] From the global weight matrix Learn the sample weights W of the current batch L Where k represents the number of global features and weights pre-stored, and it is the same as the batch size:

[0021]

[0022] In the formula, B is the batch size and w represents the sample weights and ω i This represents the weight of each sample, and the sum of these weights equals the number of sample weights. Representing feature Z O：，i and Z O：，j The corresponding weighted partial cross covariance matrix is ​​obtained by the Random Fourier Features (RFF) mapping function, which measures general independence.

[0023] Then, the features and weights of the current batch are integrated with the previous global features and weights, then:

[0024]

[0025]

[0026] In the formula, for each set of global information α i To smooth the parameters used for global information analysis, when α i When α is large, it is used to analyze long-term memory in global information. i Smaller for analyzing short-term memory in global information; all Replace with This serves as the initialization for the next training batch.

[0027] (2) Construct a hybrid enhancement module;

[0028] Define data augmentation operations, including maximizing image contrast, equalizing the image histogram, changing the value of image pixels, rotation, cropping, and translation; sample k augmentation chains, each consisting of one to three randomly selected augmentation operations.

[0029] The image obtained from the enhancement chain is the result of forming the enhancement chain by using convex combination of elements, where the k-dimensional vector (ω1, ω2, ..., ω) of the convex coefficients is a convex coefficient vector. k If is randomly sampled from the Dirichlet(α, α, ..., α) distribution, then:

[0030]

[0031] In the formula, (ω1,ω2,...,ωk )~Dirichlet(α, α, ..., α), chain is a chain for each augmentation chain pair of input image X org The result of the execution; after image blending through the enhancement chain, the result of the enhancement chain is combined with the original image using a second random convex combination sampled from the Beta(α, α) distribution via skip connections to form the enhanced image X. auggmix ,but:

[0032] X augmix =mX org +(1-m)X aug ;

[0033] In the formula, m represents the convex combination coefficient and m ~ Beta(α, α).

[0034] (3) Construct a deep learning network that combines stable learning and hybrid reinforcement;

[0035] For each batch of samples, the input data (X) L Y L ), X L Two views X are obtained through the hybrid enhancement module. augmix1 and X augmix2 This makes the overall view of the input network consist of three parts: X L X augmix1 and X augmix2 Normalize all input views to (0, 1) using the formula X = X / 255.0; standardize each channel to (-1, 1) using the formula X = X - mean / std, where mean is the mean and std is the standard deviation.

[0036] Sample X L The feature map Z is obtained after feature extractor f. L =f(X) L The sample weights W are calculated by the sample weighting module. L Feature map Z L The output logits are obtained after classifier g. L =g(Z) L And calculate the cross-entropy loss, along with the sample weights W. L Element-wise multiplication yields the first loss for optimizing the feature extractor f and the classifier g. Then:

[0037]

[0038] In the formula, B is the batch size, and L(·,·) returns the cross-entropy loss for each sample in the batch. This represents the sample weight corresponding to sample i in the batch. This represents the label corresponding to sample i in the batch.

[0039] View X augmix1 and X augmix2 The output logits are obtained by passing the feature extractor f and the classifier g respectively. augmix1 =g(f(X) augmix1 )) and logits augmix2 =g(f(X) augmix2 Then calculate the posterior distribution p. L =softmax(logits) L ), p augmix1 =softmax(logits) augmix1 ) and p augmix2 =softmax(logits) augmix2 ); by minimizing the original sample X L And enhanced variant X augmix1 and X augmix2 The Jensen-Shannon consistency loss JS(p) between posterior distributions L ;p augmix1 ;p augmix2 We obtain a second loss for optimizing the feature extractor f and the classifier g, used to keep the model stable across different input ranges. Then:

[0040]

[0041] In the formula, M=(p L +p augmix1 +p augmix2 ) / 3, KL[·] represents the Kullback-Leibler divergence.

[0042] If the feature extractor f and classifier g are updated using the following total loss, then:

[0043]

[0044] In the formula, a, b ∈ [0, 1] and This is the cross-entropy loss function, which returns the average cross-entropy loss of the entire batch of samples.

[0045] Furthermore, step three involves training the network using the training set, validating the network using the validation set during training, and selecting the network model, including:

[0046] (1) Train the network using the training set. Input the training set into the network and calculate the total loss. Then update the parameters of the feature extractor f and the classifier g using the total loss;

[0047] (2) Use the validation set to test and select the network model;

[0048] The validation set undergoes data processing, including normalization and standardization. During validation, the data skips the sample weighting module and directly passes through the feature extractor f and classifier g to predict the output label y. pred ;

[0049]

[0050] In the formula, z is the output of the classifier, and C is the number of categories.

[0051] Predict label y pred The validation set accuracy is calculated by comparing it with the true labels, and an early stopping strategy is used. When the validation set accuracy no longer increases after a certain number of training epochs e, the training stops and the best model on the validation set during the training process is saved.

[0052] Furthermore, in step four, the test set and external validation set are input into the selected network model to obtain the pathological image classification results, including:

[0053] (1) Input the test set and external validation set into the model selected in step three. Data processing includes normalization and standardization. When testing the two datasets, the data skips the sample weighting module and directly passes through the feature extractor f and classifier g to predict the output label.

[0054] (2) Calculate the classification accuracy of the test set and the external validation set.

[0055] Another object of the present invention is to provide a pathological image classification system combining stable learning and hybrid enhancement, which applies the aforementioned method for classifying pathological images using stable learning and hybrid enhancement. The pathological image classification system combining stable learning and hybrid enhancement includes:

[0056] The image data acquisition module is used to acquire two pathological image datasets containing the same category but prepared at different times.

[0057] The data preprocessing module is used to divide the first dataset into training set, validation set and test set according to a certain ratio, use the second dataset as external validation set, and preprocess the dataset.

[0058] The network building module is used to build stable learning networks and hybrid reinforcement modules, and to build deep learning networks that combine stable learning and hybrid reinforcement.

[0059] The network training module is used to train the network using the training set, and to validate the network and select the network model during the training process using the validation set.

[0060] The image classification module is used to input the test set and external validation set into the selected network model to obtain the pathological image classification results.

[0061] Another object of the present invention is to provide a computer device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the steps of the pathological image classification method combining stable learning and hybrid enhancement.

[0062] Another object of the present invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the pathological image classification method combining stable learning and hybrid enhancement.

[0063] Another objective of this invention is to provide an information data processing terminal for implementing the aforementioned pathological image classification system that combines stable learning and hybrid enhancement.

[0064] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0065] First, addressing the technical problems existing in the prior art and the difficulty of solving them, this paper closely analyzes, in conjunction with the technical solution to be protected by this invention and the results and data obtained during the research and development process, how the technical solution of this invention solves the technical problems, and the inventive technical effects brought about by solving these problems. The specific description is as follows:

[0066] This invention employs a deep learning network combining stable learning and hybrid enhancement to classify pathological images. The network uses stable learning to learn the weights of training samples to eliminate dependencies between features and mitigate the impact of distribution variations between training and test data. Hybrid enhancement improves the model's robustness and uncertainty measurement, thereby increasing the accuracy of disease diagnosis. This invention yields a well-fitting pathological image classification model, effectively addressing the weakness of existing deep learning models in generalizing to domain-biased data, improving the recognition accuracy of independent and identically distributed data, and significantly enhancing the accuracy of disease diagnosis.

[0067] Second, considering the technical solution as a whole or from a product perspective, the technical effects and advantages of the technical solution to be protected by this invention are specifically described as follows:

[0068] The pathological image classification method combining stable learning and hybrid enhancement provided by this invention can effectively improve the overfitting problem of traditional machine learning models and deep learning models. It not only improves the recognition accuracy of independent and identically distributed data, but also improves the impact of domain offset, better identifies domain offset data, and improves the robustness and generalization ability of the model as well as the diagnostic accuracy of pathological images.

[0069] Third, as supplementary evidence of the inventive step of the claims of this invention, it is also reflected in the following important aspects:

[0070] (1) The expected benefits and commercial value of the technical solution of this invention after transformation are as follows:

[0071] This invention aims to improve the domain offset problem in pathological images, which will bring many expected benefits and commercial value, improve the accuracy and reliability of pathological diagnosis, enable medical institutions to diagnose and process cases more quickly, reduce patient waiting time, improve institutional efficiency and productivity, and also help medical institutions better manage medical resources, provide better medical services, and promote the innovation and development of medical technology.

[0072] (2) The technical solution of the present invention solves a technical problem that people have long wanted to solve but have never been able to solve successfully:

[0073] This invention improves the problem of domain shift caused by differences between pathological images. Traditional machine learning or deep learning methods cannot effectively improve this problem. This invention enables the original model to more accurately identify domain shift data caused by image corruption or adversarial noise, thereby making disease diagnosis more accurate. Attached Figure Description

[0074] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 This is a flowchart of a pathological image classification method combining stable learning and hybrid enhancement provided in an embodiment of the present invention;

[0076] Figure 2 This is a schematic diagram of the pathological image classification method combining stable learning and hybrid enhancement provided in the embodiments of the present invention;

[0077] Figure 3 This is a schematic diagram of a deep learning network structure combining stable learning and hybrid enhancement provided in an embodiment of the present invention;

[0078] Figure 4 This is a flowchart illustrating the model's identification process for validation set, test set, and external validation set data, as provided in this embodiment of the invention. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0080] To address the problems existing in the prior art, this invention provides a pathological image classification method and system that combines stable learning and hybrid enhancement. The invention will be described in detail below with reference to the accompanying drawings.

[0081] like Figure 1 As shown, the pathological image classification method combining stable learning and hybrid enhancement provided in this embodiment of the invention includes the following steps:

[0082] S101, Obtain the pathological image dataset, divide the pathological image dataset into training set, validation set, test set and external validation set, and preprocess the dataset;

[0083] S102, Construct a deep learning network that combines stable learning and hybrid reinforcement and train it using the training set, and obtain the optimal deep learning network model using the validation set.

[0084] S103 inputs the test set and external validation set into the optimal deep learning network model and outputs the pathological image classification results.

[0085] As a preferred embodiment, such as Figure 2 As shown, the pathological image classification method combining stable learning and hybrid enhancement provided in this embodiment of the invention specifically includes the following steps:

[0086] (1) Obtain two pathological image datasets containing the same category but with different preparation times. Divide the first dataset into a training set, a validation set, and a test set according to a certain ratio. Use the second dataset as an external validation set. Finally, preprocess the datasets to meet the input size of the deep learning network.

[0087] One method for preprocessing the pathology image dataset is to resize each image in the dataset to 224 pixels * 224 pixels to match the input size of the deep learning network.

[0088] (2) Construct a stable learning network and a hybrid reinforcement module, and construct a deep learning network that combines stable learning and hybrid reinforcement, specifically including:

[0089] (2.1) Establish a stable learning network:

[0090] (2.1.1) The stable learning network consists of a feature extractor f, a classifier g, and a sample weighting module. The network uses the sample weighting module to eliminate dependencies between features in the representation space and measures the general independence between features using random Fourier features. During the training phase, the stable learning network uses the sample weighting module to assign weights to samples in each batch.

[0091] (2.1.2) The sample weighting module calculates the sample weights while saving global information, as described below:

[0092] Sample X L The feature map Z is obtained after feature extractor f. L =f(X) L Then the sample weighting module will process the feature map Z... L With the pre-saved global feature matrix Connect, forming the result Z after the connection. O ,but:

[0093]

[0094] From the global weight matrix Learn the sample weights W of the current batch L Where k represents the number of global features and weights pre-stored, and it is the same as the batch size:

[0095]

[0096] In the formula, B is the batch size and w represents the sample weight matrix and ω i This represents the weight of each sample, and the sum of these weights equals the number of sample weights. Representing feature Z O：，i and Z O：，j The corresponding weighted partial cross covariance matrix is ​​obtained by the Random Fourier Features (RFF) mapping function, which measures general independence.

[0097] Then, the features and weights of the current batch are integrated with the previous global features and weights, then:

[0098]

[0099]

[0100] In the formula, for each set of global information αi To smooth the parameters used for global information analysis, when α i When α is large, it is used to analyze long-term memory in global information. i Smaller for analyzing short-term memory in global information; all Replace with This serves as the initialization for the next training batch.

[0101] (2.2) Construct a hybrid enhancement module;

[0102] (2.21) Define data augmentation operations, including maximizing image contrast, equalizing the image histogram, changing the values ​​of image pixels, rotation, cropping, and translation. Then sample k augmentation chains, each consisting of one to three randomly selected augmentation operations.

[0103] (2.22) The images obtained from these enhancement chains are formed by using element-wise convex combination, where the k-dimensional vector of the convex coefficients (ω1, ω2, ..., ω...) k ) is randomly sampled from the Dirichlet(a, α, ..., α) distribution, and is expressed by the following formula:

[0104]

[0105] Among them, (ω1,ω2,...,ω k )~Dirichlet(a, α, ..., α), chain is a chain for each augmentation chain pair of input image X org The result of the execution. After image blending through the enhancement chain, the result of the enhancement chain and the original image are combined using a "skip connection" through a second random convex combination sampled from the Beta(a, α) distribution to form the final enhanced image X. augmix It can be expressed as the following formula:

[0106] X augmix =mX org +(1-m)X aug

[0107] Where m represents the convex combination coefficient and m ~ Beta(α, α).

[0108] (2.3) Construct a deep learning network that combines stable learning and hybrid reinforcement (see...) Figure 3 ).

[0109] (2.31) For the input data (X) of each batch of samples L Y L ), X L First, two views X are obtained through the Hybrid Enhancement module. augmix1 and X augmix2This results in three overall views of the input network, namely X L X augmix1 and X augmix2 Then, all input views are normalized to (0, 1), expressed as X = X / 255.0. Each channel is then standardized to (-1, 1), expressed as X = X - mean / std, where mean is the mean and std is the standard deviation.

[0110] (2.32) Sample X L The feature map Z is obtained after feature extractor f. L =f(X) L The sample weights W are calculated by the sample weighting module. L Meanwhile, feature map Z L The output logits are obtained after classifier g. L =g(Z) L Then calculate the cross-entropy loss, and then combine it with the sample weights W. L The first loss of the optimized feature extractor f and classifier g is obtained by performing element-wise multiplication, and is expressed as the following formula:

[0111]

[0112] Where B is the batch size, and L(·,·) returns the cross-entropy loss for each sample in the batch.

[0113] (2.33) View X augmix1 and X augmix2 The output logits are obtained by passing the feature extractor f and the classifier g respectively. augmix1 =g(f(X) augmix1 )) and logits augmix2 =g(f(X) augmix2 Then calculate the posterior distribution p. L =softmax(logits) L ), p augmix1 =softmax(logits) augmix1 ) and p augmix2 =softmax(logits) augmix2 Finally, by minimizing the original sample machine and its enhanced variant X... augmix1 and X augmix2 The Jensen-Shannon consistency loss JS(p) between posterior distributions L ;p augmix1 ;p augmix2 The second loss, which optimizes the feature extractor f and classifier g, is used to keep the model stable across different input ranges and is expressed as follows:

[0114]

[0115] Where M = (p L +p augmix1 +p augmix2 ) / 3, KL[·] represents the Kullback-Leibler divergence.

[0116] (2.34) To balance the impact of stable learning and hybrid augmentation on network training, this method uses the following total loss to update the feature extractor f and classifier g:

[0117]

[0118] Where a, b ∈ [0, 1] and This is the cross-entropy loss function, which returns the average cross-entropy loss of the entire batch of samples.

[0119] (3) Train the network using the training set, and validate the network using the validation set during the training process and select the network model.

[0120] (3.1) Train the network using the training set. Input the training set into the network and calculate the total loss. Then update the parameters of the feature extractor f and the classifier g using the total loss;

[0121] (3.2) Use the validation set to test and select the network model.

[0122] (3.21) The validation set is processed in the same way as the training set, including normalization and standardization, but without the hybrid augmentation. During validation, the data skips the sample weighting module and directly passes through the feature extractor f and classifier g to predict the output label y. pred It can be expressed as the following formula:

[0123]

[0124] Where z is the output of the classifier, and C is the number of classes.

[0125] (3.22) Predict the label y pred The validation set accuracy is calculated by comparing it with the true labels, and an early stopping strategy is used, that is, when the validation set accuracy no longer increases after a certain number of training epochs e, the training stops and the best model on the validation set during the training process is saved.

[0126] (4) Input the test set and external validation set into the selected network model to obtain the pathological image classification results, specifically including:

[0127] (4.1) Input the test set and external validation set into the model selected in step (3). Data processing includes normalization and standardization. When testing the two datasets, the data skips the sample weighting module and directly passes through the feature extractor f and classifier g to predict the output label.

[0128] (4.2) Calculate the classification accuracy of the test set and the external validation set.

[0129] The identification process of the network model for the validation set, test set, and external validation set data provided in this embodiment of the invention is as follows: Figure 4 As shown.

[0130] The pathological image classification system combining stable learning and hybrid enhancement provided in this invention includes:

[0131] The image data acquisition module is used to acquire two pathological image datasets containing the same category but prepared at different times.

[0132] The data preprocessing module is used to divide the first dataset into training set, validation set and test set according to a certain ratio, use the second dataset as external validation set, and preprocess the dataset.

[0133] The network building module is used to build stable learning networks and hybrid reinforcement modules, and to build deep learning networks that combine stable learning and hybrid reinforcement.

[0134] The network training module is used to train the network using the training set, and to validate the network and select the network model during the training process using the validation set.

[0135] The image classification module is used to input the test set and external validation set into the selected network model to obtain the pathological image classification results.

[0136] To demonstrate the inventiveness and technical value of the technical solution of this invention, this section provides specific product or related technology application examples of the technical solution claimed.

[0137] This invention, using a primary algorithm, is applied to a computer-aided diagnostic system for endometrial cancer. This system accepts two-dimensional histopathological image data and labels as training input. The training data is then fed into a network combining stable learning and hybrid reinforcement to train a classification model, which is stored within the computer-aided diagnostic system for endometrial cancer. The system also accepts two-dimensional histopathological image data as test input. The test data is then fed into the trained deep learning classification model to obtain a predicted output. Finally, the computer-aided diagnostic system for endometrial cancer outputs the predicted category of the image: normal endometrium, endometrial adenocarcinoma, endometrial polyp, or endometrial hyperplasia.

[0138] As a preferred embodiment, such as Figure 2 As shown, the pathological image classification method combining stable learning and hybrid enhancement provided in this embodiment of the invention specifically includes the following steps:

[0139] (1) Obtain two pathological image datasets containing the same category but with different preparation times. Divide the first dataset into a training set, a validation set, and a test set according to a certain ratio. Use the second dataset as an external validation set. Finally, preprocess the datasets to meet the input size of the deep learning network.

[0140] In a preferred embodiment, this invention obtains two publicly available endometrial histopathological image datasets containing the same categories. The two datasets contain the same categories, the second dataset was collected after the first dataset, and the pixel size differs from the first dataset. The first dataset contains 3302 digital image patches (640×480 pixels), including four categories of digital image patches: 1333 normal endometrial (NE) images (21 from the menstrual phase, 600 from the luteal phase, and 712 from the follicular phase), 636 endometrial polyps (EP), 798 hyperplasia images (516 simple hyperplasia and 282 complex hyperplasia), and 535 endometrial adenocarcinoma (EA). The second dataset contains four categories of digital image patches (1280×960 pixels), including 74 normal endometrial images, 12 endometrial polyps, 55 hyperplasia images, and 59 endometrial adenocarcinoma images.

[0141] Then, the first dataset was divided into a training set, a validation set, and a test set in an 8:1:1 ratio, and the second dataset was used as an external validation set. All images were resized to 224 pixels * 224 pixels to meet the input size of the deep learning network.

[0142] (2) Construct a stable learning network and a hybrid reinforcement module, and construct a deep learning network that combines stable learning and hybrid reinforcement;

[0143] (2.1) Establish a stable learning network:

[0144] (2.1.1) The stable learning network consists of a feature extractor f, a classifier g, and a sample weighting module. The network uses the sample weighting module to eliminate dependencies between features in the representation space and measures the general independence between features using random Fourier features. During the training phase, the stable learning network uses the sample weighting module to assign weights to samples in each batch.

[0145] In a preferred embodiment, the present invention uses a ResNet50 network with ImageNet pre-trained weights as the feature extractor f; the classifier g contains three fully connected layers with sizes of 512, 512 and 4, respectively, and uses batch normalization (BN) and ReLU activation functions between the layers.

[0146] (2.1.2) The sample weighting module calculates the sample weights while saving global information, as described below: Sample X L The feature map Z is obtained after feature extractor f. L =f(X) L Then the sample weighting module will process the feature map Z... L With the pre-saved global feature matrix Connect, forming the result Z after the connection. O ,but:

[0147]

[0148] From the global weight matrix Learn the sample weights W of the current batch L Where k represents the number of global features and weights pre-stored, and it is the same as the batch size:

[0149]

[0150] In the formula, F represents the calculation of the partial cross covariance matrix at the feature level, and B is the batch size. w represents the sample weight matrix and ω i This represents the weight of each sample, and the sum of these weights equals the number of sample weights. Representing feature Z O：，i and Z O：，j The corresponding weighted partial cross covariance matrix is ​​obtained by the Random Fourier Features (RFF) mapping function, which measures general independence.

[0151] Then, the features and weights of the current batch are integrated with the previous global features and weights, then:

[0152]

[0153]

[0154] In the formula, for each set of global information α i To smooth the parameters used for global information analysis, when α iWhen α is large, it is used to analyze long-term memory in global information. i Smaller for analyzing short-term memory in global information; all Replace with This serves as the initialization for the next training batch.

[0155] In a preferred embodiment, the batch size B used in this invention is 16, α i The value of is 0.9. Initially, the global feature matrix is ​​16×2048 in size and all values ​​are 0. The global weight matrix is ​​16×1 in size and all values ​​are 1. When learning sample weights, the SGD optimizer is used, the number of training rounds is 20, the learning rate is 1, and the momentum is 0.9.

[0156] (2.2) Construct a hybrid enhancement module;

[0157] (2.21) Define data augmentation operations, including maximizing image contrast, equalizing the image histogram, changing the values ​​of image pixels, rotation, cropping, and translation. Then sample k augmentation chains, each consisting of one to three randomly selected augmentation operations.

[0158] (2.22) The images obtained from these enhancement chains are formed by using element-wise convex combination, where the k-dimensional vector of the convex coefficients (ω1, ω2, ..., ω...) k ) is randomly sampled from the Dirichlet(α, α, ..., α) distribution, and is expressed by the following formula:

[0159]

[0160] Among them, (ω1,ω2,...,ω k )~Dirichlet(α, α, ..., α), chain is a chain for each augmentation chain pair of input image X org The result of the execution. After image blending through the enhancement chain, the result of the enhancement chain and the original image are combined using a "skip connection" through a second random convex combination sampled from the Beta(α, α) distribution to form the final enhanced image X. augmix It can be expressed as the following formula:

[0161] X augmix =mX org +(1-m)X aug

[0162] Where m represents the convex combination coefficient and m ~ Beta(α, α).

[0163] In a preferred embodiment, the present invention uses a value of 3 for k and a value of 1 for α.

[0164] (2.3) Construct a deep learning network that combines stable learning and hybrid reinforcement.

[0165] (2.31) As Figure 3 As shown, for each batch of samples, the input data (X) L Y L ), X L First, two views X are obtained through the Hybrid Enhancement module. augmix1 and X augmix2 This results in three overall views of the input network, namely X L X augmix1 and X augmix2 Then, all input views are normalized to (0, 1), expressed as X = X / 255.0. Each channel is then standardized to (-1, 1), expressed as X = X - mean / std, where mean is the mean and std is the standard deviation.

[0166] (2.32) Sample X L The feature map Z is obtained after feature extractor f. L =f(X) L The sample weights W are calculated by the sample weighting module. L Meanwhile, feature map Z L The output logits are obtained after classifier g. L =g(Z) L Then calculate the cross-entropy loss, and then combine it with the sample weights W. L The first loss of the optimized feature extractor f and classifier g is obtained by performing element-wise multiplication, and is expressed as the following formula:

[0167]

[0168] Where B is the batch size, and L(·,·) returns the cross-entropy loss for each sample in the batch.

[0169] (2.33) View X augmix1 and X augmix2 The output logits are obtained by passing the feature extractor f and the classifier g respectively. augmix1 =g(f(X) augmix1 )) and logits augmix2 =g(f(X) augmix2 Then calculate the posterior distribution p. L =softmax(logits) L ), p augmix1 =softmax(logits) augmix1 ) and p augmix2 =softmax(logits) augmix2Finally, by minimizing the original sample X... L and its enhanced variant X augmix1 and X augmix2 The Jensen-Shannon consistency loss JS(p) between posterior distributions L ;p augmix1 ;p augmix2 The second loss, which optimizes the feature extractor f and classifier g, is used to keep the model stable across different input ranges and is expressed as follows:

[0170]

[0171] Where M = (p L +p augmix1 +p augmix2 ) / 3, KL[·] represents the Kullback-Leibler divergence.

[0172] (2.34) To balance the impact of stable learning and hybrid augmentation on network training, this method uses the following total loss to update the feature extractor f and classifier g:

[0173]

[0174] Where a, b ∈ [0, 1] and This is the cross-entropy loss function, which returns the average cross-entropy loss of the entire batch of samples.

[0175] In a preferred embodiment, the present invention uses a value of 0.5 for mean, 0.5 for std, 0.75 for a, and 0.5 for b.

[0176] (3) Train the network using the training set, and validate the network using the validation set during the training process and select the network model:

[0177] (3.1) Train the network using the training set. Input the training set into the network and calculate the total loss. Then update the parameters of the feature extractor f and the classifier g using the total loss;

[0178] (3.2) Use the validation set to test and select the network model.

[0179] (3.21) The validation set is processed in the same way as the training set, including normalization and standardization, but without the hybrid augmentation. During validation, the data skips the sample weighting module and directly passes through the feature extractor f and classifier g to predict the output label y. pred It is expressed as the following formula.

[0180]

[0181] Where z is the output of the classifier, and C is the number of classes.

[0182] (3.22) Predict the label y pred The validation set accuracy is calculated by comparing it with the true labels, and an early stopping strategy is used, that is, when the validation set accuracy no longer increases after a certain number of training epochs e, the training stops and the best model on the validation set during the training process is saved.

[0183] In a preferred embodiment, the value of C is 4 and the value of e is 30 in this invention.

[0184] (4) Input the test set and external validation set into the selected network model to obtain the pathological image classification results, specifically including:

[0185] (4.1) Input the test set and external validation set into the model selected in step (3). Data processing includes normalization and standardization. When testing the two datasets, the data skips the sample weighting module and directly passes through the feature extractor f and classifier g to predict the output label.

[0186] (4.2) Calculate the classification accuracy of the test set and the external validation set.

[0187] The evaluation criterion for the pathological image classification method provided in this embodiment of the invention is the average accuracy of the test set and the external validation set, to balance the prediction accuracy of the model for independent and identically distributed data and data with domain offset. For comparison with this embodiment, different combinations of values ​​for a and b are used, including (0, 1), (1, 0), and (0, 0). When a, b = (0, 1), this invention is the same as using the Augmix scheme; when a, b = (1, 0), this invention is the same as using the StableNet scheme; when a, b = (0, 0), this invention is the same as using the traditional neural network ResNet50 classification scheme. Furthermore, this invention is also compared with two single-source domain generalization methods, CNSN and RSC. As shown in Table 1, the experimental results show that the traditional network has low recognition accuracy on the external validation set, while the pathological image classification method combining stable learning and hybrid enhancement provided in this embodiment of the invention achieves the best experimental results on both the test set and the external validation set of the publicly available endometrial tissue pathology dataset, indicating that the classification method of this invention has good generalization ability and robustness.

[0188] Table 1 Comparison of experimental results

[0189] method Validation set test set External validation set average value a=1, b=0 (StableNet) 0.791541 0.817073 0.675 0.746036 a = 0, b = 1 (Augmix) 0.779456 0.789634 0.805 0.797317 a=0, b=0 (ResNet50) 0.773414 0.765244 0.615 0.690122 CNSN 0.788520 0.765244 0.845 0.805122 RSC(dropf=1 / 3) 0.782477 0.807927 0.69 0.748963 a = 0.75, b = 0.5 (This invention) 0.800604 0.810976 0.845 0.827988

[0190] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0191] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method of pathological image classification that combines stable learning with hybrid augmentation, the method comprising: The pathological image classification method combining stable learning and hybrid augmentation includes: acquiring a pathological image dataset, dividing the dataset into a training set, a validation set, a test set, and an external validation set, and preprocessing the dataset; constructing a deep learning network combining stable learning and hybrid augmentation and training it using the training set, obtaining the optimal deep learning network model using the validation set; inputting the test set and the external validation set into the optimal deep learning network model, and outputting the pathological image classification results. The pathological image classification method combining stable learning and hybrid enhancement includes the following steps: Step 1: Obtain two pathological image datasets containing the same category but prepared at different times. Divide the first dataset into a training set, a validation set, and a test set according to a certain ratio. Use the second dataset as an external validation set and preprocess the datasets. Step 2: Construct a stable learning network and a hybrid reinforcement module, and then construct a deep learning network that combines stable learning and hybrid reinforcement. Step 3: Train the network using the training set, and validate the network using the validation set during the training process to select the network model. Step four: Input the test set and external validation set into the selected network model to obtain the pathological image classification results; Step two involves constructing a stable learning network and a hybrid reinforcement module, and building a deep learning network that combines stable learning and hybrid reinforcement, including: (1) Establish a stable learning network; Stable learning networks include feature extractors Classifier And a sample weighting module, in which the network eliminates the dependencies between features in the representation space and measures the general independence between features through random Fourier features; during the training phase, the stable learning network uses the sample weighting module to assign weights to samples in each batch, and the sample weighting module saves global information while calculating the sample weights; sample After feature extractor Obtain feature map Subsequently, the sample weighting module will process the feature map. With the pre-saved global feature matrix Connect, forming the result of the connection. ,but: ； From the global weight matrix Learn the sample weights of the current batch ,in This indicates the number of global features and weights to be pre-stored, and it is the same as the batch size. ； In the formula, Batch size and ; Describes the sample weight matrix and , This represents the weight of each sample, and the sum of these weights equals the number of sample weights. Representation of features and The corresponding weighted partial cross covariance matrix is ​​obtained by the Random Fourier Features (RFF) mapping function that measures general independence; Then, the features and weights of the current batch are integrated with the previous global features and weights, then: ； ； In the formula, for each set of global information , To smooth the parameters used for global information analysis, when When the value is large, it is used to analyze long-term memory in global information. Smaller for analyzing short-term memory in global information; all Replace with As initialization for the next training batch; (2) Construct a hybrid enhancement module; Define data augmentation operations, including maximizing image contrast, equalizing the image histogram, changing the values ​​of image pixels, rotation, cropping, and translation; sampling. Each enhancement chain consists of one to three randomly selected enhancement operations; The image obtained from the enhancement chain is the result of forming the enhancement chain using convex combination of elements, where the convex coefficients are... dimensional vector From If the sample is randomly selected from the distribution, then: ； In the formula, , For each augmentation chain, pair the input image The result of the execution; after image blending through the enhancement chain, skip connections are used from... The result of the second random convex combination enhancement chain of distributed sampling and the original image form the enhanced image. ,but: ； In the formula, Denotes the coefficients of a convex combination and ; (3) Construct a deep learning network that combines stable learning and hybrid reinforcement; For the input data of each batch of samples , Two views are obtained through the hybrid enhancement module. and This results in the overall view of the input network comprising three parts: and Normalize all input views to (0, 1), the formula is: Standardize each channel to (-1, 1), as shown in the formula. ,in The mean, Standard deviation; sample After feature extractor Obtain feature map The sample weights are calculated by the sample weighting module. Feature map After classifier Get output And calculate the cross-entropy loss, along with the sample weights. Element-wise multiplication yields an optimized feature extractor. and classifier The first loss is: ； In the formula, The returned value is the cross-entropy loss for each sample in the batch; This represents the sample weight corresponding to sample i in the batch. This represents the label corresponding to sample i in the batch; view and Each passed through a feature extractor and classifier Get output and Then the posterior distribution is calculated. , and By minimizing the original sample and enhanced variants and Jensen-Shannon consistency loss between posterior distributions Optimized feature extractor and classifier The second loss, used to ensure the model remains stable across different input ranges, is as follows: ； In the formula, , Indicates the Kullback-Leibler divergence; Update the feature extractor using the following total loss. and classifier ,but: ； In the formula, and The returned value is the average cross-entropy loss of the entire batch of samples.

2. The pathological image classification method combining stable learning and hybrid enhancement as described in claim 1, characterized in that, Step one, the preprocessing of the pathological image dataset, includes: Each image in the pathology image dataset was resized to 224 pixels by 224 pixels to match the input size of the deep learning network.

3. The pathological image classification method combining stable learning and hybrid enhancement as described in claim 1, characterized in that, Step three involves training the network using the training set, validating the network using the validation set during training, and selecting the network model, including: (1) Train the network using the training set. Input the training set into the network and calculate the total loss. And update the feature extractor using the total loss. and classifier Parameters; (2) Use the validation set to test and select the network model; The validation set undergoes data processing, including normalization and standardization; during validation, the data skips the sample weighting module and passes directly through the feature extractor. and classifier Predicted output Predicted label ; ； In the formula, The output of the classifier, Number of categories; Predict labels The validation set accuracy is calculated by comparing it with the true labels, and an early stopping strategy is used when the validation set accuracy reaches a certain threshold after a certain number of training epochs. When the model stops improving, training stops, and the best model obtained on the validation set during training is saved.

4. The pathological image classification method combining stable learning and hybrid enhancement as described in claim 1, characterized in that, Step four involves inputting the test set and external validation set into the selected network model to obtain the pathological image classification results, including: (1) Input the test set and external validation set into the model selected in step three. Data processing includes normalization and standardization. When testing the two datasets, the data skips the sample weighting module and goes directly through the feature extractor. and classifier Predict the output labels; (2) Calculate the classification accuracy of the test set and the external validation set.

5. A pathological image classification system combining stable learning and hybrid enhancement, applying the pathological image classification method combining stable learning and hybrid enhancement as described in any one of claims 1 to 4, characterized in that, Pathological image classification systems that combine stable learning with hybrid enhancement include: The image data acquisition module is used to acquire two pathological image datasets containing the same category but prepared at different times. The data preprocessing module is used to divide the first dataset into training set, validation set and test set according to a certain ratio, use the second dataset as external validation set, and preprocess the dataset. The network building module is used to build stable learning networks and hybrid reinforcement modules, and to build deep learning networks that combine stable learning and hybrid reinforcement. The network training module is used to train the network using the training set, and to validate the network and select the network model during the training process using the validation set. The image classification module is used to input the test set and external validation set into the selected network model to obtain the pathological image classification results.

6. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the pathological image classification method combining stable learning and hybrid enhancement as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the pathological image classification method combining stable learning and hybrid enhancement as described in any one of claims 1 to 4.

8. An information data processing terminal, characterized in that, The information data processing terminal includes the pathological image classification system that combines stable learning and hybrid enhancement as described in claim 5.

Images