WSI image organization classification method based on domain adaptive learning technology

Abstract

The invention relates to a WSI image tissue classification method based on a domain adaptive learning technology. Compared with the prior art, the defect that the WSI image tissue classification effect is poor due to pathological sample process differences is overcome. The method comprises the following steps: acquiring a WSI image and preprocessing the WSI image; constructing a WSI image organization classification model; training a WSI image organization classification model; carrying out benefit reduction correction on the WSI image tissue classification model; obtaining a WSI image to be classified; and obtaining a WSI image organization classification result. The tissue microstructure and texture information are extracted by using the convolutional neural network, and different tissue space structure combinations and microenvironment changes are learned from a multi-resolution angle by using a pyramid structure, so that accurate tissue classification and recognition are performed on the WSI image, the tissue pathology image analysis efficiency is improved, and meanwhile, the tissue pathology analysis accuracy is improved. On the basis of a domain adaptive method, reduction and benefit correction is carried out on network performance degradation caused by cross-data-set application, and accurate classification of WSI image organizations is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a WSI image tissue classification method based on domain adaptive learning technology. Background Art

[0002] Pathological diagnosis refers to the accurate diagnosis and classification of diseases through pathological examinations, including tissue biopsy, cytological examination, etc. Pathological diagnosis can provide clinicians with accurate disease information, which helps to formulate treatment plans and evaluate prognosis. Traditional pathological diagnosis requires doctors to manually observe tissue sections through a microscope, which is easily affected by subjective factors and human errors. Digital pathology is a discipline in the medical field that uses digital image processing technology to diagnose and study tissue sections.

[0003] Among them, WSI (Whole Slide Imaging) technology is an important technology in digital pathology. It generates a large amount of digital image data by scanning the entire tissue slice at high resolution. Combined with image processing and machine learning algorithms, it can achieve high-precision classification and diagnosis of tissue slice images, reduce human errors, speed up diagnosis, and support remote consultation and medical research. The application of this technology not only improves the accuracy and efficiency of medical diagnosis, but also promotes the development of medical science and the digital management of pathology archives, providing doctors and researchers with rich information to diagnose diseases and conduct scientific research, which has a positive impact on medical diagnosis, disease research and clinical practice.

[0004] With the development of deep learning, neural networks have been widely used in WSI tissue classification. By automatically learning the abstract feature representation in the image and learning the rich subtle structure and texture information of different human tissues, accurate identification of different types of tissues can be achieved.

[0005] The application of pyramid structure in WSI (Whole Slide Imaging) image classification is of great significance. This structure processes images in layers, including low-resolution and high-resolution layers. By processing layer by layer, image features can be effectively extracted, and rapid analysis and classification of large-size WSI images can be achieved. The use of pyramid structure can overcome the problems of large WSI image size and complex information, and improve the accuracy and robustness of classification. However, there are also some challenges in the application of pyramid structure, including the need to consume more computing resources and time, high requirements for algorithm design and optimization, and may also cause some information loss or error accumulation. Therefore, when using the pyramid structure for WSI image classification, it is necessary to comprehensively consider its advantages and limitations, and select appropriate processing methods and technical strategies based on actual conditions to improve the effect and application value of image classification.

[0006] In actual applications, it is found that due to differences in scanners, staining methods, and slide preparation techniques in different laboratories, the image performance of the same tissue in different datasets may be different. This difference leads to a decrease in the performance of the model when applied across datasets, limiting its breadth and versatility in actual application.

[0007] Therefore, how to design a method that can overcome sample differences and improve the WSI image tissue classification effect has become a technical problem that needs to be solved urgently. Summary of the invention

[0008] The purpose of the present invention is to solve the defect in the prior art that the WSI image tissue classification effect is poor due to the process differences of pathological samples, and to provide a WSI image tissue classification method based on domain adaptive learning technology to solve the above problem.

[0009] In order to achieve the above object, the technical solution of the present invention is as follows:

[0010] A WSI image organization classification method based on domain adaptive learning technology includes the following steps:

[0011] Acquiring WSI images and preprocessing: Acquiring WSI images and preprocessing them to generate WSI tissue classification source domain images;

[0012] Constructing a WSI image organization classification model: Constructing a WSI image organization classification model based on domain adaptive learning technology;

[0013] Training of WSI image tissue classification model: Using WSI tissue classification source domain images to train the WSI image tissue classification model;

[0014] Loss correction of WSI image tissue classification model: Using unlabeled target domain images to perform loss correction on the WSI image tissue classification model based on domain adaptation method;

[0015] Acquisition of the WSI image to be classified: acquiring the WSI image to be classified and performing preprocessing to generate the WSI image to be classified;

[0016] Obtaining WSI image tissue classification results: Input the WSI image to be classified into the trained WSI image tissue classification model to obtain the WSI image tissue classification results.

[0017] The WSI image acquisition and preprocessing comprises the following steps:

[0018] Acquire the WSI image, scale the annotated WSI tissue classification label image, and scale the label image size from [W, H] to [W / (R / R1), H / (R / R1)], where R is the original image acquisition ratio, W is the original image width, H is the original image height, and R1 is the first target magnification;

[0019] Use the square search box to traverse the WSI organization classification label images;

[0020] Calculate the proportion of the labeled tissue in the search box. If the proportion of the tissue is greater than r1, where r1 is the area occupied by the tissue / the area of ​​the search box, then capture the tissue image block in the WSI image.

[0021] Use the background detection algorithm to detect the background of the captured tissue image block and calculate the background ratio; if the obtained background ratio is less than r2, where r2 is the area occupied by the background / the area of ​​the search box, then save the image block and its associated label information as image I1 at R1 magnification;

[0022] Taking the center coordinate point of the R1x image as the center, intercept the images of size 256×256 at the second target magnification R2 and the third target magnification R3 as the image I2 at the R2 magnification and the image I3 at the R3 magnification;

[0023] The image I1 at R1 magnification, the image I2 at R2 magnification, and the image I3 at R3 magnification are used as labeled source domain images.

[0024] The construction of the WSI image organization classification model includes the following steps:

[0025] Based on the domain adaptive learning technology, the WSI image tissue classification model is set to include a generator G, a classifier C and a discriminator D. The generator G includes a pyramid feature extraction network and a dynamic feature fusion network. The output network is used as the classifier C. The classifier C and the additional nuclear norm operator are used as the discriminator D. The nuclear norm operator uses the nuclear norm first-order Wasserstein distance W1(μ,v), which is expressed as follows:

[0026]

[0027] Where Γ(μ,v) is the set of all possible probability measures of the two distributions μ and v, x and y are samples drawn from μ and v respectively, |xy| is the distance between sample x and sample y, γ is the joint distribution, and inf is the infimum;

[0028] The pyramid feature extraction network is set to include feature encoders F1, F2, and F3 of three scales with the same structure, wherein F1 is a feature encoder of R1 magnification, F2 is a feature encoder of R2 magnification, and F3 is a feature encoder of R3 magnification, and its inputs are respectively image I1 at R1 magnification, image I2 at R2 magnification, and image I3 at R3 magnification;

[0029] The feature encoder is set to use the ResNeXt partial network. The feature encoder changes the output dimension of the last fully connected layer from the default 1000 to 768, and changes the last activation function from the default Softmax to GELU. The network structure of the feature encoding is as follows:

[0030] Feature encoder input layer: The input is a feature matrix with a scale of (3, 256, 256);

[0031] RN1-ResNet pre-order network: ResNet pre-order network RN1 extracts the input feature matrix to obtain a feature vector with a scale of (2048,);

[0032] FC2-Fully connected layer: The fully connected layer FC2 passes the output feature vector of RN1 through the fully connected layer to obtain a feature vector with a scale of (768,);

[0033] A3-Activation layer: The activation layer A3 obtains a feature vector with a scale of (768,) by using the nonlinear transformation GELU to transform the feature vector output by FC2;

[0034] Feature encoder output layer: Output three feature vectors with a scale of (768,) obtained by activation layer A3, which are the first feature matrix, the second feature matrix and the third feature matrix;

[0035] Set up a dynamic feature fusion network, which includes a dynamic feature fusion network input layer, Concat1-concatenation layer, MSA2-multi-head self-attention module, R3-residual layer, FC4-fully connected layer, A5-activation layer, FC6-fully connected layer, R7-residual layer and dynamic feature fusion network output layer;

[0036] Among them, the input of the dynamic feature fusion network input layer is the feature vectors of three scales (768,) output by the pyramid feature extraction network;

[0037] Concat1-The concatenation layer Concat1 adds one dimension to the three feature vectors of the input layer in the first dimension, turning them into (1,768), and obtains a feature matrix with a scale of (3,768) through the concatenation operation in the first dimension;

[0038] MSA2 - Multi-head self-attention module The multi-head self-attention module MSA2 uses the matrix output by Concat1 through multi-head self-attention to obtain a feature matrix with a scale of (3,768);

[0039] R3 - residual layer The residual layer R3 adds the feature matrix output by MSA2 and the feature matrix output by Concat1 to obtain a feature matrix with a scale of (3,768);

[0040] FC4-Fully connected layer The fully connected layer FC4 passes the output feature matrix of R3 through the fully connected layer to obtain a feature matrix with a scale of (3,768);

[0041] A5-Activation layer A5 converts the output feature matrix of FC4 through nonlinear transformation GELU to obtain a feature matrix with a scale of (3,768);

[0042] FC6-Fully connected layer The fully connected layer FC6 passes the output feature matrix of A5 through the fully connected layer to obtain a feature matrix with a scale of (3,768);

[0043] R7-Residual layer R7 adds the output feature matrix of FC6 and the output feature matrix of residual layer R3 to obtain a feature matrix with a scale of (3,768);

[0044] The output of the dynamic feature fusion network output layer is the feature matrix obtained by the residual layer R7, that is, the first eigenvector;

[0045] Set the output network, which includes the output network input layer, FC1-fully connected layer, A2-activation layer, FC3-fully connected layer, A4-activation layer, FC5-fully connected layer and output network output layer;

[0046] Output network input layer: The input is the feature matrix of the scale (3,768) output by the dynamic feature fusion network;

[0047] FC1-Fully connected layer The fully connected layer FC1 changes the feature matrix scale of the output network input layer to (2304,) and obtains the feature vector with a scale of (768,) through the fully connected layer;

[0048] A2-Activation layer The activation layer A2 transforms the feature vector obtained by the fully connected layer FC1 through the nonlinear transformation LeakyReLU to obtain a feature vector with a scale of (2304,), where the LeakyReLU leakage rate is 0.2;

[0049] FC3-Fully connected layer The fully connected layer FC3 passes the feature vector obtained by the A2-activation layer through the fully connected layer to obtain a feature vector with a scale of (768,);

[0050] A4-Activation layer The activation layer A4 transforms the feature vector obtained by FC3-Fully Connected Layer through nonlinear transformation LeakyReLU to obtain a feature vector with a scale of (2304,), where the LeakyReLU leakage rate is 0.2;

[0051] FC5 - fully connected layer The fully connected layer FC5 passes the feature vector obtained by A4 through the fully connected layer to obtain a feature vector of scale (num,), where num is the target category of WSI tissue classification;

[0052] Output network output layer: Output the feature vector obtained by FC5, that is, the prediction matrix P.

[0053] The training of the WSI image tissue classification model includes the following steps:

[0054] Obtain source domain images including image I1 at R1 magnification, image I2 at R2 magnification, and image I3 at R3 magnification and corresponding tissue category labels Y, and convert the three magnification images I 1、 I 2、 I3 input WSI image tissue classification model;

[0055] The source domain images of image I1 at R1 magnification, image I2 at R2 magnification, and image I3 at R3 magnification are respectively input into the pyramid feature extraction network, and the first feature matrix, the second feature matrix, and the third feature matrix are respectively obtained by feature encoders F1, F2, and F3;

[0056] Inputting the first feature matrix, the second feature matrix and the third feature matrix into a dynamic feature fusion network to obtain a first feature vector;

[0057] Input the first eigenvector into the output network to obtain the prediction matrix P;

[0058] Construct the loss function based on P and Y. The loss function is the cross entropy L, and its expression is:

[0059] L = -Y·log(P);

[0060] According to the loss function, all parameters of the WSI image organization classification model are updated through the back-propagation mechanism, and the model parameter weights with the best indicators are saved.

[0061] The impairment correction of the WSI image tissue classification model includes the following steps:

[0062] For the unlabeled target domain dataset {X t}Preprocessing is performed to obtain unlabeled target domain images at R1, R2, and R3 magnifications

[0063] Label the organization as Y s The source domain image and Input them into the generator G to obtain the third eigenvector and the fourth eigenvector respectively;

[0064] Input the third feature vector into the classifier C to obtain a first prediction vector;

[0065] Using the first prediction vector and Y s Construct the first loss function L cls ,

[0066] Use cross entropy loss L ce Construct the first loss function L cls , which is expressed as follows:

[0067]

[0068] in, is the source domain image set The i-th group of images at R1, R2, and R3 magnifications, is the tissue classification label corresponding to the image, G is the generator, and C is the classifier;

[0069] Input the third eigenvector and the fourth eigenvector into the discriminator D to obtain the fifth eigenvector and the sixth eigenvector;

[0070] Use the fifth and sixth eigenvectors to construct the second loss function L nwd ;

[0071] The second loss function L is constructed using the nuclear norm first-order Wasserstein distance nwd , which is expressed as follows:

[0072]

[0073] Among them, N s and N t are the number of samples of the source domain dataset and the target domain dataset respectively;

[0074] Use the first loss function L cls and the second loss function L nwd Construct the total loss function L total ;

[0075] According to the total loss function L total All parameters of the WSI image tissue classification model are updated through the back-propagation mechanism. After the update is completed, the generator G and classifier C after the loss correction are generated and applied to the tissue classification of the WSI image to be classified.

[0076] Beneficial Effects

[0077] Compared with the prior art, the WSI image tissue classification method based on domain adaptive learning technology of the present invention uses a convolutional neural network to extract tissue fine structure and texture information, and uses a pyramid structure to learn different tissue spatial structure combinations and microenvironment changes from a multi-resolution perspective, so as to accurately classify and identify WSI images and improve the efficiency of tissue pathology image analysis. At the same time, based on the domain adaptive method, the network performance attenuation caused by cross-dataset application is corrected, thereby achieving accurate classification of WSI image tissues.

[0078] The present invention applies domain adaptation technology to WSI tissue classification, and uses the knowledge learned in the source domain (training data) by domain adaptation to improve the performance of the model in the target domain (test data); through domain adaptation, the model can recognize and adapt to the differences between different data distributions, thereby maintaining stable classification performance on WSI data sets generated by different laboratories or equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 is a method sequence diagram of the present invention;

[0080] Figure 2 The flowchart of making WSI organization classification pyramid data set of the present invention;

[0081] Figure 3 Schematic diagram of the WSI image organization classification model involved in the present invention;

[0082] Figure 4 It is a network structure diagram of a feature encoder of the present invention;

[0083] Figure 5 It is a dynamic feature fusion network structure diagram of the present invention;

[0084] Figure 6 is the output network structure diagram of the present invention;

[0085] Figure 7 The WSI organization prediction flow chart of the present invention;

[0086] FIG8 is a visualization example diagram of the present invention;

[0087] Fig. 9 This is a diagram showing the evaluation indicators of the present invention in the five-classification task of pancreatic ductal adenocarcinoma WSI tissue;

[0088] Fig.10 This is a flow chart of the model loss correction of the present invention. DETAILED DESCRIPTION

[0089] In order to have a further understanding and recognition of the structural features and the effects achieved by the present invention, a preferred embodiment and accompanying drawings are used for detailed description as follows:

[0090] like Figure 1 As shown, the WSI image organization classification method based on domain adaptive learning technology described in the present invention includes the following steps:

[0091] The first step is to obtain WSI images and preprocess them: Figure 2 As shown, a WSI image is obtained and preprocessed to generate a WSI tissue classification source domain image.

[0092] (1) Acquire a WSI image and scale the annotated WSI tissue classification label image. The label image size is scaled from [W, H] to [W / (R / R1), H / (R / R1)], where R is the original image acquisition ratio, W is the original image width, H is the original image height, and R1 is the first target magnification.

[0093] (2) Use the square search box to traverse the WSI organization classification label images.

[0094] (3) Calculate the proportion of the labeled tissue in the search box. If the proportion of the tissue is greater than r1, where r1 is the area occupied by the tissue / the area of ​​the search box, then capture the tissue image block in the WSI image.

[0095] (4) Use a background detection algorithm to detect the background of the captured tissue image block and calculate the background ratio; if the obtained background ratio is less than r2, where r2 is the area occupied by the background / the area of ​​the search box, then save the image block and its associated label information as image I1 at R1 magnification.

[0096] (5) Taking the central coordinate point of the R1x image as the center, images of size 256×256 at the second target magnification R2 and the third target magnification R3 are intercepted as the image I2 at the R2 magnification and the image I3 at the R3 magnification.

[0097] (6) Take image I1 at R1 magnification, image I2 at R2 magnification, and image I3 at R3 magnification as labeled source domain images.

[0098] Here, we generate multi-rate pyramid images in order to jointly optimize feature design when performing domain adaptation. This technique allows the model to extract and integrate features at multiple scales, thereby establishing a more accurate feature correspondence between the source and target domains. Multi-rate pyramid images can capture tissue details and global structural information at different resolutions, which is crucial for domain adaptation. Jointly optimizing these multi-scale features not only enhances the model's ability to map features between different domains, but also improves the model's classification accuracy on unlabeled target domain data. In addition, this feature design method enables the model to better adapt to the unique characteristics of the target domain while maintaining the knowledge learned in the source domain, thereby achieving more effective domain adaptation learning.

[0099] The second step is to build a WSI image tissue classification model: Based on domain adaptive learning technology, a WSI image tissue classification model is built. The goal is to build a WSI image tissue classification model that aims to improve the efficiency and accuracy of pathological diagnosis, reduce human errors, and achieve diagnostic standardization. In addition, without the need for target domain data annotation, the model can overcome the differences in data distribution in different data centers, realize cross-data center model sharing and collaborative research, and further improve the accuracy of WSI image tissue classification.

[0100] (1) Figure 3 As shown in the figure, a WSI image tissue classification model is set based on domain adaptive learning technology, that is, a pyramid dynamic feature fusion network for WSI tissue classification, including a generator G, a classifier C and a discriminator D. The generator G includes a pyramid feature extraction network and a dynamic feature fusion network, and the output network is used as the classifier C. C and the additional nuclear norm operator are used as the discriminator D.

[0101] The nuclear norm operator uses the nuclear norm first-order Wasserstein distance W1(μ,v), which is expressed as:

[0102]

[0103] where Γ(μ,v) is the set of all possible probability measures for the two distributions μ and v, x and y are samples drawn from μ and v respectively, |xy| is the distance between sample x and sample y, γ is the joint distribution, and inf is the infimum.

[0104] (2) The pyramid feature extraction network is set to include feature encoders F1, F2, and F3 of three scales with the same structure, where F1 is a feature encoder of R1 scale, F2 is a feature encoder of R2 scale, and F3 is a feature encoder of R3 scale, and their inputs are respectively image I1 at R1 scale, image I2 at R2 scale, and image I3 at R3 scale;

[0105] like Figure 4 As shown in the figure, the feature encoder is set to use the ResNeXt partial network. The feature encoder changes the output dimension of the last fully connected layer from the default 1000 to 768, and changes the last activation function from the default Softmax to GELU. The network structure of the feature encoding is as follows:

[0106] Feature encoder input layer: The input is a feature matrix with a scale of (3, 256, 256);

[0107] RN1-ResNet pre-order network: ResNet pre-order network RN1 extracts the input feature matrix to obtain a feature vector with a scale of (2048,);

[0108] FC2-Fully connected layer: The fully connected layer FC2 passes the output feature vector of RN1 through the fully connected layer to obtain a feature vector with a scale of (768,);

[0109] A3-Activation layer: The activation layer A3 obtains a feature vector with a scale of (768,) by using the nonlinear transformation GELU to transform the feature vector output by FC2;

[0110] Feature encoder output layer: The output is obtained by activating layer A3 with three feature vectors of scale (768,), which are the first feature matrix, the second feature matrix and the third feature matrix.

[0111] (3) Set up a dynamic feature fusion network. Figure 5 As shown, the dynamic feature fusion network includes a dynamic feature fusion network input layer, Concat1-concatenation layer, MSA2-multi-head self-attention module, R3-residual layer, FC4-fully connected layer, A5-activation layer, FC6-fully connected layer, R7-residual layer and a dynamic feature fusion network output layer;

[0112] Among them, the input of the dynamic feature fusion network input layer is the feature vectors of three scales (768,) output by the pyramid feature extraction network;

[0113] Concat1-The concatenation layer Concat1 adds one dimension to the three feature vectors of the input layer in the first dimension, turning them into (1,768), and obtains a feature matrix with a scale of (3,768) through the concatenation operation in the first dimension;

[0114] MSA2 - Multi-head self-attention module The multi-head self-attention module MSA2 uses the matrix output by Concat1 through multi-head self-attention to obtain a feature matrix with a scale of (3,768);

[0115] R3 - residual layer The residual layer R3 adds the feature matrix output by MSA2 and the feature matrix output by Concat1 to obtain a feature matrix with a scale of (3,768);

[0116] FC4-Fully connected layer The fully connected layer FC4 passes the output feature matrix of R3 through the fully connected layer to obtain a feature matrix with a scale of (3,768);

[0117] A5-Activation layer A5 converts the output feature matrix of FC4 through nonlinear transformation GELU to obtain a feature matrix with a scale of (3,768);

[0118] FC6-Fully connected layer The fully connected layer FC6 passes the output feature matrix of A5 through the fully connected layer to obtain a feature matrix with a scale of (3,768);

[0119] R7-Residual layer R7 adds the output feature matrix of FC6 and the output feature matrix of residual layer R3 to obtain a feature matrix with a scale of (3,768);

[0120] The output of the dynamic feature fusion network output layer is the feature matrix obtained by the residual layer R7, that is, the first eigenvector.

[0121] (4) Set the output network. Figure 6 As shown, the output network includes an output network input layer, FC1-fully connected layer, A2-activation layer, FC3-fully connected layer, A4-activation layer, FC5-fully connected layer and output network output layer;

[0122] Output network input layer: The input is the feature matrix of the scale (3,768) output by the dynamic feature fusion network;

[0123] FC1-Fully connected layer The fully connected layer FC1 changes the feature matrix scale of the output network input layer to (2304,) and obtains the feature vector with a scale of (768,) through the fully connected layer;

[0124] A2-Activation layer The activation layer A2 transforms the feature vector obtained by the fully connected layer FC1 through the nonlinear transformation LeakyReLU to obtain a feature vector with a scale of (2304,), where the LeakyReLU leakage rate is 0.2;

[0125] FC3-Fully connected layer The fully connected layer FC3 passes the feature vector obtained by the A2-activation layer through the fully connected layer to obtain a feature vector with a scale of (768,);

[0126] A4-Activation layer The activation layer A4 transforms the feature vector obtained by FC3-Fully Connected Layer through nonlinear transformation LeakyReLU to obtain a feature vector with a scale of (2304,), where the LeakyReLU leakage rate is 0.2;

[0127] FC5 - fully connected layer The fully connected layer FC5 passes the feature vector obtained by A4 through the fully connected layer to obtain a feature vector of scale (num,), where num is the target category of WSI tissue classification;

[0128] Output network output layer: Output the feature vector obtained by FC5, that is, the prediction matrix P.

[0129] The third step is to train the WSI image organization classification model: Figure 7 As shown, the WSI image tissue classification model is trained using the WSI tissue classification source domain images.

[0130] (1) Obtain source domain images including image I1 at R1 magnification, image I2 at R2 magnification, and image I3 at R3 magnification and corresponding tissue category labels Y, and convert the three magnifications I1, I2, and I3 into 1、 I 2、 I3 Input WSI image tissue classification model.

[0131] (2) The source domain images of image I1 at R1 magnification, image I2 at R2 magnification, and image I3 at R3 magnification are respectively input into the pyramid feature extraction network, and the first feature matrix, the second feature matrix, and the third feature matrix are respectively obtained by feature encoders F1, F2, and F3.

[0132] (3) The first feature matrix, the second feature matrix and the third feature matrix are input into the dynamic feature fusion network to obtain the first feature vector.

[0133] (4) Input the first eigenvector into the output network to obtain the prediction matrix P;

[0134] (5) Construct a loss function based on P and Y. The loss function is the cross entropy L, which is expressed as:

[0135] L = -Y·log(P).

[0136] (6) All parameters of the WSI image tissue classification model are updated through the back-propagation mechanism according to the loss function, and the model parameter weights with the best indicators are saved.

[0137] After training, the model has been able to show good WSI image tissue classification performance in the source domain dataset with the same distribution as the training data. In order to further improve the model's WSI image tissue classification ability in the target domain dataset with different distribution, we adopted unsupervised domain adaptation technology.

[0138] The fourth step is to correct the loss of the WSI image tissue classification model: Fig.10 As shown in the figure, the WSI image organization classification model is corrected based on the domain adaptation method using the unlabeled target domain image. The generalization ability of the pyramid-based WSI image organization classification model is enhanced through the domain adaptation method, so that it can maintain or improve the classification performance on the unlabeled target domain images, so as to better generalize to new data sets. At the same time, the pyramid structure can capture the characteristics of WSI images at different scales. When combined with the domain adaptation method, the model can more effectively handle multi-scale feature changes in the target domain. This strong adaptability enables the pyramid model to cope with unknown distribution characteristics in the target domain, which is crucial for dealing with the data diversity encountered in practical applications.

[0139] (1) For the unlabeled target domain dataset {X t}Preprocessing is performed to obtain unlabeled target domain images at R1, R2, and R3 magnifications

[0140] (2) Label the organization as Y s The source domain image and Input them into generator G respectively to obtain the third eigenvector and the fourth eigenvector.

[0141] (3) Input the third feature vector into classifier C to obtain the first prediction vector.

[0142] (4) Using the first prediction vector and Y s Construct the first loss function L cls ,

[0143] Use cross entropy loss L ce Construct the first loss function L cls , which is expressed as follows:

[0144]

[0145] in, is the source domain image set The i-th group of images at R1, R2, and R3 magnifications, is the tissue classification label corresponding to the image, G is the generator, and C is the classifier.

[0146] (5) The third eigenvector and the fourth eigenvector are input into the discriminator D to obtain the fifth eigenvector and the sixth eigenvector.

[0147] (6) Use the fifth eigenvector and the sixth eigenvector to construct the second loss function L nwd ;

[0148] The second loss function L is constructed using the nuclear norm first-order Wasserstein distance nwd , which is expressed as follows:

[0149]

[0150] Among them, N s and N t are the number of samples in the source domain dataset and the target domain dataset, respectively.

[0151] (7) Using the first loss function L cls and the second loss function L nwd Construct the total loss function L total .

[0152] (8) According to the total loss function L total All parameters of the WSI image tissue classification model are updated through the back-propagation mechanism. After the update is completed, the generator G and classifier C after the loss correction are generated and applied to the tissue classification of the WSI image to be classified.

[0153] Step 5: Obtaining the WSI image to be classified: Obtain the WSI image to be classified and perform preprocessing to generate the WSI image to be classified.

[0154] Step 6: Obtaining the WSI image tissue classification result: Input the WSI image to be classified into the trained WSI image tissue classification model to obtain the WSI image tissue classification result.

[0155] Here, the tissue prediction probability values ​​are color mapped to realize the visualization of tissue classification. The visualization example is shown in Figure 8, where Figure 8(a) is the original image before prediction, Figure 8(b) is the cancer tissue prediction image, and Figure 8(c) is the weighted visualization image of the prediction image and the original image. Fig. 9 As shown in the figure, the pyramid dynamic feature fusion network can achieve accurate tissue classification in the pancreatic ductal adenocarcinoma WSI tissue classification task with high evaluation indicators.

[0156] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A WSI image organization classification method based on domain adaptive learning technology, characterized in that: The following steps are involved: 11) Acquiring WSI images and preprocessing: Acquiring WSI images and preprocessing them to generate WSI tissue classification source domain images; 12) Constructing a WSI image organization classification model: Constructing a WSI image organization classification model based on domain adaptive learning technology; 13) Training of WSI image tissue classification model: Using WSI tissue classification source domain images to train the WSI image tissue classification model; 14) Loss correction of WSI image tissue classification model: The loss correction of WSI image tissue classification model is performed based on the domain adaptation method using the unlabeled target domain image; 15) Obtaining the WSI image to be classified: obtaining the WSI image to be classified and performing preprocessing to generate the WSI image to be classified; 16) Obtaining WSI image tissue classification results: Input the WSI image to be classified into the trained WSI image tissue classification model to obtain the WSI image tissue classification results.

2. According to claim 1, a WSI image organization classification method based on domain adaptive learning technology is characterized in that: The WSI image acquisition and preprocessing comprises the following steps: 21) Acquire a WSI image, and scale the annotated WSI tissue classification label image. The label image size is scaled from [W, H] to [W / (R / R1), H / (R / R1)], where R is the original image acquisition ratio, W is the original image width, H is the original image height, and R1 is the first target magnification; 22) Use the square search box to traverse the WSI organization classification label images; 23) Calculate the proportion of the labeled tissue in the search box. If the proportion of the tissue is greater than r1, where r1 is the area occupied by the tissue / the area of ​​the search box, then intercept the tissue image block in the WSI image; 24) Using a background detection algorithm to detect the background of the captured tissue image block and calculate the background ratio; if the obtained background ratio is less than r2, where r2 is the area occupied by the background / the area of ​​the search box, then saving the image block and its associated label information as image I1 at R1 magnification; 25) Taking the central coordinate point of the R1x image as the center, intercept the images of size 256×256 at the second target magnification R2 and the third target magnification R3 as the image I2 at the R2 magnification and the image I3 at the R3 magnification; 26) Image I1 at R1 magnification, image I2 at R2 magnification, and image I3 at R3 magnification are used as labeled source domain images.

3. The WSI image organization classification method based on domain adaptive learning technology according to claim 1 is characterized in that: The construction of the WSI image organization classification model includes the following steps: 31) Based on domain adaptive learning technology, the WSI image tissue classification model is set to include a generator G, a classifier C and a discriminator D. The generator G includes a pyramid feature extraction network and a dynamic feature fusion network. The output network is used as the classifier C. The classifier C and the additional nuclear norm operator are used as the discriminator D. The nuclear norm operator uses the nuclear norm first-order Wasserstein distance W1(μ,v), which is expressed as: Where Γ(μ,v) is the set of all possible probability measures of the two distributions μ and v, x and y are samples drawn from μ and v respectively, |xy| is the distance between sample x and sample y, γ is the joint distribution, and inf is the infimum; 32) The pyramid feature extraction network is set to include feature encoders F1, F2, and F3 of three scales with the same structure, wherein F1 is a feature encoder of R1 magnification, F2 is a feature encoder of R2 magnification, and F3 is a feature encoder of R3 magnification, and its inputs are respectively image I1 at R1 magnification, image I2 at R2 magnification, and image I3 at R3 magnification; The feature encoder is set to use the ResNeXt partial network. The feature encoder changes the output dimension of the last fully connected layer from the default 1000 to 768, and changes the last activation function from the default Softmax to GELU. The network structure of the feature encoding is as follows: Feature encoder input layer: The input is a feature matrix with a scale of (3, 256, 256); RN1-ResNet pre-order network: ResNet pre-order network RN1 extracts the input feature matrix to obtain a feature vector with a scale of (2048,); FC2-Fully connected layer: The fully connected layer FC2 passes the output feature vector of RN1 through the fully connected layer to obtain a feature vector with a scale of (768,); A3-Activation layer: The activation layer A3 obtains a feature vector with a scale of (768,) by using the nonlinear transformation GELU to transform the feature vector output by FC2; Feature encoder output layer: Output three feature vectors with a scale of (768,) obtained by activation layer A3, which are the first feature matrix, the second feature matrix and the third feature matrix; 33) Setting a dynamic feature fusion network, the dynamic feature fusion network includes a dynamic feature fusion network input layer, Concat1-concatenation layer, MSA2-multi-head self-attention module, R3-residual layer, FC4-fully connected layer, A5-activation layer, FC6-fully connected layer, R7-residual layer and dynamic feature fusion network output layer; Among them, the input of the dynamic feature fusion network input layer is the feature vectors of three scales (768,) output by the pyramid feature extraction network; Concat1-The concatenation layer Concat1 adds one dimension to the three feature vectors of the input layer in the first dimension, turning them into (1,768), and obtains a feature matrix with a scale of (3,768) through the concatenation operation in the first dimension; MSA2 - Multi-head self-attention module The multi-head self-attention module MSA2 uses the matrix output by Concat1 through multi-head self-attention to obtain a feature matrix with a scale of (3,768); R3 - residual layer The residual layer R3 adds the feature matrix output by MSA2 and the feature matrix output by Concat1 to obtain a feature matrix with a scale of (3,768); FC4-Fully connected layer The fully connected layer FC4 passes the output feature matrix of R3 through the fully connected layer to obtain a feature matrix with a scale of (3,768); A5-Activation layer A5 converts the output feature matrix of FC4 through nonlinear transformation GELU to obtain a feature matrix with a scale of (3,768); FC6-Fully connected layer The fully connected layer FC6 passes the output feature matrix of A5 through the fully connected layer to obtain a feature matrix with a scale of (3,768); R7-Residual layer R7 adds the output feature matrix of FC6 and the output feature matrix of residual layer R3 to obtain a feature matrix with a scale of (3,768); The output of the dynamic feature fusion network output layer is the feature matrix obtained by the residual layer R7, that is, the first eigenvector; 34) Setting the output network, the output network includes an output network input layer, an FC1-fully connected layer, an A2-activation layer, an FC3-fully connected layer, an A4-activation layer, an FC5-fully connected layer and an output network output layer; Output network input layer: The input is the feature matrix of the scale (3,768) output by the dynamic feature fusion network; FC1-Fully connected layer The fully connected layer FC1 changes the feature matrix scale of the output network input layer to (2304,) and obtains the feature vector with a scale of (768,) through the fully connected layer; A2-Activation layer The activation layer A2 transforms the feature vector obtained by the fully connected layer FC1 through the nonlinear transformation LeakyReLU to obtain a feature vector with a scale of (2304,), where the LeakyReLU leakage rate is 0.2; FC3-Fully connected layer The fully connected layer FC3 passes the feature vector obtained by the A2-activation layer through the fully connected layer to obtain a feature vector with a scale of (768,); A4-Activation layer The activation layer A4 transforms the feature vector obtained by FC3-Fully Connected Layer through nonlinear transformation LeakyReLU to obtain a feature vector with a scale of (2304,), where the LeakyReLU leakage rate is 0.2; FC5 - fully connected layer The fully connected layer FC5 passes the feature vector obtained by A4 through the fully connected layer to obtain a feature vector of scale (num,), where num is the target category of WSI tissue classification; Output network output layer: Output the feature vector obtained by FC5, that is, the prediction matrix P.

4. The WSI image organization classification method based on domain adaptive learning technology according to claim 1 is characterized in that: The training of the WSI image tissue classification model includes the following steps: 41) Obtain source domain images including image I1 at R1 magnification, image I2 at R2 magnification, and image I3 at R3 magnification and corresponding tissue category labels Y, and convert the three magnification images I 1、 I 2、 I3 inputs the WSI image tissue classification model; 42) The source domain images of the image I1 at R1 magnification, the image I2 at R2 magnification, and the image I3 at R3 magnification are respectively input into the pyramid feature extraction network, and the first feature matrix, the second feature matrix, and the third feature matrix are respectively obtained by the feature encoders F1, F2, and F3; 43) inputting the first feature matrix, the second feature matrix and the third feature matrix into a dynamic feature fusion network to obtain a first feature vector; 44) Inputting the first eigenvector into the output network to obtain a prediction matrix P; 45) Construct a loss function based on P and Y. The loss function is the cross entropy L, and its expression is: L = -Y·log(P); 46) All parameters of the WSI image tissue classification model are updated through the back-propagation mechanism according to the loss function, and the model parameter weights with the best indicators are saved.

5. The WSI image organization classification method based on domain adaptive learning technology according to claim 1 is characterized in that: The impairment correction of the WSI image tissue classification model includes the following steps: 51) For the unlabeled target domain dataset {X t }Preprocessing is performed to obtain unlabeled target domain images at R1, R2, and R3 magnifications 52) Label the organization as Y s The source domain image and Input them into the generator G to obtain the third eigenvector and the fourth eigenvector respectively; 53) Inputting the third feature vector into the classifier C to obtain a first prediction vector; 54) Using the first prediction vector and Y s Construct the first loss function L cls , Use cross entropy loss L ce Construct the first loss function L cls , which is expressed as follows: in, is the source domain image set The i-th group of images at R1, R2, and R3 magnifications, is the tissue classification label corresponding to the image, G is the generator, and C is the classifier; 55) Input the third eigenvector and the fourth eigenvector into the discriminator D to obtain the fifth eigenvector and the sixth eigenvector; 56) Use the fifth and sixth eigenvectors to construct the second loss function L nwd ; The second loss function L is constructed using the nuclear norm first-order Wasserstein distance nwd , which is expressed as follows: Among them, N s and N t are the number of samples of the source domain dataset and the target domain dataset respectively; 57) Use the first loss function L cls and the second loss function L nwd Construct the total loss function L total ; 58) According to the total loss function L total All parameters of the WSI image tissue classification model are updated through the back-propagation mechanism. After the update is completed, the generator G and classifier C after the loss correction are generated and applied to the tissue classification of the WSI image to be classified.

Images