Pathological image color standardization method and device based on structure decomposition and restaining

Abstract

The application discloses a pathological image color standardization method and device based on structure decomposition and restaining, and the method comprises the following steps: acquiring a pathological image dataset and performing pretreatment, and cutting to obtain a patch image dataset; performing color space conversion on the patch image dataset, extracting L channel data in the Lab color space for standardization pretreatment to obtain a training set; converting the training set from the RGB space to the HED space through color deconvolution, extracting H and E channels and combining to obtain an image structure channel; building a ResNet18 neural network, constructing a loss function and training to obtain a color transfer matrix; modifying the neural network and training; inputting the image structure channel and the color transfer matrix into the trained neural network to obtain a generated image and calculating network loss. The method can completely maintain the structure of the image through image structure decomposition and restaining, obtain a color standardized image, maintain the original structure information of the image, and solve the problems of image structure deformation and incorrect color transfer.

Description

Technical Field

[0001] This invention belongs to the technical field of digital medical image processing, specifically relating to a method and apparatus for color standardization of pathological images based on structural decomposition and restaining. Background Technology

[0002] Currently, there are various methods for cancer diagnosis, but in clinical practice, pathology reports from pathological slides are generally considered the most reliable standard for cancer diagnosis. During manual analysis of pathological slides, the number of images that clinical pathologists can analyze per unit of time (hour) is often limited. Compared to the massive amount of digital pathology images, the number of images that clinical pathologists can analyze is significantly insufficient; moreover, the one-to-many model inevitably leads to overwork for clinicians, and analyzing a large number of pathology images over a long period undoubtedly challenges the diagnostic accuracy of the report. Therefore, with the development of algorithms, computer-aided diagnostic systems have been developed for rapid analysis of pathology images, making the analysis of a large number of images in a short time a trend. However, during the preparation of pathology slides, manual operations and experimental environments can affect the final color. For example, non-standard procedures in staining time, ambient temperature, and rinsing time can lead to color differences between pathology images. In particular, even scanning with different digital pathology slide scanning systems in the same laboratory can produce pathology images with different colors. In computer-aided diagnostic systems, most system designs rely on machine learning to uncover the data characteristics implicit in the dataset. Data characteristics, as prior knowledge inherent in the data itself, are determined by the quality of the data, which in turn determines the amount of information in the prior knowledge. Color differences between pathological images are part of image quality. If the impact of image color differences is ignored when designing the analysis system, the prediction accuracy will be lower than the expected accuracy.

[0003] To reduce systematic errors caused by color differences in pathological images, previous studies have proposed various countermeasures: data augmentation and color enhancement, color ignoring, and color standardization. Data augmentation and color enhancement methods directly alter the color distribution of the source image in the color space by randomly changing the brightness, saturation, and contrast of the training set images. However, they cannot accurately generate images with unknown color distributions. Typically, pathological images from different laboratories exhibit significant color distribution differences; therefore, this method has limitations and cannot simulate the color distribution of images under real-world conditions. Color ignoring methods usually ignore the RGB color information of the source image, converting it to a grayscale image, an optical density image, or a color separation matrix before further analysis. However, color information is a crucial image feature, vital for boundary differentiation and object recognition in image segmentation and classification. Therefore, this method's limitation lies in ignoring the image's color information. Color standardization is a relatively new research direction, which can be roughly divided into three categories: (1) color standardization based on chromatic separation; (2) statistical template color matching; and (3) color standardization based on deep learning. Color standardization based on chromatic separation decomposes an RGB image into multiple independent color channels and color matrices. The standardized image is obtained by multiplying the color channels of the source image with the color matrix of the target image. Statistical template color matching uses the color spectrum of RGB to match the gray values ​​of the source image with the gray values ​​of the target image, which involves color mapping from the source image to the target image. Color standardization based on deep learning usually relies on the development of generative adversarial networks. It uses the generator to learn the color distribution of the target color domain and the discriminator to judge the generated image. Adversarial generation can approach the image of the real color distribution. However, the most commonly used color normalization method based on staining separation in current research is color deconvolution. While empirical matrices obtained statistically from a certain number of pathological images offer the advantage of rapid computation, they are limited by the experimental conditions at the time and cannot yield more representative color matrices. Therefore, existing research employs non-negative matrix factorization (NMF) to separate image color channels. However, the extensive matrix operations and solutions result in high algorithm time complexity, making it somewhat inefficient for processing pathological images with resolutions reaching tens of thousands of pixels. Statistical template color matching methods rely on the color distribution of the template image. Using a single template image to represent the color distribution of the entire dataset has limitations, and reprocessing is required if the template image changes.Deep learning-based color normalization methods introduce adversarial generative networks, which lead to structural loss and image distortion problems in the generated new images. Image distortion is fatal for medical images, as changes in image structure affect clinicians' diagnosis of diseases. Therefore, color normalization algorithms that rely on paired images are limited in their applicability, making it difficult to obtain paired pathological images. Summary of the Invention

[0004] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a method and apparatus for color standardization of pathological images based on structural decomposition and restaining. This method decomposes a given pathological image by color deconvolution and obtains the image structure channels by combination. The color transfer matrix generated by the network is linearly combined with the image structure channels to obtain the restaining image. This method ensures that the source image structure does not change to the greatest extent, solves the problems of image structure deformation and incorrect color transfer, and greatly helps in the transmission and protection of image structure.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] The primary objective of this invention is to provide a method for color standardization of pathological images based on structural decomposition and restaining, characterized by comprising the following steps:

[0007] The pathological image dataset is acquired and preprocessed. The preprocessed pathological image dataset is then segmented to obtain the patch image dataset.

[0008] The images in the patch image dataset are transferred from the RGB color space to the Lab color space via the XYZ color space. The L channel data in the Lab color space is extracted and standardized preprocessed to obtain the training set.

[0009] The training set images are converted from the RGB color space to the HED color space by color deconvolution, and the H and E channels in the HED color space are extracted and combined to obtain the image structure channel.

[0010] A ResNet18 neural network was built based on the PyTorch deep learning framework. A loss function was constructed and the ResNet18 neural network was trained to obtain a color transfer matrix. The ResNet18 neural network G(x) includes 8 residual learning modules. Each residual learning module includes 2 convolutional layers and a ReLU activation function.

[0011] Modify the ResNet18 neural network structure and change the single fully connected layer of the ResNet18 neural network to a double fully connected layer, and train the modified ResNet18 neural network.

[0012] The image structure channels and color transfer matrix are input into the trained ResNet18 neural network to obtain the generated image and the network loss is calculated.

[0013] As a preferred technical solution, the pathological image dataset uses RGB pathological images at 40x magnification from the MITOS-ATYPIA 14 dataset; the data preprocessing involves registering the pathological image dataset using a non-rigid registration method.

[0014] The segmentation of the preprocessed pathological image dataset specifically involves dividing the images in the preprocessed pathological image dataset into several patch images with a width and height of 256 pixels each, using a step size of 256, to obtain the patch image dataset.

[0015] As a preferred technical solution, the XYZ color space is represented as follows:

[0016]

[0017] Among them, I XYZ For an image in the XYZ color space, I RGB The image is in the RGB color space, where X, Y, and Z are the color channels in the XYZ color space, respectively.

[0018] The Lab color space is represented as follows:

[0019]

[0020] Among them, I Lab The image is in the Lab color space, where L, a, and b are the color channels in the Lab color space, and X is the color channel. n Y n Z n For a given constant, the formula for calculating f(x) is:

[0021]

[0022] As a preferred technical solution, the standardization preprocessing specifically includes:

[0023] The extracted L-channel data is standardized using z-score preprocessing, with the following formula:

[0024]

[0025] Where I′ is the image obtained after normalization preprocessing, I is the image output by the L channel in the Lab color space, and μ and σ are the image mean and variance, respectively.

[0026] As a preferred technical solution, obtaining the image structure channel specifically involves:

[0027] Converting the training set images from the RGB color space to the HED color space using color deconvolution is represented as follows:

[0028]

[0029] Among them, I HED The image is in the HED color space, where H, E, and D are the color channels in the HED color space, respectively.

[0030] Extract I HED I in H and I E And obtain I through combination. HH =I H *I H I EE =I E *I E I HE =I H *I E ;

[0031] Combine I H I E I HE I HH and I EE To obtain image structure channel I str =[H,E,HE,HH,EE].

[0032] As a preferred technical solution, the construction of the loss function and training of the ResNet18 neural network to obtain the color transfer matrix are specifically as follows:

[0033] The loss function is constructed using the squared absolute error (MAE) and the structural similarity index (SSIM).

[0034] The squared absolute error (MAE) is used to calculate the absolute value of the error between the predicted value and the actual value. The calculation formula is as follows:

[0035]

[0036] Where N is the number of samples in the training set. For the predicted value, I y The actual value;

[0037] The Structural Similarity Index (SSIM) evaluates images by calculating the structural feature differences between them. The calculation formula is as follows:

[0038]

[0039] in, These are the predicted values. The mean and variance; The true value I is respectively y The mean and variance; c1 and c2 are constants used to avoid division by zero;

[0040] Training set I L The input to the ResNet18 neural network G(x) yields the network output G(I). L ), where G(I L M is a 3x5 color transition matrix. 3×5 , represented as:

[0041]

[0042] Where 'a' represents the weight coefficient of the network output.

[0043] As a preferred technical solution, the dual fully connected layer uses the Tanh function as the activation function, expressed as:

[0044]

[0045] Among them, e x Let x be an exponential function, and x be an eigenvector.

[0046] The modified ResNet18 neural network is specifically trained as follows:

[0047] Initialize the weights of the ResNet18 neural network;

[0048] The ResNet18 neural network was trained using an adaptive learning method and iteratively updated.

[0049] Set a random seed and perform the computation on the GPU.

[0050] As a preferred technical solution, the calculation of network loss specifically involves:

[0051] The color transfer matrix M 3×5 With image structure channel I str Multiplication yields the color-transferred pathological image I output , represented as:

[0052] I output =M 3×5 ×I str

[0053] Based on the color-transferred pathological image and its corresponding RGB color space image, the loss of the ResNet18 neural network is calculated using the following formula:

[0054] Loss=α*MAE(I output ,I rgb )+β*SSIM(I outpit ,I rgb )

[0055] Where α and β are the weighting parameters that control MAE and SSIM.

[0056] Another objective of this invention is to provide a pathological image color standardization system based on structural decomposition and restaining, characterized in that it includes a data acquisition and processing module, an L-channel extraction module, a structural channel acquisition module, a color matrix acquisition module, a network modification module, and a training and calculation module.

[0057] The data acquisition and processing module is used to acquire pathological image datasets and perform data preprocessing, and to segment the preprocessed image datasets to obtain patch image datasets.

[0058] The L-channel extraction module is used to transfer the images in the patch image dataset from the RGB color space through the XYZ color space to the Lab color space, extract the L-channel data in the Lab color space and perform standardized preprocessing to obtain the training set.

[0059] The structure channel acquisition module converts the training set image from the RGB color space to the HED color space through color deconvolution, extracts the H channel and E channel data in the HED color space, and combines them to obtain the image structure channel.

[0060] The color matrix acquisition module builds a ResNet18 neural network based on the PyTorch deep learning framework, constructs a loss function, and trains the ResNet18 neural network to obtain the color transfer matrix.

[0061] The network modification module is used to modify the ResNet18 neural network structure, changing the single fully connected layer of the ResNet18 neural network into a double fully connected layer, and training the modified ResNet18 neural network.

[0062] The training and computation module inputs the image structure channels and color transfer matrix into the trained ResNet18 neural network to generate the image and calculate the network loss.

[0063] Another objective of this invention is to provide a computer-readable storage medium storing a program, characterized in that, when the program is executed by a processor, it implements the above-described method for color standardization of pathological images based on structural decomposition and restaining.

[0064] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0065] 1. The pathological image color normalization method based on structure decomposition and restaining provided in this invention is designed based on a self-supervised learning method. Compared with other weakly supervised or fully supervised color normalization methods, this method has strong practicality because it does not require paired target domain images during training. In clinical practice, obtaining pathological images is extremely difficult, and the sample size is limited by different medical institutions. Different medical institutions protect patient privacy data, making sample transfer difficult between institutions. This is especially true for paired pathological images, which are usually scanned using a single digital pathology scanner. If a supervised color normalization method is used, its applicability would be severely limited. Therefore, this method uses a self-supervised learning method to fit the color domain of the image dataset, eliminating the need to prepare additional paired target color domain images.

[0066] 2. The image color decomposition method used in this invention can effectively decompose the nuclear and cytoplasmic structures, which are represented by the H and E channels, respectively. Furthermore, by performing matrix transformations between structures, the contrast between different structures is increased to some extent. This structural decomposition method utilizes the original image structural information and recolors it based on the original image structure, preserving the original structural information to the greatest extent possible and avoiding structural loss. In contrast, previous pathological image color normalization methods mostly rely on generative adversarial networks to generate new images. The generation of new images inevitably involves structural loss, and without restrictions during training, structural distortion may even occur.

[0067] 3. This invention can not only solve the problem of color difference caused by different machine models, but also solve the color standardization of different tissue sections. It has wide applicability and no excessive restrictions. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 This is a flowchart of a pathological image color normalization method based on structural decomposition and restaining in an embodiment of the present invention;

[0070] Figure 2 This is a structural diagram of the pathological image color normalization method based on structural decomposition and restaining in an embodiment of the present invention;

[0071] Figure 3This is a schematic diagram of the residual learning module in the ResNet18 neural network in an embodiment of the present invention;

[0072] Figure 4 This is a schematic diagram illustrating the modification of a single fully connected layer to a double fully connected layer in the ResNet18 neural network in an embodiment of the present invention.

[0073] Figure 5 This is a structural diagram of the pathological image color normalization system based on structural decomposition and restaining in an embodiment of the present invention;

[0074] Figure 6 This is a schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Detailed Implementation

[0075] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0076] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0077] During the preparation of pathological slides, human or environmental factors often lead to color variations and stylistic inconsistencies in digital pathology images, especially those obtained from different laboratories. Furthermore, in digital pathology scanners designed by different manufacturers, inconsistencies in image processing algorithms can cause even the same pathological image to exhibit different color appearances. Directly incorporating these color-inconsistent pathological images into machine learning or deep learning-based analysis models, and then applying the optimal model trained on one dataset to another for validation and testing, can result in performance falling short of expectations. Therefore, to address this technical problem, this invention proposes a pathological image color standardization method based on structural decomposition and restaining.

[0078] like Figure 1 , Figure 2As shown, this embodiment uses a pathological image color normalization method based on structural decomposition and restaining, which includes the following steps:

[0079] S1. Obtain the pathological image dataset and perform data preprocessing. Segment the preprocessed image dataset to obtain the patch image dataset.

[0080] More specifically, the pathological image dataset in this embodiment uses 40x magnification RGB pathological images from the MITOS-ATYPIA 14 dataset. The pathological sections in this dataset are prepared from breast cancer biopsy tissue sections stained with standard hematoxylin-eosin dye. The RGB pathological images were obtained by scanning at 40x magnification using both AperioScanscope XT and Hamamatsu Nanozoomer 2.0-HT scanners, meaning that the same section corresponds to pathological images obtained by scanning with two different scanners. Data preprocessing includes registering the pathological image dataset using non-rigid registration techniques, the purpose of which is to calculate the consistency index of related images in the training set and test set and to verify performance.

[0081] Due to the limitations of convolutional neural networks and computer memory on input data, pathological images need to be segmented into several patch images of appropriate size. Therefore, the images in the preprocessed pathological image dataset are segmented into several patch images with a stride of 256 and a width and height of 256 pixels, resulting in a patch image dataset.

[0082] S2. Transfer the images in the patch image dataset from the RGB color space to the Lab color space via the XYZ color space, extract the L channel data in the Lab color space and perform normalization preprocessing to obtain the training set.

[0083] First, each patch image is transferred from the RGB color space to the Lab color space via the XYZ color space. The XYZ color space is represented as follows:

[0084]

[0085] Among them, I XYZ For an image in the XYZ color space, I RGB The image is in the RGB color space, where X, Y, and Z are the color channels in the XYZ color space, respectively.

[0086] The Lab color space is represented as:

[0087]

[0088] Among them, I LabThe image is in the Lab color space, where L, a, and b are the color channels in the Lab color space, and X is the color channel. n Y n Z n For a given constant, the formula for calculating f(x) is:

[0089]

[0090] In this embodiment, X n =96.422, Y n =100.000, Z n =82.522.

[0091] This invention uses only the L-channel data in the Lab color space as input data for the neural network. The L-channel represents the brightness information of the image, is device-independent, and reduces the impact of color differences in the source image on the neural network. Before inputting the L-channel data into the neural network, z-score normalization preprocessing is required. This data normalization preprocessing eliminates the influence of data dimensions and allows the loss to decrease slowly and steadily during training, stabilizing the transmission of network gradient information. The formula is:

[0092]

[0093] Where I′ is the image obtained after normalization preprocessing, I is the image output by the L channel in Lab color space, and μ and σ are the image mean and variance, respectively;

[0094] After standardization preprocessing, the training set I is obtained. L .

[0095] S3. Convert the training set image from RGB color space to HED color space by color deconvolution, extract the H channel and E channel data in HED color space, and combine them to obtain the image structure channel.

[0096] In the HED color space, the H channel typically represents the nuclear structure of cells in a pathological image, and the E channel represents the cytoplasmic structure of a pathological image. The starting point of the color standardization of this invention is to re-stain the pathological structure diagram of a given image, so the data of the H channel and E channel in the HED color space are extracted.

[0097] The HED color space is represented as:

[0098]

[0099] Among them, I HED The image is located in the XYZ color space, and H, E, and D are the color channels in the HED color space, respectively.

[0100] Extract IHED I in H and I E And obtain I through combination. HH =I H *I H I EE =I E *I E I HE =I H *I E ;

[0101] Combine I H I E I HE I HH and I EE To obtain image structure channel I str =[H,E,HE,HH,EE].

[0102] S4. Build a ResNet18 neural network based on the PyTorch deep learning framework, construct the loss function and train the ResNet18 neural network to obtain the color transfer matrix.

[0103] The ResNet18 neural network G(x) includes 8 residual learning modules, such as Figure 3 As shown, the structure of the residual learning module includes two convolutional layers and a ReLU activation function, which increases depth and fitting ability while reducing computational cost and avoiding the "gradient vanishing" phenomenon. The ResNet18 neural network reduces training time and computational cost while maintaining network complexity, becoming the key to color standardization of large-scale pathological images and accelerating the analysis efficiency of computer-aided diagnostic systems.

[0104] The loss function is the fitting target of a deep learning algorithm, used to measure the difference between the network's predicted output and the true value. A decrease in the loss value corresponds to an improvement in the network's prediction performance on the training set. Therefore, when designing a loss function, multiple loss functions can be combined to form an overall loss function. This method adds constraints, which is beneficial for training the network and accelerating convergence.

[0105] During the training of the ResNet18 neural network, Mean Absolute Error (MAE) and Structural Similarity (SSIM) were added as loss functions. The MAE function, also known as the L1 loss function, is used to calculate the absolute value of the error between the predicted value and the true value. The SSIM function takes image quality evaluation as its starting point and evaluates the image from an objective perspective by calculating the difference in structural features between images. When calculating the SSIM function, an N×N window slides across the entire image to calculate the local SSIM value, and after traversing the entire image, the average value is taken as the global SSIM value.

[0106] The formula for calculating the squared absolute error (MAE) is:

[0107]

[0108] Where N is the number of samples in the training set. For the predicted value, I y The actual value;

[0109] The formula for calculating the structural similarity index (SSIM) is as follows:

[0110]

[0111] in, These are the predicted values. The mean and variance; The true value I is respectively y The mean and variance; c1 and c2 are constants used to avoid division by zero;

[0112] Training set I L The input to the ResNet18 neural network G(x) yields the network output G(I). L ), where G(I L M is a 3x5 color transition matrix. 3×5 , represented as:

[0113]

[0114] Where 'a' represents the weight coefficient of the network output.

[0115] S5. Modify the ResNet18 neural network structure by changing the single fully connected layer of the ResNet18 neural network to a double fully connected layer, and train the modified ResNet18 neural network.

[0116] like Figure 4As shown, the single fully connected layer in the original ResNet18 neural network structure is modified into a double fully connected layer, and the Tanh function is used as the activation function, expressed as:

[0117]

[0118] Among them, e x Let x be an exponential function, and x be an eigenvector.

[0119] Next, the modified ResNet18 neural network is trained, specifically as follows:

[0120] S51. Initialize the weights of the ResNet18 neural network; In this embodiment, the weights of the ResNet18 neural network are initialized to satisfy the normal distribution assumption with a mean of 0 and a standard deviation of 0.02, which is beneficial to the convergence of the network and the improvement of its performance.

[0121] S52. Train the ResNet18 neural network using an adaptive learning method and perform iterative updates; select an appropriate optimizer for iteratively updating the network and control the rate of gradient descent.

[0122] In this embodiment, the adaptive learning rate Adam algorithm is selected, and the learning rate is set to 10. -4 The exponential decay rate beta1 = 0.05, and the exponential decay rate beta2 = 0.99;

[0123] S53. Set a random seed for the reproducibility study of the experiment; all calculations related to network parameters are performed in a GPU (a graphics processing unit, commonly used for fast, large-scale computations in deep learning neural networks).

[0124] In this embodiment, all the data that needs to be used in the training of the ResNet18 neural network is transferred to CUDA.

[0125] S6. Input the image structure channels and color transfer matrix into the trained ResNet18 neural network to obtain the generated image and calculate the network loss.

[0126] The color transfer matrix M 3×5 With image structure channel I str Multiplication yields a linear combination of the structural images, i.e., the pathological image I after color transfer. output , represented as:

[0127] I output =M 3×5 ×I str

[0128] Based on the color-transferred pathological image and its corresponding RGB color space image, the loss of the ResNet18 neural network is calculated using the following formula:

[0129] Loss=α*MAE(I output ,I rgb )+β*SSIM(I output ,I rgb )

[0130] Where α and β are the weighting parameters that control MAE and SSIM.

[0131] In this embodiment, α = 1 and β = 1, meaning that MAE loss and SSIM loss contribute equally to the ResNet18 neural network during the training phase, and are of equal importance.

[0132] The starting point of this invention for color standardization of pathological images is to decompose a given pathological image using a color deconvolution method and obtain image structure channels through combination. The color transfer matrix generated by the network is then linearly combined with the image structure channels to obtain the recolored image. This operation greatly helps in the transfer and preservation of image structure. Existing methods merely use color deconvolution to convert the image to HE channels, and then use the HE channels as network input for end-to-end image generation. End-to-end image generation methods suffer from image structure distortion and incorrect color transfer. The image structure decomposition and recoloring method proposed in this invention completely preserves the image structure, eliminating the risk of structural distortion. Furthermore, based on fitting the color distribution of the entire dataset, the trained network can effectively represent the data distribution, and the standardized image color distribution is similar to the target color distribution.

[0133] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0134] Based on the same idea as the pathological image color standardization method based on structural decomposition and restaining in the above embodiments, the present invention also provides a pathological image color standardization system based on structural decomposition and restaining. This system can be used to execute the above-described pathological image color standardization method based on structural decomposition and restaining. For ease of explanation, the structural schematic diagram of the embodiment of the pathological image color standardization system based on structural decomposition and restaining only shows the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0135] like Figure 5 As shown, another embodiment of the present invention provides a pathological image color normalization system based on structural decomposition and restaining, comprising the following modules:

[0136] The data acquisition and processing module is used to acquire pathological image datasets and perform data preprocessing. The preprocessed image datasets are then segmented to obtain patch image datasets.

[0137] The L-channel extraction module is used to transfer the images in the patch image dataset from the RGB color space through the XYZ color space to the Lab color space, extract the L-channel data in the Lab color space and perform normalization preprocessing to obtain the training set.

[0138] The structure matrix acquisition module converts the training set image from the RGB color space to the HED color space through color deconvolution, extracts the H and E channels in the HED color space, and obtains the image structure channels.

[0139] The color matrix acquisition module is based on the PyTorch deep learning framework to build a ResNet18 neural network, construct a loss function and train the ResNet18 neural network to obtain the color transfer matrix.

[0140] The network modification module is used to modify the ResNet18 neural network structure, changing the single fully connected layer of the ResNet18 neural network to a double fully connected layer, and training the modified ResNet18 neural network.

[0141] The training computation module inputs the image structure channels and color transfer matrix into the trained ResNet18 neural network to obtain the generated image and calculates the network loss.

[0142] It should be noted that the pathological image color standardization system based on structural decomposition and restaining of the present invention corresponds one-to-one with the pathological image color standardization method based on structural decomposition and restaining of the present invention. The technical features and beneficial effects described in the embodiments of the pathological image color standardization method based on structural decomposition and restaining are applicable to the embodiments of the pathological image color standardization system based on structural decomposition and restaining. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0143] Furthermore, in the embodiments of the pathological image color standardization system based on structural decomposition and restaining described above, the logical division of each program module is merely illustrative. In practical applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or the convenience of software implementation. That is, the internal structure of the pathological image color standardization system based on structural decomposition and restaining can be divided into different program modules to complete all or part of the functions described above.

[0144] like Figure 6 As shown, in one embodiment, a computer-readable storage medium is provided, storing a program in a memory. When executed by a processor, the program implements the aforementioned pathological image color normalization method based on structural decomposition and restaining, specifically:

[0145] The pathological image dataset is acquired and preprocessed. The preprocessed pathological image dataset is then segmented to obtain the patch image dataset.

[0146] The images in the patch image dataset are transferred from the RGB color space to the Lab color space via the XYZ color space. The L channel data in the Lab color space is extracted and standardized preprocessed to obtain the training set.

[0147] The training set images are converted from the RGB color space to the HED color space by color deconvolution, and the H and E channels in the HED color space are extracted and combined to obtain the image structure channel.

[0148] A ResNet18 neural network was built based on the PyTorch deep learning framework. A loss function was constructed and the ResNet18 neural network was trained to obtain a color transfer matrix. The ResNet18 neural network G(x) includes 8 residual learning modules. Each residual learning module includes 2 convolutional layers and a ReLU activation function.

[0149] Modify the ResNet18 neural network structure by changing the single fully connected layer of the ResNet18 neural network to a double fully connected layer, and train the modified ResNet18 neural network.

[0150] The image structure channels and color transfer matrix are input into the trained ResNet18 neural network to obtain the generated image and the network loss is calculated.

[0151] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0152] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0153] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for color normalization of pathological images based on structural decomposition and restaining, characterized in that, Includes the following steps: The pathological image dataset is acquired and preprocessed. The preprocessed pathological image dataset is then segmented to obtain the patch image dataset. The images in the patch image dataset are transferred from the RGB color space to the Lab color space via the XYZ color space. The L channel data in the Lab color space is extracted and standardized preprocessed to obtain the training set. The training set images are converted from the RGB color space to the HED color space using color deconvolution, and the H and E channels of the HED color space are extracted. H I E And obtain I through combination. HH =I H *I H I EE =I E *I E I HE =I H *I E Combined with I H I E I HE I HH and I EE To obtain image structure channel I str =[H,E,HE,HH,EE]; A ResNet18 neural network was built using the PyTorch deep learning framework. A loss function was constructed, and the ResNet18 neural network was trained to obtain a color transfer matrix. The ResNet18 neural network G(x) includes 8 residual learning modules. Each residual learning module includes 2 convolutional layers and a ReLU activation function. The loss function is jointly constructed using the squared absolute error (MAE) and the structural similarity index (SSIM). The color transfer matrix is ​​a 3x5 matrix M. 3×5 ; Modify the ResNet18 neural network structure and change the single fully connected layer of the ResNet18 neural network to a double fully connected layer, and train the modified ResNet18 neural network. The dual fully connected layer uses the Tanh function as the activation function; The image structure channels and color transfer matrix are input into a trained ResNet18 neural network to obtain a generated image, and the network loss is calculated; the generated image is composed of the color transfer matrix M. 3×5 With image structure channel I str Multiply them to get the result.

2. The pathological image color normalization method based on structural decomposition and restaining according to claim 1, characterized in that, The pathological image dataset uses RGB pathological images at 40x magnification from the MITOS-ATYPIA 14 dataset; the data preprocessing involves registering the pathological image dataset using a non-rigid registration method. The segmentation of the preprocessed pathological image dataset specifically involves dividing the images in the preprocessed pathological image dataset into several patch images with a width and height of 256 pixels each, using a step size of 256, to obtain the patch image dataset.

3. The pathological image color normalization method based on structural decomposition and restaining according to claim 1, characterized in that, The XYZ color space is represented as follows: ， Among them, I XYZ For an image in the XYZ color space, I RGB The image is in the RGB color space, where X, Y, and Z are the color channels in the XYZ color space, respectively. The Lab color space is represented as follows: ， Among them, I Lab The image is in the Lab color space, where L, a, and b are the color channels in the Lab color space, and X is the color channel. n Y n Z n For a given constant, the formula for calculating f(x) is: 。 4. The pathological image color normalization method based on structural decomposition and restaining according to claim 3, characterized in that, The standardization preprocessing specifically involves: The extracted L-channel data is standardized using z-score preprocessing, with the following formula: ， Among them, I ' The image is obtained after standardization preprocessing. I is the image output from the L channel in the Lab color space, and μ and σ are the image mean and variance, respectively.

5. The pathological image color normalization method based on structural decomposition and restaining according to claim 4, characterized in that, The process of converting the training set images from the RGB color space to the HED color space using color deconvolution is represented as follows: ， Among them, I HED The image is in the HED color space, where H, E, and D are the color channels in the HED color space.

6. The pathological image color normalization method based on structural decomposition and restaining according to claim 5, characterized in that, The process of constructing the loss function and training the ResNet18 neural network to obtain the color transfer matrix is ​​as follows: The loss function is constructed using the squared absolute error (MAE) and the structural similarity index (SSIM). The squared absolute error (MAE) is used to calculate the absolute value of the error between the predicted value and the actual value. The calculation formula is as follows: ， Where N is the number of samples in the training set. For the predicted value, I y The actual value; The Structural Similarity Index (SSIM) evaluates images by calculating the structural feature differences between them. The calculation formula is as follows: ， in, , These are the predicted values. The mean and variance; , The true value I is respectively y The mean and variance; c1 and c2 are constants used to avoid division by zero; Training set I L The input to the ResNet18 neural network G(x) yields the network output G(I). L ), where G(I L M is a 3x5 color transition matrix. 3×5 , represented as: , Where 'a' represents the weight coefficient of the network output.

7. The pathological image color normalization method based on structural decomposition and restaining according to claim 6, characterized in that, The dual fully connected layer is represented as follows: ， Among them, e x Let x be an exponential function, and x be an eigenvector. The modified ResNet18 neural network is specifically trained as follows: Initialize the weights of the ResNet18 neural network; The ResNet18 neural network was trained using an adaptive learning method and iteratively updated. Set a random seed and perform the computation on the GPU.

8. The pathological image color normalization method based on structural decomposition and restaining according to claim 7, characterized in that, The calculation of network loss specifically involves: The color transfer matrix M 3×5 With image structure channel I str Multiplication yields the color-transferred pathological image I output , represented as: I output =M 3×5 ×I str , Based on the color-transferred pathological image and its corresponding RGB color space image, the loss of the ResNet18 neural network is calculated using the following formula: Loss=α*MAE(I output ,I rgb )+β*SSIM(I output ,I rgb ), Where α and β are the weighting parameters that control MAE and SSIM.

9. A pathological image color normalization system based on structural decomposition and restaining, characterized in that, It includes a data acquisition and processing module, an L-channel extraction module, a structure channel acquisition module, a color matrix acquisition module, a network modification module, and a training calculation module; The data acquisition and processing module is used to acquire pathological image datasets and perform data preprocessing, and to segment the preprocessed image datasets to obtain patch image datasets. The L-channel extraction module is used to transfer the images in the patch image dataset from the RGB color space through the XYZ color space to the Lab color space, extract the L-channel data in the Lab color space and perform standardized preprocessing to obtain the training set. The structure channel acquisition module converts the training set images from the RGB color space to the HED color space through color deconvolution, and extracts the H and E channel data in the HED color space. H I E And obtain I through combination. HH =I H *I H I EE =I E *I E I HE =I H *I E Combined with I H I E I HE I HH and I EE To obtain image structure channel I str =[H,E,HE,HH,EE]; The color matrix acquisition module is based on a PyTorch deep learning framework to build a ResNet18 neural network, constructs a loss function, and trains the ResNet18 neural network to obtain the color transfer matrix. The ResNet18 neural network G(x) includes 8 residual learning modules. Each residual learning module includes 2 convolutional layers and a ReLU activation function. The loss function is jointly constructed by the squared absolute error (MAE) and the structural similarity index (SSIM). The color transfer matrix is ​​a 3x5 matrix M. 3×5 ; The network modification module is used to modify the ResNet18 neural network structure, changing the single fully connected layer of the ResNet18 neural network into a double fully connected layer, and training the modified ResNet18 neural network. The dual fully connected layer uses the Tanh function as the activation function; The training and computation module inputs the image structure channels and color transfer matrix into the trained ResNet18 neural network to obtain the generated image and calculates the network loss; the generated image is composed of the color transfer matrix M. 3×5 With image structure channel I str Multiply them to get the result.

10. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the pathological image color normalization method based on structural decomposition and restaining as described in any one of claims 1-8.

Images