Pathological image color correction method and device and readable storage medium thereof

Abstract

The present application provides a pathological image color correction method and device and readable storage medium thereof, aiming at the problems that the traditional linear method in the prior art is difficult to fit the nonlinear color mapping and depends on manual parameter adjustment, the deep learning method has poor adaptability and insufficient real-time performance, a three-level progressive deep learning network is designed: a general color correction network learns high-order parameters through a complex structure to realize full color correction; a special color correction network realizes automatic color grouping by combining a compatible color loss function based on the feature constraint of the general network, reduces parameters to adapt to limited hardware; a lightweight color correction network further simplifies the structure and realizes efficient real-time correction based on grouped parameters. The three-level network can be flexibly selected according to hardware resources, realizes the adaptation to various staining and devices, balances the correction accuracy and real-time performance, improves the reliability of pathological image analysis, and is suitable for various pathological sections and scanner scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital image processing, in particular to color correction of pathological images, and is especially suitable for color calibration and optimization of digital pathological images obtained by a pathological section scanner. BACKGROUND

[0002] After chemical fixation, paraffin embedding and section staining, the pathological section is obtained by a digital section scanner. Due to the differences in spectral response of the optical system of the scanner, as well as factors such as staining batch, reagent type and section thickness, there is a significant deviation between the color of the digital image and the true color under the microscope, which directly affects the judgment of the pathologist on the lesion area, the degree of tissue differentiation and the expression strength of immunohistochemistry.

[0003] The traditional color correction method mainly uses a linear color correction matrix (CCM). This method linearly maps the original RGB value to the target color space through a 3x3 or 3x4 matrix, which is simple to calculate and easy to deploy. However, pathological section staining is highly nonlinear, and a linear matrix cannot accurately model the complex spectral-color mapping, often leading to over-saturation of some color tones and still deviation of another part of the color tone; at the same time, the CCM parameters need to be adjusted manually according to the color card, which is dependent on experience and low in efficiency.

[0004] In recent years, a nonlinear color correction method based on convolutional neural network has been proposed. This kind of method can learn complex nonlinear mapping through end-to-end training, which significantly improves the correction accuracy. However, the existing deep learning solutions still have the following shortcomings:

[0005] 1. A single network with the same set of parameters is used to deal with all staining and device differences, which is difficult to meet the diversified needs;

[0006] 2. Some methods only correct the color of local areas, ignoring the global color consistency, resulting in obvious color difference in different areas of the same image;

[0007] 3. The deep model has large parameter quantity and high computational complexity, which is difficult to meet the low delay requirement in the embedded hardware of the scanner or real-time reading scene;

[0008] 4. When new staining or optical system is added, a large amount of paired data needs to be collected and the entire network needs to be retrained, which is high in maintenance cost.

[0009] Therefore, how to ensure the correction accuracy while realizing parameter lightweight, model adaptive update and real-time inference has become a technical problem to be solved in the field of digital pathology. SUMMARY

[0010] The embodiment of the present application provides a pathological image color correction method, device and readable storage medium thereof, aiming at the problems that the existing linear CCM is difficult to handle nonlinear staining difference, and the existing deep learning scheme uses a single heavy network to cope with all scenes, resulting in that precision, real-time performance and universality cannot be achieved simultaneously.

[0011] The core technology of the present application is to propose a "general-purpose-special-purpose-lightweight" three-level progressive deep learning framework, which realizes an extensible, customizable and real-time deployable pathological image color correction system through automatic color grouping and model distillation during training.

[0012] In the first aspect, the present application provides a pathological image color correction method, which comprises the following steps:

[0013] A multi-level color correction network is constructed, which comprises a general-purpose color correction network, a special-purpose color correction network and a lightweight color correction network in turn;

[0014] The general-purpose color correction network is trained based on a loss function of the difference between the predicted image and the target image through a deep learning network structure comprising an encoding module and a decoding module, to learn high-order color correction parameters suitable for all colors;

[0015] The special-purpose color correction network is trained based on the training results of the general-purpose color correction network through a deep learning network structure comprising a simplified encoding module and a decoding module, and in the training process, the features extracted by the special-purpose color correction network are consistent with the features extracted by the general-purpose color correction network through feature constraint, and the sample weight is dynamically adjusted through the compatible color loss function to reduce the influence of samples with poor correction effect, color grouping is realized, and multiple sets of correction parameters adapted to different color groups are obtained;

[0016] The lightweight color correction network is trained through a further simplified network structure based on the color grouping data generated by the special-purpose color correction network, to obtain lightweight correction parameters corresponding to each color group, and the lightweight correction parameters are used for realizing real-time color correction based on simple matrix operation;

[0017] According to the hardware resources and application scenarios of the pathological section scanner, the general-purpose color correction network, the special-purpose color correction network or the lightweight color correction network is selected to correct the color of the pathological image;

[0018] Among them, the special-purpose color correction network and the lightweight color correction network determine the color group to which the pathological image belongs through a classification model or manual setting when applied, and call the correction parameters of the corresponding group.

[0019] Further, the encoding modules of the general color correction network, the special color correction network and the lightweight color correction network each consist of several convolution modules, and the convolution module includes a convolution layer, a batch normalization layer and an activation function layer; the decoding module of the general color correction network and the special color correction network consists of several convolution modules and an up-sampling module.

[0020] Further, the encoding module and the decoding module of the general color correction network each contain 4 convolution modules; the encoding module and the decoding module of the special color correction network each contain 2 convolution modules.

[0021] The lightweight color correction network adopts a network structure containing 3 1x1 convolution kernels, and the loss function for training is consistent with that of the general color correction network.

[0022] Further, the loss function of the general color correction network is a loss function based on the mean square error of the predicted image and the target image.

[0023] Further, the feature constraint of the special color correction network is achieved by measuring the similarity of the feature vectors extracted by the special color correction network and the general color correction network through the cosine distance.

[0024] Further, in the compatible color loss function of the special color correction network, the sample weight is dynamically adjusted with training iterations, and the adjustment mode is calculated based on the maximum value, the minimum value and the hyperparameter of the correction loss of the previous iteration.

[0025] Further, the color grouping of the special color correction network is achieved by repeatedly selecting samples with correction loss exceeding a threshold value as new grouping training samples until the loss variance of the remaining samples is less than a threshold value, and the threshold value is the sum of the mean value and the standard deviation of the loss of all samples.

[0026] In a second aspect, the present application provides a pathological image color correction device, comprising:

[0027] A network construction module is configured to construct the general color correction network, the special color correction network and the lightweight color correction network in sequence.

[0028] A training module is configured to train the general color correction network, the special color correction network and the lightweight color correction network respectively, wherein the special color correction network realizes color grouping based on the feature constraint and the compatible color loss function of the general color correction network, and the lightweight color correction network is trained based on the color grouping data of the special color correction network.

[0029] A correction execution module is configured to select the general color correction network, the special color correction network or the lightweight color correction network to perform color correction on the pathological image according to the hardware resources and the application scenarios.

[0030] In a third aspect, the present application provides an electronic device comprising a memory and a processor, the memory storing a computer program, and the processor being configured to execute the computer program to perform the pathological image color correction method described above.

[0031] In a fourth aspect, the present application provides a readable storage medium storing a computer program, the computer program comprising program codes for controlling a process to perform the process, the process comprising the pathological image color correction method described above.

[0032] The main contributions and innovations of the present application are as follows:

[0033] 1. Precision improvement: In the training phase, the compatible color loss function is used to automatically divide the color space into several compatible subspaces, and each group uses a dedicated lightweight parameter. The correction accuracy is significantly better than a single global linear matrix and a traditional single network deep model.

[0034] 2. Real-time performance: In the inference phase, a lightweight network (3x3 convolution equivalent to a traditional CCM) or a dedicated network can be flexibly selected according to hardware resources to meet the real-time processing requirements of embedded scanners. A general network can still be used on a high-end server to ensure the highest accuracy.

[0035] 3. Efficient parameters: Through knowledge distillation and grouping compression from general to dedicated to lightweight, the final lightweight network parameter quantity is reduced by more than 90% compared with the general network, while maintaining better correction effect than the traditional CCM.

[0036] 4. Scalability: When adding new staining or optical systems, only a small amount of data needs to be supplemented to correspond to the grouping, without changing the overall framework. The system supports online or offline incremental updates, with low maintenance cost.

[0037] 5. Global consistency: The correction process is always performed on the full image level, avoiding local area color difference problems, ensuring that the color of the entire digital slice is consistent with the real color under the microscope, and improving the accuracy of lesion identification and the repeatability of quantitative analysis.

[0038] The details of one or more embodiments of the present application are presented in the following drawings and description to make other features, objects and advantages of the present application more apparent. BRIEF DESCRIPTION OF DRAWINGS

[0039] The drawings described herein are intended to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their descriptions serve to explain the present application, and do not constitute an improper limitation of the present application. In the drawings:

[0040] Figure 1 is a structure diagram of a multi-module color correction model according to an embodiment of the present application;

[0041] Figure 2 is a schematic diagram of a convolution module according to an embodiment of the present application;

[0042] Figure 3 is a schematic diagram of an encoding module according to an embodiment of the present application;

[0043] Figure 4 is a schematic diagram of a decoding module according to an embodiment of the present application;

[0044] Figure 5 is a flow chart of a use process of a multi-module color correction model according to an embodiment of the present application;

[0045] Figure 6 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers are used to indicate the same or similar elements, unless otherwise represented. The embodiments described in the following exemplary embodiments do not represent all the implementations consistent with one or more embodiments of the present specification. Rather, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of the present specification, as detailed in the appended claims.

[0047] It should be noted that the steps of the corresponding method are not necessarily performed in the order shown and described in the present specification in other embodiments. In some other embodiments, the steps included in the method can be more or less than described in the present specification. In addition, a single step described in the present specification can be divided into multiple steps for description in other embodiments, and multiple steps described in the present specification can be combined into a single step for description in other embodiments.

[0048] The existing linear CCM is difficult to handle nonlinear staining differences, and the existing deep learning scheme uses a single heavy network to deal with all scenarios, resulting in a trade-off between accuracy, real-time performance and versatility.

[0049] Based on this, the present application solves the problems existing in the prior art based on a "general-purpose-special-purpose-lightweight" three-level progressive deep learning framework.

[0050] Embodiment one

[0051] The present application aims to propose a pathological image color correction method, specifically, referring to Figures 1-5 , the method comprises the following steps:

[0052] Step one, construct a multi-level color correction network, such as Figure 1As shown in the figure, the multi-stage color correction network includes a general color correction network, a special color correction network and a lightweight color correction network in turn;

[0053] In this embodiment, the encoding modules of the general color correction network, the special color correction network and the lightweight color correction network are each composed of a plurality of convolution modules, and the convolution module includes a convolution layer, a batch normalization layer and an activation function layer; the decoding modules of the general color correction network and the special color correction network are each composed of a plurality of convolution modules and an up-sampling module. The encoding module and the decoding module of the general color correction network each contain 4 convolution modules; the encoding module and the decoding module of the special color correction network each contain 2 convolution modules; the lightweight color correction network adopts a network structure containing 3 1x1 convolution kernels, and the loss function for training is consistent with that of the general color correction network.

[0054] As shown in the figure, Figure 2 As shown in the figure, the convolution module is the basic unit of the encoding and decoding modules, which is composed of convolution, batch normalization and activation function Relu. Figure 2 In the formula, k, s, p and c are several important hyperparameters. k represents the size of the convolution kernel. s is the step, which represents the interval of the convolution kernel sliding on the input data. p represents the number of additional pixels added around the boundary of the input data. c_in represents the number of channels of the input data or the convolution kernel. The dimension c_out of the output data is the number of convolution kernels. The batch normalization is to normalize each feature dimension respectively, so that the mean is 0 and the variance is 1.

[0055] As shown in the figure, Figure 3 As shown in the figure, the encoding module is composed of a plurality of convolution modules. As shown in the figure, Figure 4 As shown in the figure, the decoding module is composed of a plurality of convolution modules and an up-sampling module. The specific number of convolution modules can be adjusted. For example: the encoding and decoding modules of the general color correction network each contain 4 convolution modules. The encoding and decoding modules of the special color correction network each contain 2 convolution modules.

[0056] In this embodiment, the general color correction network is trained based on the loss function of the difference between the predicted image and the target image through the deep learning network structure containing the encoding module and the decoding module, to learn the high-order color correction parameters suitable for all colors; specifically including:

[0057] First, the general color correction network learns high-order color correction parameters in a high-dimensional space. The model converts the original image into a target image through a series of convolution modules. The color correction network needs to correct various colors. For an image, the loss function is defined as:

[0058]

[0059] In the formula, I predis the corrected (also called rectified) pixel value output by the network, I gt is the pixel value of the corresponding position of the predetermined target image (ideal color image). The formula first calculates the square of the difference value of the corresponding pixels of the two images, and then takes the average (avg) of all pixels to obtain the correction loss P of a single image. The smaller the loss P, the smaller the difference between the corrected image and the target image, and the better the color correction parameters learned by the network. The core of network training is to optimize the parameters by minimizing this loss.

[0060] For the i-th training image, is the image color output by the network after rectification, is the target color (real color in ideal state) corresponding to the image, and the square of the difference value of the corresponding pixels is calculated, and then the average of all pixels is taken, that is, the rectification loss P of a single image is obtained, and is the rectification loss of the i-th image, and the loss function L1 of all image rectifications is:

[0061]

[0062] The loss function Loss1 is the average of the rectification losses of all training images: the single loss of the N images in the training set is summed and then divided by the total number of images N to obtain the average loss of the entire training set.

[0063] In this embodiment, the special color correction network is trained based on the training results of the general color correction network through a deep learning network structure containing a simplified encoding module and a decoding module. During the training process, the features extracted by the special color correction network are consistent with the features extracted by the general color correction network through feature constraints, and the influence of samples with poor rectification effect is reduced through dynamic adjustment of sample weights by a compatible color loss function, color grouping is realized, and multiple sets of correction parameters suitable for different color groups are obtained; specifically including:

[0064] In the case of limited hardware resources, the size of the network needs to be reduced. However, reducing the number of network layers may result in difficulty in using one parameter to accommodate all colors, so a special color correction network is designed to handle different colors. After the general color correction network is trained, the network weights of the module are fixed, and the training of the special color correction network is constrained by the features extracted by the general module, that is, the features extracted by the two modules should be the same. Therefore, the cosine distance is used to measure the similarity of the two feature vectors, and the calculation formula is as follows:

[0065]

[0066] D = 1 N å i=1 D i where D represents the average similarity of the feature vectors extracted by the two networks; N is the total number of training images; i corresponds to the ith training image; is the feature vector extracted by the general color correction network for the ith image (the "reference feature" after training); is the feature vector extracted by the special color correction network for the same image (the "target feature" to be optimized); is a cosine similarity function used to calculate the directional consistency of two feature vectors.

[0067] The core logic of the cosine similarity is to measure the similarity of two vectors by calculating the cosine value of the included angle between them, and the result ranges from -1 to 1. The closer the value is to 1, the more consistent the directions of the two vectors (the more similar the features); the closer the value is to -1, the more opposite the directions (the greater the difference between the features).

[0068] The average value of the cosine similarity for all training images (i.e. ) is obtained, and D reflects the overall consistency of the feature extraction capabilities of the two networks. During the training of the special network, the difference between the feature vectors is minimized (i.e., D is maximized) to ensure that the special network can maintain similar feature extraction capabilities to the general network after simplifying the network structure, thereby inheriting the color correction effect of the general network and laying a foundation for subsequent color grouping and parameter optimization.

[0069] In order to let the network learn the parameters that meet the color correction of most images, the influence of a small part of special color images needs to be reduced. Since the worse the correction effect is, the higher the correction loss is, a weighted loss function is adopted, and the compatible color loss function is calculated as follows:

[0070]

[0071] wherein, is the weight of the ith image, and its core function is to adjust the contribution of the sample in the total loss. For regular samples with good correction effect (w is small), the value is larger, so that it accounts for a higher proportion in the loss calculation; for special samples with poor correction effect (w is large), the value is smaller, reducing its influence on the total loss; the weighted total loss E is obtained by summing the "weight x single loss" and then taking the average.

[0072] The weight of all images is 1 in the first training iteration, and the weight is calculated from the (t-1)th iteration in each subsequent iteration t. The calculation method of w is as follows:

[0073]

[0074] ​in is the maximum and minimum loss of all samples in the current iteration, It is a hyperparameter that decreases linearly from 1 to 0.1 with each iteration. The number of iterations is usually set to 100.

[0075] This dynamic adjustment makes the sample with worse correction effect ( The closer ),That The smaller it is, the weaker the contribution to the loss E; and the sample with good correction effect ( near ), The closer to 1, the stronger the contribution. Through this weighted loss function E, the specialized color correction network can "focus" on the correction needs of most common color images during training, reducing the interference of special color images, thereby learning correction parameters that are more suitable for most scenes. At the same time, this also lays the foundation for subsequent color grouping - special samples with consistently high loss are identified as independent groups, achieving the effect of "compatible colors share a set of parameters, and special colors are grouped separately."

[0076] Therefore, the total loss function of the dedicated color correction network is defined as:

[0077]

[0078] Here, D is the feature consistency constraint, which is the average cosine distance between the feature vectors extracted by the universal color correction network and the specialized color correction network (as described above). Its core function is to ensure that despite the simplified structure of the specialized network (e.g., the number of convolutional modules is reduced), the extracted features remain highly consistent with those of the universal network. This constraint allows the specialized network to "inherit" the color correction capabilities of the universal network. Because the universal network has already learned precise color mapping patterns through its complex structure, the specialized network, by aligning features, can preserve these patterns in a simplified structure, thus avoiding the degradation of correction effectiveness caused by network simplification.

[0079] E stands for the compatible color grouping constraint. E is the compatible color loss function (as described above). By dynamically weighting sample losses, the specialized network focuses on learning the correction parameters for the majority of common color images while reducing the interference from a few special color images. The core of this constraint lays the foundation for "color grouping": by weakening the influence of special samples, the network prioritizes the correction effect of common samples. Special samples that are consistently difficult to correct (high loss) are subsequently identified as independent color groups, ultimately achieving the goal of "compatibility colors share a set of parameters, while special colors are grouped separately."

[0080] b1 and b2: balance hyperparameters b1 and b2 are preset hyperparameters (default values are 1 and 0.1 respectively) for adjusting the weight of D and E in the total loss, balancing the priority of "feature consistency" and "color grouping". Larger b1 ensures that the dedicated network and the general network have high feature consistency, ensuring the inheritance of correction effect; reasonable b2 ensures that the network can effectively distinguish between regular and special colors, providing a basis for subsequent grouping.

[0081] Preferably, the core purpose of color grouping is to separate "difficult to correct with the same set of parameters" colors into different groups, so that the colors in each group can be accurately corrected by the same set of parameters, ultimately reducing the number of parameter sets and improving adaptability. The specific steps are as follows:

[0082] Step 1: Determine the grouping threshold

[0083] In the last iteration of the dedicated color correction network training, calculate the mean and standard deviation of the loss value (based on compatible color loss E) of all training samples, and set the threshold as T = mean + std.

[0084] Where, the mean (mean) reflects the average correction effect of the current parameters on all samples; the standard deviation (std) reflects the dispersion degree (fluctuation range) of the sample loss; the significance of threshold T: the correction effect of the sample whose loss exceeds T is significantly worse than the average level, which belongs to "special color difficult to correct with the current parameters".

[0085] Step 2: Screening and forming new groups

[0086] The samples with loss values greater than T are screened out as the training samples of the next set of color correction parameters. For this new set of samples, retrain the dedicated color correction network (keep the network structure unchanged, only optimize the parameters for the new samples) to obtain new parameters that adapt to the color of this group.

[0087] Step 3: Repeat grouping until termination

[0088] For the remaining samples that have not been screened (loss ≤ T), calculate the mean (mean') and standard deviation (std') of their loss again, determine the new threshold T' = mean' + std', repeat step 2, and screen out the samples with loss greater than T' as the next set of training data.

[0089] This process continues to iterate until the loss variance of the remaining samples is less than the current threshold (i.e. the fluctuation range of the sample loss is small, indicating that the color characteristics of these samples are similar and can be accurately corrected by the same set of parameters), at which point the remaining samples are taken as the last group.

[0090] Through the above process, a number of color groups will be obtained, each corresponding to a set of color correction parameters:

[0091] The sample loss fluctuation within each group is small (color characteristic compatibility), and the same set of parameters can be used for accurate correction.

[0092] Different groups of samples are separated into independent groups due to large color characteristic differences (difficult to correct with the same parameters).

[0093] This automatic grouping mechanism solves the problem of "single parameter difficult to adapt to all colors", and reduces the number of parameter sets through dynamic screening (only separates truly incompatible colors), balancing correction accuracy and the convenience of parameter maintenance.

[0094] In the present embodiment, the lightweight color correction network is trained based on the color grouping data generated by the special color correction network through a further simplified network structure, and the loss function used in the training is Loss1, so as to obtain lightweight correction parameters corresponding to each color group. The lightweight correction parameters are used to realize real-time color correction based on simple matrix operations; preferably, as shown in Figure 1 The rightmost one in the above formula is the lightweight color correction network, which can be set with different numbers and types of convolutions. In the present embodiment, three 1-size convolution kernels are used for convolution, and the calculation process of correction is the same as that of the traditional ccm.

[0095] Preferably, the lightweight color correction network is used to perform color correction with a lighter network structure than the special color correction network to meet the real-time requirement. In order to apply the parameters to the traditional ccm method (linear transformation and nonlinear transformation), the lightweight color correction network structure is variable. In the present embodiment, a single-layer network with three 3x3 convolutions is used as the lightweight color correction network, and the parameters obtained through training correspond to the parameters of linear transformation. (Linear transformation: the three channels of the original image pixel are multiplied by the parameters [r1, r2, r3] to obtain the red channel pixel value after correction, and similarly, [b1, b2, b3] and [g1, g2, g3] are used for blue and green, respectively).

[0096] The lightweight color correction network only trains different color parameters according to the color grouping of the special color correction network, and does not further group.

[0097] Step 2: According to the hardware resources and application scenarios of the pathological section scanner, select the general color correction network, the special color correction network or the lightweight color correction network to perform color correction on the pathological image; wherein, when the special color correction network and the lightweight color correction network are applied, the color group to which the pathological image belongs is determined through a classification model or manual setting, and the correction parameters corresponding to the group are called.

[0098] As shown in Figure 5 , the following steps are taken:

[0099] 1. Input: pathological section

[0100] The process starts with the pathological section to be processed (such as H&E stained section, there is color deviation due to different staining conditions, which needs to be standardized and corrected).

[0101] 2. Judgment: Determine the color category

[0102] First, determine whether the color grouping of the section is known (identified by manual annotation or pre-trained classification model):

[0103] Yes: Enter the "grouping adaptation process" (use grouping parameters to improve efficiency);

[0104] No: Enter the "general correction process" (no grouping information, rely on the first network (general color correction network) to cover the bottom).

[0105] 3. Branch A: Color category can be determined (grouping is known)

[0106] 3.1 Match color parameters:

[0107] Determine the color grouping to which the section belongs and the corresponding correction parameters by manual selection (manual assignment of grouping) or automatic identification by classification network.

[0108] 3.2 Judgment: Whether to scan the section and correct the color at the same time

[0109] Determine whether real-time processing is required (such as digital pathology scanner needs to "scan and output corrected image at the same time"):

[0110] Yes: Call the third network (lightweight color correction network) to complete "scanning + correction" simultaneously (meet real-time requirements) using its fast inference speed;

[0111] No: Scan the section first (get the original image), then use the second network (special color correction network) to correct (low real-time requirement, priority to ensure accuracy).

[0112] 3.3 Lightweight network correction & accuracy verification:

[0113] Determine whether the color is accurate (judged by comparison with standard template, error index, etc.) for the correction result of the lightweight network:

[0114] Yes: Directly output the "color corrected image", the process is completed;

[0115] No: The accuracy of the lightweight network is insufficient, upgrade to call the second network (special color correction network) (optimized for this grouping, accuracy is better than the lightweight network).

[0116] 3.4 Special network correction & accuracy verification:

[0117] The correction result of the second network (special color correction network) is used to determine whether the color is accurate again:

[0118] → Yes: output the "color corrected image", and the process ends.

[0119] → No: the special network still cannot be adapted, and finally the first network (general color correction network) is called (the structure is complex and the precision is the highest, serving as a bottom solution).

[0120] 4. Branch B: color category cannot be determined (group unknown)

[0121] Directly execute the scanning slice (get the original image), and then call the first network (general network):

[0122] Without group information, the general network covers all colors through a set of parameters to ensure basic correction effect (sacrifice part of efficiency for universality).

[0123] 5. Final output: color corrected image

[0124] No matter which network is used, the final output is a standardized pathological image (meeting the requirement of consistency of nuclear, cytoplasm and other colors for diagnosis).

[0125] Preferably, the third network will be used for pathological slice scanning. Before scanning, it is necessary to determine which set of color parameter determination method to use: manually select parameters or train a classification model with grouped color data. Focusing will be performed before scanning, and multiple focus images similar to the training data tiles can be obtained. Classify these focus images and select the parameters corresponding to the most categories for color correction.

[0126] After scanning, the original color image and the corrected image will be saved. When the original color becomes significantly different from the training image due to changes in optical systems, dyes and other factors, the third network may not correct the color correctly. At this time, the second network is used and the original image is manually selected for color correction.

[0127] When a new color is encountered, the first network is directly used for correction. If the correction effect is not good, the image of the new color can be added as training data to retrain the three networks. A new set of second or third network parameters can also be trained for the new color.

[0128] Embodiment two

[0129] Based on the same idea, the present application also proposes a pathological image color correction device, comprising:

[0130] A network construction module is used to construct a general color correction network, a special color correction network and a lightweight color correction network in sequence.

[0131] a training module configured to train a general color correction network, a special color correction network and a lightweight color correction network respectively, wherein the special color correction network is implemented for color grouping based on a feature constraint and a compatible color loss function of the general color correction network, and the lightweight color correction network is trained based on color grouping data of the special color correction network;

[0132] a correction execution module configured to select the general color correction network, the special color correction network or the lightweight color correction network to perform color correction on a pathological image according to hardware resources and application scenarios.

[0133] Embodiment Three

[0134] The embodiment also provides an electronic device, referring to Figure 6 including a memory 404 and a processor 402, the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to perform the steps in any of the above method embodiments.

[0135] Specifically, the processor 402 can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0136] The memory 404 can include a mass storage that stores data or instructions. For example, and without limitation, the memory 404 can include a Hard Disk Drive (HDD), a floppy disk drive, a Solid State Drive (SSD), a flash drive, a Compact Disc Read Only Memory (CD-ROM), a magneto-optical disk, a magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. The memory 404 can be removable and / or non-removable (or fixed) as appropriate. The memory 404 can be internal or external as appropriate. In particular embodiments, the memory 404 is a Non-Volatile memory. In particular embodiments, the memory 404 includes a Read-Only Memory (ROM) and a Random Access Memory (RAM). The ROM can be a mask-programmed ROM, a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), an Electrically Alterable ROM (EAROM), or a FLASH memory, or a combination of two or more of these, as appropriate. The RAM can be a Static Random-Access Memory (SRAM) or a Dynamic Random Access Memory (DRAM), which can be a Fast Page Mode Dynamic Random Access Memory (FPMDRAM), an Extended Data Output Dynamic Random Access Memory (EDODRAM), a Synchronous Dynamic Random-Access Memory (SDRAM), or the like, as appropriate.

[0137] The memory 404 can be used to store or buffer various data files needed for processing and / or communication, and possible computer program instructions executed by the processor 402.

[0138] The processor 402 implements any one of the pathological image color correction methods in the above embodiments by reading and executing the computer program instructions stored in the memory 404.

[0139] Optionally, the electronic device described above can further include a transmission device 406 connected with the processor 402 and an input / output device 408 connected with the processor 402.

[0140] The transmission device 406 can be used to receive or send data via a network. Specific examples of the network can include a wired or wireless network provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (NIC) which can be connected with other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 406 can be a radio frequency (RF) module which is used to communicate with the Internet in a wireless manner.

[0141] The input / output device 408 is used to input or output information.

[0142] Embodiment Four

[0143] The embodiment also provides a readable storage medium, and the readable storage medium stores a computer program. The computer program includes program codes for controlling a process to execute the process. The process includes the pathological image color correction method according to Embodiment One.

[0144] It should be noted that specific examples in the embodiment can refer to examples described in the above embodiments and optional implementation manners, and the embodiment will not be described here.

[0145] Generally, various embodiments can be implemented in hardware or special-purpose circuitry, software, logic or any combination thereof. Some aspects of the application can be implemented in hardware, while other aspects can be implemented using firmware or software executed by a controller, microprocessor or other computing device, but the application is not limited thereto. Although various aspects of the application can be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein can be implemented in hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

[0146] Embodiments of the application can be implemented by computer software executable by a data processor of the mobile device such as in the processor entity, or by hardware, or by a combination of software and hardware. Computer software or program, also called program product, including software routines, applets and / or macros, can be stored in any apparatus-readable data storage medium and they include program instructions to implement certain tasks. The program product can include one or more computer-executable components such as the components as shown in the logic flow of FIG. 10. The one or more computer-executable components can be one or more Figure 5 Any block in the logic flow of the methods described herein, including those of the flowchart illustrations and / or the diagrams, can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that the

[0147] It should be understood that all the technical features of the above embodiments can be combined in any manner, and in order to make the description concise, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not contradict, they should be considered as the scope of the present application.

[0148] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the present application. It should be pointed out that for those skilled in the art, a number of modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A pathological image color correction method, characterized in that: The following steps are involved: Constructing a multi-stage color correction network, wherein the multi-stage color correction network includes a universal color correction network, a dedicated color correction network, and a lightweight color correction network in a progressive manner; The universal color correction network is trained using a deep learning network structure comprising an encoding module and a decoding module based on a loss function of the difference between the predicted image and the target image to learn high-order color correction parameters applicable to all colors; The dedicated color correction network is trained based on the training results of the universal color correction network through a deep learning network structure including a simplified encoding module and a decoding module. During the training process, feature constraints are used to ensure that the features extracted by the dedicated color correction network are consistent with the features extracted by the universal color correction network. Sample weights are dynamically adjusted using a compatible color loss function to reduce the influence of samples with poor correction effects, thereby achieving color grouping and obtaining multiple sets of correction parameters suitable for different color groupings. The lightweight color correction network is trained based on the color grouping data generated by the dedicated color correction network through a further simplified network structure to obtain lightweight correction parameters corresponding to each color group. The lightweight correction parameters are used to implement real-time color correction based on simple matrix operations. The color grouping of the dedicated color correction network is achieved by screening samples with correction losses exceeding a threshold as new group training samples, and repeating until the loss variance of the remaining samples is less than the threshold, where the threshold is the sum of the mean and standard deviation of the losses of all samples. According to the hardware resources and application scenarios of the pathology slide scanner, the general color correction network, the dedicated color correction network or the lightweight color correction network is selected to perform color correction on the pathology image; Wherein, when the dedicated color correction network and the lightweight color correction network are applied, the color group to which the pathological image belongs is determined through a classification model or manual setting, and the correction parameters of the corresponding group are called.

2. The pathological image color correction method according to claim 1, wherein: The encoding modules of the universal color correction network, the dedicated color correction network and the lightweight color correction network are all composed of several convolution modules, and the convolution modules include convolution layers, batch normalization layers and activation function layers; the decoding modules of the universal color correction network and the dedicated color correction network are composed of several convolution modules and upsampling modules.

3. The pathological image color correction method according to claim 1, wherein: The encoding module and decoding module of the general color correction network each include 4 convolution modules; the encoding module and decoding module of the dedicated color correction network each include 2 convolution modules; The lightweight color correction network adopts a network structure including three 1×1 convolution kernels, and its training loss function is consistent with the loss function of the universal color correction network.

4. The pathological image color correction method according to claim 1, wherein: The loss function of the universal color correction network is a loss function based on the mean square error between the predicted image and the target image.

5. The pathological image color correction method according to claim 1, wherein: The feature constraint of the dedicated color correction network is implemented by measuring the similarity of feature vectors extracted by the dedicated color correction network and the universal color correction network through cosine distance.

6. The pathological image color correction method according to claim 1, wherein: In the compatible color loss function of the dedicated color correction network, the sample weights are dynamically adjusted with training iterations, and the adjustment method is calculated based on the maximum and minimum values ​​of the correction loss and hyperparameters of the previous iteration.

7. A pathological image color correction device, characterized in that: include: Network construction module, used to construct a general color correction network, a dedicated color correction network, and a lightweight color correction network in sequence; A training module, configured to respectively train the universal color correction network, the dedicated color correction network, and the lightweight color correction network, wherein the universal color correction network is trained based on a loss function of the difference between the predicted image and the target image through a deep learning network structure including an encoding module and a decoding module, so as to learn high-order color correction parameters applicable to all colors; The dedicated color correction network is trained based on the training results of the universal color correction network through a deep learning network structure including a simplified encoding module and a decoding module. During the training process, feature constraints are used to ensure that the features extracted by the dedicated color correction network are consistent with the features extracted by the universal color correction network. Sample weights are dynamically adjusted using a compatible color loss function to reduce the influence of samples with poor correction effects, thereby achieving color grouping and obtaining multiple sets of correction parameters suitable for different color groupings. The lightweight color correction network is trained based on the color grouping data generated by the dedicated color correction network through a further simplified network structure to obtain lightweight correction parameters corresponding to each color group. The lightweight correction parameters are used to implement real-time color correction based on simple matrix operations. The color grouping of the dedicated color correction network is achieved by screening samples with correction losses exceeding a threshold as new group training samples, and repeating until the loss variance of the remaining samples is less than the threshold, where the threshold is the sum of the mean and standard deviation of the losses of all samples. A correction execution module, configured to select the general color correction network, the dedicated color correction network, or the lightweight color correction network to perform color correction on the pathological image according to the hardware resources and application scenario of the pathological slide scanner; Wherein, when the dedicated color correction network and the lightweight color correction network are applied, the color group to which the pathological image belongs is determined through a classification model or manual setting, and the correction parameters of the corresponding group are called.

8. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to run the computer program to perform the pathological image color correction method according to any one of claims 1 to 6.

9. A readable storage medium, characterized in that: The readable storage medium stores a computer program, wherein the computer program includes a program code for controlling a process to execute a process, wherein the process includes the pathological image color correction method according to any one of claims 1 to 6.

Images