Pathological image color correction method and device and readable storage medium thereof

Abstract

The invention provides a pathological image color correction method and device and a readable storage medium, and aims to solve the problems that in the prior art, a traditional linear method is difficult to fit nonlinear color mapping and depends on manual parameter adjustment, and a deep learning method is poor in adaptability and insufficient in real-time performance. According to the invention, a three-stage progressive deep learning network is designed; a general color correction network learns high-order parameters through a complex structure to realize full-color correction; the special color correction network realizes automatic color grouping based on universal network feature constraints in combination with a compatible color loss function, and parameters are reduced to adapt to limited hardware; the lightweight color correction network further simplifies the structure, and realizes efficient real-time correction based on packet parameters. The three-level network can be flexibly selected according to hardware resources, adaptation to various dyeing and equipment is achieved, correction accuracy and real-time performance are balanced, reliability of pathological image analysis is improved, and the method is suitable for various pathological section and scanner scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and in particular to color correction of pathological images, and is particularly suitable for color calibration and optimization of digital pathological images acquired by pathological slice scanners. Background Art

[0002] After chemical fixation, paraffin embedding, and staining, pathology sections are scanned with digital slide scanners to obtain full-field digital images. Due to variations in the spectral response of the scanner's optical system, as well as factors such as staining batches, reagent types, and section thickness, the colors in the digital images can differ significantly from the true colors seen under the microscope. This directly impacts the pathologist's assessment of the lesion area, degree of tissue differentiation, and the strength of immunohistochemical expression.

[0003] Traditional color correction methods primarily use a linear color correction matrix (CCM). This method linearly maps the original RGB values ​​to the target color space using a 3×3 or 3×4 matrix. This approach is computationally simple and easy to deploy. However, pathological slide staining exhibits a highly nonlinear distribution, and linear matrices cannot accurately model the complex spectrum-color mapping, often resulting in oversaturation of some hues while others remain deviated. Furthermore, CCM parameters require repeated manual adjustment based on color charts, relying on experience and inefficiency.

[0004] In recent years, nonlinear color correction methods based on convolutional neural networks have been proposed. These methods can learn complex nonlinear mappings through end-to-end training, significantly improving correction accuracy. However, existing deep learning solutions still have the following shortcomings: 1. Using a single network and the same set of parameters to handle all dyeing and equipment differences makes it difficult to meet diverse needs; 2. Some methods only perform color correction on local areas, ignoring global color consistency, resulting in significant color differences in different areas of the same image; 3. Deep models have large number of parameters and high computational complexity, making it difficult to meet low latency requirements in scanner embedded hardware or real-time film reading scenarios. 4. When adding new dyes or replacing the optical system, a large amount of paired data needs to be re-collected and the entire network retrained, which results in high maintenance costs.

[0005] Therefore, how to achieve parameter lightweighting, model adaptive updating and real-time inference while ensuring correction accuracy has become a technical challenge that needs to be solved urgently in the field of digital pathology. Summary of the Invention

[0006] The embodiments of the present invention provide a method and device for color correction of pathological images and a readable storage medium thereof. These methods address the problems that existing linear CCMs are difficult to handle with nonlinear staining differences, while existing deep learning solutions use a single heavyweight network to handle all scenarios, resulting in the inability to achieve both accuracy, real-time performance, and versatility.

[0007] The core technology of this invention is to propose a three-level progressive deep learning framework of "general-purpose-specialized-lightweight". Through automatic color grouping and model distillation during training, a scalable, customizable, and real-time deployable pathological image color correction system is realized.

[0008] In a first aspect, the present invention provides a method for color correction of a pathological image, the method comprising the following steps: Construct a multi-level color correction network, which includes a general color correction network, a dedicated color correction network, and a lightweight color correction network in sequence; The universal color correction network is trained through a deep learning network structure consisting of an encoding module and a decoding module based on a loss function that measures the difference between the predicted image and the target image to learn high-order color correction parameters applicable to all colors. The dedicated color correction network is trained based on the training results of the universal color correction network through a deep learning network structure that includes simplified encoding and decoding modules. During the training process, feature constraints are used to ensure that the features extracted by the dedicated color correction network are consistent with those extracted by the universal color correction network. The sample weights are dynamically adjusted using a compatible color loss function to reduce the influence of samples with poor correction effects, thereby achieving color grouping and obtaining multiple sets of correction parameters suitable for different color groupings. The lightweight color correction network is based on the color grouping data generated by the dedicated color correction network. Through a further simplified network structure, it is trained to obtain lightweight correction parameters corresponding to each color group. The lightweight correction parameters are used to achieve real-time color correction based on simple matrix operations. According to the hardware resources and application scenarios of the pathology slide scanner, select a general color correction network, a dedicated color correction network, or a lightweight color correction network to perform color correction on pathology images; Among them, when the dedicated color correction network and the lightweight color correction network are applied, the color group to which the pathological image belongs is determined through a classification model or manual settings, and the correction parameters of the corresponding group are called.

[0009] Furthermore, the encoding modules of the general color correction network, the dedicated color correction network and the lightweight color correction network are all composed of several convolution modules, which include convolution layers, batch normalization layers and activation function layers; the decoding modules of the general color correction network and the dedicated color correction network are composed of several convolution modules and upsampling modules.

[0010] Furthermore, the encoding module and decoding module of the general color correction network each contain 4 convolution modules; the encoding module and decoding module of the dedicated color correction network each contain 2 convolution modules; The lightweight color correction network adopts a network structure containing three 1×1 convolution kernels, and its training loss function is consistent with the loss function of the general color correction network.

[0011] Furthermore, the loss function of the universal color correction network is a loss function based on the mean square error between the predicted image and the target image.

[0012] Furthermore, the feature constraint of the dedicated color correction network is implemented by measuring the similarity of the feature vectors extracted by the dedicated color correction network and the universal color correction network through the cosine distance.

[0013] Furthermore, in the compatible color loss function of the dedicated color correction network, the sample weights are dynamically adjusted with training iterations, and the adjustment method is calculated based on the maximum and minimum values ​​of the correction loss and hyperparameters of the previous iteration.

[0014] Furthermore, the color grouping of the dedicated color correction network is achieved by screening samples whose correction loss exceeds a threshold as new group training samples, and repeating until the loss variance of the remaining samples is less than a threshold, where the threshold is the sum of the mean and standard deviation of the losses of all samples.

[0015] In a second aspect, the present invention provides a pathological image color correction device, comprising: Network construction module, used to construct a general color correction network, a dedicated color correction network, and a lightweight color correction network in sequence; A training module is used to train a universal color correction network, a dedicated color correction network, and a lightweight color correction network respectively. The dedicated color correction network implements color grouping based on the feature constraints and compatible color loss function of the universal color correction network, and the lightweight color correction network is trained based on the color grouping data of the dedicated color correction network. The correction execution module is used to select a general color correction network, a dedicated color correction network or a lightweight color correction network to perform color correction on pathological images based on hardware resources and application scenarios.

[0016] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the above-mentioned pathological image color correction method.

[0017] In a fourth aspect, the present invention provides a readable storage medium storing a computer program, wherein the computer program includes a program code for controlling a process to execute a process, wherein the process includes the pathological image color correction method described above.

[0018] The main contributions and innovations of the present invention are as follows: 1. Improved accuracy: During the training phase, the color space is automatically divided into several compatible subspaces using a compatible color loss function. Each group uses dedicated lightweight parameters, resulting in significantly better correction accuracy than a single global linear matrix and traditional single-network deep models.

[0019] 2. Real-time performance: During the inference phase, lightweight networks (3×3 convolution is equivalent to traditional CCM) or dedicated networks can be flexibly selected based on hardware resources to meet the real-time processing requirements of embedded scanners. On high-end servers, general-purpose networks can still be used to ensure the highest accuracy.

[0020] 3. Parameter efficiency: Through knowledge distillation and group compression from "general → specialized → lightweight", the number of parameters in the final lightweight network is reduced by more than 90% compared to the general network, while maintaining a correction effect that is superior to traditional CCM.

[0021] 4. Scalability: When adding a new dyeing or optical system, only a small amount of data needs to be added and regrouped, without changing the overall framework; the system supports online or offline incremental updates, with low maintenance costs.

[0022] 5. Global consistency: The correction process is always performed at the full image level to avoid localized color differences and ensure that the color of the entire digital slide is consistent with the actual color under the microscope, improving the accuracy of lesion identification and the repeatability of quantitative analysis.

[0023] The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below so that other features, objects, and advantages of the invention are more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 is a structural diagram of a multi-module color correction model according to an embodiment of the present invention; Figure 2 is a schematic diagram of a convolution module according to an embodiment of the present invention; Figure 3 is a schematic diagram of an encoding module according to an embodiment of the present invention; Figure 4 is a schematic diagram of a decoding module according to an embodiment of the present invention; Figure 5 is a flowchart of using a multi-module color correction model according to an embodiment of the present invention; Figure 6 FIG. 4 is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments are not intended to represent all implementations consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of one or more embodiments of this specification, as detailed in the appended claims.

[0026] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than those described in this specification. In addition, a single step described in this specification may be broken down into multiple steps for description in other embodiments, and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0027] Existing linear CCMs have difficulty handling nonlinear coloring differences, while existing deep learning solutions use a single heavyweight network to handle all scenarios, resulting in a lack of accuracy, real-time performance, and versatility.

[0028] Based on this, the present invention solves the problems existing in the prior art based on a three-level progressive deep learning framework of "general-purpose-special-lightweight".

[0029] Example 1 The present invention aims to propose a color correction method for pathological images, specifically, referring to Figure 1-Figure 5 , the method comprises the following steps: Step 1: Build a multi-level color correction network, such as Figure 1 As shown, the multi-stage color correction network includes a general color correction network, a dedicated color correction network and a lightweight color correction network in sequence; In this embodiment, the encoding modules of the general color correction network, the specialized color correction network, and the lightweight color correction network are all composed of several convolutional modules, each comprising a convolutional layer, a batch normalization layer, and an activation function layer. The decoding modules of the general color correction network and the specialized color correction network are composed of several convolutional modules and an upsampling module. The encoding and decoding modules of the general color correction network each contain four convolutional modules; the encoding and decoding modules of the specialized color correction network each contain two convolutional modules. The lightweight color correction network uses a network structure containing three 1×1 convolutional kernels, and its training loss function is consistent with the loss function of the general color correction network.

[0030] Among them, Figure 2As shown in Figure 1, the convolution module is the basic unit of the encoding and decoding modules, which consists of convolution, batch normalization and activation function ReLU. Figure 2 In this article, k, s, p, and c are important hyperparameters. k represents the size of the convolution kernel. s is the stride, which indicates the interval at which the convolution kernel slides over the input data. p represents the number of extra pixels added around the input data boundary. c_in represents the number of channels in the input data or convolution kernel. The dimension of the output data, c_out, is the number of convolution kernels. Batch normalization normalizes each feature dimension individually to have a mean of 0 and a variance of 1.

[0031] like Figure 3 As shown in , the encoding module consists of several convolution modules. Figure 4 As shown, the decoding module consists of several convolutional modules and an upsampling module. The specific number of convolutional modules can be adjusted. For example, the encoding and decoding modules of the general color correction network each contain four convolutional modules. The encoding and decoding modules of the specialized color correction network each contain two convolutional modules.

[0032] In this embodiment, a universal color correction network is trained using a deep learning network structure comprising an encoding module and a decoding module based on a loss function representing the difference between a predicted image and a target image to learn high-order color correction parameters applicable to all colors. Specifically, the network includes: First, the universal color correction network learns high-order color correction parameters in a high-dimensional space. The model transforms the original image into the target image through a series of convolutional modules. The color correction network needs to correct various colors. For an image, the loss function is defined as:

[0033] Among them, I pred It is the pixel value after correction (also called rectification) output by the network, I gt is the pixel value at the corresponding position in the predetermined target image (ideal color image). This formula first calculates the square of the difference between the corresponding pixels in the two images and then takes the average value (avg) of all pixels to obtain the correction loss P for a single image. The smaller the loss P, the smaller the difference between the corrected image and the target image, and the better the color correction parameters learned by the network. The core of network training is to optimize the parameters by minimizing this loss.

[0034] For the i-th training image, is the output image color after network correction, is the target color corresponding to the image (the real color under ideal conditions), and is obtained by calculating the square difference between the corresponding pixels of the two , and then take the average of the values ​​of all pixels to get the correction loss P of a single image, and is the correction loss of the i-th image, then the loss function L1 of all image corrections is:

[0035] The loss function Loss1 averages the correction loss of all training images: the loss of each image in the training set is After summing, divide it by the total number of images N to get the average loss of the entire training set.

[0036] In this embodiment, a dedicated color correction network is trained based on the training results of a universal color correction network using a deep learning network structure comprising simplified encoding and decoding modules. During the training process, feature constraints are used to ensure that the features extracted by the dedicated color correction network are consistent with those extracted by the universal color correction network. A compatible color loss function is used to dynamically adjust sample weights to reduce the impact of samples with poor correction effects, thereby achieving color grouping and obtaining multiple sets of correction parameters suitable for different color groupings. Specifically, the following steps are involved: Given limited hardware resources, the network size needs to be reduced. However, reducing the number of network layers may make it difficult to use a single parameter for all colors. Therefore, a dedicated color correction network is designed to handle different colors. After the general color correction network is trained, the network weights of this module are fixed, and the features extracted by the general module are used to constrain the training of the dedicated color correction network. That is, the features extracted by the two modules should be the same. Therefore, the cosine distance is used to measure the similarity of two feature vectors. The calculation formula is as follows:

[0037] Where D represents the average similarity of the feature vectors extracted by the two networks; N is the total number of training images; i corresponds to the i-th training image; is the feature vector extracted by the universal color correction network for the i-th image (the trained “benchmark feature”); is the feature vector extracted by the dedicated color correction network for the same image (the "target feature" to be optimized); is the cosine similarity function, which is used to calculate the directional consistency of two feature vectors.

[0038] The core logic of cosine similarity is to measure the similarity between two vectors by calculating the cosine of the angle between them. The result range is [-1, 1]. The closer the value is to 1, the more consistent the two vectors are (the more similar their features are); the closer it is to -1, the more opposite the two vectors are (the more different their features are).

[0039] The cosine similarity of all training images is averaged (i.e. ), the resulting D reflects the overall consistency of the two networks' feature extraction capabilities. During specialized network training, by minimizing the difference in feature vectors (i.e., maximizing D), the specialized network maintains similar feature extraction capabilities to the general network even after simplifying its structure. This ensures that the specialized network inherits the general network's color correction effects, laying the foundation for subsequent color grouping and parameter optimization.

[0040] In order for the network to learn parameters that meet the color correction requirements for most images, it is necessary to reduce the impact of a small number of images with special colors. Since the worse the correction effect, the higher the correction loss. Therefore, a weighted loss function is used. The calculation formula for the compatible color loss function is:

[0041] in, is the weight of the i-th image, and its core function is to adjust the contribution of the sample in the total loss - for conventional samples with good correction effect ( Small), The larger the value, the higher the proportion in the loss calculation; for special samples with poor correction effect ( big), The smaller the value, the lower the impact on the total loss; the overall average is calculated by summing the weight × single loss ( ), and get the weighted total loss E.

[0042] In the first training iteration, all images have a weight of 1. For each subsequent iteration t, the weight is calculated from t-1 iterations. The calculation method of w is:

[0043] in is the maximum and minimum loss of all samples in the current iteration, It is a hyperparameter that decreases linearly from 1 to 0.1 with each iteration. The number of iterations is usually set to 100.

[0044] This dynamic adjustment makes the sample with worse correction effect ( The closer ),That The smaller it is, the weaker the contribution to the loss E; and the sample with good correction effect ( near ), The closer to 1, the stronger the contribution. Through this weighted loss function E, the specialized color correction network can "focus" on the correction needs of most common color images during training, reducing the interference of special color images, thereby learning correction parameters that are more suitable for most scenes. At the same time, this also lays the foundation for subsequent color grouping - special samples with consistently high loss are identified as independent groups, achieving the effect of "compatible colors share a set of parameters, and special colors are grouped separately."

[0045] Therefore, the total loss function of the dedicated color correction network is defined as:

[0046] Here, D is the feature consistency constraint, which is the average cosine distance between the feature vectors extracted by the universal color correction network and the specialized color correction network (as described above). Its core function is to ensure that despite the simplified structure of the specialized network (e.g., the number of convolutional modules is reduced), the extracted features remain highly consistent with those of the universal network. This constraint allows the specialized network to "inherit" the color correction capabilities of the universal network. Because the universal network has already learned precise color mapping patterns through its complex structure, the specialized network, by aligning features, can preserve these patterns in a simplified structure, thus avoiding the degradation of correction effectiveness caused by network simplification.

[0047] E stands for the compatible color grouping constraint. E is the compatible color loss function (as described above). By dynamically weighting sample losses, the specialized network focuses on learning the correction parameters for the majority of common color images while reducing the interference from a few special color images. The core of this constraint lays the foundation for "color grouping": by weakening the influence of special samples, the network prioritizes the correction effect of common samples. Special samples that are consistently difficult to correct (high loss) are subsequently identified as independent color groups, ultimately achieving the goal of "compatibility colors share a set of parameters, while special colors are grouped separately."

[0048] b1 and b2: Balance hyperparameters b1 and b2 are preset hyperparameters (default values ​​are 1 and 0.1, respectively) that adjust the weight of D and E in the total loss, balancing the priorities of "feature consistency" and "color grouping." A large b1 ensures high feature consistency between the specialized and general networks, ensuring the continuity of the correction effect; a reasonable b2 ensures that the network can effectively distinguish between common and special colors, providing a basis for subsequent grouping.

[0049] Preferably, the core purpose of color grouping is to separate colors that are "difficult to calibrate with the same set of parameters" into different groups, so that the colors within each group can be accurately calibrated with the same set of parameters, ultimately reducing the number of parameter sets and improving adaptability. The specific steps are as follows: Step 1: Determine the grouping threshold In the last iteration of training the dedicated color correction network, the mean (mean) and standard deviation (std) of the loss values ​​(based on the compatible color loss E) of all training samples are calculated, and the threshold is set to T=mean+std.

[0050] Among them, the mean reflects the average correction effect of the current parameters on all samples; the standard deviation (std) reflects the discrete degree (fluctuation range) of the sample loss; the significance of the threshold T is that the correction effect of samples whose loss exceeds T is significantly worse than the average level, and they belong to "special colors that are difficult to correct with the current parameters."

[0051] Step 2: Filter and form new groups Samples with loss values ​​greater than T are selected as training samples for the next set of color correction parameters. For this new set of samples, the dedicated color correction network is retrained (keeping the network structure unchanged and optimizing parameters only for the new samples) to obtain new parameters that are suitable for this set of colors.

[0052] Step 3: Repeat grouping until termination For the remaining unscreened samples (loss ≤ T), calculate the mean (mean') and standard deviation (std') of their loss again, determine the new threshold T'=mean'+std', repeat step 2, and screen out samples with loss greater than T' as the next set of training data.

[0053] This process continues iterating until the loss variance of the remaining samples is less than the current threshold (that is, the fluctuation range of sample loss is very small, indicating that the color characteristics of these samples are similar and can be accurately corrected by the same set of parameters). At this time, the remaining samples are taken as the last group.

[0054] Through the above process, we will eventually get several color groups, each of which corresponds to a set of color correction parameters: The sample loss fluctuation within each group is small (color characteristics are compatible) and can be accurately corrected using the same set of parameters; Samples from different groups were separated into independent groups because of their large differences in color characteristics (difficult to correct using the same parameters).

[0055] This automatic grouping mechanism solves the problem of "a single parameter is difficult to adapt to all colors". At the same time, it reduces the number of parameter sets through dynamic screening (only separating truly incompatible colors), taking into account both correction accuracy and convenience of parameter maintenance.

[0056] In this embodiment, the lightweight color correction network is trained based on the color grouping data generated by the dedicated color correction network through a further simplified network structure. The training loss function uses Loss1 to obtain lightweight correction parameters corresponding to each color group. The lightweight correction parameters are used to achieve real-time color correction based on simple matrix operations. Preferably, Figure 1 The rightmost part of the image is a lightweight color correction network, which can be configured with different numbers and types of convolutions. The present invention uses three convolution kernels of size 1 for convolution. The correction calculation process is the same as that of traditional CCM.

[0057] Preferably, the lightweight color correction network performs color correction using a more lightweight network structure than dedicated color correction networks to meet real-time requirements. To enable the application of parameters to traditional CCM methods (linear and nonlinear transformations), the lightweight color correction network structure is variable. The present invention uses three single-layer networks with 3x3 convolutions as the lightweight color correction network. The trained parameters correspond precisely to the parameters of the linear transformation. (Linear transformation: The three channels of the original image pixel are multiplied by the parameters [r1, r2, r3] to obtain the corrected red channel pixel value for the corresponding pixel. Similarly, [b1, b2, b3] and [g1, g2, g3] are used for blue.)

[0058] The lightweight color correction network will only train different color parameters according to the grouped colors of the dedicated color correction network without further grouping.

[0059] Step 2: Based on the hardware resources and application scenarios of the pathology slide scanner, select a general color correction network, a dedicated color correction network, or a lightweight color correction network to perform color correction on the pathology image. When the dedicated color correction network and the lightweight color correction network are applied, the color group to which the pathology image belongs is determined through a classification model or manual settings, and the correction parameters of the corresponding group are called.

[0060] like Figure 5 As shown, the following steps: 1. Input: Pathology slides The process starts with the pathological slides to be processed (such as H&E stained slides, which have color deviations due to differences in staining conditions and need to be standardized and corrected).

[0061] 2. Judgment: Determine the color category First, determine whether the color grouping of the slices is known (identified by manual annotation or pre-trained classification model): →Yes: Enter the "Group Adaptation Process" (use group parameters to improve efficiency); → No: Enter the "Universal Calibration Process" (no grouping information, relying on the first network (Universal Color Calibration Network) as a backup).

[0062] 3. Branch A: Color categories can be determined (grouping is known) 3.1 Matching color parameters: The color group to which the slice belongs and the corresponding correction parameters are determined by manual selection (manually specified grouping) or automatic identification by the classification network.

[0063] 3.2 Determination: whether to scan slices and correct colors at the same time Determine whether real-time processing is required (e.g., a digital pathology scanner needs to output corrected images while scanning): →Yes: Call the third network (the lightweight color correction network) and use its fast inference speed to complete "scanning + correction" simultaneously (meeting real-time requirements); → No: Scan the slice first (to get the original image), and then use the second network (dedicated color correction network) for correction (low real-time requirements, priority to ensure accuracy).

[0064] 3.3 Lightweight network calibration & accuracy verification: For the correction results of the lightweight network, determine whether the color is accurate (by comparing with the standard template, error indicators, etc.): →Yes: Directly output the "color-corrected image" and the process ends; → No: The lightweight network is not accurate enough, so the second network (dedicated color correction network) is upgraded and called (optimized for this group and more accurate than the lightweight network).

[0065] 3.4 Dedicated network calibration & accuracy verification: For the correction results of the second network (dedicated color correction network), we can again judge whether the color is accurate: →Yes: Output the "color-corrected image" and the process ends; → No: The dedicated network still cannot be adapted, and the first network (universal color correction network) is finally called (with a complex structure and the highest accuracy, serving as a backup solution).

[0066] 4. Branch B: Color category cannot be determined (grouping is unknown) Directly perform the scan slice (get the original image), and then call the first network (general network): Since there is no grouping information, the general network covers all colors through a set of parameters to ensure the basic correction effect (sacrificing some efficiency in exchange for universality).

[0067] 5. Final output: color-corrected image Regardless of which network is used, the final output is a standardized pathological image (which meets the diagnostic requirements for color consistency of the nucleus, cytoplasm, etc.).

[0068] Pathology slide scanning preferably utilizes a third network. Before scanning, a decision must be made regarding which color parameter determination method to use: manually selecting parameters or training a classification model using grouped color data. Focusing is performed before scanning, resulting in multiple focused images similar to the training data tiles. These focused images are then classified, and the parameters corresponding to the most common category are selected for color correction.

[0069] After scanning is complete, both the original color image and the corrected image are saved. If factors such as optical system and dyes cause the original color to differ significantly from the training image, the corrected color produced by the third network may be incorrect. In this case, the second network is used with manually selected parameters to perform color correction on the original image.

[0070] When encountering a new color, the first network is used for correction. If the correction is not effective, images with the new color can be added as training data and all three networks can be retrained. Alternatively, a new set of parameters for the second or third network can be trained for the new color.

[0071] Example 2 Based on the same concept, the present invention also proposes a pathological image color correction device, comprising: Network construction module, used to construct a general color correction network, a dedicated color correction network, and a lightweight color correction network in sequence; A training module is used to train a universal color correction network, a dedicated color correction network, and a lightweight color correction network respectively. The dedicated color correction network implements color grouping based on the feature constraints and compatible color loss function of the universal color correction network, and the lightweight color correction network is trained based on the color grouping data of the dedicated color correction network. The correction execution module is used to select a general color correction network, a dedicated color correction network or a lightweight color correction network to perform color correction on pathological images based on hardware resources and application scenarios.

[0072] Example 3 This embodiment also provides an electronic device, referring to Figure 6 , includes a memory 404 and a processor 402, wherein the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to perform the steps in any of the above method embodiments.

[0073] Specifically, the processor 402 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits for implementing the embodiments of the present invention.

[0074] Memory 404 may include a large-capacity memory 404 for data or instructions. By way of example, and not limitation, memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 404 may include removable or non-removable (or fixed) media. Where appropriate, memory 404 may be internal or external to the data processing device. In certain embodiments, memory 404 is non-volatile memory. In certain embodiments, memory 404 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory (FLASH), or a combination of two or more of these. In appropriate circumstances, the RAM may be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM may be a fast page mode dynamic random access memory 404 (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0075] The memory 404 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 402 .

[0076] The processor 402 reads and executes computer program instructions stored in the memory 404 to implement any one of the pathological image color correction methods in the above embodiments.

[0077] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408 , wherein the transmission device 406 is connected to the processor 402 , and the input / output device 408 is connected to the processor 402 .

[0078] Transmission device 406 can be used to receive or transmit data via a network. Specific examples of such networks may include wired or wireless networks provided by the electronic device's communications provider. In one embodiment, the transmission device includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, transmission device 406 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0079] The input / output device 408 is used to input or output information.

[0080] Example 4 This embodiment further provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute the process. The process includes the pathological image color correction method according to the first embodiment.

[0081] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be repeated here.

[0082] In general, various embodiments may be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects of the invention may be implemented in hardware, while other aspects may be implemented in firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or a controller or other computing device, or some combination thereof.

[0083] The embodiments of the present invention may be implemented by computer software that is executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets and / or macros may be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. A computer program product may include one or more computer executable components that are configured to perform an embodiment when the program is run. One or more computer executable components may be at least one software code or a portion thereof. In addition, it should be noted at this point that, for example, Figure 5 Any block of the logic flow in the program may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media such as memory chips or memory blocks implemented within the processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants, CDs, etc. Physical media are non-transitory media.

[0084] Those skilled in the art should understand that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0085] The above embodiments merely illustrate several embodiments of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of the present invention. Therefore, the scope of the present invention shall be determined by the appended claims.

Claims

1. A pathological image color correction method, characterized in that: The following steps are involved: Constructing a multi-stage color correction network, wherein the multi-stage color correction network includes a universal color correction network, a dedicated color correction network, and a lightweight color correction network in a progressive manner; The universal color correction network is trained using a deep learning network structure comprising an encoding module and a decoding module based on a loss function of the difference between the predicted image and the target image to learn high-order color correction parameters applicable to all colors; The dedicated color correction network is trained based on the training results of the universal color correction network through a deep learning network structure including a simplified encoding module and a decoding module. During the training process, feature constraints are used to ensure that the features extracted by the dedicated color correction network are consistent with the features extracted by the universal color correction network. Sample weights are dynamically adjusted using a compatible color loss function to reduce the influence of samples with poor correction effects, thereby achieving color grouping and obtaining multiple sets of correction parameters suitable for different color groupings. The lightweight color correction network is trained based on the color grouping data generated by the dedicated color correction network through a further simplified network structure to obtain lightweight correction parameters corresponding to each color group. The lightweight correction parameters are used to achieve real-time color correction based on simple matrix operations; According to the hardware resources and application scenarios of the pathology slide scanner, the general color correction network, the dedicated color correction network or the lightweight color correction network is selected to perform color correction on the pathology image; Wherein, when the dedicated color correction network and the lightweight color correction network are applied, the color group to which the pathological image belongs is determined through a classification model or manual setting, and the correction parameters of the corresponding group are called.

2. The pathological image color correction method according to claim 1, wherein: The encoding modules of the universal color correction network, the dedicated color correction network and the lightweight color correction network are all composed of several convolution modules, and the convolution modules include convolution layers, batch normalization layers and activation function layers; the decoding modules of the universal color correction network and the dedicated color correction network are composed of several convolution modules and upsampling modules.

3. The pathological image color correction method according to claim 1, wherein: The encoding module and decoding module of the general color correction network each include 4 convolution modules; the encoding module and decoding module of the dedicated color correction network each include 2 convolution modules; The lightweight color correction network adopts a network structure including three 1×1 convolution kernels, and its training loss function is consistent with the loss function of the universal color correction network.

4. The pathological image color correction method according to claim 1, wherein: The loss function of the universal color correction network is a loss function based on the mean square error between the predicted image and the target image.

5. The pathological image color correction method according to claim 1, wherein: The feature constraint of the dedicated color correction network is implemented by measuring the similarity of feature vectors extracted by the dedicated color correction network and the universal color correction network through cosine distance.

6. The pathological image color correction method according to claim 1, wherein: In the compatible color loss function of the dedicated color correction network, the sample weights are dynamically adjusted with training iterations, and the adjustment method is calculated based on the maximum and minimum values ​​of the correction loss and hyperparameters of the previous iteration.

7. The pathological image color correction method according to any one of claims 1 to 6, wherein: The color grouping of the dedicated color correction network is achieved by screening samples whose correction loss exceeds a threshold as new group training samples, and repeating until the loss variance of the remaining samples is less than the threshold, where the threshold is the sum of the mean and standard deviation of the losses of all samples.

8. A pathological image color correction device, characterized in that: include: Network construction module, used to construct a general color correction network, a dedicated color correction network, and a lightweight color correction network in sequence; a training module, configured to respectively train the universal color correction network, the dedicated color correction network, and the lightweight color correction network, wherein the dedicated color correction network implements color grouping based on the feature constraints and compatible color loss function of the universal color correction network, and the lightweight color correction network is trained based on the color grouping data of the dedicated color correction network; The correction execution module is used to select the universal color correction network, the dedicated color correction network or the lightweight color correction network to perform color correction on the pathological image according to hardware resources and application scenarios.

9. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to run the computer program to perform the pathological image color correction method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, wherein the computer program includes a program code for controlling a process to execute a process, wherein the process includes the pathological image color correction method according to any one of claims 1 to 7.

Images