A pathological image color restoration method and scanner based on deep learning

Abstract

The application discloses a pathological image color restoration method and a scanner based on deep learning, and the method comprises the following steps: reducing the resolution of input data and target data based on a bilateral grid downsampling technology; training a neural network model for the first time by using the input data and the target data after the resolution is reduced; improving the resolution of the input data and the target data after the resolution is reduced based on a bilateral grid upsampling technology; and training the neural network model for the second time by using the input data and the target data after the resolution is improved, so that the training speed of the neural network model is improved, and the staining effect of the pathological image collected by a digital pathology scanner is improved.

Description

Technical Field

[0001] This application relates to the field of pathological image processing technology, and in particular to a deep learning-based method for color restoration of pathological images and a scanner. Background Technology

[0002] Pathological images are high-resolution images of glass slides containing cells taken under a scanner. Pathological images help doctors make diagnoses of patients. Through pathological images, the specific condition of tumor cells can be observed, such as whether there is infiltration, lymph node metastasis, and the degree of differentiation. They are very helpful for the diagnosis, prognosis, grading, and staging of tumors.

[0003] Generally, pathological images acquired by digital pathology scanners are dark red or light red, while those observed under a real microscope are reddish-blue. Therefore, related techniques, such as hematoxylin and eosin, perform image signal processing on the pathological images acquired by digital pathology scanners to ensure color consistency with those observed under a real microscope. However, these techniques often produce poor staining results for the acquired pathological images, frequently leading to inconsistencies between the colors. Using images acquired by digital pathology scanners for diagnosis can result in erroneous results and delay patient treatment. Summary of the Invention

[0004] This application provides a deep learning-based method for color restoration of pathological images and a scanner to improve the staining effect of pathological images acquired by digital pathology scanners.

[0005] In a first aspect, this application provides a deep learning-based method for color restoration of pathological images, comprising: determining input data based on pathological images acquired by a digital pathology scanner; determining target data based on pathological images acquired under a real microscope; reducing the resolution of the input data and target data using a bilateral grid downsampling technique; constructing a neural network model, using the reduced-resolution input data as training data and the reduced-resolution target data as supervision data to train the neural network model for the first time; the supervision data is the result to be achieved by the input data after training during the training process; the neural network model includes: an intermediate illumination layer, multiple convolutional kernels located before the intermediate illumination layer, and multiple convolutional kernels located after the intermediate illumination layer, wherein the input of the intermediate illumination layer is obtained based on the output features of at least one of the preceding convolutional kernels of the intermediate illumination layer, and the input of at least one of the following convolutional kernels of the intermediate illumination layer is obtained based on the output features of the intermediate illumination layer; increasing the resolution of the reduced-resolution input data and the reduced-resolution target data using a bilateral grid upsampling technique; using the increased-resolution input data as training data and the increased-resolution target data as supervision data to train the neural network model for the second time; and performing staining processing on the pathological images acquired by the digital pathology scanner based on the trained neural network model.

[0006] In the above embodiments, the resolution of the input and target data is reduced using a bilateral grid-based downsampling technique. The neural network model is then trained using the reduced-resolution input and target data, thus learning an image-to-color mapping function in the low-resolution domain. Next, the resolution of the input and target data is increased using the same bilateral grid-based downsampling technique. This increased-resolution input and target data are then used to further train the neural network model. Since the neural network model already possesses an image-to-color mapping function, its training speed is faster and more efficient compared to directly using high-resolution input and target data. Furthermore, the neural network model retains image-to-color mapping functions at various resolutions, making it suitable for pathological images acquired by digital pathology scanners of different resolutions. Simultaneously, using target data to supervise the training of the neural network model provides a clear objective for the input data, resulting in highly accurate prediction and classification results, thus improving the staining effect of the neural network model on pathological images acquired by digital pathology scanners. Meanwhile, an intermediate illumination layer was added to the neural network model, and multiple supervisory signals, i.e. target data, were added to the middle of the neural network model. This allows the intermediate layers of the neural network model to learn more robust feature representations, thereby associating the input data with the target data and improving the staining effect of the neural network model on pathological images acquired by digital pathology scanners.

[0007] In conjunction with some embodiments of the first aspect, in some embodiments, the neural network model further includes:

[0008] The input layer is used to receive training data;

[0009] The first symmetric convolutional layer is used to perform convolution operations on the received training data to obtain intermediate data;

[0010] The second symmetric convolutional layer is used to perform convolution operations on the intermediate data and combine it with the received training data to obtain the predicted coloring data.

[0011] The output layer is used to output the predicted staining data.

[0012] In the above embodiments, the first symmetric convolutional layer is used to extract features from the training data, and the second symmetric convolutional layer is used to further extract features from the training data while retaining the features extracted by the first symmetric convolutional layer. This improves the feature extraction capability of the neural network model and accelerates its training speed. Furthermore, the first and second symmetric convolutional layers have the same parameters due to their vertical symmetry. This symmetry effectively reduces the number of parameters in the neural network model, thereby improving its computational efficiency and generalization performance.

[0013] In conjunction with some embodiments of the first aspect, the first symmetric convolutional layer specifically includes:

[0014] The first convolutional layer is used to perform convolution operations on the received training data to extract the first feature information;

[0015] The first normalization layer is used to perform batch normalization processing on the first feature information;

[0016] The first activation layer is used to process the normalized first feature information using an activation function to obtain intermediate data.

[0017] The second symmetric convolutional layer specifically includes:

[0018] The second convolutional layer is used to perform convolution operations on the intermediate data to extract the second feature information, and then combine the second feature information with the received training data.

[0019] The second normalization layer is used to perform batch normalization processing on the combined second feature information;

[0020] The second activation layer is used to process the normalized second feature information using an activation function to obtain the predicted coloring data.

[0021] In the above embodiments, the first normalization layer and the second normalization layer can accelerate the training of the neural network model and improve the generalization ability of the neural network model. The first activation layer and the second activation layer can increase the nonlinearity of the neural network model, making it more suitable for staining pathological images acquired by digital pathology scanners.

[0022] In conjunction with some embodiments of the first aspect, the neural network model is trained for the first time using input data with reduced resolution as training data and target data with reduced resolution as supervision data, and for the second time using input data with increased resolution as training data and target data with increased resolution as supervision data. This further includes: calculating the pixel difference between the predicted stained data and the supervision data using a pixel loss function; calculating the color difference between the predicted stained data and the supervision data using a color loss function; calculating the smoothness difference between the predicted stained data and the supervision data using a smoothness loss function; and completing the training of the neural network model when the pixel difference, color difference, and smoothness difference between the predicted stained data and the supervision data are all less than preset thresholds.

[0023] In the above embodiments, by constructing pixel loss function, color loss function and smoothness loss function, the distance and deviation between the neural network model's predicted color data and the supervised data can be measured to improve the neural network model, so that the predicted color data can obtain better contrast, saturation and clarity.

[0024] In conjunction with some embodiments of the first aspect, the pixel loss function is specifically as follows:

[0025]

[0026] In the formula, n represents the total number of pixels; y i To supervise the value of the i-th pixel in the data, x i Let f(x) be the value of the i-th pixel in the training data. i ) represents the value of the i-th pixel in the predicted staining data; loss(x,y) represents the mean square error between the supervised data and the predicted staining data;

[0027] The color loss function is as follows:

[0028]

[0029] In the formula, To predict the i-th pixel in the coloring data, The i-th pixel in the supervised data; To predict the angle between the i-th pixel in the stained data and the i-th pixel in the supervised data;

[0030] The smoothness loss function is as follows:

[0031]

[0032] In the formula, S p To predict the p-th pixel in the stained data, To predict the gradient of the p-th pixel in the x-direction in the stained data, To predict the gradient of the p-th pixel in the y-direction in the stained data, c is the weight for spatial variation smoothness. To predict the weight of the p-th pixel in the x-direction in the stained data, To predict the weight of the p-th pixel in the y-direction in the stained data.

[0033] In the above embodiments, the pixel loss function penalizes predicted color data with larger errors more severely and predictive color data with smaller errors less severely. The color loss function is used to measure the difference between the predicted color data and the supervised data in the feature space, and the smoothness loss function is used to measure the difference in smoothness between the predicted color data and the supervised data, thereby improving the neural network model and enabling the predicted color data to achieve better contrast, saturation, and sharpness.

[0034] In conjunction with some embodiments of the first aspect, after determining input data based on pathological images acquired by a digital pathology scanner and before reducing the resolution of the input data and target data using a bilateral grid downsampling technique, and after determining target data based on pathological images acquired under a real microscope, the method further includes: dividing the input data into multiple sub-input data; using a template matching algorithm to perform template matching between the sub-input data and the target data to find the sub-input data most similar to the target data; and registering the most similar sub-input data with the target data according to the SIFT algorithm.

[0035] In the above embodiments, the input data is divided into multiple sub-input data, and then a template matching algorithm is used to find the sub-input data most similar to the target data, thereby reducing the amount of data to be registered. Simultaneously, the SIFT algorithm is used to register the most similar sub-input data with the target data, finding the same local features in different images (i.e., input data and target data), i.e., the registered sub-input data and the registered target data. This addresses the problem of inconsistent field of view between the input and target data images, increases the range of selectable input data, increases training data, and improves the staining effect of the neural network model on pathological images acquired by the digital pathology scanner.

[0036] In conjunction with some embodiments of the first aspect, after registering the most similar sub-input data with the target data according to the SIFT algorithm, the method further includes: removing the black borders of the registered sub-input data and target data.

[0037] In the above embodiments, black borders generated during image registration are removed to ensure the integrity and accuracy of the registered sub-input data and the registered target data.

[0038] Secondly, this application also provides a deep learning-based pathological image color restoration scanner, which includes:

[0039] The data collection module is used to determine input data based on pathological images acquired by a digital pathology scanner and to determine target data based on pathological images acquired under a real microscope.

[0040] The first resolution adjustment module is used to reduce the resolution of input and target data based on a two-sided grid downsampling technique; the first neural network training module is used to build a neural network model, using the reduced-resolution input data as training data and the reduced-resolution target data as supervision data to train the neural network model for the first time; the supervision data is the result that the input data is expected to achieve after training.

[0041] The neural network model includes: an intermediate illumination layer, multiple convolutional kernels before the intermediate illumination layer, and multiple convolutional kernels after the intermediate illumination layer. The input of the intermediate illumination layer is obtained based on the output features of at least one of the convolutional kernels before the intermediate illumination layer, and the input of at least one of the convolutional kernels after the intermediate illumination layer is obtained based on the output features of the intermediate illumination layer.

[0042] The second resolution adjustment module is used to improve the resolution of the input data and the target data after the resolution reduction based on the double-sided grid upsampling technique.

[0043] The second neural network training module is used to train the neural network model a second time by using the input data with improved resolution as training data and the target data with improved resolution as supervision data.

[0044] The staining module is used to stain pathological images acquired by the digital pathology scanner based on the trained neural network model.

[0045] In conjunction with some embodiments of the second aspect, in some embodiments, the neural network model further includes:

[0046] The input layer is used to receive training data;

[0047] The first symmetric convolutional layer is used to perform convolution operations on the received training data to obtain intermediate data;

[0048] The second symmetric convolutional layer is used to perform convolution operations on the intermediate data and combine it with the received training data to obtain the predicted coloring data.

[0049] The output layer is used to output the predicted staining data.

[0050] In conjunction with some embodiments of the second aspect, in some embodiments, the first symmetric convolutional layer specifically includes:

[0051] The first convolutional layer is used to perform convolution operations on the received training data to extract the first feature information;

[0052] The first normalization layer is used to perform batch normalization processing on the first feature information;

[0053] The first activation layer is used to process the normalized first feature information using an activation function to obtain intermediate data.

[0054] The second symmetric convolutional layer specifically includes:

[0055] The second convolutional layer is used to perform convolution operations on the intermediate data to extract the second feature information, and then combine the second feature information with the received training data.

[0056] The second normalization layer is used to perform batch normalization processing on the combined second feature information;

[0057] The second activation layer is used to process the normalized second feature information using an activation function to obtain the predicted coloring data.

[0058] In conjunction with some embodiments of the second aspect, in some embodiments, the scanner further includes:

[0059] The pixel loss module is used to calculate the pixel difference between the predicted coloring data and the supervision data using the pixel loss function.

[0060] The color module is used to calculate the color difference between the predicted color data and the supervised data using a color loss function;

[0061] The loss module is used to calculate the smoothness difference between the predicted coloring data and the supervised data using the smoothness loss function;

[0062] The termination module is used to complete the training of the neural network model when the pixel difference between the predicted color data and the supervised data, the color difference between the predicted color data and the supervised data, and the smoothness difference between the predicted color data and the supervised data are all less than the corresponding preset thresholds.

[0063] In conjunction with some embodiments of the second aspect, in some embodiments, the pixel loss function is specifically as follows:

[0064]

[0065] In the formula, n represents the total number of pixels; y i To supervise the value of the i-th pixel in the data, x i Let f(x) be the value of the i-th pixel in the training data. i ) represents the value of the i-th pixel in the predicted staining data; loss(x,y) represents the mean square error between the supervised data and the predicted staining data;

[0066] The color loss function is as follows:

[0067]

[0068] In the formula, To predict the i-th pixel in the coloring data, The i-th pixel in the supervised data; To predict the angle between the i-th pixel in the stained data and the i-th pixel in the supervised data; the smoothness loss function is as follows:

[0069]

[0070] In the formula, S p To predict the p-th pixel in the coloring data, To predict the gradient of the p-th pixel in the x-direction in the stained data, To predict the gradient of the p-th pixel in the y-direction in the stained data, c is the weight for spatial variation smoothness. To predict the weight of the p-th pixel in the x-direction in the stained data, To predict the weight of the p-th pixel in the y-direction in the stained data.

[0071] In conjunction with some embodiments of the second aspect, in some embodiments, the scanner further includes:

[0072] The equal-division module is used to divide the input data into multiple sub-input data;

[0073] The template matching algorithm module is used to perform template matching between sub-input data and target data to find the sub-input data that is most similar to the target data.

[0074] The SIFT algorithm module is used to register the most similar sub-input data with the target data according to the SIFT algorithm.

[0075] In conjunction with some embodiments of the second aspect, in some embodiments, the scanner further includes:

[0076] The black border removal module is used to remove black borders from the registered sub-input data and target data.

[0077] Thirdly, embodiments of this application provide an electronic device, which includes: one or more processors and a memory;

[0078] The memory is coupled to the one or more processors and is used to store computer program code, which includes computer instructions that the one or more processors call to cause the electronic device to perform the methods described in the first aspect and any possible implementation thereof.

[0079] Fourthly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0080] Fifthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0081] Understandably, the ground-penetrating scanner 3D imaging display audio device provided in the second aspect, the electronic device provided in the third aspect, the computer program product provided in the fourth aspect, and the computer storage medium provided in the fifth aspect are all used to execute the wireless hotspot connection method provided in the embodiments of this application. Therefore, the beneficial effects it can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0082] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0083] 1. The deep learning-based pathological image color restoration method provided in this application reduces the resolution of input and target data using a bilateral grid-based downsampling technique. The reduced-resolution input and target data are then used to train a neural network model, allowing the model to learn an image-to-color mapping function in the low-resolution domain. Next, the resolution of the input and target data is increased using the bilateral grid-based downsampling technique. This increased-resolution input and target data are then used to further train the neural network model. Since the neural network model already possesses the image-to-color mapping function, its training speed is faster and more efficient compared to directly using high-resolution input and target data. Furthermore, the neural network model retains the image-to-color mapping function at various resolutions, making it suitable for pathological images acquired by digital pathology scanners of different resolutions. Additionally, using target data to supervise the training of the neural network model provides a clear objective for the input data, resulting in highly accurate prediction and classification results, thus improving the staining effect of the neural network model on pathological images acquired by digital pathology scanners. Meanwhile, an intermediate illumination layer was added to the neural network model, and multiple supervisory signals, i.e. target data, were added to the middle of the neural network model. This allows the intermediate layers of the neural network model to learn more robust feature representations, thereby associating the input data with the target data and improving the staining effect of the neural network model on pathological images acquired by digital pathology scanners.

[0084] 2. The deep learning-based pathological image color restoration method provided in this application can measure the distance and deviation between the predicted staining data and the supervised data by constructing pixel loss function, color loss function and smoothness loss function, thereby improving the neural network model and making the predicted staining data have better contrast, saturation and clarity.

[0085] 3. The deep learning-based pathological image color restoration method provided in this application reduces the amount of data required for registration by dividing the input data into multiple sub-input data and then using a template matching algorithm to find the sub-input data most similar to the target data. Simultaneously, the SIFT algorithm is used to register the most similar sub-input data with the target data, finding the same local features in different images (i.e., input and target data). This registration of the sub-input data and the registered target data addresses the inconsistency in the field of view between the input and target images, increasing the range of selectable input data, thereby increasing training data and improving the staining effect of the neural network model on pathological images acquired by digital pathology scanners. Attached Figure Description

[0086] Figure 1An exemplary scenario diagram of a training neural network model for the deep learning-based pathological image color restoration method provided in this application.

[0087] Figure 2 A flowchart illustrating the deep learning-based pathological image color restoration method provided in this application.

[0088] Figure 3 A schematic diagram of the data structure of a neural network model for the deep learning-based pathological image color restoration method provided in this application.

[0089] Figure 4a This is an exemplary scenario diagram of a first-time trained neural network model for the deep learning-based pathological image color restoration method provided in this application.

[0090] Figure 4b An exemplary scenario diagram of a second-trained neural network model for the deep learning-based pathological image color restoration method provided in this application.

[0091] Figure 5 An exemplary scenario diagram of another training neural network model for the deep learning-based pathological image color restoration method provided in this application.

[0092] Figure 6 Another flowchart illustrating the deep learning-based pathological image color restoration method provided in this application.

[0093] Figure 7 A schematic diagram of the modular virtual device for a deep learning-based pathological image color restoration scanner provided in this application.

[0094] Figure 8 A schematic diagram of the physical device of the deep learning-based pathological image color restoration scanner provided in this application. Detailed Implementation

[0095] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0096] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0097] In some embodiments, for ease of description, the deep learning-based pathological image color restoration method in this application may also be referred to as a pathological image staining method, or may be referred to by other names, which are not limited here.

[0098] In some embodiments, both the input data and the target data are datasets consisting of multiple images. However, for better description and understanding, the processing of the input data and the target data below will be performed using one image from the input data and the target data as an example. Other images should also be processed in accordance with the above operations.

[0099] Please see Figure 1 , Figure 1 An exemplary scenario diagram of a training neural network model for the deep learning-based pathological image color restoration method provided in this application.

[0100] Please see Figure 1 In the first part, the input data and target data are prepared, and the resolution of the input data and target data is reduced to obtain the input data and target data with reduced resolution. Then, the neural network model is trained with the input data and target data with reduced resolution to obtain the image-to-color mapping function learned in the low-resolution domain.

[0101] The input data is based on pathological images acquired by a digital pathology scanner, while the target data is based on pathological images acquired under a real microscope.

[0102] It should be noted that the pathological images acquired by the digital pathology scanner are structurally similar to those observed under a real microscope. The biggest difference is color; the pathological images acquired by the digital pathology scanner are dark red or light red, while the pathological images observed under a real microscope are reddish-blue. However, to comply with the requirements of the accompanying diagrams in the instruction manual, neither the pathological images acquired by the digital pathology scanner nor the pathological images observed under a real microscope display color.

[0103] Please see Figure 1The lower half of the process involves increasing the resolution of the input and target data after the resolution reduction, resulting in improved input and target data. This improved input and target data are then used to further train the neural network model. Based on the image-to-color mapping function learned in the low-resolution domain, the neural network model further optimizes the image-to-color mapping function.

[0104] It is understood that the above scenario is only an example, and in practical applications, other content or forms may also be used, which are not limited here.

[0105] It is evident that this deep learning-based method for color restoration of pathological images enables the neural network model to learn the image-to-color mapping function in the low-resolution domain and then optimizes the image-to-color mapping function in the high-resolution domain. The training speed of this neural network model is faster and more efficient compared to directly utilizing high-resolution input and target data.

[0106] Please see Figure 2 , Figure 2 A flowchart illustrating the deep learning-based pathological image color restoration method provided in this application.

[0107] S201: Determine input data based on pathological images acquired by a digital pathology scanner; and determine target data based on pathological images acquired under a real microscope;

[0108] In this embodiment, the input data is the pathological images acquired by the digital pathology scanner, which are processed to facilitate subsequent training of the neural network model. For example, the format of the pathological images acquired by the digital pathology scanner, such as JPEG, PNG, BMP, etc., is converted into vector image formats, such as SVG, AI, etc. Of course, in other embodiments, other methods can be used, or the pathological images acquired by the digital pathology scanner can be used directly. This is not limited here.

[0109] Similarly, the operations on the target data are the same as those on the input data.

[0110] S202: The bilateral grid-based downsampling technique reduces the resolution of both input and target data;

[0111] In this embodiment, the downsampling step of the bilateral grid is as follows: all pixels of the input data are mapped to a finite value range to generate a discretized image; a bilateral grid is then constructed, and the discretized image is mapped onto the grid; the input data is downsampled using downsampling techniques to reduce the resolution of the input data. Similarly, the resolution reduction operation for the target data is completed.

[0112] In layman's terms, downsampling technology converts pixels within a certain range into a single pixel, and so on, to reduce the resolution of an entire image.

[0113] In this embodiment, the input data is an image of size M*N, which is downsampled by a factor of s to obtain a (M / s)*(N / s) resolution image. This means that all pixels within an s*s window of the input data are converted into a single pixel, and the value of this single pixel is the average value of all pixels within the window. This process is repeated for all pixels within the entire s*s window, thus reducing the resolution of the input data. Similarly, the resolution of the target data is reduced. Of course, other methods may be used in other embodiments, which are not limited here.

[0114] S203: Construct a neural network model, using the input data with reduced resolution as training data and the target data with reduced resolution as supervision data to train the neural network model for the first time.

[0115] Supervised data refers to the desired outcome of training the input data. Please refer to [link / reference]. Figure 3 , Figure 3 This is a schematic diagram of the data structure of a neural network model for a deep learning-based pathological image color restoration method provided in this application. The neural network model includes: an intermediate illumination layer, multiple convolutional kernels before the intermediate illumination layer, and multiple convolutional kernels after the intermediate illumination layer. The input to the intermediate illumination layer is obtained based on the output features of at least one of the preceding convolutional kernels of the intermediate illumination layer, and the input to at least one of the following convolutional kernels of the intermediate illumination layer is obtained based on the output features of the intermediate illumination layer.

[0116] The intermediate illumination layer, also known as the intermediate layer with illumination function, selectively applies illumination and reflection to the pathological images acquired by the digital pathology scanner. This balances the light intensity in the acquired pathological images, resulting in a more balanced brightness distribution. Simultaneously, it adjusts the contrast of the acquired pathological images, making the brightness differences between different areas more pronounced. Furthermore, it corrects color deviations in the acquired pathological images, ensuring more accurate and saturated colors.

[0117] In this embodiment, the neural network model learns the prediction function by inputting correct supervised data, that is, the function that maps the training data to the supervised data, which is also the image-to-color mapping function described in the above embodiment. By continuously optimizing the parameters of the neural network model, it is able to make accurate predictions for new and unseen inputs.

[0118] Supervised data provides supervisory signals, enabling the neural network model to continuously optimize its parameters to make the training data more closely resemble the supervised data. Through continuous and iterative training and optimization, the neural network model can make accurate predictions on subsequent unseen new data, namely the pathological images acquired by the digital pathology scanner in subsequent embodiments.

[0119] While this neural network model is capable of staining pathological images acquired by digital pathology scanners, the training and supervision data used by this neural network model are low-resolution images. Therefore, it is only suitable for low-resolution fields and has poor adaptability to pathological images acquired by digital pathology scanners in high-resolution fields.

[0120] S204: The upsampling technique based on bilateral grids improves the resolution of both the input data and the target data after the resolution reduction.

[0121] In actual use, although the resolution of the input data and target data obtained in step S201 is higher than that of the input data and target data after the resolution is reduced, digital pathology scanners generally do not acquire ultra-high resolution pathology images. Therefore, the resolution of the input data and target data obtained in step S201 is much lower than the highest resolution theoretically supported by digital pathology scanners.

[0122] Therefore, the upsampling technique of bilateral grids is used to increase the resolution of the input data and the target data after the resolution reduction, so as to approach the highest resolution supported by the digital pathology scanner, making the training of the subsequent neural network model more comprehensive.

[0123] In this embodiment, the upsampling step for the bilateral grid is as follows:

[0124] By mapping all pixels of the input data to a finite value range, a discretized image is produced.

[0125] Then construct a two-sided grid and map the discretized image onto the grid;

[0126] Next, new pixels are inserted between the pixels of the input data using a suitable interpolation algorithm to complete the resolution improvement operation of the target data. Similarly, the resolution improvement operation of the target data is completed. Of course, other methods can be used in other embodiments, which are not limited here.

[0127] S205: Use the input data with improved resolution as training data and the target data with improved resolution as supervision data to train the neural network model a second time.

[0128] The steps used in this embodiment are based on the same concept as those used in the above embodiments. The specific implementation process is detailed in step S203, and will not be repeated here.

[0129] S206: Use the trained neural network model to stain pathological images acquired by the new digital pathology scanner.

[0130] This deep learning-based pathological image color restoration method reduces the resolution of input and target data using a bilateral grid-based downsampling technique. The neural network model is then trained using these reduced-resolution input and target data, allowing it to learn an image-to-color mapping function in the low-resolution domain. Next, the resolution of both input and target data is increased using the same bilateral grid-based downsampling technique. This increased resolution is then used to further train the neural network model. Since the neural network model already possesses the image-to-color mapping function, its training speed is faster and more efficient compared to directly using high-resolution input and target data. Furthermore, the neural network model retains the image-to-color mapping function at various resolutions, making it suitable for pathological images acquired by digital pathology scanners of different resolutions. Additionally, using target data to supervise the training of the neural network model provides a clear objective for the input data, resulting in highly accurate prediction and classification results, thus improving the staining effect of the neural network model on pathological images acquired by digital pathology scanners. Meanwhile, an intermediate illumination layer was added to the neural network model, and multiple supervisory signals, i.e. target data, were added to the middle of the neural network model. This allows the intermediate layers of the neural network model to learn more robust feature representations, thereby associating the input data with the target data and improving the staining effect of the neural network model on pathological images acquired by digital pathology scanners.

[0131] In the above embodiments, simply using the input data with reduced or increased resolution as training data and the target data with reduced or increased resolution as supervision data for the first training of the neural network model can improve the training speed. However, in practical applications, the neural network model cannot quantify the difference between the predicted and colored data and the supervision data, thus making it impossible to evaluate the performance and accuracy of the neural network model. This makes the optimization and tuning of the neural network model extremely difficult. The following section will combine... Figure 4a and Figure 4b The illustrated embodiment, taking one method of measuring the distance and deviation between the staining data predicted by the neural network model and the supervised data as an example, specifically describes the deep learning-based pathological image color restoration method in this application embodiment:

[0132] Please see Figure 4a, Figure 4a This is an exemplary scenario diagram of a first-time trained neural network model for the deep learning-based pathological image color restoration method provided in this application.

[0133] Prepare input and target data, reduce the resolution of the input and target data to obtain the reduced-resolution input and target data, and then train a neural network model with the reduced-resolution input and target data; thus obtaining the low-resolution domain-learned image-to-color mapping function.

[0134] The steps used in this embodiment are based on the same concept as those used in the above embodiments, and the specific implementation process is detailed in [link to implementation details]. Figure 1 The embodiments described herein will not be repeated here.

[0135] During the training of the neural network model, pixel loss function, color loss function, and smoothness loss function are constructed.

[0136] The pixel loss function is used to determine the pixel difference between the predicted colored data and the supervised data by the neural network model, and convert it into specific numerical values ​​for intuitive display. Based on the pixel difference between the predicted colored data and the supervised data, it is decided whether to continue to improve the neural network model.

[0137] Similarly, the color loss function is used to determine the color difference between the color data predicted by the neural network model and the supervised data, and based on this pixel difference, it is decided whether to continue to improve the neural network model.

[0138] Similarly, the smoothness loss function is used to determine the smoothness difference between the predicted colored data and the supervised data by the neural network model, and based on this smoothness difference, it is decided whether to continue to improve the neural network model.

[0139] Training of the neural network model will only stop when the pixel differences between the predicted color data and the supervised data, the color differences between the predicted color data and the supervised data, and the smoothness differences between the color difference predicted color data and the supervised data all meet the requirements.

[0140] Please see Figure 4b , Figure 4b An exemplary scenario diagram of a second-trained neural network model for the deep learning-based pathological image color restoration method provided in this application.

[0141] The steps used in this embodiment are based on the same concept as those used in the above embodiments, and the specific implementation process is detailed in [link to implementation details]. Figure 4b The embodiments shown are not described in detail here.

[0142] It is evident that this deep learning-based pathological image color restoration method, by constructing pixel loss functions, color loss functions, and smoothness loss functions, can measure the distance and deviation between the neural network model's predicted staining data and the supervised data, thereby improving the neural network model and enabling the predicted staining data to achieve better contrast, saturation, and clarity.

[0143] In the above embodiments, only pixel loss functions, color loss functions, and smoothness loss functions are needed to predict better contrast, saturation, and clarity in stained data. However, in practical applications, the field of view of pathological images acquired by digital pathology scanners differs from that of pathological images acquired under real microscopes. Inconsistent field of view in training and supervision data can easily lead to errors in the training of neural network models, resulting in deviations in the training results. The following description, using the embodiment shown in Figure 4 as an example, illustrates a method for aligning the field of view of pathological images acquired by digital pathology scanners with those acquired under real microscopes, to specifically describe the deep learning-based pathological image color restoration method in this application.

[0144] Please see Figure 5 , Figure 5 An exemplary scenario diagram of another training neural network model for the deep learning-based pathological image color restoration method provided in this application.

[0145] like Figure 5 As shown in (a), the field of view of pathological images acquired by digital pathology scanners is different from that of pathological images acquired under a real microscope.

[0146] It should be noted that the pathological images acquired by the digital pathology scanner are the input data, while the pathological images acquired under a real microscope are the target data. In this embodiment, the field of view of the pathological images acquired under the real microscope is significantly smaller than that of the pathological images acquired by the digital pathology scanner. Of course, in other embodiments, the field of view of the pathological images acquired under the real microscope may be larger than that of the pathological images acquired by the digital pathology scanner, which is not limited here.

[0147] like Figure 5 As shown in (b), the pathological images acquired by the digital pathology scanner are divided into multiple sub-pathological images. In this embodiment, the pathological images acquired by the digital pathology scanner are divided into sub-pathological image 1, sub-pathological image 2, sub-pathological image 3, and sub-pathological image 4.

[0148] It should be noted that the sub-pathological images are sub-input data.

[0149] In other embodiments, the pathological images acquired under a real microscope can be larger than the field of view between pathological images acquired by a digital pathology scanner. In this case, the pathological images acquired under the real microscope are divided equally to complete subsequent operations.

[0150] In other embodiments, the images may be divided into other numbers of sub-pathological images, which are not limited here.

[0151] like Figure 5 As shown in (c), a template matching algorithm is used to find the sub-pathological image that is most similar to the pathological image acquired under the real microscope by comparing the sub-pathological image acquired by the pathological scanner with the pathological image acquired under the real microscope. In this embodiment, sub-pathological image 3 is most similar to the pathological image acquired under the real microscope.

[0152] It is evident that although sub-pathological image 3 is most similar to the pathological image acquired under a real microscope, there are still significant differences between sub-pathological image 3 and the pathological image acquired under a real microscope.

[0153] Of course, in practical applications, there are still many differences between the sub-pathological images that are most similar to the pathological images collected under a real microscope and the pathological images collected under a real microscope. The specific differences and solutions will be described later and will not be repeated here.

[0154] In other embodiments, the images may be divided into other numbers of sub-pathological images, which are not limited here.

[0155] like Figure 5 As shown in (d), the sub-pathological image that is most similar to the pathological image acquired under a real microscope is registered with the pathological image acquired under a real microscope.

[0156] In the figure, the solid-lined area represents the sub-pathological image most similar to the pathological image acquired under a real microscope, while the dashed-lined area represents the pathological image acquired under a real microscope. The overlapping area represents the sub-pathological image most similar to the pathological image acquired under a real microscope, and the pathological image that is completely identical to the pathological image acquired under a real microscope, i.e., the registered image.

[0157] like Figure 5 As shown in (e), the black borders of the pathological images acquired by the registered digital pathology scanner and the pathological images acquired under a real microscope are removed to obtain pathological images acquired by the digital pathology scanner and the pathological images acquired under a real microscope with the same field of view.

[0158] Of course, in practical applications, the black borders of pathological images acquired by registered digital pathology scanners may vary from those acquired under a real microscope, but this is not a limitation here.

[0159] This deep learning-based pathological image color restoration method reduces the amount of data required for registration by dividing the input data into multiple sub-input data and then using a template matching algorithm to find the sub-input data most similar to the target data. Simultaneously, the SIFT algorithm is used to register the most similar sub-input data with the target data, finding the same local features in different images (i.e., the input and target data), thus resolving the inconsistency in the field of view between the input and target images. This increases the range of selectable input data, expands the training data, and improves the staining effect of the neural network model on pathological images acquired by digital pathology scanners.

[0160] The above embodiments illustrate various application scenarios of the deep learning-based pathological image color restoration method. The following will combine these examples with... Figure 6 The illustrated embodiment provides a detailed description of the deep learning-based pathological image color restoration method described in this application:

[0161] Please see Figure 6 This is another flowchart illustrating the deep learning-based pathological image color restoration method provided in this application.

[0162] S601: Determine input data based on pathological images acquired by a digital pathology scanner; and determine target data based on pathological images acquired under a real microscope;

[0163] The steps used in this embodiment are based on the same concept as those used in the above embodiments, and the specific implementation process is detailed in [link to implementation details]. Figure 1 The embodiments and steps S201 described herein will not be repeated here.

[0164] The potential illumination areas of a digital pathology scanner and a physical microscope are not identical, resulting in discrepancies between the pathological images produced by the digital scanner and those produced by the physical microscope. Because digital pathology scanners are more convenient to control than physical microscopes, the field of view of a digital pathology scanner image is significantly larger than that of a physical microscope image for the same cellular tissue. This prevents the digital pathology scanner image from including the physical microscope image.

[0165] S602: Divide the input data into multiple sub-input data;

[0166] In some embodiments, the input data is split into images with similar dimensions to the sub-input data to facilitate subsequent template matching operations; of course, there are other methods, which are not limited here.

[0167] In some embodiments, the input data is split into several 1200x1200 images. Of course, other formats also exist, which are not limited here.

[0168] S603: Use a template matching algorithm to perform template matching between the sub-input data and the target data, and find the sub-input data that is most similar to the target data;

[0169] In this embodiment, a template image is first selected from the target data. The shape of this template image should be similar to that of the sub-input data image.

[0170] For each pixel location, the matching degree between it and the template image is calculated in the sub-input data using a window the size of the template image as the region. In some embodiments, the matching degree is calculated using the squared difference, while in other embodiments, the matching degree is calculated using the correlation coefficient. Of course, there are other methods as well, which are not limited here.

[0171] The calculated matching degree can find the best matching position, which is the position in the sub-input data that is most similar to the template image. The sub-input data with the best matching position is the sub-input data that is most similar to the target data.

[0172] In practical applications, template matching algorithms can also take other forms, which are not limited here.

[0173] S604: Register the most similar sub-input data with the target data according to the SIFT algorithm;

[0174] Since the target data and the sub-input data may differ in terms of rotation, scaling, translation, etc., the SIFT algorithm is used. The main feature of the SIFT algorithm is that it has good invariance to image transformations such as rotation, scaling, and translation, and can find the same local features in different images.

[0175] In this embodiment, firstly, the Gaussian difference algorithm is used to detect the most similar sub-input data and the local extrema of the target data in different scale spaces.

[0176] Based on the extreme points, Gaussian curvature space is used to further locate key points and find their precise location and scale.

[0177] Within the region surrounding the key point, its main direction is determined by the gradient direction histogram.

[0178] Image features are constructed using pixels surrounding keypoints.

[0179] Image matching and target recognition are achieved by matching the feature vectors of key points in the most similar sub-input data with the feature vectors in the target data.

[0180] In practical applications, the SIFT algorithm can also be used in other ways, which are not limited here.

[0181] S605: Remove the black borders from the registered sub-input data and target data to obtain input data and target data with consistent image field of view;

[0182] In the process of image registration, it is usually necessary to perform certain transformations such as rotation, translation, and scaling on the most similar sub-input data and target data. These transformations may cause changes in the pixel positions of the image boundaries, resulting in black borders.

[0183] In some embodiments, if the sub-input data is translated, then the boundary pixels may be shifted out of the range of the atomic input data.

[0184] In some embodiments, if the sub-input data is rotated, then boundary pixels may be rotated out of the range of the sub-input data.

[0185] In some embodiments, the sub-input data is scaled, which may result in boundary pixels being stretched or compressed.

[0186] To avoid the impact of black borders, the black borders of the registered sub-input data and target data are removed. Alternatively, in some embodiments, image processing techniques, such as extrapolation or boundary filling, are used to fill in the removed black borders to ensure the integrity and accuracy of the registered sub-input data and target data. This is not a limitation here.

[0187] S606: The bilateral grid-based downsampling technique reduces the resolution of input and target data with consistent image field of view;

[0188] The steps used in this embodiment are based on the same concept as those used in the above embodiments, and the specific implementation process is detailed in step S202, which will not be repeated here.

[0189] S607: Construct a neural network model and build pixel loss functions, color loss functions, and smoothness loss functions; please refer to [link / reference]. Figure 3 , Figure 3 A schematic diagram of the data structure of a neural network model for the deep learning-based pathological image color restoration method provided in this application;

[0190] A neural network model includes: an input layer, which receives training data;

[0191] The first symmetric convolutional layer, connected to the input layer, is used to perform convolution operations on the received training data to obtain intermediate data;

[0192] The first symmetric convolutional layer specifically includes: a first convolutional layer, a first normalization layer, and a first activation layer;

[0193] The first convolutional layer is used to perform convolution operations on the received training data to extract the first feature information;

[0194] The first normalization layer is used to perform batch normalization processing on the first feature information;

[0195] The first activation layer is used to process the normalized first feature information using an activation function to obtain intermediate data.

[0196] Multiple convolutional kernels preceding the intermediate illumination layer are connected to the first symmetric convolutional layer and are used for feature extraction and transformation of the intermediate data;

[0197] The intermediate illumination layer, connected to the front convolutional kernel, is used to perform more complex and abstract feature extraction and transformation on the intermediate data;

[0198] Multiple convolutional kernels following the intermediate illumination layer are connected to the intermediate illumination layer and are used to extract and transform features from the intermediate data.

[0199] The second symmetric convolutional layer is used to perform convolution operations on the intermediate data and combine it with the received training data to obtain the predicted coloring data.

[0200] The second symmetric convolutional layer specifically includes: a second convolutional layer, a second normalization layer, and a second activation layer.

[0201] The second convolutional layer is used to perform convolution operations on the intermediate data to extract the second feature information, and then combine the second feature information with the received training data.

[0202] The second normalization layer is used to perform batch normalization processing on the combined second feature information;

[0203] The second activation layer is used to process the normalized second feature information using an activation function to obtain the predicted coloring data.

[0204] The output layer is used to output the predicted staining data.

[0205] The pixel loss function is as follows:

[0206]

[0207] In the formula, n represents the total number of pixels; y i To supervise the value of the i-th pixel in the data, x i Let f(x) be the value of the i-th pixel in the training data. i ) represents the value of the i-th pixel in the predicted staining data; loss(x,y) represents the mean square error between the supervised data and the predicted staining data;

[0208] The color loss function is as follows:

[0209]

[0210] In the formula, To predict the i-th pixel in the coloring data, The i-th pixel in the supervised data; To predict the angle between the i-th pixel in the stained data and the i-th pixel in the supervised data; the smoothness loss function is as follows:

[0211]

[0212] In the formula, S p To predict the p-th pixel in the stained data, To predict the gradient of the p-th pixel in the x-direction in the stained data, To predict the gradient of the p-th pixel in the y-direction in the stained data, c is the weight for spatial variation smoothness. To predict the weight of the p-th pixel in the x-direction in the stained data, To predict the weight of the p-th pixel in the y-direction in the stained data.

[0213] Of course, the pixel loss function, color loss function, and smoothness loss function can also be other formulas, which are not limited here.

[0214] S608: The neural network model is trained for the first time using the input data with reduced resolution as training data and the target data with reduced resolution as supervision data. The training of the neural network model is completed when the pixel difference between the predicted color data and the supervision data, the color difference between the predicted color data and the supervision data, and the smoothness difference between the predicted color data and the supervision data are all less than the corresponding preset thresholds.

[0215] The pixel loss function is used to determine the pixel difference between the predicted colored data and the supervised data by the neural network model, and convert it into specific numerical values ​​for intuitive display. Based on the pixel difference between the predicted colored data and the supervised data, it is decided whether to continue to improve the neural network model.

[0216] Similarly, the color loss function is used to determine the color difference between the color data predicted by the neural network model and the supervised data, and based on this pixel difference, it is decided whether to continue to improve the neural network model.

[0217] Similarly, the smoothness loss function is used to determine the smoothness difference between the predicted colored data and the supervised data by the neural network model, and based on this smoothness difference, it is decided whether to continue to improve the neural network model.

[0218] The training of the neural network model ends only when the pixel difference between the predicted color data and the supervised data, the color difference between the predicted color data and the supervised data, and the smoothness difference between the predicted color data and the supervised data are all less than the corresponding preset thresholds.

[0219] S609: The upsampling technique based on a double-sided grid increases the resolution of both the input data and the target data after the resolution reduction.

[0220] The steps used in this embodiment are based on the same concept as those used in the above embodiments, and the specific implementation process is detailed in step S204, which will not be repeated here.

[0221] S610: The neural network model is trained a second time using the input data with improved resolution as training data and the target data with improved resolution as supervision data. The training of the neural network model is completed when the pixel difference between the predicted color data and the supervision data, the color difference between the predicted color data and the supervision data, and the smoothness difference between the predicted color data and the supervision data are all less than the corresponding preset thresholds.

[0222] The steps used in this embodiment are based on the same concept as those used in the above embodiments, and the specific implementation process is detailed in step S205, which will not be repeated here.

[0223] As can be seen from the above technical solution, the deep learning-based pathological image color restoration method provided in this application reduces the resolution of input and target data using a bilateral grid-based downsampling technique. The reduced-resolution input and target data are then used to train the neural network model, allowing the model to learn an image-to-color mapping function in the low-resolution domain. Next, the resolution of the input and target data is increased using the bilateral grid-based downsampling technique. This increased-resolution input and target data are then used to further train the neural network model. Since the neural network model already possesses an image-to-color mapping function, its training speed is faster and more efficient compared to directly using high-resolution input and target data. Furthermore, the neural network model retains image-to-color mapping functions at various resolutions, making it suitable for pathological images acquired by digital pathology scanners of different resolutions. Simultaneously, using target data to supervise the training of the neural network model ensures that the input data has a clear objective, leading to highly accurate prediction and classification results and improving the staining effect of the neural network model on pathological images acquired by digital pathology scanners. Meanwhile, an intermediate illumination layer was added to the neural network model, and multiple supervisory signals, i.e. target data, were added to the middle of the neural network model. This allows the intermediate layers of the neural network model to learn more robust feature representations, thereby associating the input data with the target data and improving the staining effect of the neural network model on pathological images acquired by digital pathology scanners.

[0224] The deep learning-based pathological image color restoration method provided in this application uses a first symmetric convolutional layer to extract features from the training data, and a second symmetric convolutional layer to further extract features from the training data while retaining the features extracted by the first symmetric convolutional layer. This improves the feature extraction capability of the neural network model and accelerates its training speed. Furthermore, the first and second symmetric convolutional layers have the same parameters due to their vertical symmetry. This symmetry effectively reduces the number of parameters in the neural network model, thereby improving its computational efficiency and generalization performance. The first and second normalization layers accelerate the training of the neural network model and enhance its generalization ability. The first and second activation layers increase the nonlinearity of the neural network model, making it more suitable for staining pathological images acquired by digital pathology scanners.

[0225] The deep learning-based pathological image color restoration method provided in this application can measure the distance and deviation between the predicted staining data and the supervised data by constructing pixel loss function, color loss function and smoothness loss function, thereby improving the neural network model and making the predicted staining data have better contrast, saturation and clarity.

[0226] The deep learning-based pathological image color restoration method provided in this application reduces the amount of data to be registered by dividing the input data into multiple sub-input data and then using a template matching algorithm to find the sub-input data most similar to the target data. Simultaneously, the SIFT algorithm is used to register the most similar sub-input data with the target data, finding the same local features in different images (i.e., input and target data), thus resolving the inconsistency in the field of view between the input and target images. This increases the range of selectable input data, expands the training data, improves the staining effect of the neural network model on pathological images acquired by digital pathology scanners, and removes black borders generated during image registration, ensuring the integrity and accuracy of the registered sub-input data and registered target data.

[0227] The following are device embodiments of this application, which can be used to execute the method embodiments of this application. For details not disclosed in the device embodiments of this application, please refer to the method embodiments of this application.

[0228] Please see Figure 7 This illustration shows a schematic diagram of a modular virtual device for a deep learning-based pathological image color restoration scanner, provided in an exemplary embodiment of this application. The scanner can be implemented as all or part of a scanner through software, hardware, or a combination of both.

[0229] The scanner includes: a data collection module 701 for determining input data based on pathological images acquired by a digital pathology scanner; and for determining target data based on pathological images acquired under a real microscope;

[0230] The first resolution adjustment module 702 is used to reduce the resolution of input data and target data based on a two-sided grid downsampling technique;

[0231] The first neural network training module 703 is used to construct a neural network model. It uses the input data with reduced resolution as training data and the target data with reduced resolution as supervision data to train the neural network model for the first time. The supervision data is the result that the input data is expected to achieve after training.

[0232] The neural network model includes 704: an intermediate illumination layer, multiple convolutional kernels before the intermediate illumination layer, and multiple convolutional kernels after the intermediate illumination layer. The input of the intermediate illumination layer is obtained based on the output features of at least one of the preceding convolutional kernels of the intermediate illumination layer, and the input of at least one of the following convolutional kernels of the intermediate illumination layer is obtained based on the output features of the intermediate illumination layer.

[0233] The second resolution adjustment module 705 is used to improve the resolution of the input data and the target data after the resolution reduction based on the double-sided grid upsampling technique.

[0234] The second neural network training module 706 is used to train the neural network model a second time by using the input data with improved resolution as training data and the target data with improved resolution as supervision data.

[0235] The staining module 707 is used to stain pathological images acquired by the digital pathology scanner according to the trained neural network model.

[0236] In other embodiments, the neural network model further includes:

[0237] The input layer is used to receive training data;

[0238] The first symmetric convolutional layer is used to perform convolution operations on the received training data to obtain intermediate data;

[0239] The second symmetric convolutional layer is used to perform convolution operations on the intermediate data and combine it with the received training data to obtain the predicted coloring data.

[0240] The output layer is used to output the predicted staining data.

[0241] In other embodiments, the first symmetric convolutional layer specifically includes:

[0242] The first convolutional layer is used to perform convolution operations on the received training data to extract the first feature information;

[0243] The first normalization layer is used to perform batch normalization processing on the first feature information;

[0244] The first activation layer is used to process the normalized first feature information using an activation function to obtain intermediate data.

[0245] The second symmetric convolutional layer specifically includes:

[0246] The second convolutional layer is used to perform convolution operations on the intermediate data to extract the second feature information, and then combine the second feature information with the received training data.

[0247] The second normalization layer is used to perform batch normalization processing on the combined second feature information;

[0248] The second activation layer is used to process the normalized second feature information using an activation function to obtain the predicted coloring data.

[0249] In other embodiments, the scanner further includes:

[0250] The pixel loss module is used to calculate the pixel difference between the predicted coloring data and the supervision data using the pixel loss function.

[0251] The color module is used to calculate the color difference between the predicted color data and the supervised data using a color loss function;

[0252] The loss module is used to calculate the smoothness difference between the predicted coloring data and the supervised data using the smoothness loss function;

[0253] The termination module is used to complete the training of the neural network model when the pixel difference between the predicted color data and the supervised data, the color difference between the predicted color data and the supervised data, and the smoothness difference between the predicted color data and the supervised data are all less than the corresponding preset thresholds.

[0254] In other embodiments, the pixel loss function is specifically:

[0255]

[0256] In the formula, n represents the total number of pixels; y i To supervise the value of the i-th pixel in the data, x i Let f(x) be the value of the i-th pixel in the training data. i ) represents the value of the i-th pixel in the predicted staining data; loss(x,y) represents the mean square error between the supervised data and the predicted staining data;

[0257] The color loss function is as follows:

[0258]

[0259] In the formula, To predict the i-th pixel in the coloring data, The i-th pixel in the supervised data; To predict the angle between the i-th pixel in the stained data and the i-th pixel in the supervised data; the smoothness loss function is as follows:

[0260]

[0261] In the formula, S p To predict the p-th pixel in the stained data, To predict the gradient of the p-th pixel in the x-direction in the stained data, To predict the gradient of the p-th pixel in the y-direction in the stained data, c is the weight for spatial variation smoothness. To predict the weight of the p-th pixel in the x-direction in the stained data, To predict the weight of the p-th pixel in the y-direction in the stained data.

[0262] In other embodiments, the scanner further includes:

[0263] The equal-division module is used to divide the input data into multiple sub-input data;

[0264] The template matching algorithm module is used to perform template matching between sub-input data and target data to find the sub-input data that is most similar to the target data.

[0265] The SIFT algorithm module is used to register the most similar sub-input data with the target data according to the SIFT algorithm.

[0266] In other embodiments, the scanner further includes:

[0267] The black border removal module is used to remove black borders from the registered sub-input data and target data.

[0268] It should be noted that the above embodiments of the apparatus are only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided above belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0269] This application also provides a computer storage medium that can store multiple instructions, which are adapted to be loaded and executed by a processor as described above. Figures 1-6 The deep learning-based pathological image color restoration method described in the illustrated embodiment can be further explained in the following steps: Figures 1-6 The specific details of the illustrated embodiments will not be elaborated here.

[0270] This application also discloses an electronic device. (See reference...) Figure 8 , Figure 8 This is a schematic diagram of the physical device of the deep learning-based pathological image color restoration scanner provided in this application. The electronic device 800 may include: at least one processor 801, at least one network interface 804, a user interface 803, a memory 805, and at least one communication bus 802.

[0271] The communication bus 802 is used to enable communication between these components.

[0272] The user interface 803 may include a display screen and a camera. Optionally, the user interface 803 may also include a standard wired interface and a wireless interface.

[0273] The network interface 808 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0274] The processor 801 may include one or more processing cores. The processor 801 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 805, and by calling data stored in the memory 805. Optionally, the processor 801 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 801 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor 801.

[0275] The memory 805 may include random access memory (RAM) or read-only memory. Optionally, the memory 805 may include a non-transitory computer-readable storage medium. The memory 805 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 805 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 805 may also be at least one storage device located remotely from the aforementioned processor 801. (Refer to...) Figure 8 The memory 405, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application for deep learning-based pathological image color restoration.

[0276] exist Figure 8 In the illustrated electronic device 800, the user interface 803 is mainly used to provide an input interface for the user and acquire user input data; while the processor 801 can be used to call the application program for deep learning-based pathological image color restoration stored in the memory 805. When executed by one or more processors 801, the electronic device 800 performs one or more of the methods described in the above embodiments. It should be noted that, for the foregoing method embodiments, for the sake of simplicity, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0277] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0278] In the various embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between apparatuses or units may be electrical or other forms.

[0279] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0280] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0281] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0282] The above description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Other embodiments of this disclosure will be readily apparent to those skilled in the art upon consideration of the specification and the disclosure of practical truths.

[0283] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A method for color restoration of pathological images based on deep learning, characterized in that, include: Input data is determined based on pathological images acquired by a digital pathology scanner; And to determine target data based on pathological images acquired under a real microscope; The downsampling technique based on a two-sided grid reduces the resolution of the input data and the target data; A neural network model is constructed, and the input data with reduced resolution is used as training data and the target data with reduced resolution is used as supervision data for the first training of the neural network model. The supervised data is the result that the input data is intended to achieve after training during the training process; The neural network model includes: an intermediate illumination layer, multiple convolutional kernels located before the intermediate illumination layer, and multiple convolutional kernels located after the intermediate illumination layer. The input of the intermediate illumination layer is obtained based on the output features of at least one of the preceding convolutional kernels of the intermediate illumination layer, and the input of at least one of the following convolutional kernels of the intermediate illumination layer is obtained based on the output features of the intermediate illumination layer. The upsampling technique based on bilateral grids increases the resolution of both the reduced-resolution input data and the reduced-resolution target data. The improved input data is used as training data, and the improved target data is used as supervision data to train the neural network model a second time. The pathological images acquired by the digital pathology scanner are stained using the trained neural network model.

2. The deep learning-based pathological image color restoration method according to claim 1, characterized in that, The neural network model also includes: The input layer is used to receive the training data; The first symmetric convolutional layer is used to perform convolution operations on the received training data to obtain intermediate data; The second symmetric convolutional layer is used to perform convolution operations on the intermediate data and combine it with the received training data to obtain the predicted coloring data. The output layer is used to output the predicted staining data.

3. The deep learning-based pathological image color restoration method according to claim 2, characterized in that, The first symmetric convolutional layer specifically includes: The first convolutional layer is used to perform convolution operations on the received training data to extract first feature information; The first normalization layer is used to perform batch normalization processing on the first feature information; The first activation layer is used to process the normalized first feature information using an activation function to obtain the intermediate data. The second symmetric convolutional layer specifically includes: The second convolutional layer is used to perform convolution operations on the intermediate data to extract the second feature information, and combine the second feature information with the received training data; The second normalization layer is used to perform batch normalization processing on the combined second feature information; The second activation layer is used to process the normalized second feature information using an activation function to obtain the predicted coloring data.

4. The deep learning-based pathological image color restoration method according to claim 2 or 3, characterized in that, The process of training the neural network model for the first time using the input data with reduced resolution as training data and the target data with reduced resolution as supervision data, and the process of training the neural network model for the second time using the input data with increased resolution as training data and the target data with increased resolution as supervision data, further includes: The pixel difference between the predicted coloring data and the supervised data is calculated using a pixel loss function; The color difference between the predicted coloring data and the supervised data is calculated using a color loss function; The smoothness difference between the predicted coloring data and the supervised data is calculated using a smoothness loss function. The training of the neural network model is completed when the pixel difference between the predicted color data and the supervised data, the color difference between the predicted color data and the supervised data, and the smoothness difference between the predicted color data and the supervised data are all less than the corresponding preset thresholds.

5. The deep learning-based pathological image color restoration method according to claim 4, characterized in that, The pixel loss function is specifically as follows: In the formula, n represents the total number of pixels; y i To supervise the value of the i-th pixel in the data, x i Let f(x) be the value of the i-th pixel in the training data. i ) represents the value of the i-th pixel in the predicted staining data; loss(x,y) represents the mean square error between the supervised data and the predicted staining data; The color loss function is specifically as follows: In the formula, To predict the i-th pixel in the coloring data, The i-th pixel in the supervised data; To predict the angle between the i-th pixel in the stained data and the i-th pixel in the supervised data; The smoothness loss function is specifically as follows: In the formula, S p To predict the p-th pixel in the stained data, To predict the gradient of the p-th pixel in the x-direction in the stained data, To predict the gradient of the p-th pixel in the y-direction in the stained data, c is the weight for spatial variation smoothness. To predict the weight of the p-th pixel in the x-direction in the stained data, To predict the weight of the p-th pixel in the y-direction in the stained data.

6. The deep learning-based pathological image color restoration method according to claim 1, characterized in that, The input data is determined based on the pathological images acquired by the digital pathology scanner; And after determining the target data based on pathological images acquired under a real microscope, the bilateral grid-based downsampling technique reduces the resolution of the input data and the target data before it is applied. Also includes: The input data is divided into multiple sub-input data; The template matching algorithm is used to perform template matching between the sub-input data and the target data to find the sub-input data that is most similar to the target data. The most similar sub-input data is registered with the target data according to the SIFT algorithm.

7. The deep learning-based pathological image color restoration method according to claim 6, characterized in that, After registering the most similar sub-input data with the target data according to the SIFT algorithm, the process also includes: Remove the black borders from the registered sub-input data and target data.

8. A pathological image color restoration scanner based on deep learning, characterized in that, include: The data collection module is used to determine the input data based on the pathological images acquired by the digital pathology scanner; And to determine target data based on pathological images acquired under a real microscope; The first resolution adjustment module is used to reduce the resolution of the input data and the target data based on a two-sided grid downsampling technique; The first neural network training module is used to build a neural network model. It uses the input data with reduced resolution as training data and the target data with reduced resolution as supervision data to train the neural network model for the first time. The supervised data is the result that the input data is intended to achieve after training during the training process; The neural network model includes: an intermediate illumination layer, multiple convolutional kernels located before the intermediate illumination layer, and multiple convolutional kernels located after the intermediate illumination layer. The input of the intermediate illumination layer is obtained based on the output features of at least one of the preceding convolutional kernels of the intermediate illumination layer, and the input of at least one of the following convolutional kernels of the intermediate illumination layer is obtained based on the output features of the intermediate illumination layer. The second resolution adjustment module is used to increase the resolution of the input data and the target data after the resolution reduction based on the double-sided grid upsampling technique; The second neural network training module is used to train the neural network model a second time by using the input data with improved resolution as training data and the target data with improved resolution as supervision data. The staining module is used to stain pathological images acquired by the digital pathology scanner based on the trained neural network model.

9. An electronic device, characterized in that, include: One or more processors and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-7.

Images