A cell image super-resolution method based on image enhancement model

Abstract

The present invention relates to a cell image super-resolution method based on an image enhancement model, comprising collecting cell images and preparing a data set; training a convolutional neural network based on the data set to obtain an image enhancement model; judging whether a required image super-resolution multiple M1 is greater than a set threshold value, and if so, performing resolution enhancement on the image to be enhanced based on the image enhancement model, otherwise performing resolution enhancement on the image using other methods. The technical solution of the present invention is based on training an image enhancement model based on a convolutional neural network, and can achieve high-magnification image super-resolution, and the super-resolved image has high clarity and is not distorted. At the same time, when super-resolution amplification is performed on the image, only one multiple amplification is performed, and super-resolution magnified images of multiple magnifications can be quickly obtained, and the image clarity is high, computing resources are saved, and the image super-resolution rate is accelerated.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a cell image super-resolution method based on an image enhancement model. Background Art

[0002] Pathological diagnosis has become an important basis for disease diagnosis. Conventional pathological diagnosis uses microscopes to observe sections. However, with the process of modernization, the digitization of pathology data has become a trend, and a large number of digital images have also been generated. Cytology and histology image browsing based on B / S architecture has become an important component.

[0003] However, due to the limitation of the scanning magnification of the scanner, the maximum scanning magnification of the general objective lens for digital images of pathological sections is 40 times. In actual use, the viewing magnification often exceeds 40 times. At this time, image super-resolution technology will be used to super-enhance the image. The existing technology will make the image display blurred when the magnification is too large, affecting the display perception of the entire image. Therefore, a new image super-resolution method is needed to improve the clarity of the image after super-resolution. Summary of the Invention

[0004] In order to solve the above-mentioned technical defects in the prior art, the present invention proposes a cell image super-resolution method based on an image enhancement model. The image enhancement model is trained based on a convolutional neural network, which can maintain the image quality and clarity under ultra-high magnification image super-resolution conditions.

[0005] The technical solution for achieving the purpose of the present invention is: a cell image super-resolution method based on an image enhancement model, comprising the following steps:

[0006] Collect cell images and create data sets;

[0007] Train a convolutional neural network based on the dataset to obtain an image enhancement model;

[0008] Determine whether the required image super-resolution multiple M1 is greater than the set threshold. If so, perform resolution enhancement on the image to be enhanced based on the image enhancement model; otherwise, use other methods to enhance the image resolution.

[0009] As a preferred implementation, the process of making the data set is:

[0010] Acquiring digital cytological slide images and dividing the digital cytological slide images into image blocks of the same size;

[0011] The image blocks are divided into training set, validation set and test set according to a certain ratio.

[0012] As a preferred embodiment, the process of training the convolutional neural network is as follows:

[0013] Preprocess the data set;

[0014] The convolutional neural network is trained based on the preprocessed training set data.

[0015] As a preferred implementation, the process of preprocessing the data set is as follows:

[0016] Based on bicubic interpolation, each original image block is first reduced by N times and then enlarged by N times to obtain a low-resolution image block corresponding to the original image block;

[0017] Convert the original image blocks and low-resolution image blocks into YCbCr format.

[0018] As a preferred embodiment, the convolutional neural network training process is as follows:

[0019] The Y channel data of the low-resolution image block in YCbCr format is used as the input image of the convolutional neural network, and the Y channel data of the original image block in YCbCr format is used as the label image corresponding to the input image;

[0020] The convolutional neural network is trained based on the input image and label map, using the mean square error value as the loss function. The end condition of the training can be set to when the accuracy of image processing reaches a set threshold or the number of iterations reaches a set upper limit.

[0021] As a preferred embodiment, the convolutional neural network has three layers, wherein the convolution kernel size of the first layer is 9*9 and the number of channels is 64;

[0022] The convolution kernel size of the second layer is 5*5 and the number of channels is 32;

[0023] The convolution kernel size of the third layer is 5*5 and the number of channels is 1;

[0024] The sliding step size of each convolution layer is set to 1, and padding operation is performed in each convolution layer.

[0025] As a preferred implementation, the convolutional neural network is trained using the Adam optimizer, with a learning rate set to 0.001 and the number of iterative training rounds set to 500.

[0026] As a preferred embodiment, the process of performing resolution enhancement on cell images based on the image enhancement model is as follows:

[0027] After equally dividing the image to be enhanced into image blocks, the image blocks are enlarged by M1 times using bicubic interpolation and converted into YCbCr format;

[0028] The Y channel data of the YCbCr format image block is input into the trained image enhancement model, and the output image replaces the original Y channel data to obtain a new YCbCr format image block. The YCbCr format image block is converted back to the original format to obtain an image with a resolution magnified N times.

[0029] As a preferred embodiment, the process of performing resolution enhancement on cell images based on the image enhancement model is as follows:

[0030] After equally dividing the image to be enhanced into image blocks, the image blocks are enlarged by M2 times using bicubic interpolation, where M2>M1, and converted into YCbCr format;

[0031] Input the Y channel data of the YCbCr format image block into the trained image enhancement model, replace the original Y channel data with the output image to obtain a new YCbCr format image block, and convert the YCbCr format image block back to the original format, that is, to obtain an image with a resolution magnified M2 times;

[0032] The image magnified M2 times is downsampled by bilinear interpolation to obtain an image magnified M1 times.

[0033] As a preferred implementation, when the required image super-resolution factor M1 is ≥ 1.5, the image enhancement model is used to perform resolution enhancement on the image to be enhanced;

[0034] When M1≤1.5, bilinear interpolation is used to enhance the image resolution.

[0035] Compared with the prior art, the present invention has the following significant advantages and beneficial effects:

[0036] (1) The traditional bilinear interpolation image magnification method will cause image distortion and quality degradation when the magnification is large. The present invention is based on a convolutional neural network to train an image enhancement model, which can achieve large-scale image super-resolution, and the super-resolved image has high clarity and no distortion;

[0037] (2) The technical solution of the present invention only needs to perform one magnification to quickly obtain super-resolution enlarged images of multiple magnifications when performing super-resolution amplification on an image. For example, when it is necessary to magnify an image by 11, 12, or 13 times, the present invention only needs to perform one amplification using the image enhancement model (for example, 20 or 30 times super-resolution amplification). Then, based on the 20 or 30 times magnified image, bilinear interpolation downsampling is used to obtain super-resolution enlarged images of different magnifications. The image clarity is high, computing resources are saved, and the image super-resolution rate is accelerated.

[0038] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description or be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like parts throughout the drawings.

[0040] Figure 1 It is a flow chart of the steps of the present invention.

[0041] Figure 2 Schematic diagram of the convolutional neural network structure used in the embodiment of the present invention. DETAILED DESCRIPTION

[0042] It is easy to understand that, based on the technical solution of the present invention, a person of ordinary skill in the art can imagine various embodiments of the present invention without changing the essential spirit of the present invention. Therefore, the following specific embodiments and drawings are merely illustrative of the technical solution of the present invention and should not be regarded as the whole of the present invention or as a limitation or limitation of the technical solution of the present invention. On the contrary, the purpose of providing these embodiments is to enable those skilled in the art to understand the present invention more thoroughly. The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the innovative concept of the present invention.

[0043] Example 1

[0044] like Figure 1 , a cell image super-resolution method based on an image enhancement model, comprising the following steps:

[0045] Collect cell images and create data sets, including:

[0046] Acquiring digital cytological slide images and dividing the digital cytological slide images into image blocks of the same size;

[0047] The image blocks are divided into a training set, a validation set, and a test set in a certain ratio. In this embodiment, the image blocks are divided into a training set, a validation set, and a test set in a ratio of 7:1:2.

[0048] Based on the dataset, a convolutional neural network is trained to obtain an image enhancement model. The specific process includes:

[0049] Preprocess the data set, namely:

[0050] Based on bicubic interpolation, each original image block is first reduced by N times and then enlarged by N times to obtain a low-resolution image block corresponding to the original image block;

[0051] The original image block and the low-resolution image block are in one-to-one correspondence. At this time, both the original image block and the low-resolution image block are converted into the YCbCr format.

[0052] The convolutional neural network is trained based on the preprocessed training set data.

[0053] The Y channel data of the low-resolution image block in YCbCr format is used as the input image of the convolutional neural network, and the Y channel data of the original image block in YCbCr format is used as the label map corresponding to the input image. Now, with the corresponding input image and label map, the convolutional neural network can be trained.

[0054] The convolutional neural network is trained based on the input image and label map, and the mean square error value is used as the loss function.

[0055] The convolutional neural network used in this embodiment has three layers, where the convolution kernel size of the first layer is 9*9 and the number of channels is 64;

[0056] The convolution kernel size of the second layer is 5*5 and the number of channels is 32;

[0057] The convolution kernel size of the third layer is 5*5 and the number of channels is 1;

[0058] The sliding step size of each convolution layer is set to 1, and padding is performed during each convolution layer so that the size of each layer is 128*128;

[0059] In this embodiment, the Adam optimizer is used in the convolutional neural network training, the learning rate is set to 0.001, and the number of iterative training rounds is set to 500 rounds.

[0060] Determine whether the required image super-resolution multiple M1 is greater than the set threshold. If so, perform resolution enhancement on the image to be enhanced based on the image enhancement model. Otherwise, use other methods to enhance the image resolution. In this embodiment, bilinear interpolation sampling is used to enhance the image resolution. The specific process is as follows:

[0061] First, a threshold for using the image enhancement model is set. Since bilinear interpolation has a relatively good image amplification effect at low magnifications and requires relatively few computing resources, in order to save computing resources, we choose to use bilinear interpolation for image amplification at low magnifications. In this embodiment, when the required image super-resolution magnification M1 is ≥ 1.5, the image enhancement model is used to enhance the resolution of the image to be enhanced.

[0062] When M1≤1.5, bilinear interpolation upsampling is used to enhance the image resolution. In actual operation, other interpolation resolution enhancement methods can also be used, such as bilinear, bilateral, spline interpolation, etc.

[0063] When using image enhancement models for image super-resolution:

[0064] First, the image to be enhanced is evenly divided into image blocks, and then the image blocks are enlarged by M1 times using bicubic interpolation and converted into YCbCr format;

[0065] The Y channel data of the YCbCr format image block is input into the trained image enhancement model, and the output image replaces the original Y channel data to obtain a new YCbCr format image block. The YCbCr format image block is converted back to the original format to obtain an image with a resolution magnified N times.

[0066] Furthermore, as a better implementation method, during the actual image browsing process, the magnification requirements for image browsing are often constantly changing. If we need to use the image enhancement model for super-resolution every time we request an image magnification, it will not only require high computing resources but also sacrifice browsing speed.

[0067] Therefore, considering that the effect of bilinear interpolation downsampling of images is often better, we can also set an integer M2 when performing image enhancement, M2>M1;

[0068] After equally dividing the image to be enhanced into image blocks, the image blocks are enlarged by M2 times using bicubic interpolation and converted into YCbCr format;

[0069] Input the Y channel data of the YCbCr format image block into the trained image enhancement model, replace the original Y channel data with the output image to obtain a new YCbCr format image block, and convert the YCbCr format image block back to the original format, that is, to obtain an image with a resolution magnified M2 times;

[0070] The image magnified M2 times is downsampled by bilinear interpolation to obtain an image magnified M1 times.

[0071] For example, when the required enhancement factor is 8 times, the image to be enhanced can be super-resolved to 20 times using the image enhancement model to obtain a 20-fold magnified image to be enhanced, and then bilinear interpolation downsampling can be used to reduce the 20-fold magnified image to 8 times the original image.

[0072] In this case, not only can an image magnified 8 times be obtained, but also, based on the image magnified 20 times, super-resolution images of all magnifications below 20 times can be obtained by bilinear interpolation downsampling.

[0073] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

[0074] It should be understood that in order to simplify the present invention and help those skilled in the art understand the various aspects of the present invention, in the above description of the exemplary embodiments of the present invention, various features of the present invention are sometimes described in a single embodiment or described with reference to a single figure. However, the present invention should not be interpreted as if all the features included in the exemplary embodiments are essential technical features of the claims of this patent.

[0075] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0076] It should be understood that the modules, units, components, etc. included in the device of one embodiment of the present invention can be adaptively changed to be installed in a device different from the embodiment. The different modules, units, or components included in the device of the embodiment can be combined into a single module, unit, or component, or they can be divided into multiple sub-modules, sub-units, or sub-components.

[0077] The modules, units or components in the embodiments of the present invention may be implemented in hardware, or in software running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) may be used in practice to implement the embodiments of the present invention. The present invention may also be implemented as a computer program product or computer-readable medium for executing part or all of the methods described herein.

Claims

1. A cell image super-resolution method based on an image enhancement model, characterized in that: The following steps are involved: Collect cell images and create data sets: Acquiring digital cytological slide images and dividing the digital cytological slide images into image blocks of the same size; Divide the image blocks into training set, validation set and test set according to a certain ratio; Train a convolutional neural network based on the dataset to obtain an image enhancement model; Determine whether the required image super-resolution multiple M1 is greater than the set threshold. If so, perform resolution enhancement on the image to be enhanced based on the image enhancement model. Otherwise, use other methods to enhance the image resolution, including: After equally dividing the image to be enhanced into image blocks, the image blocks are enlarged by M1 times using bicubic interpolation and converted into YCbCr format; Input the Y channel data of the YCbCr format image block into the trained image enhancement model, replace the original Y channel data with the output image to obtain a new YCbCr format image block, and convert the YCbCr format image block back to the original format to obtain an image with a resolution magnified N times; or: After equally dividing the image to be enhanced into image blocks, the image blocks are enlarged by M2 times using bicubic interpolation, where M2>M1, and converted into YCbCr format; Input the Y channel data of the YCbCr format image block into the trained image enhancement model, replace the original Y channel data with the output image to obtain a new YCbCr format image block, and convert the YCbCr format image block back to the original format, that is, to obtain an image with a resolution magnified M2 times; Perform bilinear interpolation downsampling on the image magnified M2 times to obtain an image magnified M1 times; The process of training the convolutional neural network is as follows: Preprocess the data set; Train the convolutional neural network based on the preprocessed training set data; The process of preprocessing the data set is as follows: Based on bicubic interpolation, each original image block is first reduced by N times and then enlarged by N times to obtain a low-resolution image block corresponding to the original image block; Convert the original image block and the low-resolution image block into YCbCr format; The process of training the convolutional neural network is as follows: The Y channel data of the low-resolution image block in YCbCr format is used as the input image of the convolutional neural network, and the Y channel data of the original image block in YCbCr format is used as the label image corresponding to the input image; The convolutional neural network is trained based on the input image and label map, and the mean square error value is used as the loss function.

2. The cell image super-resolution method according to claim 1, wherein: The convolutional neural network has three layers, where the convolution kernel size of the first layer is 9*9 and the number of channels is 64; The convolution kernel size of the second layer is 5*5 and the number of channels is 32; The convolution kernel size of the third layer is 5*5 and the number of channels is 1; The sliding step size of each convolution layer is set to 1, and padding operation is performed in each convolution layer.

3. The cell image super-resolution method according to claim 1, wherein: The convolutional neural network was trained using the Adam optimizer, with a learning rate set to 0.001 and the number of iterative training rounds set to 500.

4. The cell image super-resolution method according to claim 1, wherein: When the required image super-resolution factor M1 is ≥ 1.5, the image enhancement model is used to enhance the resolution of the image to be enhanced. When M1≤1.5, bilinear interpolation is used to enhance the image resolution.

Images