A Hyperspectral Image Classification Method for Red Blood Cells Based on a Fusion Network

Abstract

A method for classifying hyperspectral images of red blood cells based on a fusion network, which uses the fusion network to classify hyperspectral images of red blood cells. The network includes a data preprocessing module, a data conversion module, a deep learning network module, and a model output module. A preprocessing module optimized by particle swarm optimization is designed based on the SG preprocessing method to eliminate random noise in the spectral signal. A fixed-length window data conversion module is designed to amplify the differences in one-dimensional spectral data and increase the dimension of the one-dimensional spectral data. For the outputs of the preprocessing module and the data conversion module, a deep learning network module that processes one-dimensional and two-dimensional data simultaneously is constructed. Finally, a model output module is designed based on the method of decision-level fusion, which can comprehensively consider the prediction results of one-dimensional and two-dimensional data, effectively improving the accuracy and robustness of the model. This method combines neural network and ensemble learning ideas, and improves the classification accuracy through model fusion technology to achieve effective classification of red blood cells.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine vision-assisted medical examinations, and particularly relates to a method for classifying hyperspectral images of red blood cells based on a fusion network. Background Art

[0002] Red blood cells are the most abundant type of blood cells in the blood. It is the main medium for transporting oxygen through the blood in vertebrates and also has an immune function. Many blood diseases or nutrient deficiencies can lead to the production of abnormal red blood cells. Determining the type of abnormal red blood cells is of great significance for doctors to diagnose specific diseases and promote the clinical diagnosis of related diseases. At present, using an artificial microscope to identify and detect red blood cells is a common method in many biomedical experimental studies. However, this method has high requirements for the professional level and clinical experience of personnel, and the operation process is time-consuming and labor-intensive. Therefore, proposing a fast and accurate method for classifying and identifying red blood cells is an increasingly important task. Currently, some researchers have used deep learning methods to conduct research and evaluation on red blood cell identification and detection and achieved good results. However, these methods are only based on grayscale images or RGB images with the spatial characteristics of red blood cells. Uneven illumination and staining may affect the shape and color of RGB, ultimately affecting the accuracy of classification. Hyperspectral imaging technology can simultaneously obtain spectral features and spatial image information and has been widely used in the identification of fluorescently labeled cells, the identification of cell types, and the identification of Salmonella. Traditional hyperspectral classification methods are based on one-dimensional spectral curves and use various classification algorithms to classify them. For example, "A Fiber Component Classification Method Based on One-Dimensional Convolutional Neural Network" (CN114119984A) proposes a fiber component classification method based on one-dimensional convolutional neural network, which extracts features from one-dimensional fiber near-infrared spectral data to determine the types of multiple fibers contained in the textile to be tested. However, this method does not explore and utilize deeper features of the spectral curve. Another method is to combine one-dimensional spectral data and the two-dimensional image information obtained by segmentation. For example, "A Hyperspectral Image Classification Method Based on a Dual-Channel Sparse Network" (CN115471677A) proposes to fuse spatial features and spectral features to obtain the output result. However, in the process of obtaining two-dimensional image information, the steps are complex, and inaccurate segmentation or too much background in the segmented target will lead to inaccurate classification results. In the present invention, a method of directly converting one-dimensional spectral data into a two-dimensional spatial image is proposed, which further magnifies the differences between different bands of the one-dimensional spectral curve, and the converted two-dimensional spatial image is fed into a two-dimensional convolution to learn and explore more deep features through the two-dimensional convolution. In addition, during the process of neural network modeling, in order to optimize the results, various algorithms are usually explored (such as using models with different structures or extracting different features, etc.), and multiple models are established for comparison, and finally the model with the best performance is selected for classification.However, this approach has two drawbacks. On the one hand, neural network training usually takes a lot of time. Eventually, only a single model is selected, wasting the time used to train other models. On the other hand, a model with good classification performance on the current test set may not be the optimal one in terms of performance on more new samples. Discarding other models may result in missing the model with the strongest generalization ability among them. Summary of the Invention

[0003] The object of the present invention is to provide a method for classifying hyperspectral images of red blood cells based on a fusion network. A hyperspectral microscope system is used to obtain hyperspectral images of red blood cells. Based on the SG preprocessing method, a data preprocessing module optimized by the particle swarm optimization (PSO) is designed. Based on the fixed-length window method, a data conversion module is designed to perform dimensionality increase on one-dimensional spectral data. For the outputs of the data preprocessing module and the data conversion module, a deep learning network module for simultaneously processing one-dimensional and two-dimensional data is constructed. At the same time, an output module based on decision-level fusion is designed.

[0004] The technical solution of the present invention:

[0005] A method for classifying hyperspectral images of red blood cells based on a fusion network includes the following steps:

[0006] S1. A hyperspectral microscopy imaging (HMI) system is used to capture hyperspectral images of the red blood cell-containing region in a blood smear. The tool ENVI5.3 is used to crop the red blood cell region from the collected hyperspectral images as the ROI region, and one-dimensional spectral data of each type of red blood cell is labeled to form a dataset with five types of red blood cells. A SG data preprocessing module based on the particle swarm optimization algorithm is designed to preprocess the one-dimensional spectral data of red blood cells to eliminate random noise in the spectral signal;

[0007] S1-1. The hyperspectral microscopy imaging system for collecting hyperspectral images of red blood cells includes a microscope with a 6V, 20W metal halide lamp, a VIS / NIR hyperspectral camera, and a display device. The hyperspectral microscopy imaging system operates in the visible and near-infrared bands, with a spectral range from 382.3 nm to 1020.2 nm, a spectral resolution of 2.8 nm, and a total of 360 spectral bands;

[0008] S1-2. The dataset consists of five types of red blood cells, namely spherocytes, elliptocytes, dacryocytes, rouleaux, and stomatocytes. The total number of hyperspectral images collected as the dataset is 1000, and each type of red blood cell includes 200 hyperspectrals;

[0009] S1-3. During the acquisition of hyperspectral image data of red blood cells, it may be affected by light intensity and dark current, resulting in a large amount of noise. The black hyperspectral calibration image and the white hyperspectral calibration image can remove this noise as much as possible. Among them, the black hyperspectral calibration image (reflectivity is about 0%) is collected by covering the camera lens with a lens cap; the white hyperspectral calibration image (reflectivity is about 100%) is obtained by moving a uniform white calibration plate to a position perpendicular to the lens;

[0010] S1-4. After the acquisition and calibration of the hyperspectral image of red blood cells, use the tool ENVI5.3 to select the ROI area on the hyperspectral image of red blood cells, and return the average value of the spectral reflectance of this ROI area, and label the categories of the spectral reflectance values of all ROI areas;

[0011] S1-5. Preprocess the one-dimensional spectral data. Use the data preprocessing module, that is, use the particle swarm optimization algorithm (PSO) to determine the optimal parameters of the polynomial order p and window size w of the SG preprocessing algorithm. Among them, the particle swarm optimization algorithm (PSO) is an optimization algorithm based on swarm intelligence. It searches for the optimal solution of the optimization problem through the mechanism of cooperation, information sharing and competition among particles. It has been widely used in practical problems due to its easy implementation and fast convergence. Then use the optimized SG preprocessing to process the spectral data.

[0012] S2. Divide the preprocessed one-dimensional spectral data in S1 into a training set and a test set according to a certain ratio;

[0013] S3. Design a data conversion module based on a fixed-length window; by segmenting with a fixed-length window, the similar bands within the same window are separated as much as possible, and the corresponding bands between different windows are arranged in a row in turn. The purpose of doing this is to magnify the differences in the one-dimensional spectral bands as much as possible and increase the dimension of the one-dimensional spectral data.

[0014] S3-1. Perform normalization processing on the output of the data preprocessing module, and map the one-dimensional data to [0,1] using the minmax normalization method;

[0015] S3-2. Traverse the scaled one-dimensional spectral data using a fixed-length window. The window width is denoted as W, and the number of windows is denoted as K. There is no overlap set between each window;

[0016] S3-3. Traverse each spectral value of the K windows in turn, denoted as:

[0017]

[0018] S3-4. After normalizing and segmenting the data, a two-dimensional matrix data of size [W, K] is obtained and saved as a grayscale image in the Python platform;

[0019] S4. Construct a deep learning network module that performs one-dimensional data convolution and two-dimensional data convolution simultaneously for the output of the data preprocessing module and the output of the data conversion module; the deep learning network module is formed by connecting two convolution models in parallel, and the outputs of the data preprocessing module and the data conversion module are introduced into the deep learning network module. The output of the data preprocessing module is connected to the one-dimensional convolution model and the data conversion module respectively, and the output of the data conversion module is connected to the two-dimensional convolution model; wherein the one-dimensional convolution model processes the one-dimensional spectral data output by the data preprocessing module, learns the reflectivity difference of the one-dimensional spectral data through the convolution layer and the pooling layer, and the softmax function of the last layer outputs the probability distribution of the sample belonging to each type; the two-dimensional convolution model processes the two-dimensional grayscale image output by the data conversion module, learns the features of the two-dimensional grayscale image through the convolution layer and the pooling layer, and the softmax function of the last layer outputs the probability distribution of the sample belonging to each type;

[0020] In the training phase, the output of the data preprocessing module is input into the one-dimensional convolution model for training until the one-dimensional convolution model converges or reaches a preset number of training times; at the same time, the output of the data conversion module is input into the two-dimensional convolution model for training until the two-dimensional convolution model converges or reaches a preset number of training times;

[0021] S5. Design an output module based on decision layer fusion. In the model prediction stage, the probability of the last softmax output of the one-dimensional convolution model and the two-dimensional convolution model is used. Each probability value is regarded as the confidence of the model for this type. The output of the one-dimensional convolution model and the two-dimensional convolution model is introduced into the output module. The probability of the samples belonging to each type in the output of the one-dimensional convolution model and the two-dimensional convolution model is weighted and fused, and then the type with the largest probability is used as the classification prediction result of the fusion network.

[0022] In step S2, the dataset is divided into two parts, including a training set and a test set, in a ratio of 7:3.

[0023] In step S3, W=24 and K=15 are set. This is mainly because in the entire band of red blood cells, the spectral reflectance values ​​between adjacent bands are not very different. Dividing these bands into the same window can maximize the difference in spectral reflectance values ​​in different windows. The two-dimensional matrix obtained in this way can ensure that the difference in one-dimensional spectral data is maximized. The generated two-dimensional image is sent to the two-dimensional convolution, which can maximize the deep features and improve the classification accuracy.

[0024] Beneficial effects of the present invention:

[0025] (1) In the present invention, an SG data preprocessing module based on the Particle Swarm Optimization (PSO) algorithm is designed. By taking advantage of the simplicity and fast convergence speed of the particle swarm algorithm, the parameters of the SG preprocessing algorithm are optimized to further improve the classification speed of the network. The optimized SG algorithm is used to smooth the spectral data, removing the noise therein and eliminating the interference factors.

[0026] (2) In the present invention, a data conversion module based on a fixed-length window is designed. By using the method of the fixed-length window, the difference of one-dimensional spectral data can be amplified as much as possible. Sending the obtained two-dimensional grayscale image into a two-dimensional convolution may enable the network to mine deeper features and improve the classification accuracy.

[0027] (3) The present invention draws on the idea of ensemble learning in machine learning, weights and fuses the probabilities belonging to each type output by the one-dimensional convolution model and the two-dimensional convolution model, and then takes the type with the maximum probability as the classification prediction result of the fusion network, which can improve the classification accuracy and the robustness of the model.

[0028] (4) The present invention can be applied to the classification of different types of cell types and has certain application value in the field of hyperspectral data processing and classification. Description of the Drawings

[0029] Figure 1 is a schematic flow chart of a hyperspectral image classification method of red blood cells based on a fusion network;

[0030] Figure 2 represents the result graph after the original spectrum is preprocessed by SG;

[0031] Figure 3 is the two-dimensional grayscale image obtained by respectively passing the one-dimensional spectral reflectance values of five types of red blood cells through the conversion module;

[0032] Figure 4 is the loss curve of the training set of an embodiment of the present invention. Detailed Embodiment

[0033] The technical solution of the present invention will be further described below with reference to the drawings and embodiments.

[0034] Embodiment

[0035] A hyperspectral image classification method of red blood cells based on a fusion network includes the following steps:

[0036] S1. Obtain hyperspectral image data of red blood cells through a hyperspectral microscopy imaging system, which altogether includes five types of abnormal red blood cells, namely spherocytes, ovalocytes, teardrop-shaped red blood cells, rouleaux, and stomatocytes. During the acquisition of hyperspectral images, it will be affected by light intensity and dark current. Using black and white plate calibration can eliminate these effects as much as possible. Use ENVI5.3 to crop the ROI area of the collected hyperspectral images. There are altogether 1000 ROI areas. Label the average reflection spectra of each ROI area to obtain 1000 red blood cell samples, with 200 samples for each type of red blood cell. SG smoothing is used to eliminate random noise in the spectral signal and improve the signal-to-noise ratio of red blood cell samples. In the data preprocessing module of the present invention, the preprocessing algorithm is set to SG. The parameter polynomial order p and window size w of SG are optimized by the particle swarm optimization algorithm (PSO) to select the best p and w. Through continuous optimization, when the PSO search parameters are specifically set as follows: the maximum population size is 30, c1 = 1.5, c2 = 1.7, the maximum number of iterations is 2000, the best p and w are 3 and 5 respectively.

[0037] S2. Divide the preprocessed one-dimensional spectral data in S1 according to a certain ratio to obtain a training set and a test set. The output of the data preprocessing module is 1000. According to the ratio of 7:3, the numbers of the training set and the test set are 899 and 101 respectively.

[0038] S3. Design a data conversion module based on a fixed-length window to increase the dimension of the output of the data preprocessing module. The process includes:

[0039] S3-1. Perform normalization processing on the output of the data preprocessing module, and use the minmax normalization method to map one-dimensional data to [0,1];

[0040] S3-2. Traverse the scaled one-dimensional spectral data using a fixed-length window, where the window width is denoted as W and the number of windows is denoted as K;

[0041] S3-3. Traverse each spectral value of K windows in turn, denoted as:

[0042]

[0043] S3-4. After normalization and data segmentation, obtain two-dimensional matrix data with a size of [W, K], and save it as a grayscale image in the Python platform.

[0044] The hyperspectral system used in the present invention includes wavelengths from 382.3 nm to 1020.2 nm, with a total of 360 bands. In the data conversion module, we set W = 24 and K = 15. This is mainly because in the entire band of red blood cells, the spectral reflection values between adjacent bands do not differ much. By dividing these bands into the same window, the spectral reflection values in different windows can be maximally different. The resulting two-dimensional matrix can ensure the maximization of the differences in one-dimensional spectral data. The generated two-dimensional image is fed into a two-dimensional convolution, which can maximize the extraction of deep features and improve the classification accuracy.

[0045] S4. Construct a deep learning network module capable of performing one-dimensional convolution and two-dimensional convolution simultaneously. Among them, one-dimensional convolution classifies one-dimensional spectral data, with an input dimension of 1×360. The one-dimensional convolution model structure used is as Figure 1 shown. It has an input layer, two convolutional layers, two pooling layers, and a flatten layer, followed by three fully connected layers at the end. Then, in the first convolution operation, the convolution kernel size is 12, the stride is 1, and the number of convolution kernels is 8. The second convolution kernel size is 3, the stride is 1, and the number of convolution kernels is 16. There is a pooling layer after each convolutional layer. Then, there is a flatten layer, and finally, through three fully connected layers, the classification information is output using the softmax activation function. Two-dimensional convolution classifies two-dimensional grayscale images. The input is the output of the data conversion module, with an input dimension of 24×15. The two-dimensional convolution model structure used is as Figure 1 shown. It has an input layer, two convolutional layers, two pooling layers, and a flatten layer, followed by three fully connected layers at the end. Then, in the first convolutional layer, the convolution kernel is 3×3, the stride is 1×1, and the number is 8. The convolution kernel size of the second convolutional layer is 3×3, the stride is 1×1, and the number is 16. There is a pooling layer after each convolutional layer. Then, there is a flatten layer, and finally, through three fully connected layers, the classification information is output using the softmax activation function.

[0046] S5. Design a model output module based on decision-level fusion. In the prediction stage, the outputs of the last layers of one-dimensional convolution and two-dimensional convolution are respectively input into the output module based on decision-level fusion, and the prediction output of the fusion network can be obtained. Among them, in the output module, the probabilities belonging to each type output by the one-dimensional convolution and two-dimensional convolution models are weighted and fused, and then the type with the highest probability is used as the classification prediction result of the fusion network. This can comprehensively consider the prediction results of the one-dimensional convolution and two-dimensional convolution models and improve the robustness of the model.

[0047] S6. Based on the above modules, a fusion network is constructed. The fusion network mainly consists of the data preprocessing module in S1, the data conversion module in S3, the deep learning module in S4, and the output module based on the decision layer in S5. In the training stage, the training set of the one-dimensional spectral data of red blood cells collected from microscopic hyperspectral data is used as the input and sent into the data preprocessing module of the fusion network. The particle swarm optimization (PSO) algorithm is used to optimize the polynomial order p and window size w of the SG algorithm, where the optimal parameters of p and w are 3 and 5 respectively. Then, the output of the data preprocessing module is sent to the one-dimensional convolution of the deep learning network module on the one hand and to the data conversion module on the other hand. The fixed-length window method is used to increase the dimension of the one-dimensional spectral data to obtain a two-dimensional grayscale image, which is input into the two-dimensional convolution of the deep learning network module. The one-dimensional convolution and two-dimensional convolution of the deep learning module are trained simultaneously until the loss of the model no longer changes or the number of iterations of the model reaches the set number, and the model is saved. In the prediction stage, the prediction set is sent into the data preprocessing module of the fusion network in the same way, and the prediction results of the one-dimensional convolution and two-dimensional convolution for the prediction set samples are obtained. The prediction results of the one-dimensional convolution and two-dimensional convolution are sent into the model output module of the fusion network. In the output module, the probabilities of the samples output by the one-dimensional convolution and two-dimensional convolution models belonging to each type are weighted and fused, and the type with the highest probability is used as the classification prediction result of the fusion network. The output result of the fusion network is compared with the label to obtain the classification result of the fusion network, as shown in Table 1.

[0048] Table 1 is the confusion matrix of the classification results of the test set in an embodiment

[0049]

Claims

1. A method for classifying hyperspectral images of red blood cells based on a fusion network, characterized in that, it includes the following steps: S1. The hyperspectral microscopy imaging system is used to capture hyperspectral images of the red blood cell-containing regions in blood smears. The tool ENVI5.3 is used to crop the red blood cell regions from the collected hyperspectral images as ROI regions, and the one-dimensional spectral data of each type of red blood cell is labeled to form a dataset with five types of red blood cells; A SG data preprocessing module based on the particle swarm optimization algorithm is designed to preprocess the one-dimensional spectral data of red blood cells to eliminate random noise in the spectral signals; S1-1. The hyperspectral microscopy imaging system for collecting hyperspectral images of red blood cells includes a microscope with a 6V, 20W metal halide lamp, a VIS / NIR hyperspectral camera, and a display device; S1-2. The dataset consists of five types of red blood cells, namely spherocytes, elliptocytes, dacryocytes, rouleaux, and stomatocytes; The total number of hyperspectral images collected as the dataset is 1000, and each type of red blood cell includes 200 hyperspectrals; S1-3. For the acquisition of the black hyperspectral calibration image, the camera lens is covered with a lens cap; For the white hyperspectral calibration image, a uniform white calibration plate is moved to a position perpendicular to the lens; S1-4. After the acquisition and calibration of the hyperspectral images of red blood cells, the tool ENVI5.3 is used to select the ROI regions on the hyperspectral images of red blood cells, and the average value of the spectral reflectance of the ROI regions is returned, and the categories of the spectral reflectance of all ROI regions are labeled; S1-5. Preprocess the one-dimensional spectral data. Using the data preprocessing module, that is, use the particle swarm algorithm (PSO) to determine the optimal parameters of the polynomial order p and window size w of the SG preprocessing algorithm; S2. Divide the preprocessed one-dimensional spectral data in S1 into a training set and a test set according to a certain ratio; S3. Design a data conversion module based on a fixed-length window; By segmenting with a fixed-length window, the similar bands within the same window are separated as much as possible, and the corresponding bands between different windows are arranged in a row in turn; S3-1. Normalize the output of the data preprocessing module, and use the minmax normalization method to map the one-dimensional data to [0,1]; S3-2. Traverse the scaled one-dimensional spectral data with a fixed-length window. The window width is denoted as W, and the number of windows is denoted as K. No overlapping parts are set between each window; S3-3. Traverse each spectral value of the K windows in turn, denoted as: S3-4. After normalization and data segmentation, a two-dimensional matrix data with a size of [W, K] is obtained and saved as a grayscale image in the Python platform; S4. Construct a deep learning network module that simultaneously performs one-dimensional data convolution and two-dimensional data convolution for the outputs of the data preprocessing module and the data conversion module. This deep learning network module is composed of two parallel convolutional models. The outputs of the data preprocessing module and the data conversion module are introduced into the deep learning network module. The output of the data preprocessing module is respectively connected to the one-dimensional convolutional model and the data conversion module, and the output of the data conversion module is connected to the two-dimensional convolutional model. Among them, the one-dimensional convolutional model processes the one-dimensional spectral data output by the data preprocessing module, learns the reflectance differences of the one-dimensional spectral data through convolutional layers and pooling layers, and the softmax function of the last layer outputs the probability distribution of the sample belonging to each type. The two-dimensional convolutional model processes the two-dimensional grayscale image output by the data conversion module, learns the features of the two-dimensional grayscale image through convolutional layers and pooling layers, and the softmax function of the last layer outputs the probability distribution of the sample belonging to each type. In the training stage, the output of the data preprocessing module is input into this one-dimensional convolutional model for training until the one-dimensional convolutional model converges or reaches the preset number of training times. At the same time, the output of the data conversion module is input into the two-dimensional convolutional model for training until the two-dimensional convolutional model converges or reaches the preset number of training times. S5. Design an output module based on decision-level fusion. In the model prediction stage, for the probabilities output by the softmax of the last layer of the one-dimensional convolutional model and the two-dimensional convolutional model, each probability value is regarded as the confidence of the model for this type. The outputs of the one-dimensional convolutional model and the two-dimensional convolutional model are introduced into the output module, and the probabilities of the sample belonging to each type in the outputs of the one-dimensional convolutional model and the two-dimensional convolutional model are weighted and fused, and then the type with the highest probability is used as the classification prediction result of the fusion network.

2. The method for classifying red blood cell hyperspectral images based on a fusion network according to claim 1, characterized in that, in step S2, the data set is divided into two parts, including a training set and a test set, and is divided according to a ratio of 7:

3.

3. The method for classifying red blood cell hyperspectral images based on a fusion network according to claim 1, characterized in that, in step S3, it is set that W = 24 and K = 15.

Images