Hyperspectrum-based bone marrow cell recognition method, system, equipment and medium

Abstract

The invention relates to a bone marrow cell recognition method, system and device based on hyperspectrum and a medium, and belongs to the technical field of cell image classification, and the method comprises the following steps: obtaining hyperspectral data of a bone marrow cell smear; preprocessing the hyperspectral data; extracting spectral characteristics and spatial information of a single bone marrow cell from the preprocessed hyperspectral data to obtain neighborhood data containing the spectral characteristics and the spatial information; inputting the neighborhood data into a pre-trained bone marrow cell classifier for feature extraction and classification to obtain a classification result of the bone marrow cells; wherein the bone marrow cell classifier is obtained based on deep learning model training, and the classification result of the bone marrow cells comprises the line type, the growth stage and the pathology of the bone marrow cells. According to the invention, automatic identification of the bone marrow cells is realized, and the accuracy and efficiency of classification of the bone marrow cells are remarkably improved.

Description

Technical Field

[0001] The present application relates to the technical field of cell image classification, and in particular to a bone marrow cell recognition method, system, device and medium based on hyperspectral. Background Art

[0002] In the medical field, especially in hematology and pathology, the identification and classification of bone marrow cells is a vital task. Bone marrow cell morphology microscopy is one of the important diagnostic methods in hematology and is widely used in the diagnosis of various diseases, such as leukemia, multiple myeloma, lymphoma, anemia and pancytopenia. Traditional bone marrow cell identification mainly relies on manual microscopy, and experienced doctors make judgments based on factors such as cell morphology, chromatin, granules, and nucleoli. However, this traditional method is time-consuming, subjective, easily affected by human factors, and difficult to meet the needs of large-scale testing. With the development of science and technology, computer-aided diagnosis technology has gradually become a trend, especially the application of machine vision and artificial intelligence, which has greatly improved the efficiency and accuracy of bone marrow cell identification.

[0003] At present, although a variety of optical imaging technologies have been applied to cell recognition, most of these technologies are limited to traditional optical microscopy and fluorescence imaging, fail to fully utilize spectral information, and perform poorly when processing complex scenes, especially in distinguishing similar types of cells or identifying abnormal cells. Therefore, how to overcome the limitations of existing technologies and quickly and accurately classify bone marrow cells is an urgent problem to be solved. Summary of the invention

[0004] In order to improve the accuracy and efficiency of bone marrow cell classification, the present application provides a bone marrow cell identification method, system, device and medium based on hyperspectral.

[0005] In a first aspect, the present application provides a method for identifying bone marrow cells based on a hyperspectral spectrum, using the following technical solution: A method for identifying bone marrow cells based on a hyperspectral spectrum, the method comprising: Acquire hyperspectral data of a bone marrow cell smear; wherein the hyperspectral data includes two-dimensional spatial information and spectral information of multiple bands; Preprocessing the hyperspectral data; Extracting spectral features and spatial information of a single bone marrow cell from the preprocessed hyperspectral data to obtain neighborhood data containing the spectral features and spatial information; The neighborhood data is input into a pre-trained bone marrow cell classifier for feature extraction and classification to obtain the classification results of the bone marrow cells; wherein the bone marrow cell classifier is trained based on a deep learning model, and the classification results of the bone marrow cells include the lineage, growth stage and pathology of the bone marrow cells.

[0006] By adopting the above technical solutions, we can obtain high-quality hyperspectral data, preprocess it, extract features, and classify it using a deep learning model. Ultimately, we can obtain important information such as the cell lineage, growth stage, and pathology. This enables automated identification of bone marrow cells, significantly improves the accuracy and efficiency of bone marrow cell classification, and provides strong support for clinical practice.

[0007] Optionally, the step of preprocessing the hyperspectral data includes: Performing image enhancement processing on the hyperspectral data; Performing dimension reduction processing on the spectral dimension of the hyperspectral data based on a principal component analysis algorithm; The hyperspectral data after dimensionality reduction are normalized.

[0008] By adopting the above technical solutions, these preprocessing steps effectively improve the accuracy and efficiency of subsequent feature extraction and classification, laying a solid foundation for the automatic identification of bone marrow cells.

[0009] Optionally, the step of extracting spectral features and spatial information of a single bone marrow cell from the preprocessed hyperspectral data to obtain neighborhood data containing the spectral features and spatial information includes: Identify a single target bone marrow cell in the hyperspectral data based on an image processing algorithm, and select the target bone marrow cell as a target center sample point; The neighborhood data of the target center sample point is extracted, and the neighborhood data is arranged in columns and spread out side by side as model input.

[0010] By adopting the above technical solution, image processing technology is used to identify a single bone marrow cell and determine it as the target center sample point, the neighborhood data of the target center sample point is extracted, and these data are arranged in columns and spread out side by side as the input of the model, which effectively ensures the accuracy of feature extraction, provides high-quality data input for subsequent bone marrow cell classification, and significantly improves the accuracy and efficiency of bone marrow cell morphological microscopy.

[0011] Optionally, the method further comprises a training step of the bone marrow cell classifier, the training step comprising: acquiring a sample spectral data set and preprocessing the data set; wherein the sample spectral data set comprises a plurality of sample spectral image data and classification results of pre-labeled bone marrow cells; Extracting sample neighborhood data from the preprocessed sample spectral data set; Dividing the sample neighborhood data into a training set, a validation set, and a test set; Inputting the training set into a pre-built three-dimensional convolutional neural network model for initial training; The three-dimensional convolutional neural network model after initial training is verified and model parameters are optimized based on the verification set and the test set to obtain the trained bone marrow cell classifier.

[0012] By adopting the above technical solution, a model that can automatically and accurately classify bone marrow cells was obtained, which improved the accuracy and efficiency of bone marrow cell classification, reduced the workload of manual classification, and provided a new and efficient tool for the diagnosis of blood diseases.

[0013] Optionally, the step of inputting the training set into a pre-built three-dimensional convolutional neural network model for initial training, and verifying and optimizing model parameters of the initially trained three-dimensional convolutional neural network model based on the verification set and the test set includes: Inputting the training set into a pre-built three-dimensional convolutional neural network model for iterative training, optimizing the model parameters and calculating the cost function of the model, until the cost function meets a preset condition or the number of iterations reaches a preset threshold, thereby obtaining a trained three-dimensional convolutional neural network model; Verifying the three-dimensional convolutional neural network model based on the verification set, evaluating the performance of the three-dimensional convolutional neural network model and adjusting model parameters according to the verification results; wherein the model parameters include the number of convolution kernels, the size of the convolution kernel, the number of neurons in the fully connected layer, and the learning rate; The prediction ability of the adjusted three-dimensional convolutional neural network model is tested based on the test set to obtain the trained bone marrow cell classifier.

[0014] By adopting the above technical solution, the model is trained, verified and tested using training sets, validation sets and test sets, and the model performance is continuously improved. Ultimately, a bone marrow cell classifier that can accurately classify bone marrow cells is obtained, which significantly improves the accuracy and efficiency of bone marrow cell classification and reduces the workload of manual classification.

[0015] Optionally, the three-dimensional convolutional neural network model includes an input layer, a convolution layer, a maximum pooling layer, a fully connected layer and an output layer, the input layer is used to receive the neighborhood data, the convolution layer is used to perform a convolution operation to extract local features in the image, the maximum pooling layer is used to perform a pooling operation to reduce the feature dimension, the fully connected layer is used to convert the extracted features into feature vectors, and the output layer is used to output the classification results of the bone marrow cells.

[0016] Optionally, performance indicators for evaluating the three-dimensional convolutional neural network model include accuracy, detection rate, specificity, coincidence rate, negative predictive value, F1 score, Kappa coefficient and area under the ROC curve.

[0017] In a second aspect, the present application provides a bone marrow cell identification system based on a hyperspectral spectrum, which adopts the following technical solution: a bone marrow cell identification system based on a hyperspectral spectrum, the system comprising: An acquisition module, used for acquiring hyperspectral data of a bone marrow cell smear; wherein the hyperspectral data includes two-dimensional spatial information and spectral information of multiple bands; A preprocessing module, used for preprocessing the hyperspectral data; A neighborhood data extraction module is used to extract the spectral characteristics and spatial information of a single bone marrow cell from the pre-processed hyperspectral data to obtain neighborhood data containing the spectral characteristics and spatial information; A bone marrow cell classification module is used to input the neighborhood data into a pre-trained bone marrow cell classifier for feature extraction and classification to obtain the classification results of the bone marrow cells; wherein the bone marrow cell classifier is trained based on a deep learning model, and the classification results of the bone marrow cells include the lineage, growth stage and pathology of the bone marrow cells.

[0018] In a third aspect, the present application provides a computer device, which adopts the following technical solution: A computer device comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method as described in the first aspect.

[0019] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium stores a computer program that can be loaded by a processor and execute any one of the methods in the first aspect.

[0020] In summary, the present application includes at least one of the following beneficial technical effects: by achieving accurate identification of bone marrow cell lineages, growth stages and pathologies, the efficiency and accuracy of bone marrow cell morphology microscopy are improved, the dependence on manual diagnosis is reduced, and the problems of cumbersome diagnostic procedures and poor diagnostic consistency existing in traditional microscopy methods are solved, providing strong support for the rapid and accurate diagnosis of blood diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a first flow chart of a bone marrow cell identification method based on hyperspectral according to one of the embodiments of the present application.

[0022] Figure 2 This is an example diagram of the data cube and spectral curve of the neutrophil segmented granulocyte according to one embodiment of the present application.

[0023] Figure 3This is a second flow chart of a bone marrow cell identification method based on hyperspectral according to one of the embodiments of the present application.

[0024] Figure 4 This is a third flow chart of a bone marrow cell identification method based on hyperspectral in one of the embodiments of the present application.

[0025] Figure 5 This is a fourth flow chart of a bone marrow cell identification method based on hyperspectral in one of the embodiments of the present application.

[0026] Figure 6 It is a structural schematic diagram of a three-dimensional convolutional neural network model of one of the embodiments of the present application.

[0027] Figure 7 This is a fifth flow chart of a method for identifying bone marrow cells based on hyperspectral spectrum according to one of the embodiments of the present application.

[0028] Figure 8 These are the results of the model performance tests on seven major categories during the actual research process of this application.

[0029] Fig. 9 These are the model performance test results for all cells during the actual research process of this application.

[0030] Fig.10 This is an example of a hyperspectral bone marrow cell detection report that was ultimately produced during the actual research process of this application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of this application more clear, the following Figure 1-10 It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0032] At present, bone marrow cell morphology microscopy is a key means of hematological diagnosis and is widely used in the diagnosis of various blood diseases, such as leukemia, multiple myeloma, etc. The annual demand in my country is as high as 3-5 million cases, but due to the reliance on manual microscopy, there are problems such as cumbersome diagnostic process, long diagnostic time, and poor diagnostic consistency.

[0033] Hyperspectral imaging technology is a new image analysis and detection technology that has developed rapidly in recent years. It is a comprehensive technology that integrates image sensors, precision optics, precision machinery and computer information processing technology. In the process of hyperspectral data acquisition, the imaging spectrometer images the detection target with nanometer-level spectral resolution and dozens or hundreds of bands, forming a digital image set of massive data (also called data cube, Data Cube), realizing the simultaneous acquisition of the target's spectral information, spatial information and radiation information. Therefore, it has great application value and broad development prospects in the field of target detection and target recognition.

[0034] This application uses hyperspectral imaging technology to perform imaging analysis on bone marrow cell smears, collects spectral information of bone marrow cells, and realizes the recognition and classification of hyperspectral bone marrow cells by establishing a bone marrow cell classifier.

[0035] The embodiment of the present application discloses a bone marrow cell identification method based on hyperspectral.

[0036] Reference Figure 1 , a bone marrow cell identification method based on hyperspectral, the method comprising: Step S101, obtaining hyperspectral data of bone marrow cell smear; Among them, hyperspectral data includes two-dimensional spatial information and spectral information of multiple bands; Specifically, in the actual collection process of hyperspectral data, spatial push-scan hyperspectral imaging technology can be used to simultaneously scan the bone marrow cell smear to be tested using a hyperspectral camera and a 4K camera connected to a microscope, and automatically generate a digital electronic smear, including 4k high-definition images and hyperspectral three-dimensional data graphs, to obtain the two-dimensional spatial information of bone marrow cells and spectral information of multiple bands for subsequent analysis and classification.

[0037] Reference Figure 2 As shown, it is an example of the data cube and spectral curve of the neutrophil granulocyte in the embodiment of the present application. The two-dimensional spatial information is the coordinate information of the nucleated cell in the figure, and the spectral information is the comprehensive spectral curve extracted from the cell. Microscopic hyperspectral imaging technology is an organic combination of spectral analysis technology and microscopic imaging technology. It can simultaneously provide two-dimensional spatial information and one-dimensional spectral information of the target to be observed, and cooperate with data analysis and processing algorithms to achieve qualitative, quantitative and localized comprehensive analysis of material components.

[0038] In the embodiment of the present application, a spatial-dimensional push-scanning type of hyperspectral imaging technology is used, so that the hyperspectral imaging system has a sufficiently high spectral resolution and a wide spectral range; in the specific implementation process, the target bone marrow cell smear on the stage is evenly illuminated by a microscope illumination light source, and the light beam passes through the target to be measured and enters the built-in scanning spectral imager, that is, it passes through the slit, collimator, dispersion element and imaging mirror in sequence, and obtains the one-dimensional spatial information and spectral information of the field target on the array detector; the data acquisition software sends a control instruction to the controller through the serial port, and the controller outputs a control signal to the driver, and the driver drives the stepper motor to rotate and then drives the precision displacement stage to move, thereby realizing the acquisition of another one-dimensional spatial information, and at the same time, the image data is collected and stored by the data acquisition software. Exemplarily, in the embodiment of the present application, the spectral range of the design system covers 400-1000nm, the number of spectral channels is greater than 200, the average spectral resolution is 3nm, the microscopic magnification is 100x, and the spatial resolution reaches 1μm.

[0039] Therefore, the acquisition of bone marrow hyperspectral data mainly consists of two parts, namely, the collection of hyperspectral data and the analysis of hyperspectral data. The hyperspectral data acquisition system mainly collects the hyperspectral data output by the array detector in real time, and sets the detector integration time, shooting frame rate, gain and other parameters. At the same time, the shooting frame rate, gain, integration time and other parameters of the high-definition array color camera are set; in addition, in order to ensure that the hyperspectral data is not distorted, the hyperspectral data acquisition system can realize the control of the motor speed in the built-in push-scan mechanism.

[0040] Step S102, preprocessing the hyperspectral data; Among them, the collected hyperspectral data are subjected to preprocessing steps such as denoising and normalization to improve the data quality and analysis accuracy, and the reliability and accuracy of subsequent analysis are improved by unifying the data scale.

[0041] Step S103, extracting the spectral characteristics and spatial information of a single bone marrow cell from the preprocessed hyperspectral data to obtain neighborhood data containing the spectral characteristics and spatial information; Specifically, the spectral characteristics and spatial information of individual bone marrow cells are extracted by analyzing the preprocessed hyperspectral data. This step can adopt the local neighborhood method to retain the local characteristics of the cells while reducing the amount of calculation and improving the efficiency of the algorithm.

[0042] For example, individual bone marrow cells can be identified and separated based on image processing techniques (such as threshold segmentation, contour detection, etc.). For each detected bone marrow cell, the small neighborhood data around it is extracted, including spectral characteristics (such as absorption intensity, reflection intensity or transmission intensity at a specific wavelength, etc.) and spatial information.

[0043] Step S104, inputting the neighborhood data into a pre-trained bone marrow cell classifier for feature extraction and classification to obtain a classification result of the bone marrow cells; Among them, the bone marrow cell classifier is trained based on a deep learning model, and the classification results of bone marrow cells include the lineage, growth stage and pathology of bone marrow cells.

[0044] Specifically, the extracted features are input into a bone marrow cell classifier based on deep learning for feature extraction and classification. The deep learning model can automatically learn complex feature representations. By utilizing its powerful feature extraction and classification capabilities, it can achieve accurate classification of bone marrow cells and improve classification accuracy. Among them, the classification results include lineage, growth stage, and pathology. These three aspects of information are very important for clinical diagnosis and treatment decisions.

[0045] Illustratively, the classification results include, for example, bone marrow nucleated cells can be divided into seven major categories: granulocytes, erythroids, lymphoid system, monocytic system, plasmacytoid system, megakaryocytes, and other systems; the classification of growth stages, for example, in the erythroid system, also includes primitive erythrocytes, proerythrocytes, mesoerythrocytes, and metaerythrocytes; the classification of pathology (i.e., whether there is an abnormality), for example, in the lymphoid system, atypical lymphocytes, lymphoma cells, etc. are all cells with pathological abnormalities.

[0046] In the above implementation, high-quality hyperspectral data is acquired, preprocessed, feature extracted, and classified using a deep learning model, ultimately obtaining important information such as cell lineage, growth stage, and pathology, thereby achieving automated identification of bone marrow cells, significantly improving the accuracy and efficiency of bone marrow cell classification, and providing strong support for clinical practice.

[0047] Reference Figure 3 As an implementation of step S102, the step of preprocessing the hyperspectral data includes: Step S201, performing image enhancement processing on the hyperspectral data; Among them, image enhancement is the process of improving image quality through a series of algorithms, with the aim of making the image clearer or highlighting certain features of interest; through image enhancement processing, the clarity of the image can be improved, making the boundaries of cells more obvious, which is conducive to subsequent cell segmentation and feature extraction.

[0048] In the embodiment of the present application, the image enhancement processing mainly includes contrast enhancement, brightness adjustment, etc., to improve the contrast and clarity of cells, so that subsequent feature extraction is more accurate.

[0049] Step S202, reducing the spectral dimension of the hyperspectral data based on a principal component analysis algorithm; Among them, principal component analysis (PCA) is a commonly used statistical method to reduce the dimension of data while retaining the main information of the original data as much as possible.

[0050] In the embodiment of the present application, PCA is used to reduce the spectral dimension in the hyperspectral data, reduce the complexity of the data by extracting the main components of the data, while maintaining the key information of the data, reducing the amount of calculation for subsequent processing, and improving processing efficiency.

[0051] Step S203, normalizing the hyperspectral data after the dimensionality reduction process.

[0052] Among them, normalization processing is to map the data to a fixed range, such as [0, 1] or [-1, 1], to eliminate the influence of the data dimension and make the comparison between different features fair.

[0053] In the embodiment of the present application, the normalization process is to make the data of different bands have the same weight, to avoid certain features dominating the results due to different dimensions, and to improve the stability of subsequent model training.

[0054] In the above implementation, these preprocessing steps effectively improve the accuracy and efficiency of subsequent feature extraction and classification, and lay a solid foundation for the automatic identification of bone marrow cells.

[0055] Reference Figure 4 As an implementation of step S103, the step of extracting the spectral features and spatial information of a single bone marrow cell from the preprocessed hyperspectral data to obtain neighborhood data containing the spectral features and spatial information includes: Step S301, identifying a single target bone marrow cell in the hyperspectral data based on an image processing algorithm, and selecting the target bone marrow cell as a target center sample point; Specifically, through image processing techniques such as threshold segmentation and edge detection, single bone marrow cells are identified from the preprocessed hyperspectral data, and the identified cells are selected as the target center sample points for subsequent extraction of the surrounding neighborhood data, ensuring the accuracy of subsequent feature extraction and avoiding the influence of non-bone marrow cell areas on the classification results.

[0056] Step S302, extracting the neighborhood data of the target center sample point, and spreading the neighborhood data in columns and rows as model input.

[0057] In one of the embodiments of the present application, neighborhood data of a certain size (e.g., 25×25 pixels) is extracted with the target center sample point as the center, and the neighborhood data contains spectral and spatial information. These neighborhood data are arranged and flattened in a certain order as input to the deep learning model, which helps to improve the accuracy of classification.

[0058] In the above implementation, image processing technology is used to identify a single bone marrow cell and determine it as the target center sample point, extract the neighborhood data of the target center sample point, and expand these data in columns and side by side as the input of the model, which effectively ensures the accuracy of feature extraction and provides high-quality data input for subsequent bone marrow cell classification, significantly improving the accuracy and efficiency of bone marrow cell morphological microscopy.

[0059] Reference Figure 5 As an implementation of the bone marrow cell classifier, the training steps of the bone marrow cell classifier specifically include: Step S401, obtaining a sample spectral data set and performing preprocessing; The sample spectral data set includes multiple sample spectral image data and pre-labeled classification results of bone marrow cells; specifically, hyperspectral data containing bone marrow cells are obtained from existing samples. These data have been labeled by morphological diagnosis experts, indicating the type of each cell (lineage, growth stage and pathology).

[0060] Among them, the preprocessing steps include image enhancement, dimensionality reduction, data normalization, etc., to improve the quality of data and the accuracy of analysis, reduce noise, and make model training more efficient and accurate.

[0061] Step S402, extracting sample neighborhood data from the preprocessed sample spectral data set; In an embodiment of the present application, an image processing algorithm is used to identify individual bone marrow cells from a preprocessed sample spectral data set, and these cells are selected as target center sample points. Sample neighborhood data (e.g., a 25×25 pixel area) around each target center sample point is extracted. These sample neighborhood data contain spectral features and spatial information. These neighborhood data can be separated and expanded side by side as model training input.

[0062] Step S403, dividing the sample neighborhood data into a training set, a validation set and a test set; In some embodiments, the extracted sample neighborhood data is divided into a 70% training set, a 15% validation set, and a 15% test set; wherein the training set is used to train the model, the validation set is used to adjust the model parameters during the training process, and the test set is used to evaluate the final performance of the model.

[0063] Step S404, inputting the training set into a pre-built three-dimensional convolutional neural network model for initial training; In one of the embodiments of the present application, combined with the characteristics of bone marrow cell hyperspectral data, according to the input (number of spectral bands) and output (bone marrow cell categories) and the number of data set samples, modeling is carried out according to the idea of ​​small neighborhood multi-spectral bands, and it is proposed to construct a three-dimensional convolutional neural network model Spec-CNN. The model can process spectral and spatial information, the data in the training set is input into the model, the model parameters are adjusted by the back propagation algorithm to minimize the loss function, and a mini-batch update strategy is used during the training process to gradually optimize the model through multiple iteration cycles. Through initial training, the model can learn the spectral characteristics and spatial characteristics of bone marrow cells, providing a basis for subsequent classification.

[0064] Among them, the three-dimensional convolutional neural network model (Spectral CNN) includes an input layer, a convolution layer, a maximum pooling layer, a fully connected layer and an output layer. The input layer is used to receive neighborhood data, the convolution layer is used to perform convolution operations to extract local features in the image, the maximum pooling layer is used to perform pooling operations to reduce feature dimensions, the fully connected layer is used to convert the extracted features into feature vectors, and the output layer is used to output the classification results of bone marrow cells.

[0065] Step S405, verifying and optimizing the model parameters of the initially trained three-dimensional convolutional neural network model based on the verification set and the test set to obtain a trained bone marrow cell classifier.

[0066] The validation set is used to evaluate the performance of the model, and the model parameters are adjusted according to the validation results, such as the learning rate, the number of convolution kernels, the size of the convolution kernel, and the number of neurons in the fully connected layer. At the same time, targeted adjustments are made according to the model convergence rate and classification effect, including the activation function type, Dropout parameter, learning rate, etc. Through continuous iterative adjustments, the model parameters are optimized and the generalization ability of the model is improved; finally, the test set is used to evaluate the performance of the final model to ensure that the model not only performs well on the training set, but also maintains a high accuracy rate on unknown data.

[0067] In the above implementation, by training a model that can automatically and accurately classify bone marrow cells, the accuracy and efficiency of bone marrow cell classification are improved, the workload of manual classification is reduced, and a new and efficient tool is provided for the diagnosis of blood diseases.

[0068] Reference Figure 6In the embodiment of the present application, the three-dimensional convolutional neural network model (Spec-CNN) includes a total of 5 layers, of which I1 is the input layer, C2 is the convolution layer, P3 is the maximum pooling layer, F4 is the fully connected layer, and O5 is the output layer of the entire convolutional neural network; the model selects a 25×25 small neighborhood and the first 5 principal components for the input of sample data. Although the first 5 principal components cannot cover all spectral dimension data, they also include most of the spectral information; after several layers of convolution and pooling processing, the input image data is converted into feature vector representation, which contains the spatial information in the 25×25 neighborhood including the central sample point. The learned spatial features are used for further classification, and finally training and fitting are performed according to the corresponding bone marrow cell true value.

[0069] Specifically, the input layer is the first layer of the neural network, responsible for receiving neighborhood data from the pre-processing stage. These data have been image enhanced, dimensionally reduced, and normalized. Each sample is a neighborhood data containing spectral and spatial information, such as a 25×25 pixel image plus several spectral bands; the convolution layer slides on the input image through the convolution kernel (filter), which can capture the morphological characteristics of bone marrow cells, such as edges, textures, etc. The maximum pooling layer reduces the dimension of the feature map by sliding the window on the feature map and selecting the maximum value in the window; the fully connected layer converts the features extracted by the convolution layer and the pooling layer into a fixed-length vector, which contains important information about bone marrow cells; the output layer is the last layer of the neural network, which is classified according to the feature vector output by the fully connected layer; the output layer uses the softmax function to convert the feature vector into a probability distribution, thereby predicting the classification results of bone marrow cells, such as cell lineage, growth stage, and pathology.

[0070] The Spec-CNN model proposed in the embodiment of the present application regards the input as a small window of data surrounding the target pixel of the bone marrow cells. Its backbone structure is a convolutional neural network CNN, which is built with the goal of extracting accurate spectral data features of blood cells. Its recognition rate is better than that of the ResNet and VGG models. Dropout and ReLU are applied to prevent the model from overfitting. At the same time, a small convolution kernel is used to perform the convolution operation. It is a lightweight model (with fewer parameters) that is less affected by local edges and is more suitable for accurate pixel-by-pixel classification of bone marrow cell spectra.

[0071] Reference Figure 7 As an implementation of step S404 and step S405, the training set is input into a pre-built three-dimensional convolutional neural network model for initial training, and the steps of verifying and optimizing model parameters of the initially trained three-dimensional convolutional neural network model based on the verification set and the test set include: Step S501, inputting the training set into a pre-built three-dimensional convolutional neural network model for iterative training, optimizing the model parameters and calculating the cost function of the model, until the cost function meets a preset condition or the number of iterations reaches a preset threshold, thereby obtaining a trained three-dimensional convolutional neural network model; Specifically, the preprocessed training set data is input into a pre-built three-dimensional convolutional neural network model for iterative training; during the training process, the model calculates the prediction results through forward propagation, and adjusts the model parameters through back propagation to minimize the cost function. The training process continues until the preset conditions are met (for example, the cost function is lower than a certain threshold) or the preset maximum number of iterations is reached.

[0072] Step S502, verifying the three-dimensional convolutional neural network model based on the verification set, evaluating the performance of the three-dimensional convolutional neural network model and adjusting the model parameters according to the verification results; Among them, the model parameters include the number of convolution kernels, the size of the convolution kernel, the number of neurons in the fully connected layer, and the learning rate; In some embodiments, the performance indicators for evaluating the three-dimensional convolutional neural network model include but are not limited to accuracy, detection rate, specificity, coincidence rate, negative predictive value, F1 score, Kappa coefficient and area under the ROC curve, etc.

[0073] Step S503, testing the prediction ability of the adjusted three-dimensional convolutional neural network model based on the test set to obtain a trained bone marrow cell classifier.

[0074] Among them, an independent test set data is used to evaluate the final performance of the model. This step does not involve model training or parameter adjustment, but is used to evaluate the model's predictive ability on unseen data.

[0075] In the above implementation, the model is trained, verified and tested using a training set, a validation set and a test set, and the model performance is continuously improved, and ultimately a bone marrow cell classifier that can accurately classify bone marrow cells is obtained, which significantly improves the accuracy and efficiency of bone marrow cell classification and reduces the workload of manual classification.

[0076] Exemplarily, the bone marrow nucleated cells used in the present application can be divided into seven major categories, namely: granulocytes, erythroids, lymphoid cells, monocytic cells, plasma cells, megakaryocytes, and other cells.

[0077] Among them, the granulocyte system includes: primitive myelocytes, primitive myelocytes with Auer bodies, promyelocytes, abnormal promyelocytes, abnormal promyelocytes with Auer bodies in the shape of bundles, neutrophil myelocytes, neutrophil metamyelocytes, neutrophil band granulocytes, neutrophil segmented granulocytes, basophils, eosinophilic myelocytes, eosinophilic metamyelocytes, eosinophilic band granulocytes, eosinophilic segmented granulocytes; the erythroid system includes: primitive erythrocytes, proerythrocytes, erythrocytes, metaerythrocytes; the lymphoid system includes: primitive lymphocytes, mature lymphocytes, atypical lymphocytes, and lymphoma cells. Among them, lymphoma cells are divided into HCL lymphoma cells, MZL lymphoma cells, FL lymphoma cells, LGLL lymphoma cells, DLBCL lymphoma cells, PLL lymphoma cells, and Burkitt lymphoma cells; the mononuclear system includes: primitive monocytes, primitive monocytes with Auer bodies, immature monocytes, and mature monocytes; the plasma system includes: mature plasma cells and abnormal plasma cells; the megakaryotic system includes: granular megakaryocytes, platelet-producing megakaryocytes, naked nucleus megakaryocytes, and small megakaryocytes; other systems include: reticular cells, platelets, degenerative cells, and mast cells.

[0078] As shown in Table 1 below, in the actual model training process of the present application, the number of selected data set samples is 306 and the number of sample cells is 64,800. Among them, the number of training set samples was 306, and the number of sample cells was 36288; the number of validation set samples was 306, and the number of sample cells was 15552; ​​the number of test set samples was 306, and the number of sample cells was 12960; the average age of the sampled population was 47.86, ranging from 2 to 86 years old, with 180 males (58.80%) and 126 females (41.19%); the disease types included 13 types: acute myeloid leukemia (AML-M2), acute abnormal promyelocytic leukemia (APL), acute monocytic leukemia (AML-M5), acute lymphocytic leukemia (ALL), chronic lymphocytic leukemia (CLL), chronic myeloid leukemia (CML), multiple myeloma (MM), myelodysplastic syndrome (MDS), viral infection, lymphoma bone marrow not invaded, donor, lymphoma bone marrow invaded, and others. Among them, the number of cases with lymphoma bone marrow invasion was the largest, with 63 cases and 13941 cells. The second was lymphoma without bone marrow invasion, which occurred in 61 cases and had 8,650 cells.

[0079] Table 1 Refer to Table 2 below, including the counts and proportions of 13 disease types in 306 samples, of which 63 cases had lymphoma bone marrow invasion, accounting for 20.59%, followed by 61 cases of lymphoma without bone marrow invasion, accounting for 19.93%. Among men, the disease type of lymphoma bone marrow invasion accounted for the most in 39 cases (21.67%), followed by 35 cases of lymphoma without bone marrow invasion (19.44%); among women, the most common disease types were lymphoma without bone marrow invasion and others, accounting for 26 cases (20.63%).

[0080] Table 2 Reference Figure 8 In the performance test process after the bone marrow cell classifier was built, the seven major lineages were analyzed separately. Among them, the highest detection rate was others (94.25%), and the lowest was erythroid (78.24%); the highest specificity was others (99.69%), and the lowest was myeloid (98.94%); the highest coincidence rate was others (92.17%), and the lowest was erythroid (71.51%); the highest negative predictive value NPV was others (99.78%), and the lowest was myeloid (99.16%); the highest accuracy was others (99.48%), and the lowest was myeloid (98.20%); the highest F1 was others (0.93), and the lowest was erythroid (0.75); the highest Kappa was megakaryocyte (0.98), and the lowest was others (0.85).

[0081] Reference Fig. 9 All cells were analyzed, among which the highest detection rate was platelets (98.21%), and the lowest was mature monocytes (65.05%); the highest specificity was platelets (99.99%), and the lowest was lymphoma cells (98.37%); the highest PPV was platelets (99.10%), and the lowest was reticular cells (71.11%); the highest negative predictive value NPV was platelets (99.98%), and the lowest was lymphoma cells (98.33%); the highest accuracy was platelets (99.98%), and the lowest was lymphoma cells (97.3%); the highest F1 was platelets (98.65%), and the lowest was mature monocytes (70.14%); the highest Kappa was platelets (0.986), and the lowest was proerythroblasts (0.7).

[0082] For the actual application process of this application, the following is explained in conjunction with the overall technical solution of hyperspectral bone marrow cell recognition: First, the hyperspectral bone marrow image analysis system previews the image on the slide, and scans the entire slide through a 20x or 10x objective lens, and uses a color camera to capture and store the slide information and megakaryocyte data. Next, enter the number of bone marrow nucleated cells N (default is 250) to be collected. Then, use a 10x objective lens to find the starting point of the monolayer cells, switch to a 100x oil lens and drip oil, and start scanning the bone marrow nucleated area. If the scanned area does not meet the requirements, the system will rescan; if it meets the requirements, cell counting will be performed. When the number of bone marrow nucleated cells counted is less than N, the system will perform a spectral scan, complete cell collection according to the turntable order (there is a reflector and multiple filters on the turntable), and store the slide information and nucleated cell data again. This process will be repeated until the number of bone marrow nucleated cells counted exceeds N, at which time the system will stop scanning and complete the scanning of the entire picture. After scanning, the nucleated cells on the slide are identified and classified by using the bone marrow cell classifier constructed in this application. Finally, these cells are counted and their percentage in all cells is calculated, and the distribution and proportion of the cells are output.

[0083] Reference Fig.10 As shown in the figure, it is an example of a hyperspectral bone marrow cell detection report finally issued during the actual research process of this application, which includes cells of different types and different growth stages, and lists the number of cells and the actual proportion in detail. At the same time, by comparing with the normal range standard, it can be determined whether the pathology is abnormal.

[0084] The embodiment of the present application also discloses a bone marrow cell identification system based on hyperspectral.

[0085] A bone marrow cell identification system based on hyperspectral, the system comprising: An acquisition module is used to acquire hyperspectral data of a bone marrow cell smear; wherein the hyperspectral data includes two-dimensional spatial information and spectral information of multiple bands; Preprocessing module, used to preprocess hyperspectral data; A neighborhood data extraction module is used to extract the spectral characteristics and spatial information of a single bone marrow cell from the preprocessed hyperspectral data to obtain neighborhood data containing the spectral characteristics and spatial information; The bone marrow cell classification module is used to input the neighborhood data into a pre-trained bone marrow cell classifier for feature extraction and classification to obtain the classification results of the bone marrow cells; wherein the bone marrow cell classifier is trained based on a deep learning model, and the classification results of the bone marrow cells include the lineage, growth stage and pathology of the bone marrow cells.

[0086] A bone marrow cell identification system based on hyperspectral in an embodiment of the present application can implement any of the above-mentioned bone marrow cell identification methods, and the specific working process of each module in the bone marrow cell identification system can refer to the corresponding process in the above-mentioned method embodiment.

[0087] In order to realize a bone marrow cell identification system based on hyperspectral in the embodiment of the present application, the relevant hardware configuration includes a spectrometer, a detector, and a scanning platform. By ensuring the compatibility and matching between the spectrometer, the detector, the scanning platform and other components, efficient data transmission and processing can be achieved. The hyperspectral imaging system can observe in hundreds of bands, with a spectral resolution of less than 10 nanometers. The system can obtain rich spectral information, which improves the accuracy and reliability of target detection. In addition, hyperspectral imaging technology not only obtains the spectral information of the target, but also obtains its two-dimensional geometric space information.

[0088] Since light plays an absolutely dominant role in the entire system, the initial acquisition requires a lot of optimized optical design to improve the imaging quality and spectral resolution of the system, including selecting the appropriate light source, lens, and adjusting parameters such as focal length and aperture. By combining hardware and software, a reliable control system is designed to achieve precise control of the scanning platform and synchronization of data acquisition. This includes writing control algorithms, designing software interfaces, etc., and developing efficient data processing software to process and analyze spectral data, perform spectral correction, noise suppression, feature extraction, etc. Finally, all the above stages and steps are integrated and optimized to build an efficient, stable and reliable system.

[0089] In addition, in order to achieve rapid data storage, the bone marrow cell identification system of the embodiment of the present application also uses a USB3.0 interface to connect to the computer, which is consistent with data digitization. In order to facilitate subsequent model upgrades, an interface is set up in the software to quickly call the model; through the relevant model developed in Python, the weight file obtained by training is packaged into an exe format file for easy use, and the subsequent upgrade process can be achieved by replacing the .exe file without making too many modifications to the code.

[0090] In the several embodiments provided in this application, it should be understood that the provided methods and systems can be implemented in other ways. For example, the system embodiments described above are only illustrative; for example, the division of a certain module is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0091] The embodiment of the present application also discloses a computer device.

[0092] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a bone marrow cell identification method based on hyperspectral as described above is implemented.

[0093] The embodiment of the present application also discloses a computer-readable storage medium.

[0094] A computer-readable storage medium stores a computer program that can be loaded by a processor and execute any one of the above-mentioned methods for identifying bone marrow cells based on hyperspectral.

[0095] Among them, computer-readable storage media can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus or device; the program code contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0096] It should be noted that in the above embodiments, the description of each embodiment has different emphases, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0097] The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any feature disclosed in this specification (including the abstract and drawings), unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

Claims

1. A bone marrow cell identification method based on hyperspectral, characterized in that: The method comprises: Acquire hyperspectral data of a bone marrow cell smear; wherein the hyperspectral data includes two-dimensional spatial information and spectral information of multiple bands; Preprocessing the hyperspectral data; Extracting spectral features and spatial information of a single bone marrow cell from the preprocessed hyperspectral data to obtain neighborhood data containing the spectral features and spatial information; The neighborhood data is input into a pre-trained bone marrow cell classifier for feature extraction and classification to obtain the classification results of the bone marrow cells; wherein the bone marrow cell classifier is trained based on a deep learning model, and the classification results of the bone marrow cells include the lineage, growth stage and pathology of the bone marrow cells.

2. A bone marrow cell identification method based on hyperspectral according to claim 1, characterized in that: The step of preprocessing the hyperspectral data includes: Performing image enhancement processing on the hyperspectral data; Performing dimension reduction processing on the spectral dimension of the hyperspectral data based on a principal component analysis algorithm; The hyperspectral data after dimensionality reduction are normalized.

3. The method for identifying bone marrow cells based on hyperspectral according to claim 1, characterized in that: The steps of extracting the spectral features and spatial information of a single bone marrow cell from the preprocessed hyperspectral data and obtaining neighborhood data containing the spectral features and spatial information include: Identify a single target bone marrow cell in the hyperspectral data based on an image processing algorithm, and select the target bone marrow cell as a target center sample point; The neighborhood data of the target center sample point is extracted, and the neighborhood data is arranged in columns and spread out side by side as model input.

4. The method for identifying bone marrow cells based on hyperspectral according to claim 1, characterized in that: The method further comprises a training step of the bone marrow cell classifier, wherein the training step comprises: Acquire a sample spectral data set and perform preprocessing; wherein the sample spectral data set includes a plurality of sample spectral image data and classification results of pre-labeled bone marrow cells; Extracting sample neighborhood data from the preprocessed sample spectral data set; Dividing the sample neighborhood data into a training set, a validation set, and a test set; Inputting the training set into a pre-built three-dimensional convolutional neural network model for initial training; The three-dimensional convolutional neural network model after initial training is verified and model parameters are optimized based on the verification set and the test set to obtain the trained bone marrow cell classifier.

5. The method for identifying bone marrow cells based on hyperspectral according to claim 4, characterized in that: The steps of inputting the training set into a pre-built three-dimensional convolutional neural network model for initial training, and verifying and optimizing model parameters of the initially trained three-dimensional convolutional neural network model based on the verification set and the test set include: Inputting the training set into a pre-built three-dimensional convolutional neural network model for iterative training, optimizing the model parameters and calculating the cost function of the model, until the cost function meets a preset condition or the number of iterations reaches a preset threshold, thereby obtaining a trained three-dimensional convolutional neural network model; Verifying the three-dimensional convolutional neural network model based on the verification set, evaluating the performance of the three-dimensional convolutional neural network model and adjusting model parameters according to the verification results; wherein the model parameters include the number of convolution kernels, the size of the convolution kernel, the number of neurons in the fully connected layer, and the learning rate; The prediction ability of the adjusted three-dimensional convolutional neural network model is tested based on the test set to obtain the trained bone marrow cell classifier.

6. The method for identifying bone marrow cells based on hyperspectral according to claim 5, characterized in that: The three-dimensional convolutional neural network model includes an input layer, a convolution layer, a maximum pooling layer, a fully connected layer and an output layer. The input layer is used to receive the neighborhood data, the convolution layer is used to perform a convolution operation to extract local features in the image, the maximum pooling layer is used to perform a pooling operation to reduce the feature dimension, the fully connected layer is used to convert the extracted features into feature vectors, and the output layer is used to output the classification results of the bone marrow cells.

7. The method for identifying bone marrow cells based on hyperspectral according to claim 5, characterized in that: The performance indicators for evaluating the three-dimensional convolutional neural network model include accuracy, detection rate, specificity, coincidence rate, negative predictive value, F1 score, Kappa coefficient and area under the ROC curve.

8. A bone marrow cell identification system based on hyperspectral, characterized in that: The system comprises: An acquisition module, used for acquiring hyperspectral data of a bone marrow cell smear; wherein the hyperspectral data includes two-dimensional spatial information and spectral information of multiple bands; A preprocessing module, used for preprocessing the hyperspectral data; A neighborhood data extraction module is used to extract the spectral characteristics and spatial information of a single bone marrow cell from the pre-processed hyperspectral data to obtain neighborhood data containing the spectral characteristics and spatial information; A bone marrow cell classification module is used to input the neighborhood data into a pre-trained bone marrow cell classifier for feature extraction and classification to obtain the classification results of the bone marrow cells; wherein the bone marrow cell classifier is trained based on a deep learning model, and the classification results of the bone marrow cells include the lineage, growth stage and pathology of the bone marrow cells.

9. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program.

10. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 7.

Images