Microscopic cell real-time recognition method and system based on embedded NPU

Abstract

The application discloses a microscopic cell real-time recognition method and system based on an embedded NPU, and particularly relates to the field of artificial intelligence image recognition, and comprises an image acquisition module, a preprocessing module, an NPU inference module, a post-processing module, a display superposition module and a storage module; the image acquisition module is used for acquiring microscopic image data through an MIPI interface; the preprocessing module is used for preprocessing the image data to generate input data suitable for NPU inference; the real-time performance of the application is significantly improved; through a lightweight model, NPU hardware acceleration and frame skipping inference, the single-frame inference delay is less than or equal to 50 ms, and the end-to-end recognition delay is less than or equal to 200 ms, thereby meeting the real-time interaction requirement of a teaching scene; the application can maintain high precision and optimize resource occupation; meanwhile, the application can realize local real-time recognition without networking, has prominent teaching value, helps junior high school students to quickly establish cell morphology cognition, and understands the application of AI in scientific research.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence image recognition technology, and more specifically, to a method and system for real-time microscopic cell recognition based on an embedded NPU. Background Technology

[0002] Microscopic cell identification is a fundamental task in biology teaching and medical testing. Traditionally, it relies on manual observation and identification, which is inefficient and highly subjective. With the development of deep learning technology, cell image classification methods based on convolutional neural networks (CNN) have gradually emerged, which can assist manual identification to a certain extent.

[0003] However, existing technical solutions mainly have the following problems: First, the deployment platform is limited. Most cell recognition methods are deployed on cloud servers or high-performance PCs, relying on powerful GPU computing power. They cannot run in real time on resource-constrained embedded terminal devices. In teaching scenarios, student terminals usually use embedded platforms with limited computing power, which cannot support complex deep learning models.

[0004] Secondly, the real-time performance is insufficient. Existing methods have high inference latency on edge devices, which makes it difficult to meet the needs of real-time interaction in teaching scenarios. For example, a study deployed a cell segmentation model on a Raspberry Pi 4B, and the single-frame inference time reached 335ms, which far exceeds the real-time response threshold acceptable to the human eye.

[0005] Third, there is a loss of accuracy due to model quantization. To adapt to embedded platforms, floating-point models need to be quantized using INT8. However, existing quantization methods often lead to a significant drop in accuracy, making it difficult to maintain high-precision recognition.

[0006] Fourth, the integration with hardware is not tight. Existing technologies have not fully considered the synergistic optimization with embedded NPUs and have failed to fully leverage the computing power advantages of NPUs.

[0007] Therefore, there is an urgent need for a real-time microscopic cell identification method and system based on an embedded NPU to solve the above problems. Summary of the Invention

[0008] To overcome the aforementioned deficiencies of existing technologies, embodiments of the present invention provide a real-time microscopic cell recognition method and system based on an embedded NPU. This invention significantly improves real-time performance by employing a lightweight model, NPU hardware acceleration, and frame-skipping inference, achieving a single-frame inference latency of ≤50ms and an end-to-end recognition latency of ≤200ms, meeting the real-time interactive requirements of teaching scenarios. This invention maintains high precision by employing a hybrid precision quantization strategy, with an accuracy loss of ≤1% after INT8 quantization. On the test set, the average recognition accuracy for the three cell types reaches 92.3%, superior to the 87.5% of the traditional uniform quantization method. This invention optimizes resource usage, with NPU utilization ≤50% and CPU utilization ≤30%, avoiding resource conflicts with concurrent dual-channel video acquisition. Deployment is simple, with a model size ≤5MB, which can be embedded in the eMMC of an embedded device. This invention achieves local real-time recognition without network connectivity, demonstrating significant educational value by helping junior and senior high school students quickly develop cell morphology awareness and understand the application of AI in scientific research, thus addressing the problems mentioned in the background.

[0009] To achieve the above objectives, the present invention provides the following technical solution: a real-time microscopic cell recognition system based on an embedded NPU, comprising an image acquisition module, a preprocessing module, an NPU inference module, a post-processing module, a display overlay module, and a storage module. The image acquisition module is used to acquire microscopic image data through the MIPI interface; The preprocessing module is used to preprocess the image data to generate input data adapted for NPU inference; The NPU inference module is used to call a lightweight cell classification model deployed on an embedded NPU for inference. The post-processing module is used to normalize the inference results; The display overlay module is used to overlay the recognition results onto the real-time video screen; The storage module is used to store the lightweight cell classification model file.

[0010] In a preferred embodiment, the image acquisition module and the NPU inference module are executed asynchronously, and data is transferred through shared memory. The NPU inference module adopts a frame skipping strategy to reduce system load.

[0011] A method for real-time microscopic cell identification based on an embedded NPU, comprising the real-time microscopic cell identification system based on an embedded NPU as described above, and the following steps: Step 1: Image acquisition step, acquiring microscopic image data from the image sensor via the MIPI interface; Step 2: Preprocessing step, the microscopic image data is normalized in size, converted in format and normalized to generate input data adapted for NPU inference; Step 3: NPU inference step, calling the lightweight cell classification model deployed on the embedded NPU to infer the input data and obtain the cell category probability; Step 4: Post-processing step, normalize the inference results, and determine cell category labels and confidence levels; Step 5: Display the overlay step, overlaying the recognition results onto the real-time video screen.

[0012] In a preferred embodiment, the lightweight cell classification model is an INT8 quantized MobileNetV2 model with a model size ≤ 5MB and an inference latency ≤ 50ms.

[0013] In a preferred embodiment, the NPU inference step employs a mixed precision quantization strategy to perform sensitivity analysis on each layer of the model, retaining FP16 precision for sensitive layers and using INT8 quantization for insensitive layers.

[0014] In a preferred embodiment, the NPU inference step employs a frame skipping inference strategy, performing inference once every 2-3 frames of images to reduce the NPU load.

[0015] In a preferred embodiment, the preprocessing step includes: S1: Scale the original image to 224×224 dimensions; S2: Convert YUYV format to RGB format; S3: Normalize the pixel values ​​to the [-1, 1] range; S4: Convert the HWC data layout to CHW layout.

[0016] In a preferred embodiment, the training steps of the lightweight cell classification model include: S1: Collect images of at least three types of cells and construct a training dataset; S2: Transfer learning based on the ImageNet pre-trained MobileNetV2 model; S3: Fine-tuning using the cross-entropy loss function and the Adam optimizer; S4: Expand the training samples through data augmentation techniques.

[0017] In a preferred embodiment, the embedded NPU is set as the NNIE engine built into the HiSilicon Hi3516DV500 chip, with a computing power of 2 TOPS.

[0018] The technical effects and advantages of this invention are as follows: This invention significantly improves real-time performance. Through a lightweight model, NPU hardware acceleration, and frame skipping inference, the single-frame inference latency is ≤50ms, and the end-to-end recognition latency is ≤200ms, meeting the real-time interactive needs of teaching scenarios. This invention maintains high accuracy by employing a hybrid precision quantization strategy, with an accuracy loss of ≤1% after INT8 quantization. On the test set, the average recognition accuracy for the three cell types reaches 92.3%, superior to the 87.5% of the traditional uniform quantization method. This invention optimizes resource usage, with NPU utilization ≤50% and CPU utilization ≤30%, avoiding resource conflicts with concurrent dual-channel video acquisition. Deployment is simple, with a model size ≤5MB, and it can be embedded in the eMMC of an embedded device. This invention can achieve local real-time recognition without a network connection, demonstrating significant educational value and helping junior and senior high school students quickly establish cell morphology recognition and understand the application of AI in scientific research. Detailed Implementation

[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] This invention provides a real-time microscopic cell recognition system based on an embedded NPU, comprising an image acquisition module, a preprocessing module, an NPU inference module, a post-processing module, a display overlay module, and a storage module. The image acquisition module is used to acquire microscopic image data through the MIPI interface; The preprocessing module is used to preprocess the image data to generate input data adapted for NPU inference; The NPU inference module is used to call a lightweight cell classification model deployed on an embedded NPU for inference. The post-processing module is used to normalize the inference results; The display overlay module is used to overlay the recognition results onto the real-time video screen; The storage module is used to store the lightweight cell classification model file.

[0021] The image acquisition module and the NPU inference module execute asynchronously, and data is transferred through shared memory. The NPU inference module adopts a frame skipping strategy to reduce system load.

[0022] A method for real-time microscopic cell identification based on an embedded NPU, comprising the real-time microscopic cell identification system based on an embedded NPU as described above, and the following steps: Step 1: Image acquisition step, acquiring microscopic image data from the image sensor via the MIPI interface; Step 2: Preprocessing step, the microscopic image data is normalized in size, converted in format and normalized to generate input data adapted for NPU inference; Step 3: NPU inference step, calling the lightweight cell classification model deployed on the embedded NPU to infer the input data and obtain the cell category probability; Step 4: Post-processing step, normalize the inference results, and determine cell category labels and confidence levels; Step 5: Display the overlay step, overlaying the recognition results onto the real-time video screen.

[0023] The lightweight cell classification model is an INT8 quantized MobileNetV2 model with a model size ≤ 5MB and inference latency ≤ 50ms.

[0024] The NPU inference step employs a mixed precision quantization strategy, performing sensitivity analysis on each layer of the model, retaining FP16 precision for sensitive layers, and using INT8 quantization for insensitive layers.

[0025] The NPU inference step adopts a frame skipping inference strategy, performing inference once every 2-3 frames of images to reduce the NPU load.

[0026] The preprocessing steps include: S1: Scale the original image to 224×224 dimensions; S2: Convert YUYV format to RGB format; S3: Normalize the pixel values ​​to the [-1, 1] range; S4: Convert the HWC data layout to CHW layout.

[0027] The training steps for the lightweight cell classification model include: S1: Collect images of at least three types of cells and construct a training dataset; S2: Transfer learning based on the ImageNet pre-trained MobileNetV2 model; S3: Fine-tuning using the cross-entropy loss function and the Adam optimizer; S4: Expand the training samples through data augmentation techniques.

[0028] The embedded NPU is set to the NNIE engine built into the HiSilicon Hi3516DV500 chip, with a computing power of 2 TOPS.

[0029] In the image acquisition and preprocessing of this invention, the image acquisition module obtains raw image data from the MIPI interface through the V4L2 framework. The parameter settings are shown in the table below: Table 1

[0030] The lightweight cell classification model used in this invention is based on the MobileNetV2 architecture, as shown in the table below: Table 2

[0031] The model training parameters are shown in the table below: Table 3

[0032] The training dataset contains at least 5,000 labeled cell images, divided into training, validation, and test sets in an 8:1:1 ratio.

[0033] The steps of the mixed-precision quantization process of this invention are as follows: a: Model sensitivity analysis: Add quantization noise layer by layer, observe the impact on accuracy, and identify sensitive layers; b: Preparation of calibration dataset: Select 200 representative images to generate quantization factors; c: Channel-by-channel quantization: Channel-by-channel INT8 quantization is used for non-sensitive layers; d: Mixed precision configuration: Configure the sensitive layer to FP16 precision, and the remaining layers to INT8 precision; e: Accuracy verification: Use the test set to verify the accuracy after quantization. If the decrease exceeds 1%, adjust the quantization strategy.

[0034] Quantization parameter calculation formula: For INT8 quantization, the scaling factor (scale) and zero point are calculated as follows: text scale = (max_value - min_value) / 255 zero_point = round(-min_value / scale) Where max_value and min_value are the activation value ranges of this layer on the calibration dataset.

[0035] The NPU inference module of this invention is implemented based on the HiSilicon NNIE (Neural Network Inference Engine) API, and the core call flow is as follows: c / / Initialize NPU HI_S32 ret = HI_MPI_SYS_Init(); / / Load model HI_HANDLE model_handle; HI_MPI_NPU_LoadModel(model_path, &model_handle); / / Create task HI_HANDLE task_handle; HI_MPI_NPU_CreateTask(model_handle, &task_handle); / / Set input HI_MPI_NPU_SetInput(task_handle, 0, input_data, input_size); / / Perform inference HI_MPI_NPU_RunTask(task_handle); / / Get output HI_MPI_NPU_GetOutput(task_handle, 0, output_data, &output_size); / / Release resources HI_MPI_NPU_DestroyTask(task_handle); HI_MPI_NPU_UnloadModel(model_handle); HI_MPI_SYS_Exit().

[0036] The post-processing module of this invention performs softmax normalization on the raw logits output by the NPU: Python def softmax(x): exp_x = np.exp(x - np.max(x)) return exp_x / np.sum(exp_x) The final category label is the category with the highest probability, and the confidence score is the corresponding probability value.

[0037] The display overlay module overlays the recognition results onto the real-time video screen as a semi-transparent overlay, in the format of "[cell type] [confidence percentage]", such as "onion epidermal cells 95%".

[0038] This invention significantly improves real-time performance. Through a lightweight model, NPU hardware acceleration, and frame skipping inference, the single-frame inference latency is ≤50ms, and the end-to-end recognition latency is ≤200ms, meeting the real-time interactive needs of teaching scenarios. This invention maintains high accuracy by employing a hybrid precision quantization strategy, with an accuracy loss of ≤1% after INT8 quantization. On the test set, the average recognition accuracy for the three cell types reaches 92.3%, superior to the 87.5% of the traditional uniform quantization method. This invention optimizes resource usage, with NPU utilization ≤50% and CPU utilization ≤30%, avoiding resource conflicts with concurrent dual-channel video acquisition. Deployment is simple, with a model size ≤5MB, and it can be embedded in the eMMC of an embedded device. This invention can achieve local real-time recognition without a network connection, demonstrating significant educational value and helping junior and senior high school students quickly establish cell morphology recognition and understand the application of AI in scientific research.

[0039] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A real-time microscopic cell identification system based on an embedded NPU, characterized in that, It includes an image acquisition module, a preprocessing module, an NPU inference module, a post-processing module, a display overlay module, and a storage module. The image acquisition module is used to acquire microscopic image data through the MIPI interface; The preprocessing module is used to preprocess the image data to generate input data adapted for NPU inference; The NPU inference module is used to call a lightweight cell classification model deployed on an embedded NPU for inference. The post-processing module is used to normalize the inference results; The display overlay module is used to overlay the recognition results onto the real-time video screen; The storage module is used to store the lightweight cell classification model file.

2. The real-time microscopic cell identification system based on an embedded NPU according to claim 1, characterized in that: The image acquisition module and the NPU inference module execute asynchronously, and data is transferred through shared memory. The NPU inference module adopts a frame skipping strategy to reduce system load.

3. A method for real-time identification of microscopic cells based on an embedded NPU, characterized in that, The system includes a real-time microscopic cell identification system based on an embedded NPU as described in any one of claims 1-2, and the following steps: Step 1: Image acquisition step, acquiring microscopic image data from the image sensor via the MIPI interface; Step 2: Preprocessing step, the microscopic image data is normalized in size, converted in format and normalized to generate input data adapted for NPU inference; Step 3: NPU inference step, calling the lightweight cell classification model deployed on the embedded NPU to infer the input data and obtain the cell category probability; Step 4: Post-processing step, normalize the inference results, and determine cell category labels and confidence levels; Step 5: Display the overlay step, overlaying the recognition results onto the real-time video screen.

4. The method for real-time microscopic cell identification based on an embedded NPU according to claim 3, characterized in that: The lightweight cell classification model is an INT8 quantized MobileNetV2 model with a model size ≤ 5MB and inference latency ≤ 50ms.

5. The method for real-time microscopic cell identification based on an embedded NPU according to claim 3, characterized in that: The NPU inference step employs a mixed precision quantization strategy, performing sensitivity analysis on each layer of the model, retaining FP16 precision for sensitive layers, and using INT8 quantization for insensitive layers.

6. The method for real-time microscopic cell identification based on an embedded NPU according to claim 3, characterized in that: The NPU inference step adopts a frame skipping inference strategy, performing inference once every 2-3 frames of images to reduce the NPU load.

7. The method for real-time microscopic cell identification based on an embedded NPU according to claim 3, characterized in that, The preprocessing steps include: S1: Scale the original image to 224×224 dimensions; S2: Convert YUYV format to RGB format; S3: Normalize the pixel values ​​to the [-1, 1] range; S4: Convert the HWC data layout to CHW layout.

8. The method for real-time identification of microscopic cells based on an embedded NPU according to claim 3, characterized in that... The training steps for the lightweight cell classification model include: S1: Collect images of at least three types of cells and construct a training dataset; S2: Transfer learning based on the ImageNet pre-trained MobileNetV2 model; S3: Fine-tuning using the cross-entropy loss function and the Adam optimizer; S4: Expand the training samples through data augmentation techniques.

9. The method for real-time microscopic cell identification based on an embedded NPU according to claim 3, characterized in that: The embedded NPU is set to the NNIE engine built into the HiSilicon Hi3516DV500 chip, with a computing power of 2 TOPS.

Images