A Malaria Blood Smear Cell Detection and Segmentation Method Based on Mask R-CNN

Abstract

The present invention discloses a malaria blood smear cell detection and segmentation method based on Mask R-CNN, which realizes the detection and segmentation of malarial parasites and red blood cells in thin blood smear images, including: using VoVNet-57 as the backbone feature extraction network to extract richer thin blood smear image features; improving the feature pyramid network structure to make full use of the cell classification information rich in high-level features to enhance the representation ability of the underlying features; using the attenuation function to improve the traditional non-maximum suppression algorithm to reduce the cases of missed detection due to cell stacking; reconstructing the cell classification loss function to improve the problem of low detection accuracy due to imbalanced cell categories. The method proposed in the present invention can effectively improve the accuracy of the Mask R-CNN network model for cell detection and segmentation in thin blood smear images, providing a scientific basis for malaria diagnosis.

Description

Technical Field

[0001] The present invention discloses a malaria blood smear cell detection and segmentation method based on Mask R-CNN, which involves using a deep neural network to perform cell detection and segmentation on thin blood smear images under a microscope and calculate the malarial parasite density, belonging to the field of target detection. Technical Background

[0002] Malaria is a contagious and fatal disease caused by Plasmodium parasites. Treatment varies depending on the parasite species and the severity of infection. Therefore, accurately distinguishing the species and calculating parasite density are prerequisites for effective malaria treatment. Manual screening of blood smears using an optical microscope is currently the most common method for malaria diagnosis. However, manual testing is cumbersome and requires extensive domain expertise and experience to accurately identify and classify parasites. Due to variations in expert knowledge, manual testing can produce biased or less objective results. When specialized testing personnel are scarce and the workload is high, manual testing is not feasible, potentially missing the optimal window for treatment and prevention.

[0003] To help inspectors make quick and accurate judgments, some well-known methods use image processing and machine learning techniques for malaria diagnosis. For example, Zhu Yuanlin (Dissertation, 2019) combined a Visual Geometry Group (VGG) network with a Support Vector Machine (SVM) to propose a VGG-SVM-based malarial parasite detection model that accurately distinguishes between infected and uninfected malarial parasite images. Liu Yinping et al. (Experimental Technology and Management, 2020) addressed issues with malaria detection models, such as long training time and redundant weight parameters. They reduced the width and depth of the ResNet-50 network and replaced its fully connected layers with a classifier. This approach improved malaria classification accuracy while shortening model training time and reducing model parameters. These methods can determine whether a person is infected with malarial parasites, but cannot effectively determine the type, location, or number of malarial parasites.

[0004] It is difficult to use machine learning methods to achieve cell detection and segmentation in thin blood smear images. The general target detection algorithm based on deep learning has problems such as difficulty in extracting sufficient and effective features due to the small proportion of malarial parasites in the image, dense cell arrangement, and imbalance of cell categories. It is prone to missed detections and false detections, resulting in less than ideal cell detection and segmentation results.

[0005] To address these issues, some well-known methods have improved general object detection algorithms to reduce missed and false detections. For example, Liu Qianyu (Computer Technology and Development, 2021) proposed an improved algorithm for malaria blood smear detection based on Faster R-CNN. This algorithm adds convolutional filters to Faster R-CNN and uses ResNet as the backbone feature extraction network. This algorithm improves malaria parasite detection by optimizing anchor point properties, but does not achieve cell segmentation. Xu Mingxing et al. (Patent 202110357678.5, 2021) used YOLO V4 to detect the location of each cell, amplify the cells, and then input them into a U-Net network for classification. By segmenting and amplifying each cell, the model enhances malaria parasite detection capabilities, but cannot effectively count red blood cells to calculate malaria parasite density. Shen Yuhong (Dissertation, 2020) proposed a cell segmentation and classification method based on FD-Mask R-CNN. On the basis of Mask R-CNN, dilated convolution was introduced to expand the convolution receptive field, better retaining the image structure information in high-level features and improving the segmentation accuracy. A regulation factor was added to the classification loss function to focus on difficult-to-classify samples, improve the problem of low classification accuracy caused by imbalanced cell categories, and realize the integration of Plasmodium classification and segmentation. However, the situation of missed detection due to cell stacking was not considered.

[0006] To overcome the shortcomings of the above-mentioned known methods, the present invention proposes a malaria blood smear cell detection and segmentation method based on Mask R-CNN. It adopts VoVNet-57, which is more suitable for small target detection tasks, as the backbone feature extraction network to extract richer thin blood smear image features; improves the Feature Pyramid Network (FPN) structure to fully utilize the rich cell classification information of high-level features to enhance the representation ability of low-level features; uses an attenuation function to improve the traditional non-maximum suppression algorithm to reduce the risk of missed cell detection; and adds loss weights and adjustment factors to the cell classification loss function to improve the problem of low detection accuracy due to imbalanced cell categories. Summary of the Invention

[0007] 1. Purpose of the Invention

[0008] Common target detection algorithms often suffer from problems such as the small proportion of malarial parasites in thin blood smear images, dense cell arrangement, and imbalanced cell types, which can lead to missed and false cell detections. To improve this, this paper proposes a malarial blood smear cell detection and segmentation method based on Mask R-CNN. This method can detect and segment malarial parasites and red blood cells in thin blood smear images, and calculate the malarial parasite density, providing a scientific basis for malaria diagnosis.

[0009] 2. Steps of the present invention

[0010] The implementation process of the present invention is divided into the following four steps:

[0011] (1) Data preprocessing: On each thin blood smear image collected by an optical microscope, different types and stages of malarial parasites and red blood cells are annotated. The annotated dataset is enhanced and the image size is unified to obtain a thin blood smear image training set.

[0012] (2) Feature extraction and processing: A modified Mask R-CNN network model is constructed to extract and process the input thin blood smear image.

[0013] (3) Model training: The thin blood smear image training set in step (1) is input into the model in step (2), the loss function is calculated and back propagation is performed, and the weights of the network model are iteratively updated.

[0014] (4) Malaria detection: The thin blood smear image to be detected is input into the model trained in step (3) for cell detection and segmentation, and the malarial parasite density is calculated.

[0015] The specific steps are as follows:

[0016] 1: Data preprocessing

[0017] 1.1: Image Annotation

[0018] Label red blood cells and malarial parasites of different types and stages in the collected thin blood smear images. Suppose the collected thin blood smear image dataset is , , For image datasets The unmarked Original image. The red blood cells and malarial parasites are outlined in a dotted manner to form a closed area, and the cell category to which they belong is marked. The marked outline, category and image information are stored in a json file to obtain the corresponding annotation data. ;right After annotating all images in the dataset, we can get the annotation dataset. , ; Image dataset and labeled datasets Merge to obtain the labeled thin blood smear image dataset , .

[0019] 1.2: Data Augmentation

[0020] Training with too few thin blood smear images will lead to underfitting of the model, and the large differences in the distribution of various types of Plasmodium collected will also lead to reduced accuracy in detecting and segmenting cells. In order to improve the generalization ability of the model, according to The malarial parasites are segmented based on the malarial parasite outline information annotated in the thin blood smear image. The segmented malarial parasites are randomly rotated at any angle and then copied and pasted to other positions in the corresponding image. This not only expands the number of malarial parasite samples but also increases the diversity of malarial parasite locations. The image after the malarial parasite samples are expanded is data enhanced by rotating and flipping to obtain the expanded thin blood smear image training set. .

[0021] 1.3: Image size scaling

[0022] right The thin blood smear images in the image are scaled so that the longest side of the image is 1024, and the scaled images are filled with pixels with a value of 0 to obtain a thin blood smear image training set with an image size of 1024×1024. .

[0023] 2: Feature extraction and processing

[0024] 2.1: Feature extraction of thin blood smear images

[0025] We use VoVNet-57 to extract features from thin blood smear images, and then combine it with FPN to fuse the extracted features at different scales. We then add a connection path on the right side of the FPN, and fuse the fused feature maps again by pixel overlay, fully utilizing the cell classification information from high-level features. The specific steps are as follows:

[0026] 2.1.1: Building the feature extraction network

[0027] To address the limited information and difficulty extracting sufficient and effective features due to the small proportion of malarial parasites in thin blood smear images, we replaced the original ResNet-101 with the VoVNet-57, which is more suitable for small object detection tasks, as the backbone feature extraction network. This strengthens the model's feature extraction capabilities and thereby improves the accuracy of cell detection and segmentation. The network structure of VoVNet-57 is shown in Table 1.

[0028] Table 1. VoVNet-57 network structure

[0029]

[0030]

[0031] VoVNet-57 consists of a Stem module and a four-stage OSA module. The Stem module consists of three 3×3 convolutional layers, and the OSA module consists of 3×3 and 1×1 convolutional layers and a tensor concatenation layer. A 3×3 maximum pooling layer with a stride of 2 is used between each OSA module stage for downsampling. The feature maps of the thin blood smear image output by the last layer of each stage from the second to the fifth stage of VoVNet-57 are denoted as .

[0032] 2.1.2: FPN feature fusion

[0033] The thin blood smear image feature map obtained in step 2.1.1 Input FPN to perform feature fusion of different scales. FPN fuses the cell classification information of the high-level feature map with the cell positioning information of the low-level feature map through a top-down path, which can effectively improve the model's detection accuracy of cells. Each top-down fusion process consists of a nearest neighbor upsampling operation and a horizontally connected 1×1 convolution layer. The nearest neighbor upsampling operation enlarges the resolution of the high-level thin blood smear image feature map to the size of the adjacent low-level thin blood smear image feature map. The horizontally connected 1×1 convolution layer is used to unify the resolution of the high-level thin blood smear image feature map. The feature map obtained by the nearest neighbor upsampling is fused with the feature map output by the 1×1 convolution layer, and then the feature aliasing effect caused by the nearest neighbor upsampling is removed by 3×3 convolution to obtain the fused thin blood smear image feature map. .

[0034] 2.1.3: Added path feature fusion

[0035] Since the high-level feature map of the thin blood smear image and The resolution is low and not suitable for cell detection, but only uses the low-level thin blood smear image feature map and Therefore, the original FPN structure is improved, and a connection path on the right is added on the basis of FPN. The thin blood smear image feature map after FPN fusion is fused again in the form of pixel superposition, making full use of the cell classification information with rich high-level features to enhance the representation ability of the underlying features. The specific steps are as follows: After four times upsampling Perform pixel superposition and fusion, and then use 3×3 convolution to remove the feature aliasing effect caused by upsampling to obtain the thin blood smear image feature map ;Will After four times upsampling Perform pixel superposition and fusion, and then use 3×3 convolution to remove the feature aliasing effect caused by upsampling to obtain the thin blood smear image feature map .

[0036] 2.2: Generation and screening of cell candidate frames

[0037] 2.2.1: Generation of cell candidate boxes

[0038] The Region Proposal Network (RPN) uses 3×3 convolution to act on the thin blood smear image feature map On the top, we can get a feature map that is easier to distinguish cells from background, and Each pixel on the image is mapped back to the original image, according to Set the corresponding cell candidate box size and aspect ratio Generate three cell candidate boxes of different sizes. The feature map obtained by 3×3 convolution is input into the RPN classification layer and regression layer: the classification layer generates a 3D feature vector through 1×1 convolution to represent the confidence that each pixel on the thin blood smear image feature map corresponds to three different sizes of cell candidate boxes containing cells, which is recorded as The regression layer generates a 12-dimensional feature vector through 1×1 convolution to represent the offset and scaling of three cell candidate boxes of different sizes for each pixel on the thin blood smear image feature map. , and Indicates the center offset of the cell candidate box, and Indicates the scaling amount of the cell candidate box length and width.

[0039] Sort the generated cell candidate boxes from high to low according to the confidence output by the RPN classification layer, and retain the top ranked ones. Cell candidate boxes (training stage 12000, detection phase is 6000), and the cell candidate box set is obtained , . Based on the offset and scaling of the regression branch output , using formula (2-1) Correct the position and size of the cell candidate box:

[0040] (2-1)

[0041] in, and Represents the center coordinates of the cell candidate box, and Indicates the length and width of the cell candidate box, and Represents the center coordinates of the corrected cell candidate box, and Indicates the length and width of the corrected cell candidate box.

[0042] The corrected cell candidate frame parameters are converted into the representation of the upper left corner coordinates and the lower right corner coordinates using formula (2-2):

[0043] (2-2)

[0044] in, and Indicates the coordinates of the upper left corner of the cell candidate box, and Indicates the coordinates of the lower right corner of the cell candidate box.

[0045] 2.2.2: Screening of cell candidate frames

[0046] Thin blood smear images contain many stacked cells. When the overlap between candidate frames generated by two cells exceeds a given threshold, the traditional non-maximum suppression algorithm will remove the cell candidate frames with lower confidence, resulting in missed detections. Using an attenuation function to improve the traditional non-maximum suppression algorithm can reduce missed cell detections. The specific steps are as follows:

[0047] (1) Take The cell candidate box with the highest confidence in the middle constitutes the comparison box , and Move into the cell candidate box result set ; traverse The remaining cell candidate boxes in the calculation of the current cell candidate box With comparison box The Intersection over Union (IoU) is calculated as follows:

[0048] (2-3)

[0049] in, For the Cell candidate boxes and comparison boxes The intersection of the areas, For the Cell candidate boxes and comparison boxes The union of the areas.

[0050] (2) If the cell candidate box With comparison box The IoU value is greater than the given threshold , then the candidate cell selection box is not directly removed , but according to formula (2-4) reduce The confidence level is , so that the cells that were originally detected will not be missed.

[0051] (2-4)

[0052] in, Cell candidate box The foreground confidence score of is the attenuation coefficient that can adjust the degree of confidence attenuation. The value is 0.5, The cell candidate box after reset The foreground confidence score of .

[0053] Repeat the above steps (1) and (2) until the cell candidate box set Empty, final The former left candidate boxes (training stage 2000, detection stage is 1000), which is the cell candidate frame after removing redundancy.

[0054] 2.3: Generation of cell candidate frame feature map

[0055] The thin blood smear image feature map after fusion in step 2.1.3 After removing redundancies from the cell candidate boxes in step 2.2.2, they are input to the Region of Interest Align (RoI Align) layer. Based on the input size constraints in the subsequent output branch network, the feature regions corresponding to the cell candidate boxes of different sizes are pooled into fixed sizes of 7×7 and 14×14. The specific steps are as follows: The feature regions corresponding to the cell candidate boxes are divided into 7×7 and 14×14 units; each unit is evenly divided into 4 small units, and the center of each small unit is set as a sampling point. The characteristic values ​​of the sampling points are calculated using bilinear interpolation; the sampling points within each unit are max-pooled to generate 7×7 and 14×14 cell candidate feature maps.

[0056] 2.4: Result prediction of cell candidate box

[0057] The fixed-size cell candidate frame feature map output in step 2.3 is input to the output branch network for cell detection and segmentation. The output branch network contains a classification regression network and a mask generation network: the classification regression network flattens the 7×7 cell candidate frame feature map obtained in step 2.3 into a one-dimensional vector, and then performs cell detection through two 1024-dimensional fully connected layers; the mask generation network passes the 14×14 cell candidate frame feature map obtained in step 2.3 through four convolution layers with a convolution kernel size of 3×3 and a number of channels of 256, and then expands the cell candidate frame feature map size to 28×28 through a 2×2 deconvolution operation, and finally passes through a convolution kernel size of 1×1 and a number of channels of The convolution layer obtains A 28×28 cell segmentation mask, is the number of cell types.

[0058] 3: Model training

[0059] 3.1: Construction and calculation of loss function

[0060] In view of the problem that the number of red blood cells is far greater than the number of malarial parasites and there is an imbalance in cell categories, considering that red blood cells are easy-to-classify samples compared to malarial parasites, a large number of easy-to-classify red blood cells dominate the direction of gradient update, resulting in low accuracy of the model for malarial parasite detection. Therefore, the loss function is reconstructed, and loss weights and adjustment factors are added to the cell classification loss function to reduce the contribution of red blood cells to the network parameters and improve the situation where the imbalance of cell categories leads to low detection accuracy. The reconstructed loss function includes the classification loss in the RPN network. and detection box offset loss , cell classification loss of the output branch network , cell detection frame offset loss and cell mask segmentation loss Five parts:

[0061] (1) RPN network classification loss , the calculation formula is as follows:

[0062] (3-1)

[0063] in, The number of sampled cell candidate boxes in a mini-batch when training the RPN network; is the index of the cell candidate box; For the The true label of the candidate box of the cell When a cell candidate box contains a cell, ; When the When the cell candidate box does not contain cells, ; For the The confidence that a cell candidate box contains a cell.

[0064] (2) RPN network detection box offset loss , the calculation formula is as follows:

[0065] (3-2)

[0066] in, For the The center offset and length and width scaling between the cell candidate frame and the cell detection frame, For the The center offset and length and width scaling between the candidate frame of each cell and the real frame of the cell, for Function, defined as follows:

[0067] (3-3)

[0068] (3) Cell classification loss , the calculation formula is as follows:

[0069] (3-4)

[0070] in, The number of sampled cell candidate boxes in a mini-batch when training the output branch network; is the number of cell categories; For the The number of candidate boxes of cell-like cells; Indicates the The cell candidate box is Cell-like confidence, Indicates the The true label of the cell candidate box, When belongs to cell-like When does not belong to cell-like; For loss weight, when the number of cells of a certain type is small, the loss weight will be relatively large, making the model pay more attention to the cell category with a small number; is the adjustment factor, the confidence level of easily classified red blood cells Close to 1, Will be close to 0, which can reduce the impact of red blood cells on the loss function. According to experience The value is 2.

[0071] (4) Cell detection frame offset loss , the calculation formula is as follows:

[0072] (3-5)

[0073] in, For the The true label of the cell candidate box, Indicates the The center offset and length and width scaling between the cell candidate frame and the cell detection frame, Indicates the The center offset and length and width scaling between the candidate frame of each cell and the real frame of the cell, for function.

[0074] (5) Cell segmentation loss , the calculation formula is as follows:

[0075] (3-6)

[0076] in, is the size of the cell mask resolution; For the cell mask The true label of each pixel, When belongs to cell-like When does not belong to cell-like; For the cell mask Pixels are Cell-like confidence.

[0077] (6) Overall loss is the classification loss in the above RPN network and detection box offset loss , cell classification loss of the output network , cell detection frame offset loss and cell mask segmentation loss The sum of the five parts is calculated as follows:

[0078] (3-7)

[0079] 3.2: Iterative update of network weights

[0080] The thin blood smear image training set in step 1 Enter the Mask R-CNN network model built in step 2 for training. When training the network model, first load the pre-trained weight file as the network model initialization parameters, and randomly initialize the parameters of the right connection path added on the basis of FPN. Then calculate the gradient value of each parameter of the overall loss function in step 3.1, and use the stochastic gradient descent algorithm (SGD) to iteratively update the weight parameters of the network model:

[0081] (3-8)

[0082] in, is the weight parameter to be updated. is the learning rate, is the overall loss function about The gradient, for Updated parameters.

[0083] 4: Malaria detection

[0084] 4.1: Cell detection and segmentation

[0085] Input the thin blood smear image to be detected into the trained model to obtain the predicted information of the cell: the cell category confidence, position coordinates and cell mask corresponding to the detection box, and set the classification threshold Detection frames whose cell category confidence does not meet the classification threshold are eliminated. Confidence decay is used to perform non-maximum suppression on cell detection frames of the same category to obtain the optimal cell detection frame. In the thin blood smear image being tested, the position of the optimal detection frame is marked as a rectangular frame. The cell category and confidence are marked above the rectangular frame, and the cells are segmented at the pixel level.

[0086] 4.2: Calculation of Plasmodium parasite density

[0087] When calculating the parasitic density of Plasmodium, more than 1000 red blood cells need to be detected, and the thin blood smear images collected using an optical microscope must have no overlapping fields of view. According to step 4.1, perform cell detection and segmentation on multiple thin blood smear images collected without overlapping fields of view, and count the number of Plasmodium and red blood cells. Calculate the parasitic density of Plasmodium according to formula (4-1) :

[0088] (4-1)

[0089] in, To detect the number of malarial parasites; To detect the number of red blood cells; It is the average number of red blood cells per microliter of blood, calculated based on 5 million red blood cells per microliter for men and 4.5 million red blood cells per microliter for women.

[0090] 3. Purpose of the Invention

[0091] The following describes the specific embodiments of the present invention in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when detailed descriptions of known functions and designs may dilute the main content of the present invention, such descriptions will be omitted here.

[0092] Example: Detection and segmentation of Plasmodium vivax and red blood cells.

[0093] 1: Data preprocessing

[0094] According to step 1.1, the thin blood smear image dataset provided by Yunnan Provincial Parasitic Disease Control and Prevention As the initial dataset, we annotate malarial parasites and red blood cells at different stages in each image to obtain an annotated thin blood smear image dataset. The dataset A total of 253 thin blood smear images, consisting of seven types of cells, namely Plasmodium vivax macrotrophozoites, Plasmodium vivax ring bodies, Plasmodium vivax immature gametocytes, Plasmodium vivax mature gametocytes, Plasmodium vivax immature schizonts, Plasmodium vivax mature schizonts and normal red blood cells.

[0095] Follow step 1.2, according to The malaria parasites are segmented based on the outline information of the malaria parasites marked in the image. The segmented malaria parasites are randomly rotated at any angle and then copied and pasted to other positions of the corresponding image to expand the number of malaria parasites. The samples after the number of malaria parasites is expanded are then data enhanced by rotation, vertical flipping and horizontal flipping. Expanded to 800 images to obtain a thin blood smear image training set .

[0096] Follow step 1.3 and By scaling and filling 0, the thin blood smear image size is unified to 1024×1024, and the thin blood smear image training set is obtained. .

[0097] 2: Feature extraction and processing

[0098] According to step 2.1, the network of VoVNet-57 and improved FPN is used to extract the features of thin blood smear images. Its structure is shown in the attached figure. Figure 2 As shown in Figure 2, there are bottom-up paths and top-down paths.

[0099] The bottom-up path uses VoVNet-57 as the base network and divides the feature extraction process into 5 stages. The size of the feature map output in each stage is smaller than that of the input thin blood smear image. , the reduction factors are The input and output feature map sizes of each stage are shown in Table 2.

[0100] Table 2. VoVNet feature extraction process

[0101]

[0102] The top-down path uses the improved FPN network to Perform feature fusion. The fusion process is shown in Table 3.

[0103] Table 3. Feature fusion process

[0104]

[0105] According to step 2.2.1, the RPN network performs a feature map on each layer. The pixel points on generate three cell candidate boxes of different sizes. and A total of 128×128×3+256×256×3=245760 cell candidate boxes are generated. Each cell candidate box corresponds to a confidence level, and the one with the highest confidence level is retained. (Training phase 12000, detection phase The candidate frames are sorted and corrected according to formulas (2-1) and (2-2). Table 4 shows the numbers, coordinates, and confidence levels of some of the candidate frames after sorting and correction in a certain detection phase.

[0106] Table 4. Some cell candidate boxes after sorting and correction

[0107]

[0108] Follow step 2.2.2 and set the IoU threshold based on the sorted and corrected cell candidate boxes. = 0.5, and non-maximum suppression is performed by attenuating the confidence of the cell candidate box according to formula (2-4), screening the best candidate box containing the cell. For example, the cell candidate box numbered 69631 is used as the comparison box, and the process of attenuating the confidence of the cell candidate box is shown in Table 5.

[0109] Table 5. The process of attenuating the confidence of cell candidate boxes

[0110]

[0111]

[0112] According to step 2.3, the filtered cell candidate boxes are input into the RoI Align layer, and the bilinear interpolation method and maximum pooling operation are used to obtain the cell candidate box feature maps of 7×7 and 14×14 sizes.

[0113] Following step 2.4, build a classification and regression network and a mask generation network. The classification and regression network uses a fully connected layer to detect cells and determine whether the target contained within the cell candidate box is a red blood cell, malarial parasite, or background. The mask generation network uses a fully convolutional neural network to segment cells and obtain the cell outline information contained within the cell candidate box. The 7×7 cell candidate box feature map is input into the classification and regression network for cell classification and target box localization. The structure of the classification and regression network is shown in Table 6.

[0114] Table 6. Structure of classification and regression network

[0115]

[0116] Among them, the classification layer outputs an 8-dimensional vector, which represents the classification confidence of 7 different cells and backgrounds, and the regression layer outputs a 28-dimensional vector, which represents the 4 coordinate vectors of the cell candidate box corresponding to each cell type.

[0117] The 14×14 cell candidate box feature map is input into the mask generation network for segmentation to obtain the cell outline. The structure of the mask generation network is shown in Table 7.

[0118] Table 7. Structure of the mask generation network

[0119]

[0120] 3: Model training

[0121] Follow step 3.1 to convert the thin blood smear image training set Input the Mask R-CNN network model for training. Compare the predicted information with the labeled real data, calculate the classification loss and detection box offset loss of the RPN network according to formulas (3-1) and (3-2), calculate the classification loss, detection box offset loss and mask segmentation loss of the output network according to formulas (3-4), (3-5) and (3-6), and calculate the overall loss function according to formula (3-7) .

[0122] According to step 3.2, set the learning rate to =0.005, according to formula (3-8), the stochastic gradient descent method is used to iteratively update the weights of the network model until the loss function value converges.

[0123] 4: Malaria detection

[0124] Follow step 4.1 to set the classification threshold , cell detection and segmentation are performed on the input thin blood smear image. Figure 3 The results of cell detection and segmentation for a thin blood smear image are shown below: Each rectangular box corresponds to a cell. The segmented cells are displayed in different colors within the box, and the cell category can be determined based on the label above the rectangular box. Represents red blood cells, Represents immature gametocytes of Plasmodium vivax.

[0125] Following step 4.2, count the number of red blood cells and malarial parasites in the multiple thin blood smear images collected, and calculate the malarial parasite density according to formula (4-1). Table 8 shows the statistical results of the number of red blood cells and malarial parasites for a male patient.

[0126] Table 8. Statistics of the number of red blood cells and malarial parasites

[0127]

[0128] The number of red blood cells detected The number of malarial parasites is 1409 For men, there are 5 million red blood cells per microliter of blood. for , then each microliter of blood contains 17743 malarial parasites:

[0129]

[0130] IV. Advantages and positive effects of the present invention compared with the prior art

[0131] (1) The present invention proposes a malaria detection method based on the Mask R-CNN model, which adopts VoVNet-57, which is more suitable for small target detection tasks, as the feature extraction network of Mask R-CNN to enhance the model's ability to extract features.

[0132] (2) The present invention proposes an improved FPN structure, which adds a right-side connection path on the basis of FPN, and further fuses the feature map after FPN fusion, enriching the cell classification information contained in the underlying feature layer, thereby improving the accuracy of the model in detecting cells.

[0133] (3) The present invention uses an attenuation function to improve the traditional non-maximum suppression algorithm. For the cell candidate frames that were originally to be discarded, the confidence is attenuated by using the attenuation function according to the intersection-over-union ratio instead of being directly eliminated, which can effectively reduce the situation of missed detection of overlapping cells.

[0134] (4) The present invention reconstructs the classification loss function of the Mask R-CNN network. By adding loss weights and adjustment factors to the cell classification loss function, the model pays more attention to the smaller number of malarial parasites and reduces the influence of red blood cells on the loss function, thereby improving the problem of low detection accuracy caused by cell class imbalance. BRIEF DESCRIPTION OF THE DRAWINGS

[0135] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0136] Figure 1 : Flow chart of the specific steps of the present invention

[0137] Figure 2 : The improved feature pyramid network structure in this invention

[0138] Figure 3 : Cell detection and segmentation results of thin blood smear images in the present invention.

Claims

1. A method for detecting and segmenting malarial blood smear cells based on Mask R-CNN, characterized in that: The following steps are involved: S1: Data preprocessing: Label different types and stages of malarial parasites and red blood cells on each thin blood smear image collected by an optical microscope. Perform data augmentation on the labeled dataset and unify the image size to obtain the thin blood smear image training set. S2: Feature extraction and processing: constructing an improved Mask R-CNN network model to extract and process the input thin blood smear image; S3: Model training: input the thin blood smear image training set in step S1 into the model in step S2, calculate the loss function and perform backpropagation, and iteratively update the weights of the network model; S3.1: Construction and calculation of loss function Reconstruct the loss function, add loss weights and adjustment factors to the cell classification loss function, reduce the contribution of red blood cells to network parameters, and improve the situation where the detection accuracy is not high due to imbalance of cell categories. The reconstructed loss function includes the classification loss in the RPN network. and detection box offset loss , cell classification loss of the output branch network , cell detection frame offset loss and cell mask segmentation loss Five parts: (1) RPN network classification loss , the calculation formula is as follows: （3-1） in, The number of sampled cell candidate boxes in a mini-batch when training the RPN network; is the index of the cell candidate box; For the The true label of the candidate box of the cell When a cell candidate box contains a cell, ; When the When the cell candidate box does not contain cells, ; For the The confidence level of cells contained in each cell candidate frame; (2) RPN network detection box offset loss , the calculation formula is as follows: （3-2） in, For the The center offset and length and width scaling between the cell candidate frame and the cell detection frame, For the The center offset and length and width scaling between the candidate frame of each cell and the real frame of the cell, for Function, defined as follows: （3-3） (3) Cell classification loss , the calculation formula is as follows: （3-4） in, The number of sampled cell candidate boxes in a mini-batch when training the output branch network; is the number of cell categories; For the The number of candidate boxes of cell-like cells; Indicates the The cell candidate box is Cell-like confidence, Indicates the The true label of the cell candidate box, When belongs to cell-like When does not belong to cell-like; For loss weight, when the number of cells of a certain type is small, the loss weight will be relatively large, making the model pay more attention to the cell category with a small number; is the adjustment factor, the confidence level of easily classified red blood cells Close to 1, Will be close to 0, which can reduce the impact of red blood cells on the loss function. According to experience The value is 2; (4) Cell detection frame offset loss , the calculation formula is as follows: （3-5） in, For the The true label of the cell candidate box, Indicates the The center offset and length and width scaling between the cell candidate frame and the cell detection frame, Indicates the The center offset and length and width scaling between the candidate frame of each cell and the real frame of the cell, for function; (5) Cell segmentation loss , the calculation formula is as follows: （3-6） in, is the size of the cell mask resolution; For the cell mask The true label of each pixel, 1 o'clock belongs to cell-like When does not belong to cell-like; For the cell mask Pixels are Cell-like confidence; (6) Overall loss is the classification loss in the above RPN network and detection box offset loss , cell classification loss of the output network , cell detection frame offset loss and cell mask segmentation loss The sum of the five parts is calculated as follows: （3-7） S3.2: Iterative Update of Network Weights The thin blood smear image training set in step S1 Input the Mask R-CNN network model built in step S2 for training. When training the network model, first load the pre-trained weight file as the network model initialization parameters, and randomly initialize the parameters of the right connection path added on the basis of FPN. Then calculate the gradient value of each parameter of the overall loss function in step S3.1, and use the stochastic gradient descent algorithm (SGD) to iteratively update the weight parameters of the network model: （3-8） in, is the weight parameter to be updated. is the learning rate and , is the overall loss function about The gradient, for Updated parameters; S4: Malaria detection: The thin blood smear image to be tested is input into the model trained in step S3 to perform cell detection and segmentation, and calculate the malarial parasite density.

Images