An automatic annotation method for fluorescence-bright field microscopic images of phytoplankton cells

Abstract

The present invention proposes an automatic annotation method for phytoplankton cell fluorescence-bright field microscopic images. The method first synchronously measures the fluorescence and bright field microscopic images of phytoplankton cells, then performs digital image morphology processing on the fluorescence image, automatically draws the outline of the phytoplankton cell bright field microscopic image through image processing technology, converts the fluorescence image into a mask image required for training instance segmentation MaskRCNN network, and finally trains the MaskRCNN network with the automatically generated annotation mask image, which provides an effective means for phytoplankton cell recognition. The automatic annotation method of the present invention greatly reduces or even replaces the workload of manual annotation.

Description

Technical Field

[0001] The invention belongs to the field of resources and environment, in particular to the field related to phytoplankton image recognition, and specifically relates to an automatic annotation method for phytoplankton cell fluorescence-bright field microscopic images. Background Art

[0002] Monitoring of phytoplankton diversity is an important part of water quality biological assessment and is of great significance to the protection of the water ecological environment. The traditional microscopic identification method of algae communities requires professional operation and is time-consuming and labor-intensive. There are very obvious morphological differences between different phytoplankton cell images. The phytoplankton monitoring method based on image recognition completes the classification of algae by extracting the morphological, color, and texture features of algae cell images, and plays a vital role in phytoplankton cell monitoring.

[0003] Image segmentation is an important step in identifying phytoplankton cell images. The segmentation results of traditional image segmentation methods such as threshold method and edge monitoring method are unstable, and they are easy to segment complete algae cells into several parts and segment background impurities. In recent years, image recognition methods based on convolutional neural networks, such as Mask RCNN, have been well applied in phytoplankton cell image segmentation. However, training convolutional neural networks requires a large amount of labeled data, and currently labeling phytoplankton cell images still relies on manually drawing algae cell outlines with a mouse in labeling tools such as LabelMe. The workload of labeling phytoplankton microscopic images is huge, which has become a bottleneck problem restricting the development of phytoplankton cell monitoring methods based on image recognition technology. Summary of the invention

[0004] The present invention discloses a method for automatically labeling a bright field microscopic image using a fluorescent microscopic image of phytoplankton, which replaces the traditional workload of manually drawing algae cell outlines in labeling tools such as LabelMe.

[0005] The technical solution of the present invention is: a method for automatically annotating phytoplankton cell fluorescence-bright field microscopic images, comprising the following steps:

[0006] Step (1) synchronously collecting a microscopic fluorescence image and a microscopic bright field image of phytoplankton cells in a fluorescence-bright field dual-channel microscopic imaging instrument;

[0007] Step (2) converting the fluorescence image into a grayscale image, and converting the grayscale fluorescence image into a binary image using a large law method;

[0008] Step (3) uses the cross-shaped structure element M to perform a morphological opening operation on the binary image to eliminate isolated points caused by noise factors:

[0009] Step (4) using a cross-shaped structure element M to perform a morphological dilation operation on the binary image;

[0010] Step (5) determining the connected regions in the binary image, thereby automatically drawing the outlines of the phytoplankton cells and generating the instance mask map required for training the Mask RCNN network;

[0011] Step (6) annotating the algae species corresponding to each instance mask image and constructing a phytoplankton cell image dataset;

[0012] Step (7) first pre-train the Mask RCNN network using the COCO dataset, and then train the Mask RCNN network using the phytoplankton cell image dataset based on the pre-trained model;

[0013] Step (8) inputs the collected phytoplankton cell image into the trained Mask RCNN model, and outputs the border, mask image and type of phytoplankton cell of the phytoplankton cell image.

[0014] Beneficial effects:

[0015] Aiming at the problem that the data annotation of the phytoplankton cell segmentation network is heavily dependent on manual annotation and the manual annotation cost is high, the present invention proposes a method for automatically annotating bright field microscopic images using fluorescent microscopic images of phytoplankton cells. The method synchronously measures the bright field image and the fluorescent image of the phytoplankton cell in a fluorescent-bright field dual-channel imaging system, performs digital image morphology processing on the fluorescent image, and then converts the fluorescent image into a mask image required for the training instance segmentation MaskRCNN model, thereby realizing effective and rapid automatic generation of mask images, replacing the workload of manual annotation, and finally training the Mask RCNN network with the automatically generated annotated mask image, providing an effective means for phytoplankton cell recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Flow chart of the method of the present invention;

[0017] Figure 2 Schematic diagram of Mask RCNN network structure;

[0018] Figure 3 Bright field image and fluorescence image effect diagram of Nostoc, a. bright field image, b. fluorescence image, c. additive fusion of bright field image and fluorescence image, d. automatically drawn mask image;

[0019] Figure 4 Bright field image and fluorescence image effect diagram of Peridinium, a. Bright field image, b. Fluorescence image, c. Additive fusion of bright field image and fluorescence image, d. Automatically drawn mask image;

[0020] Figure 5Bright field image and fluorescence image effect diagram of lake-dwelling oocysts; a. bright field image, b. fluorescence image, c. additive fusion of bright field image and fluorescence image, d. automatically drawn mask image;

[0021] Figure 6 The effect diagram after the automatic labeling method of the present invention is integrated into the LabelMe labeling tool, a. Nostoc, b. Peridinium, c. Lacustrine Oocystis;

[0022] Figure 7 Mask RCN segmentation effect diagram; a. Nostoc, b. Polydinium, c. Lake Oocystis. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0024] The present invention discloses a method for automatically marking the outline of a bright field microscopic image using a fluorescent image of a phytoplankton cell, such as Figure 1 As shown, the method comprises the following steps:

[0025] Step (1) synchronously collecting a microscopic fluorescence image and a microscopic bright field image of phytoplankton cells in a fluorescence-bright field dual-channel microscopic imaging instrument;

[0026] Step (2) converts the fluorescence image into a grayscale image, and uses the law to convert the grayscale fluorescence image into a binary image. In the binary image B processed by this step, the bright area is the area where the phytoplankton cells are located, and the black area is the background area;

[0027] Step (3) uses the cross-shaped structure element M to perform a morphological opening operation on the binary image B to eliminate isolated points caused by factors such as noise:

[0028]

[0029] Step (4) To ensure that the automatically drawn outline can contain the entire algae cell, a cross-shaped structure element M is used to binarize the image B. 1 Perform a morphological dilation operation:

[0030]

[0031] Step (5) Determine the binary image B using the cv2.findContours method in the Python-Opencv image processing library 2The connected areas in the image are automatically drawn to automatically draw the outlines of the phytoplankton cells and generate the instance mask map required for training the Mask RCNN network. The network structure is as follows: Figure 2 As shown in the figure, Mask RCNN is an instance segmentation model developed based on Faster RCNN. By using the fully connected network FCN at the output end, it can simultaneously output the bounding box of the phytoplankton cell, the segmentation mask map, and the classification result;

[0032] Step (6) using LabelMe software to label the algae species of each mask instance and construct a phytoplankton cell image dataset;

[0033] Step (7) The COCO dataset is a large-scale image dataset for object detection and segmentation. To accelerate the convergence of the model, the Mask RCNN network is first pre-trained using the COCO dataset, and then the network is trained using the phytoplankton cell image dataset based on the pre-trained model;

[0034] Step (8) inputs the collected phytoplankton cell image into the trained Mask RCNN model, and outputs the border, mask image and type of phytoplankton cell of the phytoplankton cell image.

[0035] See also Figure 3 The following are the effects of automatic annotation, which show the bright field image and fluorescent image of Nostoc. Figure 3 a is a bright field image, b is a fluorescent image, c is the additive fusion of the bright field image and the fluorescent image, and d is an automatically drawn mask image;

[0036] The present invention performs digital image processing on the fluorescent image to automatically obtain the mask image of the algae cells. In order to facilitate the labeling of the types of bright field images, the source code of the labeling tool LabelMe is modified, such as Figure 6 As shown in a, when the bright field image is opened, the fluorescent image is automatically loaded according to the path and the mask image is drawn. Then right-click the area where the algae cells are located and click the "Edit Label" button to mark the type of algae cells.

[0037] See also Figure 4 , which are the bright field image and fluorescence image effect diagram of Peridinium, where a is the bright field image, b is the fluorescence image, c is the additive fusion of the bright field image and the fluorescence image, and d is the automatically drawn mask image.

[0038] See also Figure 5 , which are the bright field image and fluorescence image effect diagram of lake-dwelling oocysts, where a is the bright field image, b is the fluorescence image, c is the additive fusion of the bright field image and the fluorescence image, and d is the automatically drawn mask image.

[0039] See also Figure 6The method of automatically drawing the outline of phytoplankton cells of the present invention is integrated into LabelMe annotation. When the bright field image is opened, the fluorescent image is automatically loaded according to the path and the edge outline is drawn. Figure 6 The user can modify the type of phytoplankton by right clicking in the pop-up box. Figure 6 This is the effect diagram after the automatic labeling method of the present invention is integrated into the LabelMe labeling tool, where a is Nostoc, b is Peridinium, and c is Lacustrine Oocystis;

[0040] See also Figure 7 , which is the Mask RCN segmentation effect diagram. After the Mask RCNN model is trained, the bright field image of the phytoplankton cells is input into the Mask RCNN model to obtain the instance segmentation result of the phytoplankton, including the type of algae, mask map, and border. The example diagram of the segmentation result is shown in Figure 7 As shown: Among them, a is Nostoc, b is Peridinium, and c is lake-dwelling Oocystis.

[0041] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, and it should be clear that the present invention is not limited to the scope of the specific embodiments, for those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A method for automatically annotating phytoplankton cell fluorescence-bright field microscopic images, characterized in that: The steps include: Step (1) synchronously collecting a microscopic fluorescence image and a microscopic bright field image of phytoplankton cells in a fluorescence-bright field dual-channel microscopic imaging instrument; Step (2) converting the fluorescence image into a grayscale image, and using the big law method to convert the grayscale fluorescence image into a binary image; Step (3) uses a cross-shaped structural element M to perform a morphological opening operation on the binary image to eliminate isolated points caused by noise factors: specifically, the following steps are performed: using a cross-shaped structural element M to perform a morphological opening operation on the binary image Perform morphological opening operations to eliminate isolated points caused by noise factors: Step (4) using the cross-shaped structure element M to perform a morphological dilation operation on the binary image; Specifically, to ensure that the automatically drawn outline can contain the entire algae cell, a cross-shaped structural element M is used to binarize the image. Perform a morphological dilation operation: Step (5) determining the connected regions in the binary image, thereby automatically drawing the outlines of the phytoplankton cells and generating the instance mask map required for training the Mask RCNN network; Step (6), labeling the algae species corresponding to each instance mask image, and constructing a phytoplankton cell image dataset; Step (7), first use the COCO dataset to pre-train the Mask RCNN network, and then based on the pre-trained model, use the phytoplankton cell image dataset to train the Mask RCNN network; Step (8) inputs the collected phytoplankton cell image into the trained Mask RCNN model, and outputs the bounding box, mask map and type of phytoplankton cell image.

2. The method for automatically annotating phytoplankton cell fluorescence-bright field microscopic images according to claim 1, characterized in that: The step (2) comprises: In the binary image B processed in this step, the bright area is the area where the phytoplankton cells are located, and the black area is the background area.

3. The method for automatically annotating phytoplankton cell fluorescence-bright field microscopic images according to claim 1, characterized in that: The step (4) uses the cross-shaped structure element M to perform a morphological dilation operation on the binary image to ensure that the automatically drawn contour includes the entire algae cell.

Images