Mendelian drosophila genetic trait batch identification method based on convolutional neural network

Abstract

The present application belongs to the technical field of genetics and image intelligent processing, and particularly relates to a Mendelian fruit fly genetic trait batch identification method based on a convolutional neural network. The method comprises the following steps: using a mobile terminal device to collect fruit fly images, manually labeling fruit fly positioning frames, fruit fly multi-traits, and fruit fly structure key point data sets; using a random enhancement and progressive enhancement method to train a model, using neural network quantification technology to make the model more suitable for mobile terminal rapid operation; using a key point auxiliary classification method to improve the model interpretability of the long-small wing task; positioning the position of each fruit fly in the image through a target detection model on the mobile terminal, and then distinguishing the long-small wing, red-white eye and female-male traits of each fruit fly through a multi-classification model. The present application can greatly facilitate the work of experimenters, reduce manual repetitive labor, save a large amount of time, and exceed the accuracy of manual classification in various trait classification tasks, thereby playing a standardization role.

Description

Technical Field

[0001] This invention belongs to the field of genetics and intelligent image processing technology, specifically relating to a method for batch identification of Mendelian Drosophila genetic traits based on deep convolutional neural networks. Background Technology

[0002] Fruit flies are a widely used model organism in classical genetics research. In Mendel's fruit fly genetics experiments, to achieve statistical results, researchers needed to artificially breed a large number of fruit flies and observe and record their genetic traits one by one. In traditional experiments, this step revealed problems such as low accuracy and long time consumption in manually judging traits. However, through deep convolutional neural network technology in machine learning, computer equipment can directly replace this part of the work.

[0003] Over the past decade, computer vision problems based on deep learning have been extensively studied and have spawned numerous applications. Image classification, as a classic computer vision problem, has yielded a wealth of mature research results, with accuracy far exceeding human capabilities in most tasks. Running lightweight deep learning models on mobile devices is also a research hotspot. Against this backdrop, this invention provides a batch classification model for fruit fly traits that can run rapidly on mobile devices, enabling researchers to quickly solve this previously laborious task with extremely low learning costs. Summary of the Invention

[0004] The purpose of this invention is to provide a method for batch identification of Mendelian fruit fly genetic traits that is accurate, efficient, and easy to use.

[0005] The method for batch identification of Mendelian Drosophila genetic traits provided by this invention is based on a deep convolutional neural network, and its specific steps are as follows.

[0006] Step 1: Collect fruit fly images and manually label them; divide the dataset into training, validation, and test sets;

[0007] After anesthetizing fruit flies with carbon dioxide, multiple images of each fruit fly (1 to 100, preferably 20-50) were taken using any mobile device camera under an optical microscope or dissecting microscope. After manual annotation, the images were divided into training and validation sets at a 3:1 ratio. The training and validation sets consisted of 110 original images and approximately 3206 individual fruit flies, collected by a total of 28 researchers. The test set consisted of 9 original images and approximately 405 individual fruit flies, collected by 6 isolated researchers to ensure effective isolation. The training set was used to train the model, the validation set was used to monitor the training effect, and the test set was used to verify the performance of the trained model. Annotations included fruit fly bounding boxes, key points, and fruit fly traits.

[0008] Step 2: Preprocess the fruit fly image to determine the coordinates of the fruit fly bounding box;

[0009] Using the EfficientDet Lite model [1] To achieve object detection, different scales of the EfficientDet Lite model are used to adapt to the hardware conditions of various mobile devices. The input image is processed by the EfficientDetLite model and a global non-maximum suppression method suitable for mobile computing, outputting fruit fly bounding box coordinates with a confidence level greater than 50%. The confidence threshold can be adjusted by the experimenters according to the actual situation. Based on the bounding box coordinates output by the object detection model, the original image is cropped to generate multiple sub-images containing a single fruit fly, which serve as the input to the deep convolutional neural network in the next step.

[0010] Step 3: Construct a deep convolutional neural network (DCNN) and train it;

[0011] The deep convolutional neural network mentioned is based on the MobileNetV3 model. [2] Built upon this foundation, the model includes removing the final fully connected layer and adding two fully connected sigmoid layers in parallel after the global average pooling layer, corresponding to the red and white eyes and long winglets traits of fruit flies, respectively. For the classification of long winglets, the model also uses a specially designed "keypoint-assisted classification module." During training, randomized augmentation and progressive learning methods are employed to improve stability and prevent overfitting.

[0012] The keypoint-assisted classification module refers to the use of keypoint data in a deep neural network to assist the model in training image classification tasks. In this invention, there are four keypoints: the tip of the head, the tip of the tail, and the tips of both wings of the fruit fly. The keypoints at the tips of the head and tail are encoded using two-dimensional coordinates to reduce the model size. Simultaneously, to avoid the model distinguishing between the left and right wings, the keypoints at the tips of the wings are encoded according to the following rules, where (x1, y1) and (x2, y2) are the original coordinates, and (a, b, c, d, e) are the encoded results:

[0013] c = |x1 - x2|, d = |y1 - y2|,

[0014] If (x1–x2)*(y1–y2)>0, then e=1; otherwise, e=0.

[0015] On the global average pooling layer of the deep convolutional network: a fully connected layer with LeakyReLU activation is used to regress the coordinates of the keypoints at the head and tail ends, as well as the above (a,b,c,d). The mean squared error is used as the loss function, denoted as Loss. Keypoints MSE The above e is classified into binary categories using fully connected layers with sigmoid activation functions in parallel, with binary cross-entropy as the loss function, denoted as Loss. Keypoints BCE Finally, all nine keypoint neurons are merged, a fully connected layer is added, and the sigmoid function is used to classify fruit flies with long wings. Binary cross-entropy is used as the loss function, denoted as Loss. Classification BCE In the early stages of training, the loss is calculated using the following weighted formula:

[0016] Loss total =Loss Keypoints MSE +ε×Loss Keypoints BCE +ε×Loss Classification BCE ,

[0017] Here, ε is a local minimum, which is set to 1 × 10 in this example. -6 Once the loss function corresponding to the keypoint converges on the validation set, the loss function corresponding to the keypoint is no longer calculated, and a small learning rate is used to train only the long wing classifier.

[0018] In this invention, the random enhancement method is specifically designed for image enhancement in deep learning. This method includes two important metrics, N and M. N refers to the number of enhancement methods used, and M refers to the intensity of each enhancement method. The specific steps are as follows: First, a value K is randomly selected from the range (N / 2, N) as the number of methods to be selected. Then, K methods are randomly selected from a pre-defined set of enhancement methods, including JPEG compression, convolution operations, edge detection, blurring, superimposed noise, pixel-covered noise, region-covered noise, channel inversion, contrast, brightness, saturation, hue, geometric transformation, histogram transformation, and superpixel transformation. Following the randomly selected set of methods and based on the intensity given by the M value, the corresponding enhancement methods are sequentially applied to the image, ultimately outputting the enhanced image for neural network training.

[0019] In this invention, the training process of the classification network model utilizes a progressive learning method and the aforementioned stochastic augmentation method. The training employs an adaptive regularization strategy, specifically including: training is divided into 80 stages, with each stage progressing to the next after 5 iterations; during the 80 stages, the N and M values ​​in the stochastic augmentation method are linearly increased, with N increasing from 6 to 8 and M increasing from 60 to 100; during the 80 stages, the resolution of the input image is linearly adjusted, increasing from 64*64 to 224*224; and a Dropout layer is added before each classifier, with a Dropout ratio of 0.5.

[0020] Step 4: Repeat step 3 until the metric on the validation set reaches its minimum, then stop training.

[0021] Step 5: To increase the model's inference speed and reduce its size, neural network quantization technology is adopted. Dynamic quantization is used for the object detection model, and floating-point optimization is used for the classification model. Fast inference is achieved on mobile devices through the TFLite interface.

[0022] This invention has the following characteristics:

[0023] (1) High efficiency: The model is specially optimized for mobile devices, saving hard disk space and running fast; the entire application installation package is only 64MB, and it only takes about 20 seconds to distinguish about 1,500 fruit fly traits using a mobile device, greatly reducing labor costs;

[0024] (2) Ease of use: Sample collection and model inference can be completed simultaneously on the mobile device, avoiding unnecessary operations and requiring no additional learning cost, which is very user-friendly for experimenters;

[0025] (3) Interpretability: A specially designed key point-assisted classification method was adopted, so the model has good interpretability in the long-winged task;

[0026] (4) High precision: When tested on an isolated test set, the target detection model has a recall and precision of about 100%, and the classification model has an accuracy of about 98.8% on the task of red and white eyes and male and female classification, and about 93.9% on the task of long small wings classification. Both are higher than the accuracy of manual classification and can play a standardization role. Attached Figure Description

[0027] Figure 1 This is an illustration of a fruit fly sample collected for this invention.

[0028] Figure 2 This diagram illustrates the difference in target detection accuracy between the two EfficientDetLite(0,4) models used in this invention.

[0029] Figure 3 To train the index curve during the training process that utilizes keypoints to assist classification.

[0030] Figure 4 This is the architecture of the key point auxiliary classification module in the long wing classification task of the Deep Convolutional Neural Network (DCNN) of this invention.

[0031] Figure 5 This is the overall architecture of the Deep Convolutional Neural Network (DCNN) of this invention.

[0032] Figure 6 This illustration demonstrates how the Grad-CAM method improves model interpretability for keypoint-assisted classification. For the same set of images, the left side of each column represents keypoint-assisted classification, and the right side represents normal classification.

[0033] Figure 7 The training metric curve is shown in the figure after the present invention employs a specially designed random augmentation method and an adaptive regularization strategy. Detailed Implementation

[0034] The technical solution of the present invention will be further described below with reference to the embodiments and accompanying drawings. Obviously, the described embodiments are only a part, and not all, of the embodiments.

[0035] Step 1: After anesthetizing the fruit flies with carbon dioxide, multiple images containing several fruit flies were taken under a dissecting microscope using a mobile device camera. In this example, a total of 150 original images were collected, of which 80 were manually labeled, containing a total of 1733 bounding boxes, representing 1733 fruit flies. The manually labeled trait classification dataset consists of three subsets, each corresponding to one of the three fruit fly traits. Each subset contains 400+400 samples. Typical samples are shown below. Figure 1 As shown, in the subset corresponding to the long wing trait, four key points were marked for each fruit fly.

[0036] Step 2: Employ both the EfficientDet Lite 0 and EfficientDet Lite 4 models to address the hardware limitations of different mobile devices. The input image is processed by the EfficientDet Lite model and a global non-maximum suppression method suitable for mobile computing, outputting fruit fly bounding box coordinates with a confidence level greater than a certain threshold. The EfficientDet Lite 4 model is approximately 5 times larger than the EfficientDet Lite 0 model and about 7 times slower inference speed, but offers better accuracy. The differences in object detection accuracy between the two models are detailed below. Figure 2 .

[0037] Step 3: Based on the bounding box coordinates output by the object detection model, the original image is cropped to generate multiple sub-images containing a single fruit fly. Building upon the MobileNetV3 model, the final fully connected layer is removed, and two fully connected sigmoid layers are added in parallel after the global average pooling layer, corresponding to the fruit fly's red and white eyes and long winglets, respectively. For the classification of long winglets, the model uses a specially designed "keypoint-assisted classification module." During training, a specially designed random augmentation and progressive learning method is employed to improve stability and prevent overfitting.

[0038] During the training process that utilizes keypoint-assisted classification, the training metric curves in the examples are as follows: Figure 3 As shown, the accuracy of the keypoint slope classifier rapidly increases to approximately 80% before remaining relatively stable, while the mean absolute error of the keypoint coordinates steadily decreases with the number of iterations. The accuracy of long wing classification also gradually improves and converges after a certain number of iterations. The specific network architecture of the keypoint auxiliary classification module is shown below. Figure 4 As shown, the overall network architecture is as follows: Figure 5 As shown, the Grad-CAM method demonstrates that keypoint-assisted classification offers better model interpretability because regions associated with judging winglets show higher activation levels, such as... Figure 6 As shown in the figure. After employing a specially designed randomized augmentation method and an adaptive regularization strategy, the training process stabilized, and the training metric curves in the examples are shown in the figure. Figure 7 As shown in the figure, after using the random augmentation method and the adaptive regularization strategy, the changes in the metrics of the training set and the validation set are relatively smooth.

[0039] Step 4: Neural network quantization technology was used. Dynamic quantization was used for the object detection model, and floating-point optimization was used for the classification model. The quantized object detection model was 3.84MB in size, and the classification model was 1.13MB in size. After writing to the metadata, the TfLite model was generated, and speed testing was performed on an Android device. The object detection model, EfficientDetLite0, could process an image in 37 milliseconds, and the classification model could process an image in 7 milliseconds. Finally, the model performance was evaluated on the test set. The object detection model, EfficientDetLite4, achieved 100% recall and precision. The classification model achieved 93.9% accuracy in classifying long wings, and 98.8% accuracy in classifying red-white eyes and gender.

[0040] References

[0041] [1]Tan,M.,Pang,R.,&Le,Q.V.(2020).Efficientdet:Scalable and efficientobject detection.In Proceedings of the IEEE / CVF conference on computer visionand pattern recognition(pp.10781-10790).

[0042] [2]Howard,A.,Sandler,M.,Chu,G.,Chen,L.C.,Chen,B.,Tan,M.,...&Adam,H.(2019).Searching for mobilenetv3.In Proceedings of the IEEE / CVF internationalconference on computer vision(pp.1314-1324)。

Claims

1. A method for batch identification of fruit fly traits based on convolutional neural networks, characterized in that, The specific steps are as follows: Step 1: Collect fruit fly images and manually label them; divide the dataset into training, validation, and test sets; After anesthetizing fruit flies with carbon dioxide, multiple images of several fruit flies were taken using a mobile device camera under an optical microscope or dissecting microscope. After manual annotation, the images were divided into training and validation sets at a 3:1 ratio. The training and validation sets consisted of 110 original images and approximately 3206 individual fruit flies, collected by a total of 28 researchers. The test set consisted of 9 original images and approximately 405 individual fruit flies, collected by 6 isolated researchers to ensure effective isolation testing. The training set was used to train the model, the validation set was used to monitor the training effect, and the test set was used to verify the performance of the trained model. Annotations included fruit fly bounding boxes, key points, and fruit fly traits. Step 2: Preprocess the fruit fly image to determine the coordinates of the fruit fly bounding box; The EfficientDet Lite model is used as the basis for object detection. Specifically, different scales of the EfficientDet Lite model are adopted to cope with the hardware conditions of different mobile devices. The input image is processed by the EfficientDetLite model and a global nonmaximum suppression method suitable for mobile computing to output fruit fly bounding box coordinates with a confidence score greater than 50%. Based on the output bounding box coordinates, the original image is cropped to generate multiple sub-images containing a single fruit fly, which are used as inputs to the deep convolutional neural network in the next step. Step 3: Construct a deep convolutional neural network and train it; The deep convolutional neural network described is built on the MobileNetV3 model. It includes removing the last fully connected layer and adding two fully connected sigmoid layers in parallel after the global flat pooling layer, corresponding to the red and white eyes and long wing traits of fruit flies, respectively. For the classification of long wings, the model also uses a keypoint auxiliary classification module specially designed for fruit flies. During the training process, random augmentation and progressive learning methods are used to improve stability and prevent overfitting. The keypoint-assisted classification module refers to the use of keypoint data in a deep neural network to assist the model in training image classification tasks. There are four keypoints: the tip of the head, the tip of the tail, and the tips of both wings of the fruit fly. The keypoints at the tips of the head and tail are encoded using two-dimensional coordinates to reduce the model size. Meanwhile, to avoid the model distinguishing between the left and right wings and to reduce redundant information in the data, the keypoints at the tips of the wings are encoded according to the following rules, where (x1, y1) and (x2, y2) are the original coordinates, and (a, b, c, d, e) are the encoded results: c=|x1-x2|,d=|y1-y2|, If (x1–x2)*(y1–y2)>0, then e=1; Otherwise, e = 0; On the global average pooling layer of the deep convolutional network: a fully connected layer with LeakyReLU activation is used to regress the coordinates of the keypoints at the head and tail ends, as well as the above (a,b,c,d). The mean squared error is used as the loss function, denoted as Loss. Keypoints MSE The above e is classified into binary categories using fully connected layers with sigmoid activation functions in parallel, with binary cross-entropy as the loss function, denoted as Loss. Keypoints BCE Finally, all nine keypoint neurons are merged, a fully connected layer is added, and the sigmoid function is used to classify fruit flies with long wings. Binary cross-entropy is used as the loss function, denoted as Loss. Classification BCE In the early stages of training, the loss is calculated using the following weighted formula: Loss total =Loss Keypoints MSE +ε×Loss Keypoints BCE +ε×Loss Classification BCE , Where ε is a local minimum value. Once the loss function corresponding to the key point converges on the validation set, the loss function corresponding to the key point is no longer calculated, and a small learning rate is used to train only the long wing classifier. Step 4: Repeat step 3 until the metrics on the validation set reach their optimal values, then stop training. Step 5: To increase the model's inference speed and reduce its size, neural network quantization technology is adopted. Dynamic quantization is used for the object detection model, and floating-point optimization is used for the classification model. Fast inference is achieved on mobile devices through the TFLite interface.

2. The method for batch identification of fruit fly traits based on convolutional neural networks according to claim 1, characterized in that, In step three, the random enhancement method is an image enhancement method used in deep learning. It includes two important metrics: N and M. N refers to the number of enhancement methods used, and M refers to the intensity of the enhancement methods used. The specific steps are as follows: a value is randomly selected from the range (N / 2, N) as the number of methods to be selected, K. K methods are randomly selected from the set of enhancement methods, which includes JPEG compression, convolution operation, edge detection, blurring, superimposed noise, pixel-covered noise, region-covered noise, channel inversion, contrast, brightness, saturation, hue, geometric transformation, histogram transformation, and superpixel transformation. According to the randomly selected set of methods and the intensity given by the value of M, the corresponding enhancement methods are applied to the image in sequence, and the enhanced image is finally output for neural network training.

3. The method for batch identification of fruit fly traits based on convolutional neural networks according to claim 2, characterized in that, In step three, the classification network model training process employs a progressive learning method and the aforementioned stochastic augmentation method. The training employs an adaptive regularization strategy, specifically including: the training is divided into 80 stages, with each stage progressing to the next after 5 iterations; Over 80 stages, the N and M values ​​in the random augmentation method are linearly increased, with N increasing from 6 to 8 and M increasing from 60 to 100; over 80 stages, the resolution of the input image is linearly adjusted, increasing from 64*64 to 224*224; and a Dropout layer is added before each classifier, with a Dropout ratio of 0.5.

Images