A method and device for analyzing river crab phenotype based on computer vision

Abstract

The present invention provides a computer vision-based method and device for analyzing river crab phenotypic characteristics. The method comprises the following steps: collecting a river crab image dataset using a river crab image acquisition device; segmenting the crab carapace from the collected river crab image using an improved Mask R-CNN image segmentation model; extracting the contours of the segmented river crab carapace image to obtain coordinate information of key phenotypic points; and calculating and saving the model output regarding the river crab phenotypic characteristics. By applying computer vision methods to carapace segmentation in river crab images, the present invention automatically extracts more comprehensive river crab phenotypic characteristics from the crab images, providing an important basis for measuring crab growth and population identification, while achieving high accuracy and fast calculation speed.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and device for analyzing river crab phenotypes based on computer vision. Background Art

[0002] The Chinese mitten crab, commonly known as the hairy crab or river crab, belongs to the genus Eriocheir, order Decapoda, order Crustacea, phylum Arthropoda. It boasts tender meat, high nutritional value, and economic value. The crab's growth status can influence its biological phenotype, and conversely, its phenotypic traits can directly reflect its growth status. Morphological analysis and evaluation can provide a basis for origin traceability and growth status evaluation, thereby helping to improve crab yield and quality. The cephalothorax, an integral part of the crab, is highly correlated with its body shape, mass, and plumpness, as its length and width, along with other body measurements. Measuring and analyzing the phenotypic characteristics of the Chinese mitten crab's cephalothorax plays a crucial role in selecting and cultivating high-yield crab varieties.

[0003] Traditional aquaculture measures and monitors the growth of these aquatic products primarily through manual measurement of morphological parameters. However, this manual measurement requires the use of vernier calipers to manually compare and measure each characteristic crab body part. This is highly subjective, labor-intensive, with large errors and low efficiency, making large-scale measurements difficult and limiting the genetic improvement of crab body traits. Therefore, rapidly collecting growth information on the biological objects being measured is a crucial issue in aquaculture science. The development of portable, efficient, integrated equipment based on computer vision and image processing technology for rapid measurement and statistics of biological phenotypic data will facilitate the further commercialization, scale-up, and development of specialized breeding technologies in the aquaculture industry.

[0004] Currently available solutions use traditional image processing methods, which involve using software to establish feature points on captured cephalothorax images or using traditional segmentation methods to segment the carapace of river crab images and manually annotate special points to obtain phenotypic data such as body length and width. These models can measure phenotypic data, but still require a lot of manual interaction, lack good end-to-end performance, and are still lacking in model production efficiency, unable to meet the requirements for accurate and efficient indicators of Chinese mitten crabs. Summary of the Invention

[0005] In response to the defects in the existing technology, the purpose of the present invention is to provide a river crab phenotypic analysis method and equipment based on computer vision. By applying computer vision methods to the carapace segmentation of river crab images, more comprehensive river crab phenotypic characteristics are automatically extracted from river crab images, thereby providing an important basis for measuring the growth condition of river crabs and population discrimination, with high accuracy and fast calculation speed.

[0006] In order to solve the above problems, the technical solution of the present invention is:

[0007] A computer vision-based phenotyping method for river crabs comprises the following steps:

[0008] Collect river crab image datasets using river crab image acquisition equipment;

[0009] The crab carapace is segmented from the collected crab images using the improved MaskR-CNN image segmentation model.

[0010] Extract the contour of the segmented crab carapace image and obtain the coordinate information of the phenotypic key points;

[0011] Calculate and save the phenotypic characteristics of the river crab output by the model.

[0012] Preferably, the step of collecting a river crab image dataset by using a river crab image collection device specifically includes the following steps:

[0013] Placing the river crab into the river crab image acquisition device, and starting the image acquisition device to acquire a top view image of the river crab sample;

[0014] The original crab images are annotated to mark the crab carapace area to be segmented, and data enhancement is performed to expand the training data set.

[0015] The training data set is randomly divided into a training set and a validation set according to proportion.

[0016] Preferably, the step of segmenting the crab carapace from the collected crab image using the improved Mask R-CNN image segmentation model specifically includes the following steps:

[0017] The PointRend module is used to replace the MaskHead module of MaskR-CNN to obtain an improved Mask R-CNN model;

[0018] Use the training dataset to train the improved Mask R-CNN model;

[0019] Use the trained improved Mask R-CNN model to segment the carapace of river crab images.

[0020] Preferably, the improved Mask R-CNN model includes: a Backbone network module, a multi-scale feature fusion module, a segmentation prediction module and a NetworksHeads module. The Backbone network module performs preliminary feature extraction on the crab image to be processed to obtain multi-scale low-resolution features and high-resolution features; the multi-scale feature fusion module performs upsampling feature filling on the low-resolution features of the Backbone network module, and fuses the upsampled low-resolution features with the high-resolution features to obtain a fused feature map; the segmentation prediction module feature map uses the region generation network to generate a candidate box expressing the position of the segmented object and performs regression calculation, and uses the ROIAlign algorithm to obtain the region of interest; the NetworksHeads module is used to classify, detect and segment the region of interest.

[0021] Preferably, the step of extracting the contour of the segmented crab carapace image and obtaining the coordinate information of the phenotypic key points specifically includes the following steps:

[0022] Binarize the segmented crab carapace image and extract the crab carapace outline;

[0023] The contour vertices are rotated and corrected, and based on the corrected contour, the minimum convex polygon belonging to the edge is calculated to obtain the convex hull point set;

[0024] Based on the convex hull point set, the phenotypic information of river crabs is obtained.

[0025] Preferably, the steps of rotating and correcting the contour vertices, and calculating the minimum convex polygon belonging to the edge based on the corrected contour to obtain the convex hull point set specifically include:

[0026] Calculate the convex defects of the contour vertices, rotate and correct the contour point set by the angle between the straight line connecting the two contour points with the largest convex defect and the x-axis, and use the corrected contour for subsequent calculations;

[0027] Use the Graham-Scan algorithm to scan the contour, calculate the minimum convex polygon belonging to the edge, and obtain the convex hull point set;

[0028] The circumscribed rectangle of the contour is calculated and divided into six regions from left to right and from top to bottom according to the midpoint of the height and the two contour points with the largest convex defects: I, II, III, IV, V, and VI. At the same time, the convex hull point set and the contour point set are divided into six regions.

[0029] Preferably, the step of obtaining the phenotypic information of river crabs based on the convex hull point set specifically includes:

[0030] Set a distance threshold d, divide the convex hull points of areas I and III into four neighborhoods according to d, select the convex hull vertices from multiple convex hull points in each neighborhood, and obtain the coordinate point pairs of the first front tooth, the second front tooth, the third front tooth, and the fourth front tooth;

[0031] Based on the outline point set described in area II, the peak and valley detection method is used to obtain the coordinate point pairs of the inner and outer frontal teeth and the coordinate point of the central notch;

[0032] In the convex hull point sets of regions IV and VI, the curvature of the points in the points is calculated, and the points with the largest curvature are selected as the trailing edge boundary angles, and the coordinate point pairs of the trailing edge boundary angles are obtained;

[0033] In region V, the intersection of the perpendicular bisector of the line segment connected by the trailing edge boundary angle coordinate point pair and the contour line is used as the trailing edge midpoint.

[0034] Preferably, the step of calculating and saving the phenotypic characteristics of the river crab output by the model specifically comprises the following steps:

[0035] The pixel length is calculated according to the formula Converted into actual length, where S real 、S pix Refer to actual length and pixel length respectively, and β is the conversion ratio;

[0036] The formula for calculating the phenotypic length of the crab's body teeth is: Where A1-A7 are the inner tooth width, outer tooth width, first front tooth width, second front tooth width, third front tooth width, fourth front tooth width (body width), and rear edge width, respectively. (x i1 ,y i1 )、(x i2 ,y i2 ) are corresponding phenotypic coordinate point pairs;

[0037] The formula for calculating the body length of river crab is: Where L is the body length, (x central notch, y central notch) is the coordinate of the central notch, (x trailing edge midpoint, y trailing edge midpoint) is the coordinate of the trailing edge midpoint;

[0038] The characteristic phenotypic parameters of river crabs calculated above are saved.

[0039] Furthermore, the present invention also provides a computer vision-based river crab phenotyping analysis device, which includes a processor and a memory for storing executable instructions of the processor, and the processor is configured to perform the computer vision-based river crab phenotyping analysis method as described above by executing the executable instructions.

[0040] Compared with the prior art, the advantages of the present invention are as follows:

[0041] 1. Combining computer vision algorithms with image processing technology enables automatic analysis of phenotypic parameters of river crabs, ensuring the accuracy of phenotypic parameter measurements while improving measurement efficiency, providing a convenient and fast evaluation solution for river crab phenotypic analysis research;

[0042] 2. An improved Mask R-CNN model was constructed to improve the accuracy and comprehensiveness of crab carapace segmentation;

[0043] 3. According to the characteristics of the carapace and cephalothorax of river crabs, an automatic detection method for the cephalothorax parameters of river crabs based on the crab contour is proposed to automatically extract the phenotypic characteristic points of river crabs and phenotypic parameters such as cephalothorax length, cephalothorax width, and posterior edge width. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0045] Figure 1 A flowchart of a computer vision-based phenotyping method for river crabs provided in an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of the structure of the improved Mask R-CNN model provided by an embodiment of the present invention;

[0047] Figure 3 Comparison of local details of the segmentation results before and after the improvement of the Mask R-CNN model;

[0048] Figure 4 This is a schematic diagram of the convex hull model of the crab carapace contour image;

[0049] Figure 5 This is the measurement point diagram of river crab body size;

[0050] Figure 6 Schematic diagram of the river crab phenotype. DETAILED DESCRIPTION

[0051] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0052] Specifically, the present invention proposes a method for analyzing river crab phenotyping based on computer vision, such as Figure 1 As shown, the method includes the following steps:

[0053] S1: Collect river crab image datasets using river crab image acquisition equipment;

[0054] Specifically, step S1 includes the following steps:

[0055] Step S11: placing the river crab into a river crab image acquisition device, and starting the image acquisition device to acquire a top view image of the river crab sample;

[0056] Specifically, the image captured by the image acquisition device has a pixel size of 4000×3000 and a picture format of JPG.

[0057] Step S12: annotating the original crab image, marking the crab carapace area to be segmented, and performing data enhancement to expand the training data set;

[0058] Step S13: randomly dividing the training data set into a training set and a validation set according to a certain proportion.

[0059] Specifically, it can be divided into training set and validation set in a ratio of 8:2.

[0060] S2: Using the improved MaskR-CNN image segmentation model, the crab carapace is segmented from the collected crab images.

[0061] Specifically, step S2 includes the following steps:

[0062] Step S21: Use the PointRend module to replace the MaskHead module of MaskR-CNN to obtain an improved MaskR-CNN model;

[0063] Specifically, the improved MaskR-CNN model structure is as follows Figure 2As shown, the improved Mask R-CNN model specifically includes: a Backbone network module, a multi-scale feature fusion module, a segmentation prediction module and a NetworksHeads module; the Backbone network module performs preliminary feature extraction on the crab image to be processed to obtain multi-scale low-resolution features and high-resolution features; the multi-scale feature fusion module performs upsampling feature filling on the low-resolution features of the Backbone network module, and fuses the up-sampled low-resolution features with the high-resolution features to obtain a fused feature map; the segmentation prediction module feature map uses the region proposal network (RPN) to generate a candidate box expressing the position of the segmented object and performs regression calculation, and uses the ROI Align algorithm to obtain the region of interest; the NetworksHeads module is used to classify, detect and segment the region of interest. The original MaskHead module is replaced by the PintRend module. The PointRend module uses a bilinear interpolation algorithm to upsample the previous prediction, and then selects the N most uncertain points on this denser grid. For example, the most uncertain points with a confidence level (confidence interval is [0,1]) lower than 0.5 are selected. Then, the PointRend network calculates the point feature representation for each of these N points and inputs the point feature representation into the multi-layer perceptron (MLP) to predict the label of each point. The above process is iteratively repeated until the required resolution is reached, and the segmentation result is finally obtained.

[0064] Step S22: using the training dataset to train the improved Mask R-CNN model;

[0065] Specifically, the stochastic gradient descent method is used for training, the initial learning rate is 0.0025, a linear increase strategy is adopted, the batch size of each training iteration is 8, the training iteration is 50 cycles, the weight parameters are saved every 5 cycles, and the weight with the highest accuracy is selected as the prediction weight.

[0066] Step S23: Use the trained improved Mask R-CNN model to segment the carapace of the river crab image.

[0067] Specifically, the comparison of the segmentation results of the MaskR-CNN model before and after improvement is as follows: Figure 3 shown.

[0068] S3: Extract the contour of the segmented crab carapace image to obtain the coordinate information of the phenotypic key points;

[0069] The step S3 specifically includes the following steps:

[0070] Step S31: binarizing the segmented crab carapace image to extract the crab carapace outline;

[0071] Step S32: Rotate and correct the contour vertices, and based on the corrected contour, calculate the minimum convex polygon belonging to the edge to obtain the convex hull point set;

[0072] The specific steps in step S32 are as follows:

[0073] 1. Calculate the convex defects of the contour vertices, rotate and correct the contour point set by the angle between the straight line connecting the two contour points with the largest convex defect and the x-axis, and use the corrected contour for subsequent calculations;

[0074] 2. Use the Graham-Scan algorithm to scan the contour, calculate the minimum convex polygon belonging to the edge, and obtain the convex hull point set;

[0075] 3. Calculate the circumscribed rectangle of the contour and divide it into the following categories from left to right and from top to bottom according to the midpoint of the height and the two contour points with the largest convex defects: Figure 4 The six regions Ⅰ, Ⅱ, Ⅲ, Ⅳ, Ⅴ, and Ⅵ shown in the figure divide the convex hull point set and the contour point set into six regions.

[0076] Step S33: Based on the convex hull point set, obtain the phenotypic information of river crabs.

[0077] The specific steps in step S33 are as follows:

[0078] 1. Set a distance threshold d, divide the convex hull points of areas I and III into four neighborhoods according to d, and select the convex hull vertices from multiple convex hull points in each neighborhood to obtain the coordinate point pairs of the first, second, third, and fourth front teeth;

[0079] 2. Based on the contour point set described in area II, the peak and valley detection method is used to obtain the coordinate point pairs of the inner and outer frontal teeth and the coordinate point of the central notch;

[0080] 3. Calculate the curvature of the convex hull points in regions IV and VI, select the point with the largest curvature as the trailing edge corner, and obtain the trailing edge corner coordinate point pair;

[0081] 4. In area V, the intersection of the perpendicular bisector of the line segment connected by the trailing edge boundary angle coordinate point and the contour line is the trailing edge midpoint.

[0082] Specifically, the obtained body measurement points are as follows: Figure 5 shown.

[0083] S4: Calculate and save the phenotypic characteristics of the river crab output by the model.

[0084] Specifically, calculate and save Figure 6The phenotypic characteristics of the river crab are shown. Step S4 comprises the following steps:

[0085] Step S41: Convert the pixel length into actual length according to the following formula:

[0086]

[0087] Where S real 、S pix Refer to actual length and pixel length respectively, β is the conversion ratio;

[0088] Step S42: The formula for calculating the crab's tooth phenotypic length is:

[0089]

[0090] Where A1-A7 are the inner frontal tooth width, outer frontal tooth width, first frontal tooth width, second frontal tooth width, third frontal tooth width, fourth frontal tooth width (body width), and rear edge width, respectively; (x i1 ,y i1 )、(x i2 ,y i2 ) are corresponding phenotypic coordinate point pairs;

[0091] The formula for calculating the body length of river crab is:

[0092]

[0093] Where L is the body length, (x 中央缺刻 ,y 中央缺刻 ) is the coordinate of the central notch, (x 后缘中点 ,y 后缘中点 ) is the coordinate of the midpoint of the trailing edge.

[0094] Step S43: Save the characteristic phenotypic parameters of the river crab calculated above.

[0095] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A computer vision-based phenotyping method for river crabs, characterized in that: The method comprises the following steps: Collect river crab image datasets using river crab image acquisition equipment; The improved Mask R-CNN image segmentation model is used to segment the crab carapace from the collected crab images. Specifically, the following steps are performed: The PointRend module is used to replace the Mask Head module of Mask R-CNN to obtain an improved Mask R-CNN model; Use the training dataset to train the improved Mask R-CNN model; Use the trained improved Mask R-CNN model to segment the carapace of river crab images; Extract the contour of the segmented crab carapace image and obtain the coordinate information of the phenotypic key points, including: Binarize the segmented crab carapace image and extract the crab carapace outline; The contour vertices are rotated and corrected, and based on the corrected contour, the minimum convex polygon belonging to the edge is calculated to obtain the convex hull point set; specifically, the following steps are involved: Calculate the convex defects of the contour vertices, rotate and correct the contour point set by the angle between the straight line connecting the two contour points with the largest convex defect and the x-axis, and use the corrected contour for subsequent calculations; Use the Graham-Scan algorithm to scan the contour, calculate the minimum convex polygon belonging to the edge, and obtain the convex hull point set; Calculate the circumscribed rectangle of the contour and divide it into six regions from left to right and from top to bottom based on the midpoint of the height and the two contour points with the largest convex defects: I, II, III, IV, V, and VI. At the same time, divide the convex hull point set and the contour point set into six regions. Based on the convex hull point set, the phenotypic information of river crabs is obtained, specifically including: Set a distance threshold d, divide the convex hull points of areas I and III into four neighborhoods according to d, select the convex hull vertices from multiple convex hull points in each neighborhood, and obtain the coordinate point pairs of the first front tooth, the second front tooth, the third front tooth, and the fourth front tooth; Based on the outline point set described in area II, the peak and valley detection method is used to obtain the coordinate point pairs of the inner and outer frontal teeth and the coordinate point of the central notch; In the convex hull point sets of regions IV and VI, the curvature of the points in the points is calculated, and the points with the largest curvature are selected as the trailing edge boundary angles, and the coordinate point pairs of the trailing edge boundary angles are obtained; In region V, the intersection of the perpendicular bisector of the line segment connected by the trailing edge boundary angle coordinate point pair and the contour line is taken as the trailing edge midpoint; Calculate and save the phenotypic characteristics of the river crab output by the model.

2. The computer vision-based phenotyping method for river crabs according to claim 1, wherein: The step of collecting a river crab image dataset by using a river crab image collection device specifically includes the following steps: Placing the river crab into the river crab image acquisition device, and starting the image acquisition device to acquire a top view image of the river crab sample; The original crab images are annotated to mark the crab carapace area to be segmented, and data enhancement is performed to expand the training data set. The training data set is randomly divided into a training set and a validation set according to proportion.

3. The computer vision-based phenotyping method for river crabs according to claim 1, wherein: The improved Mask R-CNN model includes: a Backbone network module, a multi-scale feature fusion module, a segmentation prediction module and a Networks Heads module. The Backbone network module performs preliminary feature extraction on the crab image to be processed to obtain multi-scale low-resolution features and high-resolution features; the multi-scale feature fusion module performs upsampling feature filling on the low-resolution features of the Backbone network module, and fuses the upsampled low-resolution features with the high-resolution features to obtain a fused feature map; the segmentation prediction module feature map uses the region generation network to generate a candidate box expressing the position of the segmented object and performs regression calculation, and uses the ROI Align algorithm to obtain the region of interest; the Networks Heads module is used to classify, detect and segment the region of interest.

4. The computer vision-based phenotyping method for river crabs according to claim 1, wherein: The step of calculating and saving the phenotypic characteristics of the river crab output by the model specifically comprises the following steps: The pixel length is calculated according to the formula Converted into actual length, where S real 、S pix Refer to actual length and pixel length respectively, and β is the conversion ratio; The formula for calculating the phenotypic length of the crab's body teeth is: Where A1-A7 are the inner tooth width, outer tooth width, first front tooth width, second front tooth width, third front tooth width, fourth front tooth width (body width), and rear edge width, respectively. (x i1 ,y i1 )、(x i2 ,y i2 ) are corresponding phenotypic coordinate point pairs; The formula for calculating the body length of river crab is: Where L is the body length, (x 中央缺刻 ,y 中央缺刻 ) is the coordinate of the central notch, (x 后缘中点 ,y 后缘中点 ) is the coordinate of the midpoint of the trailing edge; The characteristic phenotypic parameters of river crabs calculated above are saved.

5. A computer vision-based crab phenotyping device, characterized in that: The device includes a processor and a memory for storing executable instructions of the processor, and the processor is configured to perform the computer vision-based crab phenotyping method as described in any one of claims 1 to 4 by executing the executable instructions.

Images