A method for recognizing characteristic points of a head and thorax shield of eriocheir sinensis

Abstract

The application discloses a kind of Eriocheir sinensis carapace feature point identification methods, comprising the following steps: using mobile phone to shoot Eriocheir sinensis carapace photo;Using target identification model detects the carapace in picture and carries out cutting to picture;Using random scaling, random rotation, random occlusion, random brightness contrast and so on Method is enhanced to image;Design a kind of 37 points Eriocheir sinensis carapace positioning method, artificial using 37 points Eriocheir sinensis carapace positioning method carries out feature point annotation to initial picture;Design end-to-end differentiable neural network model, using annotated picture is trained;Using the model that training is completed, the picture that is shot is annotated with feature heat and feature point.The application significantly reduces the workload of artificial in Eriocheir sinensis culture observation, quality detection and the like, and greatly improves work efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the application field of deep learning technology in the fishery, and particularly relates to a method for recognizing characteristic points of a carapace of Eriocheir sinensis. BACKGROUND

[0002] Eriocheir sinensis, a kind of arthropod, crustacean, decapod, crawling suborder, Eriocheir, is continuously favored by the market because of its unique and wonderful taste. As early as the early 20th century, Eriocheir sinensis began to be introduced from China to Germany. Subsequently, in the 1920s-1930s, the number of Eriocheir sinensis increased rapidly, and its distribution range rapidly expanded to many northern European rivers and estuaries. In 1992, commercial shrimp trawlers in the southern San Francisco Bay collected the first batch of Eriocheir sinensis on the west coast. Eriocheir sinensis has become a dish all over the world, with great economic value and potential. In China, according to the growth environment, Eriocheir sinensis is divided into Yangtze River population, Liaohe River population, Oujiang River population, Yellow River population and other populations. In order to study the manifestation of various properties of Eriocheir sinensis in morphology, scholars in various fields have carried out relatively sufficient research on the morphology of crabs. However, there is no convenient and fast classification method that is easy to operate in batches. The computer image recognition technology based on feature point detection can provide a new technical path for the solution of this problem.

[0003] Feature point detection is an important branch of computer vision. Feature point detection has been widely applied to target matching, target tracking, three-dimensional reconstruction and other fields. Traditional feature point detection often uses corner points as important point features, and widely used feature point detection algorithms include Harris corner detection and SIFT feature detection. With the development of deep learning, neural network-based feature point detection methods have gradually become one of the mainstream methods of feature point detection. At present, neural network-based feature point detection methods are mainly divided into full connection regression method and Gaussian Heatmap regression method. The full connection regression method usually adds a full connection layer at the end of the convolutional neural network to map the feature map to the coordinate point. Intuitively, the full connection regression method has a significant advantage that the full connection layer can directly connect the global features to the coordinates of the feature points. However, the full connection regression method greatly damages the spatial generalization ability of the network. In an extreme case, when all the crabs in the training data are in the upper left corner of the image, the activation weights of the full connection layer are mainly concentrated in the first half of the one-dimensional vector after reshaping into a one-dimensional feature vector. It is difficult to achieve good results when testing a picture with a crab located in the lower right corner using this network. Therefore, the full connection regression method relies too much on the distribution of the training data, and is more likely to cause overfitting. The Gaussian Heatmap regression method is the main method in the field of human pose estimation. The feature map of the Gaussian Heatmap is large and has strong spatial generalization ability. At the same time, the Gaussian Heatmap also has problems such as high memory occupation, slow training and inference speed, and the gradient flow of this method is not end-to-end. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a method for recognizing characteristic points of the carapace of Eriocheir sinensis, which can quickly calculate and label the coordinates of 37 characteristic points of the carapace of Eriocheir sinensis through a carapace photo of Eriocheir sinensis taken by a mobile phone.

[0005] The present application is realized by the following technical solutions:

[0006] A method for recognizing characteristic points of the carapace of Eriocheir sinensis, specifically comprising the following steps:

[0007] (1) Taking a photo of a collected Eriocheir sinensis sample by using a mobile phone, and using a general target detection model through transfer learning to crop the carapace part of the Eriocheir sinensis, and enhancing the cropped data;

[0008] (2) Designing a 37-point Eriocheir sinensis, and using 37 characteristic points to locate the carapace contour of the Eriocheir sinensis;

[0009] (3) Labeling the picture based on the 37-point characteristic point positioning method, and storing the labeling file in xml format;

[0010] (4) Designing an end-to-end differentiable convolutional neural network, inputting the enhanced image into the network for training until the training parameters converge;

[0011] (5) Saving the trained model, inputting a new carapace picture, and automatically generating a characteristic point heat map and labeling the characteristic points by the program.

[0012] As a preferred embodiment, the mobile phone in the step (1) is above the crab, and the left-right or front-back inclination angle is controlled within 10° as much as possible to ensure that all the characteristic points are photographed, and the natural light is experienced from 9 am to 6 pm in the evening during the photographing process.

[0013] As a preferred embodiment, the data enhancement in the step (1) adopts a data enhancement mode of random combination of multiple enhancement schemes such as random occlusion, random rotation, random brightness contrast, etc.

[0014] As a preferred embodiment, the data enhancement in the step (1) calculates the corresponding transformed label file at the same time, and manual labeling is not needed again.

[0015] As a preferred embodiment, the 37-point positioning method in step (2) complies with the following provisions: for the 12 teeth of the head and chest armor (4 frontal teeth, 4 left front teeth, and 4 right front teeth), the 37-point positioning method uses three feature points of starting point, tooth peak, and ending point for positioning each tooth, shares one feature point between two adjacent teeth, uses three feature points for positioning the back edge, and positions the M-shaped neck groove by seven feature points.

[0016] As a preferred embodiment, the file labeled in step (3) is stored in an xml file in a tree structure, and each picture uses a label to determine, a file attribute of the label is a file name, a <label>The tags store the number of the Chinese mitten crab, each <part>The label stores a feature point, and in the part label, the name, x and y attributes represent the number of feature points, horizontal offset and vertical offset respectively.

[0017] As a preferred embodiment, the step (4) of designing the end-to-end differentiable convolutional neural network comprises the following sub-steps:

[0018] (41) design a full convolutional neural network containing 7 convolutional modules, in which a global deep convolution (GDConv) is used to replace the traditional global average pooling operation, and the calculation process of the GDConv is G m =∑K i，j，m ·F i，j，m , F is the input of the GDConv layer, K is the convolution kernel, and G is the output of the GDConv layer, if the size of F is w x h x m, the size of K is also w x h x m, and the size of G is 1 x 1 x m;

[0019] (42) after the full convolutional neural network, the Softmax normalization is performed on each channel Z′ i，j =exp(Z i，j ), define X and T matrices, wherein The coordinates (x, y) of the calculation inference are x = <Z', X> F , y = <Z', Y> F , wherein m and n represent the width and height of the feature matrix respectively, <A, B> F The corresponding elements in the matrices A and B are multiplied and then added, finally, the Loss value of the predicted feature point position (x, y) and the coordinates in the label file is calculated, and the parameters are updated.

[0020] The design principle of the application is: first, a 37-point Chinese Eriocheir sinensis carapace positioning method is proposed, that is, 37 feature points are used to express a carapace. Secondly, the data shot is randomly enhanced such as distortion, rotation and occlusion, which increases the data set and also improves the generalization ability of the model from the data. The data is divided into a training set, a validation set and a test set, and the data in the training set is all input into the network for training. After the training parameters converge, the model is saved. Finally, the test set data is used to test the effect of the saved model, and a feature point heat map and a feature point annotation map are generated.

[0021] Beneficial effects: compared with the prior art, the application has lower requirements on input data, and the carapace pictures taken in each link of the eriocheir sinensis production line can be directly input into the trained model. Compared with the existing feature point positioning method, the application greatly improves the positioning accuracy and positioning speed, can save a large amount of labor cost for the eriocheir sinensis industry, and significantly reduces the workload of artificial observation, quality detection and other links of the eriocheir sinensis culture, and greatly improves the work efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a flowchart of the application;

[0023] Figure 2 is a flowchart of data enhancement of the application;

[0024] Figure 3 is a schematic diagram of the 37-point positioning method of the carapace of eriocheir sinensis proposed by the application;

[0025] Figure 4 is an effect picture of data enhancement of the application;

[0026] Figure 5 is a full convolutional neural network structure diagram of the application;

[0027] Figure 6 is a test effect picture of the application. DETAILED DESCRIPTION

[0028] The embodiments of the application will be described in detail below with reference to the accompanying drawings: the embodiments are implemented on the premise of the technical scheme of the application, and give detailed implementation modes and specific operation processes, but the protection scope of the application is not limited to the following embodiments.

[0029] As shown in Figure 1 , 2 A method for identifying characteristic points of the carapace of eriocheir sinensis, specifically comprising the following steps: using a mobile phone to take pictures of collected eriocheir sinensis samples, using a general object detection model through transfer learning to crop part of the carapace of eriocheir sinensis, and enhancing the cropped data. A 37-point eriocheir sinensis is designed, and 37 feature points are used to locate the contour of the carapace of eriocheir sinensis. Based on the 37-point feature point positioning method, the pictures are labeled, and the labeling file is stored in xml format. An end-to-end differentiable convolutional neural network is designed, the enhanced images are input into the network for training until the training parameters converge. The trained model is saved, and a new carapace picture is input, and the program automatically generates a feature point heat map and labels the feature points.

[0030] The application will be further described below through a specific embodiment:

[0031] 1. Data collection:

[0032] Hongze Lake is the fourth largest freshwater lake in China, located in the lower reaches of the Huaihe River in Jiangsu Province (between 33°06' and 33°40' north latitude, 118°10' and 118°52' east longitude), and is also one of the main production areas of Eriocheir sinensis. In the present invention, 50 crabs were collected from Hongze Lake, and 40-50 images were taken for each crab. The shooting device is a plus 6T smart phone, including a main camera and a secondary camera. The main camera has 16 million pixels, and the sensor is Sony IMX519 with a pixel size of 1.22m and an f / 1.7 large aperture. The secondary camera has 20 million pixels, and the sensor is Sony IMX376K with a pixel size of 1.0m and an f / 1.7 large aperture. During shooting, the phone is directly above the crab, and the left-right or forward-backward tilt angle is controlled within 10° to ensure that all feature points are captured. The present invention finally uses 2300 images of 50 crabs. All images were taken on the same day without additional lighting, but the shooting process experienced natural light from 9am to 6pm.

[0033] 2. Determining the feature point positioning method:

[0034] Eriocheir sinensis has 4 frontal teeth on the front side of the carapace, 4 lateral teeth on each side, and a relatively flat rear edge. There is a relatively obvious M-shaped neck groove in the middle. Based on the above characteristics of Eriocheir sinensis, a 37-point feature point positioning method is designed as shown in Figure 3

[0035] 3. Data augmentation:

[0036] The above image shooting angles are relatively uniform, and most obstacle blocking problems are deliberately avoided during shooting. However, in actual application, there may be many influencing factors such as shooting equipment, shooting environment, shooting angle, and obstacles. Therefore, data augmentation is necessary for this study. As shown in Figure 4 , A, B, and C represent normal images, relatively blurred images, and images with large angle deviations, respectively. Column 1 represents the original image, Column 2 represents the image after random Gaussian blur processing, Column 3 represents the image after random brightness and contrast processing, Column 4 represents the image after random rotation, Column 5 represents the image after random occlusion, and Column 6 represents the image after random combination of various data augmentation methods. After data augmentation, the picture data amount is 4600.

[0037] 4. Design of neural network model:

[0038] ​First, a fully convolutional neural network is designed. Structurally, the main difference between this fully convolutional neural network and MobileNetV2 is that global average pooling is optimized into global depthwise convolution (GDConv). Learnable weights are added to each location. The calculation method of GDConv is shown in Equation (1). The kernel size of the GDConv layer is the same as the input dimension.

[0039] G m =∑K i，j，m ·F i，j，m (1)

[0041] In formula (1), F is the input of the GDConv layer, K is the convolution kernel, and G is the output of the GDConv layer. If the size of F is w×h×m, then the size of K is also w×h×m, and the size of G is 1×1×m. In designing the network, this invention also uses grouped convolution and inverse residual modules to reduce computational load and achieve fast downsampling.

[0042] Overall network design such as Figure 5 As shown, Figure 5 The arrows indicate the size of the feature vector after passing through the preceding modules. Each block is labeled with some important parameters and its sub-blocks. In the diagram, DW Conv represents the Depthwise Convolution operation. The Inverted Res module represents the inverse residual module; the dw_num parameter in the Inverted Res module indicates the number of depthwise separable convolution operations within that module. The groups in the Conv module indicate the number of groups in the grouped convolution. After the fully connected layer, the feature vector is 1×78. This is because, in addition to the 37 feature points, the top-left and bottom-right corners of the rectangle containing the Chinese mitten crab are also considered as two feature points, resulting in 39 two-dimensional coordinates, totaling 78 numbers.

[0043] After completing the design of the fully convolutional neural network, this invention adds a depthwise separable convolutional module after the fully convolutional neural network to achieve differentiability. Image samples are converted into matrix variables of size 3×512×512, and then... Figure 5 The full convolutional neural network (but not through the full connection layer) obtains a feature matrix with a size of 128x32x32. On the basis of the feature matrix, a convolution operation is performed again, so that the size of the matrix becomes 39x32x32, and 39 represents 39 feature points that need to be regressed. The feature matrix (39x32x32) is input into the DSNT module to obtain an output Z matrix. The Z matrix is globally normalized by using the Softmax method to obtain Z'. Two matrices X and Y with the same dimension as Z are defined, and their values are standardized to-1 to 1. That is, the coordinate points in the X and Y dimensions are converted to-1 to 1. Z' is subjected to F norm (point-by-point multiplication and addition) on the X and Y matrices to obtain the x and y coordinate prediction values under each channel. Finally, the loss between the prediction value and the true value is calculated, and the parameters of the network are updated. The inference process in this way is as follows:

[0044]

[0045] 5. Training model:

[0046] The enhanced data is input into the neural network model for training. The training environment is a supercomputer device equipped with an NVIDIA TeslaV100 32GB high-performance GPU and Ubuntu 18.04. After 8.33 hours of training for 300 rounds, the parameters converge, and the trained model is saved. The parameter amount of the trained model is 0.84M, the calculation amount is 8.34G, and the model size is 3.67MB.

[0047] 6. Result visualization:

[0048] The test data is input into the saved model to obtain the heat distribution diagram of the Chinese mitten crab carapace feature points and the feature point annotation diagram. The heat map reflects the heat distribution of the carapace feature points, and the feature point annotation diagram is the accurate coordinate value calculated and superimposed on the original image. The visualization results of part of the test set are as follows Figure 6 .

[0049] The above shows and describes the basic principles and main features of the present application and the advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above examples, and the above examples and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.< / part> < / label>

Claims

1. A method for recognizing feature points of a Chinese mitten crab carapace, characterized in that, Specifically comprising the following steps: (1) using a mobile phone to take pictures of the collected Eriocheir sinensis samples, using a general object detection model to crop the carapace part of Eriocheir sinensis through transfer learning, and enhancing the cropped data; (2) designing a 37-point Eriocheir sinensis, and using 37 feature points to locate the carapace contour of Eriocheir sinensis; (3) labeling the pictures based on the 37-point feature point positioning method, and storing the label file in xml format; (4) designing an end-to-end differentiable convolutional neural network, inputting the enhanced image into the network for training until the training parameters converge; The step (4) of designing an end-to-end differentiable convolutional neural network comprises the following sub-steps: (41) designing a full convolutional neural network comprising 7 convolutional modules, wherein a global depth convolution GDConv is used to replace the traditional global average pooling operation in the full convolutional neural network, and the calculation process of the GDConv is G m =∑K i，j，m ·F i，j，m , F is the input of the GDConv layer, K is the convolution kernel, and G is the output of the GDConv layer, if the size of F is w x h x m, the size of K is also w x h x m, and the size of G is 1 x 1 x m; (42) After the full convolutional neural network, Softmax normalization is performed on each channel Z′ i，j = exp(Z i，j ), define X, Y matrix, where Calculate the coordinates (x, y) of the inference, x = <Z′, X> F , y = <Z′, Y> F , where m and n represent the width and height of the feature matrix respectively, <A, B> F The result of multiplying the corresponding elements in matrix A and B and then adding them together, finally, calculate the Loss value of the predicted feature point position (x, y) and the coordinates in the label file, and update the parameters; (5) saving the trained model, inputting a new carapace picture, and automatically generating a feature point heat map and labeling feature points by the program.

2. The method for recognizing characteristic points of the carapace of Eriocheir sinensis according to claim 1, characterized in that, In the step (1), the shooting mobile phone is directly above the crab, and the left-right or front-back inclination angle is controlled within 10° to ensure that all feature points are captured, and the shooting process experiences natural light from 9 am to 6 pm.

3. The method for recognizing feature points of a Chinese mitten crab carapace according to claim 1, characterized in that, In the step (1), the data enhancement adopts a data enhancement method of randomly combining multiple enhancement schemes such as random occlusion, random rotation, and random brightness contrast.

4. The method for recognizing feature points of a Chinese mitten crab carapace according to claim 1, characterized in that, In the step (1), the data enhancement calculates the corresponding label file after transformation at the same time, without the need for manual re-labeling.

5. The method for recognizing feature points of a Chinese mitten crab carapace according to claim 1, characterized in that, In the step (2), the 37-point positioning method complies with the following provisions: the 37-point positioning method uses three feature points of starting point, tooth peak and end point to locate each tooth of the 12 teeth of the carapace, 4 frontal teeth, 4 left front teeth and 4 right front teeth, shares one feature point between two adjacent teeth, and uses three feature points for positioning of the rear edge, and the M-shaped neck groove is positioned by seven feature points.

6. The method for recognizing feature points of a Chinese mitten crab carapace according to claim 1, characterized in that, The file marked by the step (3) is stored in an xml file in a tree structure, and each picture uses tag determination, The file attribute of the tag is a file name, The <label>The tags store the number of the Chinese mitten crab, each <part>The label stores a feature point, and in the part label, the name, x and y attributes represent the number of feature points, horizontal offset and vertical offset, respectively.< / part> < / label>

Images