A Pig Counting Method Based on Instance Segmentation

Abstract

The present invention belongs to the field of computer vision and machine learning, and discloses a method for counting live pigs based on instance segmentation. It makes up for the shortcomings of the existing algorithms, such as high labeling cost, low precision and poor robustness. The technology first uses an image segmentation network to obtain the category prediction result of each pixel in the image, and then calculates the image segmentation loss, instance splitting loss, positioning loss and regression loss respectively, and finally weightedly fuses each loss and trains the network as supervision information. The technology of the present invention only needs to mark the center point of the pig, and can output a roughly segmented area for each pig instance, and then obtain the accurate location and quantity information of the pig instance, solving the problem of counting and positioning live pigs in complex environments.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and machine learning, and in particular relates to a pig counting method based on instance segmentation. Background Art

[0002] Pig counting is a key task in large-scale farming management. Accurate pig counting can not only improve the management level of pig breeding and pig house construction, but also help reduce breeding costs, detect abnormal situations such as missing pigs in time, and improve the competitiveness of farms. However, the current pig population statistics still rely on manual counting, which not only consumes a lot of labor costs but also has low efficiency. For example, it usually takes about three days to count the number of pigs in a medium-sized farm. In addition, due to factors such as frequent transactions and sow production, the number of pigs often changes, making the pig counting task more difficult.

[0003] In recent years, various computer vision algorithms have been widely used in agriculture and agricultural automation, but they still face two problems in practical applications: 1. Due to problems such as overlapping occlusion of pigs, light fluctuations, similar appearance of pigs, shape deformation and obstacle occlusion, the use of traditional computer vision technology in pig counting tasks is limited. 2. In order to achieve the goal of counting objects, existing deep learning-based methods usually need to use target detection or image segmentation technology, which requires bounding box annotation and pixel-level annotation of objects respectively. Not only is the annotation cost expensive, but it is also unable to cope with scenes with highly dense pigs and small pig targets.

[0004] Pig counting in complex scenes remains a challenging task. Therefore, automatic pig counting and localization using computer vision techniques based on instance segmentation is a low-cost, intensive and promising solution, provided that only the center points of the pigs need to be annotated. Summary of the invention

[0005] The present invention overcomes the deficiencies of the prior art and aims to solve the technical problem of providing a pig counting method based on instance segmentation.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A pig counting method based on instance segmentation comprises the following steps:

[0008] Step 1, obtaining a plurality of sample original images containing a group of pigs, and marking the center point positions of the live pig instances therein, to obtain sample annotated images corresponding to the sample original images;

[0009] Step 2, inputting the sample original image into the encoder for feature extraction, and upsampling the image features through a decoder module composed of a deconvolution layer through the feature encoding extracted by the encoder to obtain a segmentation result of the sample original image;

[0010] Step 3, input the segmentation result into the classifier, determine the category of each pixel, and obtain the pixel category prediction matrix S, which represents the probability that pixel i belongs to category c (c represents the category of the object instance. For the pig counting task, there are only two categories: pigs and background, c = {0, 1}).

[0011] Step 4, based on the pixel category prediction matrix and the sample annotated image, the image segmentation loss is calculated; the sample annotated image Y has a label c at each pixel position, c=1 indicates that the pixel is marked as a pig object, and c=0 indicates that the pixel is not marked as a pig object; according to the pixel category prediction result S obtained in step 3 and the sample annotated image Y, the image segmentation loss is calculated according to formula (1);

[0012]

[0013] Step 5, based on the pixel category prediction matrix and the sample annotated image, calculate the instance splitting loss; the specific steps are:

[0014] 5-1) Marking connected domains; generating a binary mask matrix F according to the pixel category prediction matrix S, if argmax k S ik > 0, then mark pixel i as a foreground pixel, otherwise mark it as a background pixel; find the connected region composed of foreground pixels with the same pixel value and adjacent positions in F and assign it a unique identifier;

[0015] 5-2) Calculate the watershed segmentation loss; Set the real annotation point as the seed point, and then apply the watershed image segmentation algorithm to the connected area obtained in step 5-1), find the boundary between instances, and then predict the boundary as the background, so as to split the connected area containing n annotation points and obtain n segmented areas. Calculate the watershed segmentation loss function according to formula (2):

[0016]

[0017] Where S i0 represents the probability that pixel i belongs to the background, U represents the pixel set composed of boundary points, α i Indicates the number of marked points in the connected area where i is located;

[0018] 5-3) Calculate the false positive loss; When executing the watershed segmentation algorithm, it is inevitable that there will be segmented areas that do not contain labeled points. Such a result is called a false positive, FP. In order to eliminate the influence of false positive segmentation results, find the pixel set that does not contain labeled points in the segmented area in step 5-2), and calculate the false positive loss function according to formula (3), guiding the model to predict the false positive area without labeled points as background:

[0019]

[0020] Among them B fp represents the set of pixels in the segmented area that does not contain the annotation point, S i0 represents the probability that pixel i belongs to the background;

[0021] 5-4) Calculate the instance splitting loss; Based on the watershed segmentation loss and the false positive loss function, the instance splitting loss is calculated according to formula (4):

[0022] L split (S,Y)=L watershed (S,Y)+L FP (S,Y) (4)

[0023] Step 6: Take the center point coordinates of the segmented area in step 5-2) as the predicted position of the pig instance, and minimize the weighted Hausdorff distance between the real annotation point set and the prediction point set according to formula (5) as the positioning loss, so that the predicted position is constantly close to the real annotation position:

[0024]

[0025] Where Ω represents the set of position coordinates predicted by the model, Y represents the set of real annotation points, d(g) can be any distance function, and Euclidean distance is used; represents the generalized mean. When α→-∞, M α [g] is equivalent to minimizing the function;

[0026] Step 7: Take the number of segmented regions in step 5-2) as the number of pigs predicted by the model, and calculate the Huber loss between the predicted number and the actual labeled number according to formula (6) as the regression loss to guide the model to output the correct number of instances:

[0027]

[0028] in represents the number of segmented regions predicted in step 5-2), that is, the predicted number of pig groups, and g represents the number of real annotation points obtained from the sample annotated image Y;

[0029] Step 8: Weighted fusion of the loss functions calculated in steps 4, 5, 6, and 7 to obtain the Segmentation-Split-Location-Regress (SSLR) composite counting loss function:

[0030] L SSR (S,Y)=λ1L seg (S,Y)+λ2L split (S,Y)+λ3L loc (S,Y)+λ4L reg (S,Y) (7)

[0031] Among them, λ1, λ2, λ3, λ4, respectively represent the weight coefficients of each loss function;

[0032] Step 9, based on the sample original image, sample annotated image, and SSLR composite counting loss function, the encoder and decoder are trained for multiple rounds to obtain the trained encoder and decoder as the pig counting model based on instance segmentation.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The present invention provides a pig counting method based on instance segmentation, which only needs to mark the center point position of each pig, and uses the center point position as supervision information to train the image segmentation model under the weak supervision paradigm. The backbone network of the method can adopt various image segmentation network structures, such as FCN8, Unet and HRNet, etc., to predict the classification label of each pixel in the image, and on this basis, predict segmentation areas of different sizes for different pig instances, so as to obtain accurate quantitative information of the pig herd.

[0035] 2. The present invention provides a new Segmentation-Split-Location-Regress (SSLR) composite counting loss, including segmentation loss, splitting loss, positioning loss and regression loss, which are respectively used to encourage the model to predict the semantic label of each pixel in the image, constrain the model to separate the predicted segmented regions between object instances, reduce the deviation between the predicted position coordinates and the true labeled position, and guide the model to directly predict the number of pigs in the image. Although this loss cannot accurately predict the size and shape information of the pig instance object, it can flexibly predict the approximate segmentation region for each pig instance, so that there is only one pig instance in each region, so that the number of segmented regions is the number of pigs.

[0036] The method of the present invention is easy to implement, and its application value is mainly reflected in the following aspects:

[0037] 1. Accurate segmentation, counting and positioning of different pig instances in the image;

[0038] 2. Relatively low annotation cost, easy to implement and deploy. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic flow chart of a pig counting method based on instance segmentation provided by an embodiment of the present invention;

[0040] Figure 2 Schematic diagram of the structure of the counting network used in the embodiment of the present invention;

[0041] Figure 3 The figures are the pig counting and positioning effect diagrams of the present invention, wherein (a) is the farm monitoring image, and (b) is the pig counting and positioning result diagram. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0043] See also Figure 1 The present invention is a pig counting method based on instance segmentation. The model is based on a fully convolutional image segmentation network structure, and its original semantic segmentation loss is extended to a counting loss function supervised by a center point for pig counting. The loss function of the present invention is expressed as a Segmentation-Split-Location-Regress (SSLR) composite counting loss. Different from other classic semantic segmentation models, the model of the present invention only requires center point supervision information during training, but does not require bounding box information and information of all pixels.

[0044] The present invention uses real-life pig pictures as example samples, with an image size of 384×512 and a total of 2000 images, to establish a pig counting model based on instance segmentation. The key steps are as follows:

[0045] A pig counting method based on instance segmentation comprises the following steps:

[0046] Step 1, obtaining a plurality of sample original images containing a group of pigs, and marking the center point positions of the live pig instances therein, to obtain sample annotated images corresponding to the sample original images;

[0047] Step 2: Input the sample original image into the encoder for feature extraction, and upsample the image features by the decoder module composed of deconvolution layers to obtain the segmentation result of the sample original image. The network structure is as follows: Figure 2 as stated;

[0048] Step 3, input the segmentation result into the classifier, determine the category of each pixel, and obtain the pixel category prediction matrix S, which represents the probability that the pixel i belongs to category c (c represents the category of the object instance. For the pig counting task, there are only two categories: pigs and background, c = {0, 1});

[0049] Step 4: Calculate the image segmentation loss based on the pixel category prediction matrix and the sample annotated image; the sample annotated image Y has a label c at each pixel position, c=1 indicates that the pixel is marked as a pig object, and c=0 indicates that the pixel is not marked as a pig object; according to the pixel category prediction result S obtained in step 3 and the sample annotated image, the image segmentation loss is calculated.

[0050] Image Y, calculate the image segmentation loss according to formula (1);

[0051]

[0052] Step 5, based on the pixel category prediction matrix and the sample annotated image, calculate the instance splitting loss; the specific steps are:

[0053] 5-1) Marking connected domains; generating a binary mask matrix F according to the pixel category prediction matrix S, if argmax k S ik > 0, then mark pixel i as a foreground pixel, otherwise mark it as a background pixel; find the connected region composed of foreground pixels with the same pixel value and adjacent positions in F and assign it a unique identifier;

[0054] 5-2) Calculate the watershed segmentation loss; Set the real annotation point as the seed point, and then apply the watershed image segmentation algorithm to the connected area obtained in step 5-1), find the boundary between instances, and then predict the boundary as the background, so as to split the connected area containing n annotation points and obtain n segmented areas. Calculate the watershed segmentation loss function according to formula (2):

[0055]

[0056] Where S i0represents the probability that pixel i belongs to the background, U represents the pixel set composed of boundary points, α i Indicates the number of marked points in the connected area where i is located;

[0057] 5-3) Calculate the false positive loss; When executing the watershed segmentation algorithm, it is inevitable that there will be segmented areas that do not contain labeled points. Such a result is called a false positive, FP. In order to eliminate the influence of false positive segmentation results, find the pixel set that does not contain labeled points in the segmented area in step 5-2), and calculate the false positive loss function according to formula (3), guiding the model to predict the false positive area without labeled points as background:

[0058]

[0059] Among them B fp represents the set of pixels in the segmented area that does not contain the annotation point, S i0 represents the probability that pixel i belongs to the background;

[0060] 5-4) Calculate the instance splitting loss; Based on the watershed segmentation loss and the false positive loss function, the instance splitting loss is calculated according to formula (4):

[0061] L split (S,Y)=L watershed (S,Y)+L FP (S,Y) (4)

[0062] Step 6: Take the center point coordinates of the segmented area in step 5-2) as the predicted position of the pig instance, and minimize the weighted Hausdorff distance between the real annotation point set and the prediction point set according to formula (5) as the positioning loss, so that the predicted position is constantly close to the real annotation position:

[0063]

[0064] Where Ω represents the set of position coordinates predicted by the model, Y represents the set of real annotation points, d(g) can be any distance function, and Euclidean distance is used; represents the generalized mean. When α→-∞, M α [g] is equivalent to minimizing the function;

[0065] Step 7: Take the number of segmented regions in step 5-2) as the number of pigs predicted by the model, and calculate the Huber loss between the predicted number and the actual labeled number according to formula (6) as the regression loss to guide the model to output the correct number of instances:

[0066]

[0067] in represents the number of segmented regions predicted in step 5-2), that is, the predicted number of pig groups, and g represents the number of real annotation points obtained from the sample annotated image Y;

[0068] Step 8: Weighted fusion of the loss functions calculated in steps 4, 5, 6, and 7 to obtain the Segmentation-Split-Location-Regress (SSLR) composite counting loss function:

[0069] L SSR (S,Y)=λ1L seg (S,Y)+λ2L split (S,Y)+λ3L loc (S,Y)+λ4L reg (S,Y) (7)

[0070] Among them, λ1, λ2, λ3, λ4, respectively represent the weight coefficients of each loss function;

[0071] Step 9, based on the sample original image, sample annotated image, and SSLR composite counting loss function, the encoder and decoder are trained for multiple rounds to obtain the trained encoder and decoder as the pig counting model based on instance segmentation.

[0072] In the example of the present invention, the Adam optimizer with an initial learning rate of 0.0001 and a momentum decay of 0.000 5 is used. During training, since the watershed segmentation algorithm does not support processing multiple images at the same time, the batch size is set to 1. At the same time, the data is effectively amplified by applying enhancement methods such as horizontal flipping, color jittering, and random erasing on each image.

[0073] The present invention is a pig counting method based on instance segmentation. It only needs to mark the center points of the pigs to predict segmentation areas of different sizes for each pig instance and obtain the accurate number of pigs. At the same time, the present invention designs a composite counting loss based on center point supervision. Under the constraint of this loss function, the pig counting task in a complex environment is realized. To verify the effectiveness of the present invention, we conducted experiments on real-world data sets collected from actual pig farms. The mean absolute error (MAE) is as low as 0.51, which is better than the competing algorithm, proving the robustness of the method of the present invention. Figure 3 This is the experimental result diagram. Figure 3 (a) is a real-time monitoring image of the farm. Figure 3 (b) is the counting and positioning result of the present invention, and it can be clearly seen that the counting and positioning result of the present invention is accurate.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A pig counting method based on instance segmentation, characterized in that: The following steps are involved: Step 1, obtaining a plurality of sample original images containing a group of pigs, and marking the center point positions of the live pig instances therein, to obtain sample annotated images corresponding to the sample original images; Step 2, inputting the sample original image into the encoder for feature extraction, and upsampling the image features through a decoder module composed of a deconvolution layer through the feature encoding extracted by the encoder to obtain a segmentation result of the sample original image; Step 3, input the segmentation result into the classifier, determine the category of each pixel, and obtain the pixel category prediction matrix S, which represents the probability that pixel i belongs to category c, where c represents the category of the object instance. For the pig counting task, there are only two categories: pigs and background, and c = {0, 1}; Step 4, based on the pixel category prediction matrix and the sample annotated image, the image segmentation loss is calculated; the sample annotated image Y has a label c at each pixel position, c=1 indicates that the pixel is marked as a pig object, and c=0 indicates that the pixel is not marked as a pig object; according to the pixel category prediction result S obtained in step 3 and the sample annotated image Y, the image segmentation loss is calculated according to formula (1); Step 5, based on the pixel category prediction matrix and the sample annotated image, calculate the instance splitting loss; the specific steps are: 5-1) Marking connected domains; generating a binary mask matrix F according to the pixel category prediction matrix S, if argmax k S ik > 0, then mark pixel i as a foreground pixel, otherwise mark it as a background pixel; find the connected region composed of foreground pixels with the same pixel value and adjacent positions in F and assign it a unique identifier; 5-2) Calculate the watershed segmentation loss; Set the real annotation point as the seed point, and then apply the watershed image segmentation algorithm to the connected area obtained in step 5-1) to find the boundary between instances, and then predict the boundary as the background, so as to split the connected area containing n annotation points, and obtain n segmented areas. Calculate the watershed segmentation loss function according to formula (2): Where S i0 represents the probability that pixel i belongs to the background, U represents the pixel set composed of boundary points, α i Indicates the number of marked points in the connected area where i is located; 5-3) Calculate the false positive loss; When executing the watershed segmentation algorithm, it is inevitable that there will be segmented areas that do not contain labeled points. Such a result is called a false positive, FP. In order to eliminate the influence of false positive segmentation results, find the pixel set that does not contain labeled points in the segmented area in step 5-2), and calculate the false positive loss function according to formula (3), guiding the model to predict the false positive area without labeled points as background: Among them B fp represents the set of pixels in the segmented area that does not contain the annotation point, S i0 represents the probability that pixel i belongs to the background; 5-4) Calculate the instance splitting loss; Based on the watershed segmentation loss and the false positive loss function, the instance splitting loss is calculated according to formula (4): L split (S,Y)=L watershed (S,Y)+L FP (S,Y) (4) Step 6: Take the center point coordinates of the segmented area in step 5-2) as the predicted position of the pig instance, and minimize the weighted Hausdorff distance between the real annotation point set and the prediction point set according to formula (5) as the positioning loss, so that the predicted position is constantly close to the real annotation position: Where Ω represents the set of position coordinates predicted by the model, Y represents the set of real annotation points, d(g) can be any distance function, and Euclidean distance is used; represents the generalized mean. When α→-∞, M α [g] is equivalent to minimizing the function; Step 7: Take the number of segmented regions in step 5-2) as the number of pigs predicted by the model, and calculate the Huber loss between the predicted number and the actual labeled number according to formula (6) as the regression loss to guide the model to output the correct number of instances: in represents the number of segmented regions predicted in step 5-2), that is, the predicted number of pig groups, and g represents the number of real annotation points obtained from the sample annotated image Y; Step 8: Weighted fusion of the loss functions calculated in steps 4, 5, 6, and 7 to obtain the Segmentation-Split-Location-Regress (SSLR) composite counting loss function: L SSR (S,Y)=λ1L seg (S,Y)+λ2L split (S,Y)+λ3L loc (S,Y)+λ4L reg (S,Y) (7) Among them, λ1, λ2, λ3, λ4, respectively represent the weight coefficients of each loss function; Step 9, based on the sample original image, sample annotated image, and SSLR composite counting loss function, the encoder and decoder are trained for multiple rounds to obtain the trained encoder and decoder as the pig counting model based on instance segmentation.

Images