An infrared pig instance segmentation method and system based on deep learning

Abstract

The application discloses a pig instance segmentation method and system based on an infrared scene. Firstly, the method can realize full-period acquisition, has real-time advantages compared with visible light image monitoring pigs; meanwhile, the infrared image can filter out some complex background interference, and is easier to segment than the visible light image; in addition, the instance segmentation model based on the infrared image can segment any pig breed with any skin color and size, and has higher generalization. Secondly, in order to solve the dense segmentation problem of the pig group lying and adhering, the DAG module is designed, the channel and spatial attention blocks are introduced in the feature pyramid respectively, the feature dependency in the channel and spatial dimensions is captured, and the outputs of the channel and spatial attention blocks are summed, so that the feature representation is further improved. Finally, in order to improve the accuracy of the segmentation candidate frame, the application method realizes all-weather pig instance segmentation with stronger robustness and better segmentation effect through cascading three detection heads.

Description

Technical Field

[0001] This invention belongs to the field of image segmentation technology, and in particular relates to a deep learning-based method for segmenting pig instances in infrared scenes. Background Technology

[0002] In intensive farming and breeding environments, increased pig density significantly increases the risk of infection and the difficulty of controlling swine fever. The health status of a herd of pigs is closely related to the economic benefits available to the farm. Machine vision-based image analysis technology provides a low-cost, non-contact, and non-destructive monitoring method for studying herds of breeding pigs. Accurate and rapid segmentation of individual pigs is beneficial for identifying abnormal behaviors and taking timely countermeasures to reduce disease incidence. However, complex light variations, pigs lying together in groups and their adherence to each other, and the rigid confinement of the pigpen make individual pig studies very difficult. Therefore, it is necessary to achieve rapid and accurate segmentation of pigs under various all-weather and multi-interference scenarios.

[0003] As early as 2008, Wathes et al. developed a computer vision-based early warning system to detect behavioral changes in livestock to monitor their health. In the individual pig domain, researchers explored multiple directions, including Nasilahmadi et al.'s research on automatic pig posture recognition, Kim et al.'s research on detecting standing and lying behaviors, and Dominiak et al.'s research on estimating water consumption in growing pigs. In recent years, deep learning-based technologies have been gradually adopted in agriculture and animal husbandry. Kamilaris et al. used deep learning for plant disease classification, Kamal et al. for fruit detection, and Marsota et al. for recognizing pig faces. However, the performance of these methods largely depends on the distinction between the pig and the background; therefore, the results are not as satisfactory as expected for complex environmental images such as those in low-light conditions and with low resolution. Furthermore, these methods primarily rely on morphological features such as color, shape, and texture, which only work within a limited range and have insufficient generalization ability. For example, when pigs are lying together in a group, it is difficult to distinguish them morphologically.

[0004] Instance segmentation can distinguish different individuals belonging to the same category. As one of the earliest network models in the field of instance segmentation, Mask R-CNN has been applied to cattle subdivision and cattle counting. However, the aforementioned Mask R-CNN-based methods only apply the original Mask R-CNN model to individual pigs without substantial model structure adjustments, thus limiting their effectiveness. Furthermore, in breeding environments, the results vary significantly due to susceptibility to lighting and complex environmental factors. In summary, current pig instance segmentation suffers from several problems: insufficient generalization of the segmentation model: pig color and appearance can change due to age, breed, and breeding environment, making it difficult for existing algorithms to achieve high robustness in pig instance segmentation.

[0005] Because infrared images are unaffected by lighting conditions and can remove most of the redundant background, the image quality remains consistent regardless of the breed or color of the pigs, resulting in a well-generalized model. Furthermore, inspired by attention mechanisms, channel and spatial attention blocks are introduced into the Feature Pyramid Network (FPN) framework to capture feature dependencies in both channel and spatial dimensions. The outputs of these attention blocks are summed to further improve feature representation, enhancing the texture detail in segmentation of pigs lying in close proximity. Integrating this dual-attention guided method into the more advanced Cascade_Mask_R-CNN enables more robust pig instance segmentation. In summary, deep learning methods hold great potential for pig instance segmentation in infrared image scenarios, but current research on pig instance segmentation in infrared scenes is limited. Summary of the Invention

[0006] This invention addresses the shortcomings of existing technologies, including low segmentation accuracy due to pigs lying together and interference from complex environments; difficulty in applying real-time monitoring across all time periods due to the influence of lighting changes on the acquired images; and poor generalization of the model to differences in pig breed and color. This invention provides a deep learning-based infrared pig instance segmentation method.

[0007] To achieve the above objectives, the technical solution provided by this invention is: a deep learning-based infrared pig instance segmentation method, comprising the following steps:

[0008] Step 1: Acquire infrared images of the pig samples and perform preprocessing;

[0009] Step 2: Perform polygon annotation on the preprocessed image and use the annotated image as input;

[0010] Step 3: Use the pre-trained convolutional neural network as the backbone network to extract features from the input image and obtain the image's feature map.

[0011] Step 4: The feature map obtained in Step 3 is used by the dual attention guidance module to obtain a more reliable feature map F. DAB The specific processing procedure is as follows:

[0012] First, the high-level feature map and the low-level feature map (F) are combined. h F l ) are fused to obtain F merge , where F h The first feature map is the output feature map of the last layer of the convolutional neural network, and the lower-level feature maps are the feature maps output after the first layer of the convolutional neural network. The fused feature map F merge The data is fed into an asymmetric convolution unit, and after the asymmetric convolution operation, an activation function operation is performed to remove irrelevant features to obtain F. ACU ;F ACU The input is fed into a dual-channel guidance unit, including spatial attention and channel attention. The outputs of the channel and spatial attention blocks are fused to obtain F, which captures feature dependencies in both the channel and spatial dimensions. DAU , will F merge and F DAU By fusing the features again, a more reliable feature map F can be obtained. DAB ;

[0013] Step 5: Use a candidate box generation algorithm to generate a series of candidate boxes from the feature map. Apply a classifier to the generated candidate boxes to determine whether each candidate box contains the target object. Candidate boxes that contain the target object are positive sample candidate boxes.

[0014] Step 6, the feature map F obtained through the dual attention guidance module DAB Bilinear interpolation is performed on the candidate boxes predicted as positive samples obtained in step 5 to more accurately define the location of the target object.

[0015] Step 7: The bilinear interpolation result is passed through multiple detectors in sequence. Except for the last detector, the outputs of the other detectors are fused with the output of the dual attention guidance module and then bilinear interpolation is performed again. The result is then input into the detector of the next layer. The output of the last detector is the final target segmentation result.

[0016] Furthermore, the preprocessing described in step 1 includes selecting the acquired images and performing data augmentation on the selected images. The data augmentation is one of random brightness variation operation and random addition of Gaussian noise operation.

[0017] The specific implementation method of random brightness change operation is as follows:

[0018] Let the new brightness be L n The original brightness is L. oIf the adjustment coefficient is r, then the formula for random brightness variation is as follows:

[0019] L n =L o *r (1)

[0020] The specific implementation of the operation of randomly adding Gaussian noise is as follows:

[0021] Let the input pixel be P. in A normal Gaussian sampling distribution formula G(d) outputs pixels P. out The formula for operating on random Gaussian noise is as follows:

[0022] P out =P in +Xmeans+sigma*G(d) (2)

[0023] Where sigma represents the standard deviation, d is a linear random number, and Xmeans represents the mean.

[0024] Furthermore, in step 2, Labelme annotation software performs polygon annotation on all pigs; in step 3, the convolutional neural network is a ResNet101 model.

[0025] Furthermore, the specific processing steps of the candidate box generation algorithm are as follows: first, anchor boxes are generated; then, binary classification is performed to determine whether each anchor box is the target or the background; a softmax classifier is used to extract possible candidate boxes, and bounding box regression is used to fine-tune the process so that the candidate boxes and the ground truth boxes are closer.

[0026] Furthermore, symmetric convolutional units include 3x3 convolutional units, 1x3 convolutional units, and 3x1 convolutional units.

[0027] Furthermore, the specific steps of the bilinear interpolation method are as follows:

[0028] Let the coordinates of the function f with unknown values ​​be (x, y), and assume that the four known points are Q. 11 = (x1, y1), Q 12 = (x1, y2), Q 21 = (x2, y1), Q 22 = (x2, y2), first perform linear interpolation on the x-axis to obtain the following formula:

[0029]

[0030]

[0031] Where R1 are two temporary points on the x-axis;

[0032] Furthermore, performing linear interpolation on the y-axis yields the following equation:

[0033]

[0034] Where P is a point on the y-axis after linear interpolation;

[0035] Combining the x-axis and y-axis results gives the final result of bilinear interpolation, as shown in the following formula:

[0036]

[0037] Furthermore, the network structures of the multiple detectors are completely identical, and the specific processing procedure is as follows:

[0038] The bilinear interpolation results are passed through the first-level detector to output a bottom-up first-level feature map and a top-down first-level feature map;

[0039] After the first-level feature map from the top down is fused with the feature map output by the dual attention guidance module, a second bilinear interpolation operation is performed.

[0040] The result of the second bilinear interpolation operation is input into the output of the second-stage detector, which is a bottom-up second-stage feature map and a top-down second-stage feature map.

[0041] After the top-down second-level feature map is fused with the feature map output by the dual attention guidance module, a third bilinear interpolation operation is performed.

[0042] The result of the third bilinear interpolation operation is input into the third-level detector to output the bottom-up third-level feature map and the top-down third-level feature map;

[0043] The third-level feature map, viewed from top to bottom, is the result of segmentation.

[0044] Furthermore, the detector includes a forward bottom-up pyramid and a reverse top-down pyramid, which output bottom-up and top-down feature maps, respectively. The two pyramids are laterally connected. Bottom-up is downsampling, and top-down is upsampling. The lateral connection interacts the shallow feature map with high resolution and the deep feature map with rich semantic information. Finally, the top-down feature map is output as the prediction result feature map.

[0045] This invention also provides a deep learning-based infrared pig instance segmentation system, comprising the following modules:

[0046] The preprocessing module is used to acquire infrared images of pig samples and perform preprocessing.

[0047] The annotation module is used to perform polygon annotations on the preprocessed image, taking the annotated image as input.

[0048] The feature extraction module is used to extract features from the input image by using a pre-trained convolutional neural network as the backbone network.

[0049] The high-confidence feature map acquisition module is used to obtain a more reliable feature map F by passing the obtained feature map through a dual-attention guidance module. DAB The specific processing procedure is as follows:

[0050] First, the high-level feature map and the low-level feature map (F) are combined. h F l ) are fused to obtain F merge , where F h The first feature map is the output feature map of the last layer of the convolutional neural network, and the lower-level feature maps are the feature maps output after the first layer of the convolutional neural network. The fused feature map F merge The data is fed into an asymmetric convolution unit, and after the asymmetric convolution operation, an activation function operation is performed to remove irrelevant features to obtain F. ACU ;F ACU The input is fed into a dual-channel guidance unit, including spatial attention and channel attention. The outputs of the channel and spatial attention blocks are fused to obtain F, which captures feature dependencies in both the channel and spatial dimensions. DAU , will F merge and F DAU By fusing the features again, a more reliable feature map F can be obtained. DAB ;

[0051] The candidate box acquisition module is used to generate a series of candidate boxes from the feature map using a candidate box generation algorithm. A classifier is applied to the generated candidate boxes to determine whether each candidate box contains the target object. Candidate boxes that contain the target object are positive sample candidate boxes.

[0052] The bilinear interpolation module is used to interpolate the feature map F obtained by the dual attention guidance module. DAB Bilinear interpolation is performed on candidate boxes predicted as positive samples to more accurately define the location of the target object;

[0053] The target segmentation module is used to pass the bilinear interpolation result through multiple detectors in sequence. Except for the last detector, the outputs of the other detectors are fused with the output of the dual attention guidance module and then bilinear interpolation is performed again before being input into the next layer detector. The output of the last layer detector is the final target segmentation result.

[0054] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows:

[0055] This invention proposes a deep learning-based infrared pig instance segmentation method. It replaces the traditional visible light image with an infrared image for pig segmentation, effectively removing most of the redundant visible light background noise. Furthermore, it is unaffected by lighting conditions, and the trained model maintains consistent performance regardless of pig breed or color, demonstrating good generalization ability. This invention also proposes a dual-channel guided feature pyramid (DAG) module for feature extraction, combining channel and spatial attention. This module improves texture detail segmentation in situations such as pigs lying together and sticking together, achieving more robust pig instance segmentation. It addresses the problems of low segmentation accuracy and difficulty in applying existing pig segmentation methods to real-time monitoring across all time periods. Attached Figure Description

[0056] Figure 1 A flowchart for creating the dataset in an embodiment of the present invention.

[0057] Figure 2 This is a general flowchart of the method in an embodiment of the present invention.

[0058] Figure 3 This is a schematic diagram of the DAG dual attention guidance principle structure in the method of this embodiment of the invention.

[0059] Figure 4 This is a schematic diagram showing the results before and after acquiring infrared images of a pigpen according to an embodiment of the present invention.

[0060] Figure 5 This is a visualization diagram of an embodiment of the present invention after image annotation based on Labelme.

[0061] Figure 6 This diagram illustrates a comparison of the accuracy of the DAG_Cascade_Mask_R-CNN instance segmentation model proposed in this embodiment of the invention with existing mainstream models.

[0062] Figure 7 This diagram illustrates the mAP comparison between the DAG_Cascade_Mask_R-CNN instance segmentation model proposed in this embodiment of the invention and existing mainstream models.

[0063] Figure 8 This is a schematic diagram of the segmentation results of the DAG_Cascade_Mask_R-CNN instance segmentation model in the presence of pig urine and pig feces in an embodiment of the present invention.

[0064] Figure 9This is a schematic diagram of the segmentation results of the DAG_Cascade_Mask_R-CNN instance segmentation model in an embodiment of the present invention when the lens has significant blur.

[0065] Figure 10 This is a schematic diagram of the segmentation results of the DAG_Cascade_Mask_R-CNN instance segmentation model in the present invention under the conditions of large and small pigs and partial occlusion.

[0066] Figure 11 This is a schematic diagram of the segmentation results of the DAG_Cascade_Mask_R-CNN instance segmentation model in the presence of people in an embodiment of the present invention.

[0067] Figure 12 This is a schematic diagram of the segmentation results of the DAG_Cascade_Mask_R-CNN instance segmentation model in the case of pigs lying together in adhesion, as described in an embodiment of the present invention.

[0068] Figure 13 This is a schematic diagram of the structure of the third-level detector in an embodiment of the present invention. Detailed Implementation

[0069] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0070] like Figure 1 As shown, this invention proposes a method for creating a pig instance segmentation dataset in an infrared scene, such as... Figure 2 The diagram illustrates the flowchart of training and testing the proposed segmentation method using a pre-prepared dataset, specifically including the following steps:

[0071] Step 1: Capture panoramic infrared images of the pigs in the pigpen using an infrared camera. The captured images are as follows: Figure 4 As shown, the left side is an infrared image, and the right side is a real-time visible light image.

[0072] Step 1.1: Place an infrared camera (model MAG64AI) about 4 meters above the pigpen to take pictures. Collect data for 1 minute every hour. The collected image resolution is 640x480 and the format is bmp.

[0073] Step 2: Select the collected images to ensure they are representative, diverse, and challenging, resulting in a total of 1150 images.

[0074] Step 2.1: The original data was collected over three months, totaling 54,000 images. The representativeness of the dataset was achieved by selecting images under different scenarios of pig distribution, temperature, and humidity. The diversity was achieved by selecting images of pig size and posture at different times. The challenge was achieved by selecting images with complex backgrounds, such as pigs lying together and images with blurred lenses.

[0075] Step 3: The original 1150 images were augmented with random brightness (0.8-1.2) or random Gaussian noise with a 50% probability, resulting in a dataset of 2300 original images.

[0076] Step 3.1: Data augmentation is a random probability operation. Here, we assume that a brightness change operation was performed randomly, and let the new brightness be L. n The original brightness is L. o If the adjustment coefficient is r (0.8~1.2), then the formula for random brightness variation operation is as follows:

[0077] L n =L o *r (1)

[0078] Step 3.2: If the brightness change operation in Step 3.1 has been performed, the Gaussian noise operation will no longer be performed; otherwise, the Gaussian noise will be randomly added. Let the input pixel be P. in A normal Gaussian sampling distribution formula G(d) outputs pixels P. out The formula for operating on random Gaussian noise is as follows:

[0079] P out =P in +Xmeans+sigma*G(d) (2)

[0080] Where sigma represents the standard deviation, d is a linear random number, and Xmeans represents the mean.

[0081] Step 4: Based on the original images of the enhanced dataset, use Labelme annotation software to perform polygon annotation on all pigs. After annotation, a JSON file of the relevant infrared images will be generated, where the visualization of each JSON file is as follows. Figure 5 As shown.

[0082] Step 5: Put all the labeled JSON files and the original images into the same folder to create a dataset similar to the COCO format.

[0083] Step 5.1: The specific method for creating a COCO-like dataset is as follows: the training set and test set are randomly allocated by code in a 4:1 ratio. The final folder named Pig contains four second-level folders. The first folder, annotations, contains information about all the individual JSON files of the integrated training and test sets, named pig_instances_train2023.json and pig_instances_val2023.json respectively. In addition, the second folder, train2023, and the third folder, val2023, contain all the original images of the training and test sets respectively. The last folder, visualization, contains all the labeled visualization images.

[0084] Step 6: The labeled dataset from the above steps is used for training and testing.

[0085] Step 6.1: If it is training, pig_instances_train2023.json and train2023 will be used as input.

[0086] Step 6.2: If it is a test, pig_instances_val2023.json and val2023 will be used as input.

[0087] Step 7: Using the pre-trained convolutional neural network ResNet101 as the backbone, extract features from the input image to obtain the image's feature map.

[0088] Step 8: Using the Region Proposal Network algorithm, a series of candidate boxes are generated from the feature map. These candidate boxes may contain the target region of interest, i.e. the pig region. In addition, the feature map generated in Step 7 will also introduce channel and spatial attention blocks into the feature pyramid through the designed dual attention guidance module (DAG module) to capture the feature dependencies in the channel and spatial dimensions and fuse the outputs of the channel and spatial attention blocks.

[0089] Step 8.1: The specific process of the Region Proposal Network algorithm is as follows: input the feature map extracted in step 7, first generate anchor boxes; then perform binary classification to determine whether each anchor box is the target or the background; use a softmax classifier to extract possible candidate boxes, and fine-tune them by bounding box regression to make the candidate boxes and ground truth boxes closer.

[0090] Step 8.2: The feature map obtained through ResNet101 will also be processed by the DAG module designed in this invention, such as... Figure 3 As shown, firstly, the high-level feature map and the low-level feature map (F) are... h F l ) are fused, where F h The first feature map is the output feature map of the last layer of ResNet101, and the lower-level feature maps are the feature maps output after the first layer of ResNet101. The fused feature map F merge The data is fed into an asymmetric convolutional unit, which includes 3x3, 1x3, and 3x1 convolutional units. After the asymmetric convolution operation, a ReLU operation is performed to remove irrelevant features to obtain F. ACU ;F ACU The input will be fed into a dual-channel guidance unit, including spatial attention and channel attention. The outputs of the channel and spatial attention blocks will be fused to obtain F, which captures feature dependencies in both the channel and spatial dimensions. DAU , will F merge and F DAU By fusing the features again, a more reliable feature map F can be obtained. DAB The specific formula is as follows:

[0091]

[0092]

[0093] Step 9: Apply a candidate box classifier to the generated candidate boxes to determine whether each candidate box contains the target object. The classifier is usually a fully connected layer network used to classify each candidate box and predict its target category.

[0094] Step 10: Convert the feature map F obtained by the DAG module DAB The candidate boxes predicted as positive samples obtained in step 9 are subjected to bilinear interpolation to further refine them and more accurately define the location of the target object.

[0095] Step 10.1: The specific steps of the bilinear interpolation method are as follows: Let the corresponding coordinates of the function relationship f of the unknown value be (x, y), and assume that the four known points are Q. 11 = (x1, y1), Q 12 = (x1, y2), Q 21 = (x2, y1), Q 22 = (x2, y2), first perform linear interpolation on the x-axis to obtain the following formula:

[0096]

[0097]

[0098] Where R1 are two temporary points on the x-axis;

[0099] Step 10.2: Further, perform linear interpolation on the y-axis to obtain the following formula:

[0100]

[0101] Where P is a point on the y-axis after linear interpolation;

[0102] Step 10.3: Combining the x-axis and y-axis results gives the final result of bilinear interpolation, as shown in the following formula:

[0103]

[0104] Step 11: Output the bilinear interpolation results as bottom-up and top-down feature maps through the first-stage detector.

[0105] Step 12: The top-down feature map output from the first stage will be bilinearly interpolated again with the feature map after feature fusion with the DAG module designed in step 8.

[0106] Step 13: Input the result of the bilinear interpolation operation into the bottom-up and top-down feature maps output by the second-stage detector.

[0107] Step 14: The top-down feature map output from the second stage will be bilinearly interpolated again with the feature map after feature fusion with the DAG module designed in step 8.

[0108] Step 15: Input the result of the bilinear interpolation operation into the bottom-up and top-down feature maps output by the third-level detector.

[0109] like Figure 13 As shown, the network structures of the first-level, second-level, and third-level detectors are identical, each including a forward bottom-up pyramid and a reverse top-down pyramid, outputting bottom-up and top-down feature maps respectively. The two pyramids are horizontally connected; bottom-up represents downsampling, and top-down represents upsampling. Horizontal connections allow interaction between high-resolution shallow feature maps and deep feature maps rich in semantic information. In this embodiment, each detector includes 4 downsampling operations and 3 upsampling operations, resulting in 12 downsampling operations and 9 upsampling operations for all three detectors.

[0110] Step 16: The bottom-up and top-down feature maps after passing through three levels of detectors are used as the basis for the segmentation result. That is, the feature map finally output by the third-level detector is the segmentation result, which is used as the segmentation result for the infrared pig instance. Under normal circumstances, the segmentation result is as follows: Figure 8 As shown, the segmentation result under the condition of lens blur is as follows: Figure 9 As shown, the segmentation results are as follows, considering the presence of pigs of different sizes and the obstruction of fences. Figure 10 As shown, the segmentation results in the case of humans are as follows: Figure 11 As shown, the segmentation results in cases where pigs are lying together and sticking together are as follows: Figure 12 As shown.

[0111] The comparative experiment used an infrared image dataset of pigs established by the unit of this invention. This database contains 2300 images, of which 460 images were used as the test set in the experiment. Several existing methods were compared in the experiment, as shown in Table 1. The best value is marked in bold. DAG_Cascade_Mask_R-CNN represents the method of this invention. The accuracy variations of each algorithm are also shown in Table 1. Figure 6 As shown, the mAP of each algorithm is compared to... Figure 7 As shown, this method is superior to other existing methods.

[0112] Table 1 Comparison of the present invention with other existing methods

[0113]

[0114]

[0115] This invention also provides a deep learning-based infrared pig instance segmentation system, comprising the following modules:

[0116] The preprocessing module is used to acquire infrared images of pig samples and perform preprocessing.

[0117] The annotation module is used to perform polygon annotations on the preprocessed image, taking the annotated image as input.

[0118] The feature extraction module is used to extract features from the input image by using a pre-trained convolutional neural network as the backbone network.

[0119] The high-confidence feature map acquisition module is used to obtain a more reliable feature map F by passing the obtained feature map through a dual-attention guidance module. DAB The specific processing procedure is as follows:

[0120] First, the high-level feature map and the low-level feature map (F) are combined. h F l ) are fused to obtain F merge , where Fh The first feature map is the output feature map of the last layer of the convolutional neural network, and the lower-level feature maps are the feature maps output after the first layer of the convolutional neural network. The fused feature map F merge The data is fed into an asymmetric convolution unit, and after the asymmetric convolution operation, an activation function operation is performed to remove irrelevant features to obtain F. ACU ;F ACU The input is fed into a dual-channel guidance unit, including spatial attention and channel attention. The outputs of the channel and spatial attention blocks are fused to obtain F, which captures feature dependencies in both the channel and spatial dimensions. DAU , will F merge and F DAU By fusing the features again, a more reliable feature map F can be obtained. DAB ;

[0121] The candidate box acquisition module is used to generate a series of candidate boxes from the feature map using a candidate box generation algorithm. A classifier is applied to the generated candidate boxes to determine whether each candidate box contains the target object. Candidate boxes that contain the target object are positive sample candidate boxes.

[0122] The bilinear interpolation module is used to interpolate the feature map F obtained by the dual attention guidance module. DAB Bilinear interpolation is performed on candidate boxes predicted as positive samples to more accurately define the location of the target object;

[0123] The target segmentation module is used to pass the bilinear interpolation result through multiple detectors in sequence. Except for the last detector, the outputs of the other detectors are fused with the output of the dual attention guidance module and then bilinear interpolation is performed again before being input into the next layer detector. The output of the last layer detector is the final target segmentation result.

[0124] The specific implementation methods of each module are the same as those of each step, and will not be described in this invention.

[0125] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A deep learning-based infrared pig instance segmentation method, characterized in that, Includes the following steps: Step 1: Acquire infrared images of the pig samples and perform preprocessing; Step 2: Perform polygon annotation on the preprocessed image and use the annotated image as input; Step 3: Use the pre-trained convolutional neural network as the backbone network to extract features from the input image and obtain the image's feature map. Step 4: The feature map obtained in Step 3 is used by the dual attention guidance module to obtain a more reliable feature map F. DAB The specific processing procedure is as follows: First, the high-level feature map and the low-level feature map (F) are combined. h F l ) are fused to obtain F merge , where F h The first feature map is the output feature map of the last layer of the convolutional neural network, and the lower-level feature maps are the feature maps output after the first layer of the convolutional neural network. The fused feature map F merge The data is fed into an asymmetric convolution unit, and after the asymmetric convolution operation, an activation function operation is performed to remove irrelevant features to obtain F. ACU ;F ACU The input is fed into a dual-channel guidance unit, including spatial attention and channel attention. The outputs of the channel and spatial attention blocks are fused to obtain F, which captures feature dependencies in both the channel and spatial dimensions. DAU , will F merge and F DAU By fusing them again, we can obtain a more reliable feature map F. DAB ; Step 5: Use a candidate box generation algorithm to generate a series of candidate boxes from the feature map. Apply a classifier to the generated candidate boxes to determine whether each candidate box contains the target object. Candidate boxes that contain the target object are positive sample candidate boxes. Step 6, the feature map F obtained through the dual attention guidance module DAB Bilinear interpolation is performed on the candidate boxes predicted as positive samples obtained in step 5 to more accurately define the location of the target object. Step 7: The bilinear interpolation result is passed through multiple detectors in sequence. Except for the last detector, the outputs of the other detectors are fused with the output of the dual attention guidance module and then bilinear interpolation is performed again. The result is then input into the detector of the next layer. The output of the last detector is the final target segmentation result.

2. The infrared pig instance segmentation method based on deep learning as described in claim 1, characterized in that: The preprocessing described in step 1 includes selecting the acquired images and performing data augmentation on the selected images. The data augmentation is one of random brightness variation operation and random addition of Gaussian noise operation. The specific implementation method of random brightness change operation is as follows: Let the new brightness be L n The original brightness is L. o If the adjustment coefficient is r, then the formula for random brightness variation is as follows: L n =L o *r (1) The specific implementation of the operation of randomly adding Gaussian noise is as follows: Let the input pixel be P. in A normal Gaussian sampling distribution formula G(d) outputs pixels P. out The formula for operating on random Gaussian noise is as follows: P out =P in +Xmeans+sigma*G(d) (2) Where sigma represents the standard deviation, d is a linear random number, and Xmeans represents the mean.

3. The infrared pig instance segmentation method based on deep learning as described in claim 1, characterized in that: In step 2, Labelme annotation software performs polygon annotation on all pigs; in step 3, the convolutional neural network is the ResNet101 model.

4. The infrared pig instance segmentation method based on deep learning as described in claim 1, characterized in that: The specific processing steps of the candidate box generation algorithm are as follows: First, anchor boxes are generated; then, binary classification is performed to determine whether each anchor box is the target or the background; a softmax classifier is used to extract possible candidate boxes, and bounding box regression is used to fine-tune the algorithm so that the candidate boxes and the ground truth boxes are closer.

5. The infrared pig instance segmentation method based on deep learning as described in claim 1, characterized in that: Symmetric convolutional units include 3x3 convolutional units, 1x3 convolutional units, and 3x1 convolutional units.

6. The infrared pig instance segmentation method based on deep learning as described in claim 1, characterized in that: The specific steps of bilinear interpolation are as follows: Let the coordinates of the function f with unknown values ​​be (x, y), and assume that the four known points are Q. 11 = (x1, y1), Q 12 = (x1, y2), Q 21 = (x2, y1), Q 22 = (x2, y2), first perform linear interpolation on the x-axis to obtain the following formula: Where R1 are two temporary points on the x-axis; Furthermore, performing linear interpolation on the y-axis yields the following equation: Where P is a point on the y-axis after linear interpolation; Combining the x-axis and y-axis results gives the final result of bilinear interpolation, as shown in the following formula:

7. The infrared pig instance segmentation method based on deep learning as described in claim 1, characterized in that: The network structures of multiple detectors are exactly the same, and the specific processing procedure is as follows: The bilinear interpolation results are passed through the first-level detector to output a bottom-up first-level feature map and a top-down first-level feature map; After the first-level feature map from the top down is fused with the feature map output by the dual attention guidance module, a second bilinear interpolation operation is performed. The result of the second bilinear interpolation operation is input into the output of the second-stage detector, which is a bottom-up second-stage feature map and a top-down second-stage feature map. After the top-down second-level feature map is fused with the feature map output by the dual attention guidance module, a third bilinear interpolation operation is performed. The result of the third bilinear interpolation operation is input into the third-level detector to output the bottom-up third-level feature map and the top-down third-level feature map; The third-level feature map, viewed from top to bottom, is the result of segmentation.

8. The infrared pig instance segmentation method based on deep learning as described in claim 1, characterized in that: The detector consists of a forward bottom-up pyramid and a reverse top-down pyramid, which output bottom-up and top-down feature maps, respectively. The two pyramids are connected laterally. Bottom-up is downsampling, and top-down is upsampling. The lateral connection interacts the shallow feature map with high resolution and the deep feature map with rich semantic information. Finally, the top-down feature map is output as the prediction result feature map.

9. A deep learning-based infrared pig instance segmentation system, characterized in that, Includes the following modules: The preprocessing module is used to acquire infrared images of pig samples and perform preprocessing. The annotation module is used to perform polygon annotations on the preprocessed image, taking the annotated image as input. The feature extraction module is used to extract features from the input image by using a pre-trained convolutional neural network as the backbone network. The high-confidence feature map acquisition module is used to obtain a more reliable feature map F by passing the obtained feature map through a dual-attention guidance module. DAB The specific processing procedure is as follows: First, the high-level feature map and the low-level feature map (F) are combined. h F i ) are fused to obtain F merge , where F h The first feature map is the output feature map of the last layer of the convolutional neural network, and the lower-level feature maps are the feature maps output after the first layer of the convolutional neural network. The fused feature map F merge The data is fed into an asymmetric convolution unit, and after the asymmetric convolution operation, an activation function operation is performed to remove irrelevant features to obtain F. ACU ;F ACU The input is fed into a dual-channel guidance unit, including spatial attention and channel attention. The outputs of the channel and spatial attention blocks are fused to obtain F, which captures feature dependencies in both the channel and spatial dimensions. DAU , will F merge and F DAU By fusing the features again, a more reliable feature map F can be obtained. DAB ; The candidate box acquisition module is used to generate a series of candidate boxes from the feature map using a candidate box generation algorithm. A classifier is applied to the generated candidate boxes to determine whether each candidate box contains the target object. Candidate boxes that contain the target object are positive sample candidate boxes. The bilinear interpolation module is used to interpolate the feature map F obtained by the dual attention guidance module. DAB Bilinear interpolation is performed on candidate boxes predicted as positive samples to more accurately define the location of the target object; The target segmentation module is used to pass the bilinear interpolation result through multiple detectors in sequence. Except for the last detector, the outputs of the other detectors are fused with the output of the dual attention guidance module and then bilinear interpolation is performed again before being input into the next layer detector. The output of the last layer detector is the final target segmentation result.

Images