Livestock image instance segmentation method based on improved Transform

Abstract

The invention relates to a livestock image instance segmentation method based on an improved Transform, and the method comprises the following steps: S1, obtaining a high-quality livestock image, carrying out the labeling and image data amplification, and constructing a training set; s2, a target detection network DETR is improved based on a multi-scale deformable attention module and a unified query representation module, so that a livestock image instance segmentation model based on an improved Transform is constructed; s3, training a livestock image instance segmentation model based on Transform according to the training set to obtain a trained segmentation model; and S4, processing the to-be-detected livestock image data according to the trained segmentation model to obtain an instance segmentation effect. The method can effectively solve the problems that when an original Transform structure is directly used for livestock image segmentation, model training is difficult, and the segmentation performance is poor.

Description

Technical Field

[0001] The present invention relates to the technical field of livestock image instance segmentation, and in particular to a livestock image instance segmentation method based on an improved Transformer. Background Art

[0002] Livestock monitoring is a critical component of large-scale livestock farming. Monitoring livestock's health, behavior, physiology, and psychological state, among other aspects, helps strengthen scientific farm management, improve animal welfare, and enhance the production efficiency and quality of livestock products. Traditional livestock monitoring technologies often rely on manual observation or contact sensors, which are costly and unsuitable for large-scale farming. In recent years, computer vision technology has demonstrated its unique advantages in livestock monitoring. It provides a contactless, automated method for daily livestock monitoring, at low cost and with minimal human burden, without causing injury or stress reactions.

[0003] When computer vision technology is applied to livestock monitoring, it is necessary to analyze and understand livestock images. The prerequisite and foundation for achieving this goal is livestock image instance segmentation. Subsequent steps, such as feature extraction of livestock objects, are based on the segmented image regions. Therefore, the quality of image segmentation plays a crucial role in achieving accurate livestock monitoring. Traditional livestock image instance segmentation methods are mostly based on convolutional neural networks. In recent years, some methods have attempted to transfer the Transformer architecture to the field of image segmentation.

[0004] However, the Transformer-based livestock image instance method has a series of problems, as follows:

[0005] (1) When the Transformer structure is directly used for computer vision tasks, the model has a large amount of computation and is difficult to train;

[0006] (2) Simply adding a mask prediction branch to the original Transformer structure to achieve image instance segmentation will prevent the model from benefiting from the learning of the original task, resulting in poor segmentation performance. Summary of the Invention

[0007] In response to the above-mentioned deficiencies in the prior art, the present invention provides a livestock image instance segmentation method based on an improved Transformer, which is used to solve the problem that model training is difficult and segmentation performance is poor when the original Transformer structure is directly used for livestock image segmentation.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A livestock image instance segmentation method based on improved Transformer, comprising the following steps:

[0010] Step S1: Obtain high-quality livestock images, label and image data augmentation, and construct a training set;

[0011] Step S2: improve the target detection network DETR based on a multi-scale deformable attention module and a unified query representation module, thereby constructing a livestock image instance segmentation model based on improved Transformer. The model contains three main parts, a backbone convolutional neural network (CNN) used to extract multi-scale image feature maps, an improved Transformer with an encoder-decoder architecture, and a unified query representation module for generating final prediction results;

[0012] Step S3: train the livestock image instance segmentation model based on Transformer according to the training set, and obtain the trained segmentation model;

[0013] Step S4: process the livestock image data to be detected according to the trained segmentation model, and obtain the instance segmentation effect.

[0014] Further, in the step S1, livestock monitoring videos are collected by high-definition cameras in the farm, and video key frames are selected according to the principle of balanced distribution of scenes of each livestock, to obtain high-quality original images with uniform size of width and height. After labeling the original images, image data augmentation processing is performed, and the image data augmentation method used includes geometric transformation, gray and color space transformation, and adding noise and filtering. The augmented livestock image dataset is divided into a training set, a validation set and a test set in a ratio of 6:2:2.

[0015] Further, the step S2 specifically comprises:

[0016] Step S21: construct a multi-scale deformable attention module, replace the self-attention of the encoder and the cross-attention of the decoder in the Transformer of the DETR with multi-scale deformable attention. Let the output result of the multi-scale deformable attention module be MSDAttn, and the calculation formula of MSDAttn is as follows:

[0017]

[0018] wherein, represents the input multi-scale feature map. Specifically, l e [1, L] represents the index of the layer where the feature map is located, since the original DETR only adopts single-scale features, for different scale image feature maps extracted by the backbone network, in addition to the position embedding, a learnable scale embedding is additionally added to distinguish different feature layers. Unlike the original Transformer, which densely samples global positions when calculating attention, this module only samples K positions (K « H l W l ) at each attention head of each feature layer when calculating attention, m e [1, M] represents the index of the attention head, and k e [1, K] represents the index of the sampling point. The sampling point is obtained by adding a learnable position offset to the reference point, and the formula is as follows: is the normalized reference point coordinate, which is obtained by the query embedding q i after passing through a learnable linear projection layer and being processed by a sigmoid function, i e [1, N q ] represents the index of the query embedding, and N q is a value much larger than the number of livestock targets to be segmented in the image. Δp mlki represents the position offset, since the sizes of different feature layers may be different, the normalized reference point coordinate is added to the position offset to obtain the sampling point coordinate, which needs to be restored to the actual coordinate of the reference point in the lth layer through operation. A mlki represents the normalized attention weight, W m , W′ m are learnable weights in multi-head attention.

[0019] Step S22: Construct a unified query representation module, that is, add a mask embedding prediction branch parallel to the original category prediction branch and positioning branch in the prediction head of DETR, so that the query embedding output by the Transformer decoder can uniformly represent and complete the three sub-tasks of classification, positioning and mask embedding prediction in parallel, and the model can also benefit from multi-task learning during training. The mask embedding prediction branch is a three-layer multi-layer perceptron (MLP) with ReLU activation function, and finally outputs the predicted mask embedding. Since the query embedding is one-dimensional, there is a dimensional gap when it is directly used to represent the two-dimensional original mask, which affects the segmentation performance, so the newly added branch predicts the one-dimensional mask embedding instead of the original mask. During model training, the original mask needs to be converted into a mask embedding, and during prediction, the mask embedding needs to be converted into the original mask. This module uses principal component analysis (PCA) encoding to convert between mask embedding and original two-dimensional mask. Let the mask vector be v, the dimension of v be d, the original mask be m, and the size of m be N x N. The calculation formula for reducing the dimension of m by PCA is as follows:

[0020] f = argmin||m-DD T ||2, s.t.DD T = I

[0021] wherein represents a conversion matrix, represents a normalization matrix. The purpose of the formula is to find D when f is minimum, and the calculation formula of mutual conversion between v and m is as follows:

[0022] v = mD.m = vD T

[0023] Further, in the step S2, the total loss function formula of the livestock image instance segmentation model based on the Transformer is as follows:

[0024] Loss = λ cls ·L cls + λ box ·L box + λ mask ·L mask

[0025] wherein, λ cls , λ box , λ mask are different weights of three losses of classification, positioning and mask embedding prediction, which are 1.5, 3.75 and 2.25 respectively.

[0026] Further, the step S3 specifically comprises:

[0027] Step S31: adopting the idea of transfer learning, using the model weight pre-trained on the large public data set on the network to initialize the weight in the network, so as to accelerate the training time and reduce the convergence difficulty of the network model.

[0028] Step S32: using the Adam optimizer to train the model, training until the network converges to obtain the optimal parameters of the model, and obtaining the trained livestock image instance segmentation model based on the Transformer.

[0029] Step S33: after the training is completed, the test set sample in the step S1 is used to test the livestock image instance segmentation model based on the Transformer after the training is completed, and the effect of the model is verified.

[0030] Overall, compared with the prior art, the above technical solutions conceived by the present application have the following advantages

[0031] Beneficial effects:

[0032] 1.The application constructs a multiscale deformable attention module, which replaces the self-attention module of the encoder and the cross-attention module of the decoder in the original Transformer, can significantly reduce the computational amount when the model calculates attention while fusing multiscale image features, and thus improves the speed of model training and reasoning.

[0033] 2.The application improves the prediction head part of DETR and constructs a unified query representation module to embed the category, position and mask of livestock instances into a unified query embedding representation, so that the model can benefit from multi-task learning, thereby completing the classification, positioning and mask embedding prediction of livestock instances in parallel, and improving the problem of poor performance when the query embedding is directly used for mask prediction. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to facilitate the understanding of those skilled in the art, the application will be further described below in conjunction with the drawings.

[0035] Figure 1 is the flowchart of the embodiment of the application;

[0036] Figure 2 is the architecture diagram of the livestock image instance segmentation model based on the improved Transformer in the embodiment of the application. DETAILED DESCRIPTION

[0037] In order to facilitate the understanding of those skilled in the art, the application will be further described below in conjunction with the drawings. The content mentioned in the embodiment is not a limitation of the application.

[0038] Please refer to Figure 1 The application provides a livestock image instance segmentation method based on an improved Transformer, which comprises the following steps:

[0039] Step S1: Obtain high-quality livestock images, perform labeling and image data augmentation, and construct a training set;

[0040] In this embodiment, a high-definition infrared gun camera is used as the acquisition device to collect monitoring videos of livestock in different behavior states at different times of the day indoors and outdoors in livestock houses in the Livestock Teaching and Experiment Base of Northwest A&F University. Key frames of the videos are extracted using FFmpeg software, and manual screening is performed to remove images of livestock that are too dark and unclear, while ensuring that the number of indoor and outdoor livestock samples is evenly distributed, with a ratio of 1:1. The obtained images are uniformly saved in a size of 512x512. After obtaining high-quality images of livestock, the images are labeled using the open-source labeling software Labelme. Subsequently, the labeled images are subjected to image data augmentation processing, which uses image data enhancement methods including geometric transformation, grayscale and color space transformation, and adding noise and filtering. In this embodiment, the geometric transformation includes random rotation and mirror flipping; the grayscale and color space transformation includes random brightness, contrast, and color space transformation; and the adding noise and filtering includes randomly adding Gaussian and salt and pepper noise, random sharpening, and blurring. The augmented livestock image dataset is divided into a training set, a validation set, and a test set in a ratio of 6:2:2.

[0041] Step S2: improving the target detection network DETR based on the multi-scale deformable attention module and the unified query representation module, thereby constructing a livestock image instance segmentation model based on the improved Transformer. The model includes three main parts: a backbone convolutional neural network (CNN) for extracting multi-scale image feature maps, an improved Transformer with an encoder-decoder architecture, and a unified query representation module for generating final prediction results. The architecture of the model is shown in Figure 2

[0042] Step S3: training the livestock image instance segmentation model based on the Transformer according to the training set to obtain a trained segmentation model.

[0043] Step S4: processing the to-be-detected livestock image data according to the trained segmentation model to obtain the instance segmentation effect.

[0044] In this embodiment, the step S2 specifically includes:

[0045] Step S21: constructing a multi-scale deformable attention module to replace the self-attention of the encoder and the cross-attention of the decoder in the Transformer of the DETR with multi-scale deformable attention. Let the output result of the multi-scale deformable attention module be MSDAttn, and the calculation formula of MSDAttn is as follows:

[0046]

[0047] wherein, ​represents the input multi-scale feature map. Specifically, l e [1, L] represents the index of the layer where the feature map is located. Since the original DETR only uses single-scale features, for different scale image feature maps extracted by the backbone network, in addition to the position embedding, a learnable scale embedding is additionally added to distinguish different feature layers. The embodiment uses ResNet-50 as the backbone network to extract the feature maps of the input image with a size of 512x512 at the c3-c5 layers, and processes the c5 layer by a 3x3 convolution with a step size of 2 as the c6 layer, a total of 4 levels of feature maps are extracted, and the channel number of the feature map is 256. Correspondingly, L = 4 is set. Unlike the original Transformer which densely samples global positions when calculating attention, only K positions (K << H l W l ) are sampled at each attention head of each feature layer when calculating attention in the module, m e [1, M] represents the index of the attention head, and k e [1, K] represents the index of the sampling point. The embodiment sets M = 8 and K = 4. The sampling point is obtained by adding a learnable position offset to the reference point, and the formula is , which is the normalized reference point coordinate, is obtained by the query embedding q i after passing through a learnable linear projection layer and being processed by a sigmoid function, i e [1, N q ] represents the index of the query embedding, and N q is a value much larger than the number of livestock targets to be segmented in the image, and the embodiment sets N q = 50. Δp mlki represents the position offset. Since the sizes of different feature layers may be different, the normalized reference point coordinate is added to the position offset to obtain the sampling point coordinate, which needs to be restored to the actual coordinate of the reference point in the l-th layer by operation. A mlki represents the normalized attention weight, and W m , W' m are learnable weights in the multi-head attention.

[0048] Step S22: constructing a unified query representation module, that is, adding a mask embedding prediction branch parallel to the original category prediction branch and positioning branch in the prediction head part of DETR, so that the query embedding output by the Transformer decoder can uniformly represent and complete the three subtasks of classification, positioning and mask embedding prediction in parallel, and the model can also benefit from multi-task learning during training. The mask embedding prediction branch is a three-layer multilayer perceptron (MLP) with ReLU activation function, and finally outputs the predicted mask embedding. Since the query embedding is one-dimensional, there is a dimensional gap when it is directly used to represent the two-dimensional original mask, which affects the segmentation performance, so the newly added branch predicts the one-dimensional mask embedding instead of the original mask. During model training, the original mask needs to be converted into mask embedding, and during prediction, the mask embedding needs to be converted into the original mask. This module uses principal component analysis (PCA) encoding method to convert between mask embedding and original two-dimensional mask. Let the mask vector be v, the dimension of v be d, the original mask be m, and the size of m be N*N. The calculation formula of reducing m by PCA is as follows:

[0049] f=argmin||m-DD T m||2,s.t.DD T =I

[0050] Wherein represents the conversion matrix, represents the normalization matrix. The purpose of the formula is to find the D when f is the smallest, and then the mutual conversion formula between v and m is as follows:

[0051] v=mD, m=vD T

[0052] In this embodiment, N=512, d=256.

[0053] Further, in the step S2, the total loss function formula of the Transformer-based livestock image instance segmentation model is as follows:

[0054] Loss=λ cls ·L cls +λ box ·L box +λ mask ·L mask

[0055] Wherein, λ cls , λ box , λ maskTo classify, locate and mask embedding prediction three different weights of loss, respectively, 1.5, 3.75, 2.25. In this embodiment, the calculation formula of classification and positioning loss is consistent with DETR, and the mask embedding loss is calculated using L1 loss function, and the calculation formula is as follows:

[0056]

[0057] In the formula is the predicted mask embedding.

[0058] In this embodiment, the step S3 specifically comprises:

[0059] Step S31: using the idea of transfer learning, using the pre-trained model weight on the large data set on the network to initialize the weight in the network, to speed up the training time, and reduce the convergence difficulty of the network model. In this embodiment, Microsoft COCO data set is used for model transfer learning.

[0060] Step S32: using Adam optimizer to train the model, in this embodiment, setting β1=0.9, β2=0.999, weight decay is 10 -4 , the initial learning rate is 2×10 -4 , the attenuation coefficient is 0.95, the attenuation speed is 5 cycles, and the batch data size (Batch size) is set to 32 during model training. Train to network convergence to obtain the optimal parameters of the model, get the trained based on the Transformer livestock image instance segmentation model.

[0061] Step S33: after training, using the test set sample in step S1 to test the based on the Transformer livestock image instance segmentation model after training, verify the effect of the model. In this embodiment, AP, AP 50 , AP 75 Three kinds of indexes are used to evaluate the test results.

[0062] The above content is only an example and description of the structure of the present application, and those skilled in the art can make various modifications or supplements or use similar ways to replace the described specific embodiments, as long as they do not deviate from the structure of the invention or exceed the scope defined by the present claims, which shall belong to the protection scope of the present application.

Claims

1. A livestock image instance segmentation method based on an improved Transformer, characterized in that: The following steps are involved: Step S1: Obtain high-quality livestock images, perform annotation and image data amplification, and construct a training set; Step S2: Improve the object detection network DETR based on the multi-scale deformable attention module and the unified query representation module to build a livestock image instance segmentation model based on the improved Transformer. The model consists of three main components: a backbone convolutional neural network (CNN) for extracting multi-scale image feature maps, an improved Transformer with an encoder-decoder architecture, and a unified query representation module for generating the final prediction results. Step S3: training the livestock image instance segmentation model based on the Transformer according to the training set to obtain a trained segmentation model; Step S4: Process the livestock image data to be detected according to the trained segmentation model to obtain instance segmentation effect.

2. The livestock image instance segmentation method based on the improved Transformer according to claim 1, characterized in that: The acquisition of high-quality livestock images requires collecting livestock monitoring videos and screening key frames based on the principle of balanced distribution of scenes in which each livestock appears. The image data enhancement methods used in the image data amplification processing include geometric transformation, grayscale and color space transformation, as well as noise addition and filtering.

3. The livestock image instance segmentation method based on the improved Transformer according to claim 1, characterized in that: The step S2 specifically includes: Step S21: Construct a multi-scale deformable attention module and replace the self-attention of the encoder and the cross-attention of the decoder in the DETR Transformer with the multi-scale deformable attention module. Let the output of the multi-scale deformable attention module be MSDAttn. The calculation formula of MSDAttn is as follows: in, Represents the multi-scale feature map of the input. Specifically, l∈[1, L] represents the index of the layer where the feature map is located. Since the original DETR only uses single-scale features, for the image feature maps of different scales extracted by the backbone network, in addition to the position embedding, an additional learnable scale embedding is added to distinguish different feature layers. Unlike the original Transformer that densely samples the global position when calculating attention, this module only samples K positions (K<<H) in each attention head of each feature layer when calculating attention. l W l ), m∈[1,M] represents the index of the attention head, and k∈[1,K] represents the index of the sampling point. The sampling point is obtained by adding a learnable position offset to the reference point. In the formula is the normalized reference point coordinate, which is embedded in the query q i After the learnable linear projection layer and the sigmoid function processing, i∈[1, N q ] represents the index of query embedding, N q is a pre-set value that is much larger than the number of livestock targets to be segmented in the image. mlki Represents position offset. Since the sizes of different feature layers may be different, the normalized reference point coordinates Before adding the position offset to get the sampling point coordinates, it is necessary to pass The operation is restored to the actual coordinates of the reference point in the lth layer. mlki represents the normalized attention weight, W m , W′ m are the learnable weights in multi-head attention. Step S22: Construct a unified query representation module, that is, add a mask embedding prediction branch in parallel with the original category prediction branch and positioning branch to the prediction head of DETR, so that the query embedding output by the Transformer decoder can uniformly represent and complete the three subtasks of classification, positioning and mask embedding prediction in parallel. The model can also benefit from multi-task learning during training. The mask embedding prediction branch is a three-layer multi-layer perceptron (MLP) with a ReLU activation function, which ultimately outputs the predicted mask embedding. Since the query embedding is one-dimensional, there is a dimensional gap when it is directly used to represent the two-dimensional original mask, which affects the segmentation performance. Therefore, the newly added branch predicts a one-dimensional mask embedding instead of the original mask. The original mask needs to be converted to a mask embedding during model training, and the mask embedding needs to be converted to the original mask during prediction. This module uses the principal component analysis (PCA) encoding method to convert between mask embedding and the original two-dimensional mask. Let the mask vector be v, the dimension of v be d, the original mask be m, the size of m is N×N, and the calculation formula for reducing the dimension of m through PCA is as follows: f=argmin||m-DD T m||2,stDD T =Iwhere represents the transformation matrix, Represents the normalized matrix. The purpose of the formula is to find D when f is the smallest. The calculation formula for the mutual conversion between v and m is as follows: v=mD,m=vD T 4. The livestock image instance segmentation method based on the improved Transformer according to claim 3, characterized in that: The total loss function formula of the Transformer-based livestock image instance segmentation model is as follows: Loss=λ cls ·L cls +λ box ·L box +λ mask ·L mask Among them, λ cls ,λ box ,λ mask Different weights of the three losses are predicted for classification, localization, and mask embedding, which are 1.5, 3.75, and 2.25 respectively.

5. The livestock image instance segmentation method based on improved Transformer according to claim 1, characterized in that: The step S3 specifically includes: Step S31: Using the idea of ​​transfer learning, the weights of the network are initialized using the pre-trained model weights on a large dataset publicly available on the Internet to speed up the training time and reduce the difficulty of convergence of the network model. Step S32: Use the Adam optimizer to train the model until the network converges to obtain the optimal parameters of the model, and obtain a trained Transformer-based livestock image instance segmentation model. Step S33: After the training is completed, the trained Transformer-based livestock image instance segmentation model is tested using the test set samples in step S1 to verify the effectiveness of the model.

Images