A Two-Stage Pig Face Recognition Method Based on Metric Learning

Abstract

The present invention discloses a two-stage pig face recognition method based on metric learning, including: 1) screening and annotating the collected pig face pictures, and dividing the training set and the test set; 2) improving the Retinaface detection model structure; 3) inputting the pictures in the training set into the improved Retinaface detection model for training; 4) cropping and subsequent processing of the pictures in the training set according to the annotation information; 5) building a Dense_Bilinear CNN pig face recognition model; 6) inputting the pictures of the frontal face data set and the profile face data set into the Dense_Bilinear CNN pig face recognition model for training respectively; 7) cascading the models according to the model parameters obtained in steps 3) and 6); 8) in the test set, selecting pictures to establish a database, and performing feature comparison on the remaining pictures. The present invention can effectively realize pig face recognition with high accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a two-stage pig face recognition method based on metric learning. Background Art

[0002] The traditional individual identification methods in the pig breeding process mainly include: ear notch method, marking method, ear tag method and electronic ear tag method. However, each of the traditional methods has its own disadvantages. For example, the ear cutting rules of the ear notch method are not unified and it is easy to get infected; the marks of the marking method are worn and overlapped; the ear tags of the ear tag method are easy to fall off; the electronic ear tag method uses RFID (radio frequency identification) to identify the tags worn on the pig ears, which is a widely used method, but there are also problems such as falling off and high cost. The traditional identification methods are limited by the inability to quickly and intelligently process data, resulting in cumbersome and untimely management of pigs in large-scale pig farms. This method of marking animal individuals has gradually been replaced by non-contact identification methods based on images or videos.

[0003] The pig face detection algorithm refers to finding the position of the pig face to be detected in a picture, then cropping and correcting it according to the prediction information to obtain a local picture with only the pig face, and then performing pig face recognition; pig face recognition is to extract the features of the pig face picture, and then map it into a high-dimensional vector, compare the features of the picture to be recognized with the pictures in the database, and judge the identity according to the similarity. The Retinaface algorithm is a classic face detection algorithm used to detect the face position and return five face key points. Bilinear CNN is a bilinear convolutional neural network for fine-grained image recognition. Triplet loss and Center loss are loss functions based on Euclidean space metric, and Arcface loss is a loss function based on angular space metric.

[0004] At present, there are few domestic and foreign research literatures on pig face recognition. The main reason is that there is no public dataset for pig faces, and almost all existing papers use the data collected by themselves. The dissertation "Research on Pig Face Recognition Based on Convolutional Neural Network" uses the classical bilinear convolutional neural network for fine-grained image recognition to classify 200 pigs; the paper "Pig Face Recognition Based on Improved YOLOv3" proposes an algorithm to complete the detection and recognition tasks simultaneously, converting pig face recognition into an object detection task; in the paper "Two-stage method based on triplet margin loss for pigface recognition", a variety of detection networks are adopted, and this two-stage network uses the KNN algorithm to classify 28 pigs during pig face recognition; the paper "Individual Pig Face Recognition Incorporating Attention Mechanism" designs a DenseNet-CBAM model that combines a deep learning network and an attention mechanism module for recognition, and the model only classifies among 8 categories of live pigs. The above pig face recognition research is based on the closed-set problem, classifying within a finite set. If new pigs appear on the farm, the classification model needs to be retrained for recognition. Moreover, the current research does not address the problem that the three-dimensional structure of the pig face causes different poses, which affects the recognition effect.

[0005] In summary, it has high practical application value to invent a method for open-set recognition problems and capable of targeting the three-dimensional structure of pig faces. Summary of the Invention

[0006] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and propose a two-stage pig face recognition method based on metric learning, which can effectively achieve pig face recognition with high accuracy.

[0007] To achieve the above purpose, the technical solution provided by the present invention is: a two-stage pig face recognition method based on metric learning, including the following steps:

[0008] 1) Obtain pig face pictures, screen and label the collected pictures, divide the training set and test set according to pig face categories, and divide the validation set according to the number of pictures in the training set;

[0009] 2) Improve the Retinaface detection model structure in combination with actual applications and detection requirements to obtain an improved Retinaface detection model. The improvement includes using channel attention and spatial attention methods in the enhanced feature extraction process to further screen features; in the key point prediction branch of the prediction module, reducing the original model branch to the number of branches suitable for the required task; in the loss function part, changing the original model's multi-task loss function with four terms to a three-task loss function of prediction classification loss, prediction facial box regression loss, and prediction key point regression loss;

[0010] 3) Input the images in the training set into the improved Retinaface detection model for training. Train according to the preset parameters, and use the model parameters with the minimum validation set loss as the model parameters for the improved Retinaface detection, denoted as R;

[0011] 4) Crop and perform subsequent processing on the images in the training set according to the annotation information. The subsequent processing targets the more three-dimensional structure of the pig face. To enable the network to learn the corresponding features, judge whether it is a frontal face or a side face according to the angle between the line connecting the midpoints of the two eyes and the line connecting the midpoints of the two nostrils and the normal line of the line connecting the two eyes, and prepare the frontal face dataset and the side face dataset;

[0012] 5) Build a Dense_Bilinear CNN pig face recognition model in combination with the actual application and recognition requirements. This model uses DenseNet121 in the backbone network; uses a bilinear convolutional neural network structure in feature extraction; in terms of the loss function, uses a combined loss function of Triplet loss, Center loss and Arcface loss;

[0013] 6) Input the images of the frontal face dataset and the side face dataset into the Dense_Bilinear CNN pig face recognition model for training respectively. Train according to the preset parameters, and use the model parameters with the minimum validation set loss as the model parameters for the Dense_Bilinear CNN pig face recognition respectively, to obtain the model parameters A for the frontal face and the model parameters B for the side face;

[0014] 7) Perform cascading of the models according to the model parameters obtained in steps 3) and 6). The specific operation is: input the input image into the improved Retinaface detection model loaded with the model parameters R to obtain prediction information, and judge whether it belongs to the frontal face according to the prediction information and the judgment criteria in step 4). If so, input the processed image into the Dense_Bilinear CNN pig face recognition model loaded with the model parameters A, otherwise input the processed image into the Dense_Bilinear CNN pig face recognition model loaded with the model parameters B. Finally, the cascaded model will output a 128-dimensional feature vector of the image;

[0015] 8) In the test set, according to the judgment criteria in step 4), the front face and side face of each pig are discriminated, and one picture is selected respectively as the pig face database. The pictures of each pig in the database also need to go through step 7) to obtain the 128-dimensional feature vectors of the front face and side face of each pig as the front face and side face features of the pig; the remaining pictures in the test set are used as the pictures to be tested and compared with the pictures in the database. The method of feature comparison is to calculate the features of the pictures to be tested through step 7), and then calculate the Euclidean distance between this feature and the features of each pig in the database. Select the pig in the database with the smallest Euclidean distance from the picture to be tested, then it is considered that the identity of the picture to be tested corresponds to this pig in the database, that is, the recognition process is completed.

[0016] Furthermore, in step 1), the collected pig face images are manually screened, and the cleaned pig face images are labeled with the Labelme tool. The labeling method is the pig face frame and four key points of the pig face, namely the left eye, right eye, left nostril and right nostril; the training set and test set are divided according to the pig face category, and the validation set is divided according to the number of pictures in the training set.

[0017] Furthermore, in step 2), the improved Retinaface detection model consists of a backbone feature extraction network, an enhanced feature extraction network and a prediction module; among them, a channel attention mechanism and a spatial attention mechanism are added to the enhanced feature extraction network to further screen the features and improve the model detection ability; the channel attention mechanism performs global max pooling and global average pooling on the input features respectively, and then sends them into two-layer shared fully connected layers respectively, and then performs an addition operation on the output, and finally passes through the Sigmoid activation operation; the spatial attention mechanism performs global max pooling and global average pooling on the input feature layer based on the channel, and then performs splicing based on the channel, and then reduces the dimension to 1 channel through a convolutional layer, and finally passes through the Sigmoid activation; in the prediction module, the face classification prediction is used to judge whether there is an object in the prior box, and there are 2 parameters; the face box prediction is used to adjust the prior box to obtain the prediction box, and there are 4 parameters; the face key point prediction is used to adjust the prior box to obtain the key points, and each key point has two parameters. Here, the Retinaface is improved, and the face key point branch is reduced to 8 1*1 convolutions, and the number of channels is 8 corresponding to the coordinates of 4 key points, and there are 8 parameters;

[0018] The loss function of the improved Retinaface detection model is calculated on three effective feature layers. The loss function of the improved Retinaface detection model includes three parts: prediction classification loss, prediction face box regression loss and prediction key point regression loss. The total loss function L1 is shown in formula (1):

[0019] L1 = λ1L cls + λ2Lbox +λ3L landmark (1)

[0020] In the formula, λ1, λ2, and λ3 are the weight coefficients of L cls , L box , L landmark respectively; where L cls is the face classification loss, specifically using the cross-entropy loss function, and L box is the face box loss, L landmark is the key point loss; it should be noted here that L box and L landmark both specifically use Smooth L1 as the loss function, L box takes the error between the true value and the predicted value of the face box parameters as the input, and L landmark takes the error between the true value and the predicted value of the key point parameters as the input.

[0021] Further, in step 3), the pictures in the training set are input into the improved Retinaface detection model for training. The overall ratio of the pictures is adjusted so that the long side of the picture is 840. If the long side and the short side of the picture are the same, the picture size is 840*840. If the short side is not 840, zero padding is performed on the short side so that the input picture is 840*840 in size; the training parameters are set, the pre-trained weights are loaded, and the SGD optimizer is used as the training optimizer, with a momentum of 0.94 and a weight decay of 5*10^-4; after the training is completed, the model parameters with the minimum validation set loss are used as the model parameters for the improved Retinaface detection, denoted as R.

[0022] Further, in step 4), for the more three-dimensional structure of the pig face, in order to let the network learn the corresponding features, it is judged whether it is a frontal face according to the angle between the line connecting the midpoints of the two eyes and the normal line of the line connecting the midpoints of the two nostrils. If the angle is less than 30 degrees, it is considered a frontal face, otherwise it is considered a side face;

[0023] After cropping the picture, no processing is done for the side face; for the frontal face, a two-point correction method is used. Taking the two eyes of the pig as two points, the rotation angle is obtained through the angle between the line connecting these two points and the horizontal direction, and then the rotation matrix is obtained in combination with the image center; finally, through the affine transformation function, the image is rotated with the center of the image as the rotation center to straighten the pig face;

[0024] Assuming the rotation angle is θ and the rotation center is (r x , r y ), the following rotation matrix M can be obtained:

[0025]

[0026] Further, in step 5), a Dense_Bilinear CNN pig face recognition model is built in combination with actual applications and recognition requirements. The DenseNet121 is adopted in the backbone network of this model; in feature extraction, a bilinear convolutional neural network structure is used, that is, two feature extractors are used for the input image, the outer product of the outputs of the two feature extractors is performed, and then pooled into a feature vector; then the features will go through dimensionality reduction and dropout operations; during the training process, the features will finally go through a linear layer with the number of categories in the training set; during the prediction process, the features will finally be mapped into a 128-dimensional feature vector; in terms of the loss function, a combined loss function of Triplet loss, Center loss and Arcface loss is adopted, aiming to project various pig faces into a discriminative metric space, making the same pig faces closer and different pig faces farther apart;

[0027] Among them, the loss functions for implementing Euclidean distance measurement are Triplet loss and Center loss, and then the classification loss softmax loss is added. Based on the deformation of softmax loss, Arcface loss is introduced to measure the angular space. The total loss function L2 is expressed as formula (2):

[0028] L2 = λ4L Triplet + λ5L Center + λ6L AF (2)

[0029] In the formula, λ4, λ5, and λ6 are the weight coefficients of L Triplet , L Center , L AF respectively; where L Triplet is Triplet loss, that is, triplet loss, and L center is Center loss, that is, center loss, and L AF is Arcface loss, that is, additive angular margin loss.

[0030] Further, in step 6), the pictures of the frontal face dataset and the side face dataset are respectively input into the Dense_Bilinear CNN pig face recognition model for training. The input pictures are adjusted to a size of 224*224, and trained according to the preset parameters. The pre-trained model is loaded, and the SGD optimizer is used for the training optimizer, with a momentum of 0.94 and a weight decay of 5*10^-4; after the training is completed, the model parameters with the minimum validation set loss are respectively used as the model parameters of the Dense_Bilinear CNN pig face recognition, and the model parameters A of the frontal face and the model parameters B of the side face are obtained.

[0031] Further, in step 7), the models are cascaded according to the model parameters obtained in step 3) and step 6). The specific operation is as follows: The input image is input into the improved Retinaface detection model loaded with the model parameter R to obtain prediction information. In the post-processing of the improved Retinaface detection model, the intersection-over-union ratio of non-maximum suppression is set to 0.5, and the face bounding boxes with a confidence level lower than 0.8 are discarded.

[0032] According to the prediction information and the judgment criteria in step 4), it is determined whether it belongs to a frontal face. If so, the processed image is input into the Dense_Bilinear CNN pig face recognition model loaded with the model parameter A; otherwise, the processed image is input into the Dense_Bilinear CNN pig face recognition model loaded with the model parameter B. Finally, the cascaded model outputs a 128-dimensional feature vector of the image.

[0033] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0034] 1. The present invention creatively prepares a pig face dataset with a face bounding box and four key points of the left eye, right eye, left nostril, and right nostril.

[0035] 2. In the improved Retinaface detection model, channel attention and spatial attention are added to improve the model detection accuracy; the branches and loss functions of the prediction network are modified to make it suitable for the pig face object of the present invention.

[0036] 3. A discriminant method for the frontal face and side face of pigs is proposed according to the three-dimensional structure of the pig face, enabling the model to identify according to the special facial structure of the pig face.

[0037] 4. The Dense_Bilinear CNN pig face recognition model is proposed, which has a good feature extraction effect on fine-grained images. The combination of Triplet loss, Center loss, and Arcface loss is used as the loss function to perform constraints in the Euclidean space and angular space, improving the inter-class separability and intra-class compactness.

[0038] 5. The Retinaface detection model and the Dense_Bilinear CNN model are cascaded to form a complete two-stage model, enabling it to achieve the pig face recognition task. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flowchart of the method of the present invention.

[0040] Figure 2 is a schematic diagram of the annotation of the pig face dataset in the embodiment of the present invention.

[0041] Figure 3 These are schematic diagrams of the front face and side face in the embodiments of the present invention. Specific embodiments

[0042] The present invention will be further described in detail below in conjunction with embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0043] As Figure 1 shown, this embodiment discloses a two-stage pig face recognition method based on metric learning, and the specific situation is as follows:

[0044] 1) Obtain pig face pictures, manually screen the collected pig face images, and use the Labelme tool to label the cleaned pig face images. As Figure 2 shown, the labeling method is the pig face frame and four key points of the pig face, namely the left eye, right eye, left nostril and right nostril; divide the training set and test set according to the pig face category, and divide the validation set according to the number of pictures in the training set.

[0045] In this embodiment, the original pig face image data is generated by the staff of a cooperative farm through a computer to capture screenshots of the pigs in the pigsty in the video they shot, generating a dataset. Subsequently, the pig face pictures taken at random angles are further cleaned, and the low-quality pig face pictures are removed, such as blurred, largely blocked, and images with only a small part of facial information.

[0046] 2) Improve the Retinaface detection model structure in combination with the actual application and detection requirements to obtain an improved Retinaface detection model, which consists of a backbone feature extraction network, an enhanced feature extraction network and a prediction module. Among them, a channel attention mechanism and a spatial attention mechanism are added to the enhanced feature extraction network to further screen the features and improve the model detection ability. The channel attention mechanism performs global max pooling and global average pooling on the input features respectively, then sends them into two-layer shared fully connected layers respectively, then performs an addition operation on the outputs, and finally passes through a Sigmoid activation operation; the spatial attention mechanism performs global max pooling and global average pooling on the input feature layer based on channels, then performs splicing based on channels, then reduces the dimension to 1 channel through a convolutional layer, and finally passes through a Sigmoid activation; in the prediction module, face classification prediction is used to judge whether there is an object in the prior box, and there are 2 parameters; face box prediction is used to adjust the prior box to obtain the prediction box, and there are 4 parameters; face key point prediction is used to adjust the prior box to obtain the key points, and each key point has two parameters. Here, the Retinaface is improved, and the face key point branch is reduced to 8 1*1 convolutions, and the number of channels is 8 corresponding to the coordinates of 4 key points, and there are 8 parameters.

[0047] The loss function of the improved Retinaface detection model is calculated on three effective feature layers. The loss function of the improved Retinaface detection model consists of three parts: prediction classification loss, prediction face box regression loss, and prediction key point regression loss. The total loss function L1 is shown in formula (1):

[0048] L1 = λ1L cls + λ2L box + λ3L landmark (1)

[0049] where λ1, λ2, λ3 are the weight coefficients of L cls , L box , L landmark respectively.

[0050] where L cls is the face classification loss, specifically the cross-entropy loss, and the formula is shown in (2):

[0051] L cls = -∑P c logQ c (2)

[0052] where P c is the true value of this category, and Q c is the predicted value.

[0053] And L box is the face box loss, and L landmark is the key point loss. It should be noted here that L box and L landmark both specifically use Smooth L1 as the loss function. L box takes the error between the true value and the predicted value of the face box parameters as the input, and L landmark takes the error between the true value and the predicted value of the key point parameters as the input. The formula of Smooth L1 is shown in (3):

[0054]

[0055] where x is the input of this function. In this embodiment, λ1, λ2, λ3 are taken as 1, 2, 1 respectively to balance the above three loss functions.

[0056] In this embodiment, the backbone network adopts Resnet50, which is implemented using the torchvision module in the deep learning framework Pytorch. Based on the transfer learning theory, other backbone networks with good effects can also be used to extract features, such as MobilenetV1, V2, V3, etc. The backbone network outputs feature layers with three different numbers of channels, which are connected to attention modules with the numbers of channels being 512, 1024, and 2048 respectively. After passing through the attention modules, the shape of the feature layers remains unchanged. In this embodiment, the channel attention module uses 1*1 convolution to replace the fully connected layer, and the convolutional kernel size in the spatial attention module must be 3*3 or 7*7.

[0057] For the feature layers after passing through the attention modules, the FPN feature pyramid structure and the SSH single-stage headless network are further used to extract features. The dimensions (width, height, channel) of the three output feature layers are 105*105*256, 53*53*256, and 27*27*256 respectively. SSH1 means dividing the original image into a 105*105 grid; SSH2 means dividing the original image into a 53*53 grid; SSH3 means dividing the original image into a 27*27 grid. There are two prior boxes on each grid, and each prior box represents a certain area on the picture. The number of input and output channels is 256. Then, the three feature layers are sent into three prediction sub-networks to complete sub-tasks, which are the face binary classification task, the face box regression task, and the face key point regression task. The three sub-networks are 1*1 convolutions added after the SSH module, and the output channels are 2, 4, and 8 respectively, and the stride is equal to 1.

[0058] 3) Input the pictures in the training set into the improved Retinaface detection model for training. Adjust the overall proportion of the pictures so that the long side of the picture is 840. If the long side and the short side of the picture are the same, the picture size is 840*840. If the short side is not 840, zero-padding is performed on the short side so that the input picture is 840*840 in size. Load the pre-trained weights, set the training parameters. The training optimizer uses the SGD optimizer, with a momentum of 0.94 and a weight decay of 5*10^-4. Train according to the preset parameters, and use the model parameters with the minimum validation set loss as the model parameters for the improved Retinaface detection, denoted as R.

[0059] In this embodiment, the specific parameters for training the improved Retinaface detection model are as follows: the training batchsize is set to 4, the initial learning rate is set to 0.01, and it is trained for 500 epochs. The parameters can be adjusted according to the specific situation. Save the training parameters every 10 epochs, and select the model parameters with the minimum validation set loss from the saved training parameters.

[0060] 4) The training set images are cropped and subsequently processed according to the annotation information. This subsequent processing is aimed at the more three-dimensional structure of the pig face. In order to allow the network to learn the corresponding features, the present invention uses the self-created judgment criteria to determine whether the image is a pig's front face or side face, such as Figure 3 As shown. The present invention determines whether a face is frontal based on the angle between the line connecting the midpoint of the line between the eyes and the midpoint of the line connecting the two nostrils, and the normal to the line connecting the eyes. If the angle is less than 30 degrees, the face is considered frontal; otherwise, it is considered a profile. (The present invention does not recognize pure profiles where only one eye is visible or no nostrils are completely invisible.) Finally, a frontal face dataset and a profile face dataset are prepared.

[0061] After cropping the image, the side profile remains unchanged. For the frontal face, a two-point correction method is used. Using the pig's eyes as two points, the angle between the line connecting these two points and the horizontal direction is used to determine the rotation angle. This is then combined with the image center to create a rotation matrix. Finally, an affine transformation function is used to rotate the image around its center, straightening the pig's face.

[0062] Assume that the rotation angle is θ and the rotation center is (r x ,r y ), we can get the following rotation matrix M:

[0063]

[0064] In this embodiment, functions in the OpenCV library are used to obtain the rotation matrix and implement the affine transformation.

[0065] 5) Build a Dense_Bilinear CNN pig face recognition model based on practical applications and recognition requirements. This model uses DenseNet121 as the backbone network and a bilinear convolutional neural network structure for feature extraction. This model applies two feature extractors to the input image, performs a matrix outer product on the outputs of the two feature extractors, and pools them into a feature vector. The features then undergo dimensionality reduction and dropout. A combined loss function of triplet loss, center loss, and arcface loss is used.

[0066] During training, features are passed through a linear layer with the same number of categories as the training set. During prediction, features are mapped into 128-dimensional feature vectors. A combined loss function of Triplet loss, Center loss, and Arcface loss is used. The goal is to project various pig faces into a discriminative metric space, making identical pig faces appear closer together and different pig faces appear further apart.

[0067] Among them, the loss functions for implementing Euclidean distance measurement are Triplet loss and Center loss. Then, the classification loss softmax loss is added, and based on the softmax loss, Arcface loss is introduced to measure the angular space. The total loss function L2 can be expressed as formula (5):

[0068] L2 = λ4L Triplet + λ5L Center + λ6L AF (5)

[0069] Among them, L Triplet is the triplet loss, and its specific formula can be expressed as formula (6):

[0070]

[0071] Among them, represents the Euclidean distance between the anchor and the positive, represents the Euclidean distance between the anchor and the negative, and α is an interval distance.

[0072] And L<I center is the center loss, and its specific formula can be expressed as formula (7):

[0073]

[0074] Among them, N is the batch size, x i is the sample feature, and c yi is the corresponding class center.

[0075] And L AF is the additive angular margin loss, and its specific formula can be expressed as formula (8):

[0076]

[0077] Among them, N is the batch size, n is the number of classes, m is the angular margin factor, θ ∈ [0, π - m], x i is the feature of the i-th sample belonging to the y i -th class, and W j is the j-th column of the weight W of the last fully connected layer.

[0078] In this embodiment, λ4, λ5, and λ6 in formula (5) are respectively set to 1, 0.01, and 0.05 to balance the above three loss functions. α in formula (6) is set to 0.5.

[0079] The triplet loss improves the intra-class compactness and inter-class separability in the Euclidean space, that is, samples of the same class are closer, and samples of different classes are more separated. The triplet data is mined online, that is, after extracting the features of the samples, they are divided into three types of samples: anchor, positive, and negative. However, only the triplet loss may result in the intra-class distance being sometimes smaller than the inter-class distance. Adding the Center loss can learn the feature center of each class, making samples of the same class close to the class center and penalizing the Euclidean distance between the feature vector and its corresponding class center. Adding the Triplet loss and the Center loss aims to constrain the sample features in the Euclidean space, enabling the model to learn a mapping relationship. The Arcface loss is derived from the softmax loss, which separates different classes on the training set and increases the inter-class distance and intra-class compactness in the angular space to further improve the discriminative ability of the recognition model.

[0080] 6) Input the pictures of the frontal face dataset and the side face dataset into the Dense_Bilinear CNN pig face recognition model for training respectively. Adjust the input pictures to a size of 224*224, and train according to the preset parameters. Load the pre-trained model. The training optimizer uses the SGD optimizer with a momentum of 0.94 and a weight decay of 5*10^-4. After the training ends, use the model parameters with the minimum validation set loss as the model parameters for Dense_Bilinear CNN pig face recognition respectively, obtaining the model parameters A for the frontal face and the model parameters B for the side face.

[0081] In this embodiment, the specific training parameters of Dense_Bilinear CNN are as follows: the training batchsize is set to 32, the initial learning rate is set to 0.01, and it is trained for 200 epochs. The parameters can be adjusted according to the specific situation. Save the training parameters every 10 epochs, and select the model parameters with the minimum validation set loss from the saved training parameters respectively.

[0082] 7) Perform model cascading according to the model parameters obtained in steps 3) and 6). The specific operation is as follows: Input the input picture into the improved Retinaface detection model loaded with the model parameters R to obtain prediction information. Determine whether it belongs to the frontal face according to the prediction information and the judgment criteria in step 4). If so, input the processed picture into the Dense_Bilinear CNN pig face recognition model loaded with the model parameters A; otherwise, input the processed picture into the Dense_Bilinear CNN pig face recognition model loaded with the model parameters B. Finally, the cascaded model will output the 128-dimensional feature vector of this picture.

[0083] In this embodiment, the improved Retinface detection model can be predicted separately. The test image is input into the improved Retinaface detection model loaded with the model parameter R. After obtaining the prediction result, decoding and adjustment are performed, and non-maximum suppression (Nms) is carried out with an intersection over union (IOU) of 0.5, and the face boxes with a confidence level lower than 0.8 are removed. Finally, the confidence level, face boxes, and key point positions are drawn on the original image. The improved Retinface detection model can also be tested using the test set in the same way. The test metrics include: mean average precision (mAP) for testing the accuracy of face boxes. Since there is only one class in this detection task, the metric is AP; and average normalized precision (NME) for testing the key point error, taking the original Retinaface paper as an example for this metric.

[0084] This embodiment can also test the Dense_Bilinear CNN loaded with model parameters A and B respectively. Using the frontal face dataset and the side face dataset, following the test format of the LFW dataset, 1000 pairs of pig faces are randomly constructed for each of the frontal face dataset and the side face dataset, where 50% are the same pigs and the remaining 50% are different pigs. Tests are carried out with a false acceptance rate (FAR) of 0.01 or 0.001. According to the size of the Euclidean distance and the threshold, test metrics such as the recognition accuracy rate, the optimal threshold, and the true acceptance rate (TAR) of the frontal face and side face data can be obtained.

[0085] 8) In the test set, according to the judgment criteria in step 4), the frontal face and side face of each pig are discriminated, and one is selected as the pig face database respectively. The pictures of each pig in the database also need to obtain the 128-dimensional feature vectors of the frontal face and side face through step 7) respectively. The remaining pictures in the test set are used as test pictures for feature comparison with the pictures in the database. The method of feature comparison is to calculate the features of the test pictures through step 7), and then calculate the Euclidean distance between this feature and the features of each pig in the database. The pig with the smallest Euclidean distance to the test picture in the database is selected, and it is considered that the identity of the test picture corresponds to this pig in the database, that is, the recognition process is completed.

[0086] In this embodiment, the remaining pictures in the test set can be used to predict this cascade system. The prediction effect is: draw the pig face box and key points in the figure, and print the pig face confidence level and the corresponding pig face name in the database, then the pig face recognition can be completed.

[0087] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. A two-stage pig face recognition method based on metric learning, characterized in that: The following steps are involved: 1) Obtain pig face images, screen and annotate the collected images, divide them into training and test sets according to pig face categories, and divide the validation set into validation sets according to the number of images in the training set; 2) The structure of the Retinaface detection model was improved based on practical applications and detection requirements, resulting in an improved Retinaface detection model. The improvements include using channel attention and spatial attention methods to further enhance feature screening during feature extraction; reducing the number of branches in the keypoint prediction branch of the prediction module to a number suitable for the required task; and changing the original model's four-term multi-task loss function to a three-term multi-task loss function consisting of prediction classification loss, predicted facial box regression loss, and predicted keypoint regression loss. 3) Input the images in the training set into the improved Retinaface detection model for training. The model parameters with the smallest loss in the validation set are used as the model parameters for the improved Retinaface detection, denoted as R. 4) The images in the training set are cropped and subsequently processed according to the annotation information. The subsequent processing is targeted at the more three-dimensional structure of the pig face. In order to allow the network to learn the corresponding features, the angle between the line connecting the midpoint of the line between the eyes and the midpoint of the line connecting the two nostrils and the normal line connecting the eyes is used to determine whether it is a frontal face or a profile face. A frontal face dataset and a profile face dataset are prepared; 5) Based on practical applications and recognition requirements, a Dense_Bilinear CNN pig face recognition model was built. This model uses DenseNet121 as the backbone network; a bilinear convolutional neural network structure is used for feature extraction; and a combined loss function of Triplet loss, Center loss, and Arcface loss is used for loss function. 6) Input the images of the front face dataset and the profile face dataset into the Dense_Bilinear CNN pig face recognition model for training respectively. The training is performed according to the preset parameters. The model parameters with the smallest loss in the validation set are used as the model parameters of the Dense_Bilinear CNN pig face recognition, and the model parameters A of the front face and B of the profile face are obtained. 7) Cascading the models according to the model parameters obtained in steps 3) and 6), specifically: inputting the input image into the improved Retinaface detection model loaded with model parameter R to obtain prediction information, and judging whether it belongs to the front face according to the prediction information and the judgment standard of step 4), if it is, inputting the processed image into the Dense_Bilinear CNN pig face recognition model loaded with model parameter A, otherwise, inputting the processed image into the Dense_Bilinear CNN pig face recognition model loaded with model parameter B, and finally the cascade model outputs a 128-dimensional feature vector of the image; 8) In the test set, the front face and side face of each pig are judged according to the judgment criteria of step 4), and one of each is selected as the pig face database. The picture of each pig in the database is also subjected to step 7) to obtain 128-dimensional feature vectors of the front face and side face as the front face and side face features of the pig; the remaining pictures in the test set are used as test pictures and feature comparison is performed with the pictures in the database. The feature comparison method is to calculate the features of the test picture through step 7), and then calculate the Euclidean distance between the features and the features of each pig in the database. The pig in the database with the smallest Euclidean distance to the test picture is selected. The identity of the test picture is considered to correspond to this pig in the database, and the recognition process is completed.

2. A two-stage pig face recognition method based on metric learning according to claim 1, characterized in that: In step 1), the collected pig face images are manually screened, and the cleaned pig face images are labeled using the Labelme tool. The labeling method is the pig face frame and the four key points of the pig face, namely the left eye, right eye, left nostril, and right nostril; the training set and test set are divided according to the pig face category, and the validation set is divided according to the number of images in the training set.

3. A two-stage pig face recognition method based on metric learning according to claim 1, characterized in that: In step 2), the improved Retinaface detection model consists of a backbone feature extraction network, an enhanced feature extraction network and a prediction module; wherein, a channel attention mechanism and a spatial attention mechanism are added to the enhanced feature extraction network to further screen the features and improve the model detection capability; the channel attention mechanism performs global maximum pooling and global average pooling on the input features, and then sends them to the shared fully connected layer of the two layers, and then performs a sum operation on the output, and finally performs a Sigmoid activation operation; the spatial attention mechanism performs global maximum pooling on the input feature layer based on the channel. And global average pooling, then channel-based splicing, and then dimension reduction to 1 channel through the convolution layer, and finally Sigmoid activation; in the prediction module, facial classification prediction is used to determine whether the prior box contains an object, with 2 parameters; facial box prediction is used to adjust the prior box to obtain the predicted box, with 4 parameters; facial key point prediction is used to adjust the prior box to obtain key points, each key point has two parameters. Here, Retinaface is improved, and the facial key point branch is reduced to 8 1*1 convolutions. The number of channels is 8, corresponding to the coordinates of 4 key points, with 8 parameters; The loss function of the improved Retinaface detection model is calculated on three effective feature layers. The loss function of the improved Retinaface detection model includes three parts: prediction classification loss, prediction face box regression loss and prediction key point regression loss. The total loss function L1 is shown in formula (1): L1=λ1L cls +λ2L box +λ3L landmark (1) Where λ1, λ2, and λ3 are L cls 、L box 、L landmark The weight coefficient of L cls is the face classification loss, specifically using the cross entropy loss function, and L box is the face box loss, L landmark is the key point loss; what needs to be explained here is L box and L landmark Specifically, they all use Smooth L1 As the loss function, L box Taking the error between the true value and the predicted value of the facial frame parameter as input, L landmark The error between the true value and the predicted value of the key point parameter is used as input.

4. A two-stage pig face recognition method based on metric learning according to claim 1, characterized in that: In step 3), the images in the training set are input into the improved Retinaface detection model for training. The overall proportion of the images is adjusted so that the long side of the image is 840. If the long side and the short side of the image are the same, the image size is 840*840. If the short side is not 840, the short side is zero-padded to make the input image size 840*840. Set the training parameters, load the pre-trained weights, use the SGD optimizer for training, the momentum is 0.94, and the weight decay is 5*10^-4; After training, the model parameters with the smallest loss in the validation set are used as the model parameters for improving Retinaface detection, denoted as R.

5. A two-stage pig face recognition method based on metric learning according to claim 1, characterized in that: In step 4), given the more three-dimensional structure of the pig face, in order to allow the network to learn the corresponding features, the angle between the line connecting the midpoint of the eye and the midpoint of the nostril line and the normal of the eye line is used to determine whether the face is frontal. If the angle is less than 30 degrees, it is considered a frontal face; otherwise, it is considered a profile face. After cropping the image, the side profile was not processed. For the front face, a two-point correction method was used. The pig's eyes were used as two points. The angle between the line connecting these two points and the horizontal direction was used to obtain the rotation angle. This was then combined with the image center to obtain the rotation matrix. Finally, an affine transformation function was used to rotate the pig's face around the center of the image to straighten it. Assume that the rotation angle is θ and the rotation center is (r x ,r y ), we can get the following rotation matrix M:

6. A two-stage pig face recognition method based on metric learning according to claim 1, characterized in that: In step 5), a Dense_Bilinear CNN pig face recognition model is built in combination with practical applications and recognition requirements. The model uses DenseNet121 on the backbone network; a bilinear convolutional neural network structure is used for feature extraction, that is, two feature extractors are used for the input image, the outputs of the two feature extractors are matrix-outer-producted, and pooled into a feature vector; then the features are subjected to dimensionality reduction and dropout operations; during the training process, the features are finally passed through a linear layer with the number of training set categories; during the prediction process, the features are finally mapped into a 128-dimensional feature vector; in terms of loss function, a joint loss function of Triplet loss, Center loss, and Arcface loss is used, the purpose of which is to project various types of pig faces into a metric space with discriminative ability, so that the same pig faces are closer and different pig faces are farther away; Among them, the loss functions used to implement the Euclidean distance measurement are Triplet loss and Center loss, and then the classification loss softmax loss is added. Based on the softmax loss, Arcface loss is introduced to measure the angle space. The total loss function L2 is expressed as formula (2): L2=λ4L Triplet +λ5L Center +λ6L AF (2) Where, λ4, λ5, and λ6 are L Triplet 、L Center 、L AF The weight coefficient of L Triplet is Triplet loss, that is, triplet loss, and L center is Center loss, and L AF Arcface loss is the additive angular margin loss.

7. A two-stage pig face recognition method based on metric learning according to claim 1, characterized in that: In step 6), the images of the front face data set and the side face data set are respectively input into the Dense_Bilinear CNN pig face recognition model for training, the input images are adjusted to 224*224 size, trained according to the preset parameters, the pre-trained model is loaded, and the training optimizer uses the SGD optimizer with a momentum of 0.94 and a weight decay of 5*10^-4; after the training, the model parameters with the smallest loss in the validation set are respectively used as the model parameters of the Dense_Bilinear CNN pig face recognition, and the model parameters A of the front face and the model parameters B of the side face are obtained.

8. A two-stage pig face recognition method based on metric learning according to claim 1, characterized in that: In step 7), the model parameters obtained in steps 3) and 6) are used to perform a model cascade. Specifically, the input image is fed into the improved Retinaface detection model loaded with the model parameters R to obtain prediction information. In the post-processing of the improved Retinaface detection model, the intersection-over-union ratio of non-maximum suppression is set to 0.5, and face frames with a confidence score lower than 0.8 are discarded. Based on the prediction information and the judgment criteria in step 4), it is determined whether it is a frontal face. If so, the processed image is input into the Dense_Bilinear CNN pig face recognition model loaded with model parameter A. Otherwise, the processed image is input into the Dense_Bilinear CNN pig face recognition model loaded with model parameter B. Finally, the cascade model outputs the 128-dimensional feature vector of the image.

Images