A method for identifying and tracking the behavior of group-reared pigs

Abstract

The present invention discloses a method for identifying and tracking the behavior of group-raised pigs, comprising the following steps: S1: establishing a video data set of group-raised pigs; S2: constructing a YOLOX‑S model; S3: constructing a YOLOX‑S+DeepSORT model, wherein the YOLOX‑S+DeepSORT model comprises a YOLOX‑S model, a Kalman filter and a Hungarian matching algorithm; S4: constructing an improved YOLOX‑S+DeepSORT model, wherein the improved YOLOX‑S+DeepSORT model comprises a YOLOX‑S+DeepSORT model and a DeepSORT optimization algorithm. The present invention can effectively improve the frequent erroneous conversion of IDs, improve the accuracy of the tracker, maintain excellent detection and tracking speed, and realize real-time monitoring and rapid and accurate tracking of multiple types of behaviors of pigs in actual pig farms.

Description

Technical Field

[0001] The present invention relates to the field of pig behavior recognition in agricultural computer vision, and in particular to a method for recognizing and tracking the behavior of group-raised pigs. Background Art

[0002] With the increasing demand for pork in my country, it is of great strategic significance to use computer vision algorithms to conduct video surveillance of group-raised pig farms to prevent diseases and increase pork production. At present, the use of non-contact computer vision systems to automatically monitor the behavior of group-raised pigs is the key to precise livestock and poultry farming management. Before the subclinical and clinical signs of many livestock diseases appear, there are usually changes in the livestock's own behavior. Therefore, daily behavior monitoring of group-raised pigs is particularly important, which can make up for the shortcomings of biosafety measures and help prevent the spread of virulent infectious diseases of pigs and zoonoses. Detecting abnormal behaviors from the behavior recognition results and issuing early warnings can improve pig welfare and breeding income. However, in the natural scene of group-raised pigs, due to the changes in light caused by day and night, the overlap between pigs, and the serious occlusion between pigs, it is difficult to accurately identify various behaviors of pigs in real time in video surveillance. Therefore, developing stable and efficient multi-target tracking technology to achieve long-term tracking of various behaviors of group-raised pigs can provide important information for pig behavior recognition and disease prevention and diagnosis.

[0003] In terms of individual behavior recognition of group-raised pigs, traditional machine learning methods and deep convolutional neural networks are currently mainly used. Publication No. CN111339838A discloses a pig behavior recognition method and system based on information fusion, which uses historical pig behavior data from previous pig activities to train a convolutional neural network; then, obtains current pig behavior data and determines the current pig behavior standard deviation; finally, identifies abnormal pig behavior based on the relationship between the standard deviation and the threshold. Publication No. CN110338092A discloses a pig behavior recognition method and system based on sound, which uses a pig bed body module to sense the sound of pigs, identify the behavior category of pigs represented by the pig sound, and promptly detect abnormal pig behaviors, such as fighting, illness, etc., to remind managers to take targeted intervention measures in a timely manner, and improve the automation and intelligence level of pig farming. Publication No. CN111382674A discloses a method for identifying pigs with aggressive behavior based on visual salience. Use visual salience technology to identify aggressive behaviors of pigs in pig pens. At present, there are very few studies on the use of multi-target tracking technology in video surveillance to achieve rapid and accurate detection of individual pigs and various behavior recognition in group-raising environments. Therefore, designing and implementing a group-raising pig behavior recognition model based on multi-target tracking has important research significance for large-scale pig farming automation. Summary of the invention

[0004] The present invention aims to solve at least one of the problems in the related art to a certain extent. To this end, the purpose of the present invention is to provide a method for identifying and tracking the behavior of group-raised pigs, which can effectively improve the frequent erroneous ID changes, improve the accuracy of the tracker, maintain excellent detection and tracking speed, and realize real-time monitoring and rapid and accurate tracking of multiple types of pig behaviors in actual pig farms.

[0005] In order to achieve the above-mentioned purpose, the present application adopts the following technical solution: a method for identifying and tracking the behavior of group-raised pigs, comprising the following steps:

[0006] S1: Establish a video dataset of group-reared pigs, annotate the behaviors of the pigs in the video dataset into four categories: lying, standing, eating, and others; divide the video dataset into a training set, a validation set, and a test set;

[0007] S2: Build a YOLOX-S model, which includes three parts in sequence: a basic backbone network, a multi-scale intermediate network, and a target detection and positioning and behavior recognition network;

[0008] S3: constructing a YOLOX-S+DeepSORT model, wherein the YOLOX-S+DeepSORT model includes a YOLOX-S model, a Kalman filter, and a Hungarian matching algorithm;

[0009] S4: constructing an improved YOLOX-S+DeepSORT model, wherein the improved YOLOX-S+DeepSORT model includes the YOLOX-S+DeepSORT model and the DeepSORT optimization algorithm.

[0010] Furthermore, step S1 specifically includes:

[0011] S11: Collect video data of group-reared pigs in natural scenes, where the video data includes group-reared pigs in three situations: uneven lighting on the pigs, severe occlusion, and no light at night;

[0012] S12: cutting the acquired group-raising pig video into images, and labeling the images by categories to form the video data set;

[0013] S13: Divide the video dataset into training set, validation set and test set;

[0014] S14: performing Gaussian noise, Gaussian blur, brightness adjustment, saturation adjustment and hue adjustment on the images in the video data set, and incorporating the adjusted images into the video data set.

[0015] Furthermore, step S2 specifically includes:

[0016] S21: The Focus benchmark network, CBS deep learning network structure, CSP deep learning network structure and SPP deep learning network structure are integrated in sequence to build a basic backbone network;

[0017] S22: Adopt feature pyramid and path aggregation network structure to fuse high-level and low-level features, generate feature maps of different scales, and build a multi-scale intermediate network;

[0018] S23: Build a target detection and positioning and behavior recognition network to achieve multi-target detection and classification.

[0019] Furthermore, step S21 specifically includes:

[0020] S211: Input the image in the training set into the Focus benchmark network to obtain the convolution kernel features of the image, and realize the vertical and horizontal interval slicing and splicing function of the image;

[0021] S212: Input the convolution kernel features into the CBS deep learning network structure, the CSP deep learning network structure and the SPP deep learning network structure to obtain feature information, and generate three feature maps at the same time; among them, the CBS deep learning network structure is composed of a convolution layer, a batch normalization layer and an activation function layer, the CSP deep learning network structure includes CSP1_1, CSP1_x and CSP2_x, where x represents the residual structure, and the SPP deep learning network structure performs multi-scale feature fusion.

[0022] Further, step S23 specifically includes:

[0023] S231: construct three decoupled head branches;

[0024] S232: Reshape the three decoupled head branches, then fuse them, and use label assignment to associate the prediction box with the target box;

[0025] S233: Head category loss and target loss are calculated using BCEWithLogitsLoss and CIoU_Loss respectively.

[0026] Furthermore, step S3 specifically includes:

[0027] S31: Kalman filter predicts the target position. The output result of the YOLOX-S model is used as the input value, and the Kalman filter method is used to predict the position of the detected target in the next frame of video;

[0028] S32: Perform data association using the Hungarian matching algorithm, and perform cascade matching and IOU matching fusion in the Hungarian matching algorithm;

[0029] S33: Kalman filter update.

[0030] Furthermore, step S31 specifically includes:

[0031] S311: Initialize the first frame target detection frame of the YOLOX-S model as a tracking frame; wherein the detection frame is represented by trajectory information u, v, r, h, confidence and appearance information, wherein u and v correspond to the target center coordinates, r is the aspect ratio, and h is the height;

[0032] The tracking box uses the state vector x in frame k k Expressed as: x k＝ (P k ,v k ),

[0033] Among them, P k and v k Represent the trajectory position and velocity respectively;

[0034] S312: During the tracking operation, predict x of the kth frame of the i-th tracking frame i,k The information of the next frame includes the mean y i,k+1 With covariance S i,k+1 , where the mean represents the target location information, and the covariance represents the uncertainty of the target location information; the Kalman filter method is used, and the prediction calculation formula is:

[0035]

[0036] Among them, F k+1 is the kinematic coefficient matrix; B k+1 is the external control matrix; u k+1 is the external control quantity; Q k+1 is the covariance matrix of external noise, F k+1 The transpose of F k+1 and The target position is predicted using a linear estimation method.

[0037] Further, step S32 specifically includes:

[0038] S321: The tracking frames are divided into two categories: confirmed tracking frames and unconfirmed tracking frames. The motion matching degree and appearance matching degree are used to measure the cost matrix of whether the current detection frame matches the prediction frame. The calculation formulas of the motion matching degree and the appearance matching degree are:

[0039]

[0040]

[0041] Among them, d (1)(i,j) Refers to the Mahalanobis distance between the jth detection box and the i-th prediction box, d j is the trajectory information u,v,r,h,S of the current detection box i is the covariance matrix input in the prediction phase, y i Represents the mean of the input in the prediction stage; d (2) (i,j) Refers to the minimum cosine value of the appearance eigenvector, r j is the feature of the detection box, r k (i) Refers to the features of the k-th prediction box in the i-th frame, R i Represents the last 100 features of the i-th prediction box;

[0042] S322: Cascade matching, after obtaining the cost matrix based on the appearance information and the Mahalanobis distance, the Hungarian matching algorithm is used on the basis of the cost matrix to obtain the optimal solution, that is, the output is the unmatched track, the unmatched detection box, and the matched track; wherein the calculation formula of the appearance information and the Mahalanobis distance is:

[0043] c i,j =λd (1) (i,j)+(1-λ)d (2) (i,j),

[0044] Among them, λ is a fixed parameter with a value of [0,1];

[0045] S323: IoU matching, taking the unmatched detection boxes and trajectories in the cascade matching and the unconfirmed prediction boxes in the Kalman filter prediction as input, and using the IoU value as the cost matrix, Hungarian matching is performed to further screen out the matching trajectories; the IoU calculation formula is:

[0046]

[0047] Among them, A and B represent the detection box or predicted trajectory box.

[0048] Further, step S33 specifically includes:

[0049] S331: First, calculate the mean error of the detection frame and the tracking frame, and then map the covariance to the detection space; the calculation formula is:

[0050] m=p-Hy,

[0051] N=HSH T +R,

[0052] Among them, p is the mean value of the detection box detected in the k+1 frame, H is called the measurement matrix, y, S are the mean and covariance matrices of the tracking box prediction stage in the kth frame; R is the noise matrix of the detector;

[0053] S332: Calculate the Kalman gain K; the calculation formula is:

[0054] K=SH T N -1 ,

[0055] S333: Update the mean and covariance according to the Kalman filter gain K; the calculation formula is:

[0056] x k+1 =y+Km,

[0057] S k+1 =(I-KH)S,

[0058] Among them, x k+1 and S k+1 The mean and covariance state matrices after the tracking box update for the k+1th frame.

[0059] Furthermore, step S4 specifically includes:

[0060] S41: In the YOLOX-S+DeepSORT model, add the variable classes, and the variable classes=4, which represent the four categories of pigs lying, standing, eating and others;

[0061] S42: construct a one-dimensional array with an array size of n, n=3, to store the number of targets detected by the current detector;

[0062] S43: for the unmatched detection frame obtained after the Hungarian matching algorithm, if the current ID number is less than the average of the one-dimensional array, the trajectory of the detection frame is initialized, otherwise the trajectory is not initialized;

[0063] S44: In case of false detection by the detector, a new track is not created for the falsely detected target according to the change of the array value; if the current target ID number remains the same in multiple consecutive frames, and then the number of targets increases in a certain frame, it is considered that the detector has a false detection and no new ID is assigned to it.

[0064] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art: the present invention improves the data association algorithm on the basic DeepSORT framework, effectively improves the frequent erroneous changes of IDs, improves the accuracy of the tracker, maintains excellent detection and tracking speed, and realizes real-time monitoring and rapid and accurate tracking of multiple types of behaviors of pigs in actual pig farms. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0067] In the attached figure:

[0068] Figure 1 It is a schematic diagram of the process of the method of the present invention;

[0069] Figure 2 It is a schematic diagram of the structure of the YOLOX-S model;

[0070] Figure 3 It is a schematic diagram of the structure of the YOLOX-S+DeepSORT model;

[0071] Figure 4 Schematic diagram for initializing and updating the target quantity array;

[0072] Figure 5 The process of creating new trajectories for false positives and negative detections by the detector;

[0073] Figure 6 This is the process of establishing the YOLOX-S+DeepSORT model;

[0074] Figure 7 This is the directory structure diagram of the YOLOX-S dataset;

[0075] Figure 8 This is the directory structure diagram of the DeepSORT dataset;

[0076] Fig. 9 Schematic diagram of the Reid dataset;

[0077] Fig.10 This is one of the recognition results of the YOLOX-S+DeepSORT model;

[0078] Fig.11 This is another recognition result of the YOLOX-S+DeepSORT model;

[0079] Fig.12 This is one of the recognition results of the improved YOLOX-S+DeepSORT model;

[0080] Fig.13 This is another recognition result of the improved YOLOX-S+DeepSORT model;. DETAILED DESCRIPTION

[0081] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, the specific embodiments of the present invention are now described in detail with reference to the accompanying drawings. In the following description, it should be understood that the directions or positional relationships indicated by "front", "back", "up", "down", "left", "right", "longitudinal", "horizontal", "vertical", "horizontal", "top", "bottom", "inside", "outside", "head", "tail", etc. are based on the directions or positional relationships shown in the accompanying drawings, are constructed and operated in a specific direction, and are only for the convenience of describing the present technical solution, rather than indicating that the mechanism or element referred to must have a specific direction, and therefore cannot be understood as a limitation to the present invention.

[0082] It should also be noted that, unless otherwise clearly specified and limited, the terms such as "installed", "connected", "connected", "fixed", "set" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. When an element is referred to as being "on" or "under" another element, the element can be "directly" or "indirectly" located on the other element, or there may be one or more intermediate elements. The terms "first", "second", "third", etc. are only for the convenience of describing the present technical solution, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first", "second", "third", etc. can explicitly or implicitly include one or more of the features. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0083] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, mechanisms, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0084] Example 1

[0085] As attached Figure 1 As shown, a method for identifying and tracking the behavior of group-raised pigs includes the following steps:

[0086] S1: Establish a video dataset of group-raised pigs, annotate the pigs’ behaviors in the video dataset into four categories: lying, standing, eating, and others; divide the video dataset into a training set, a validation set, and a test set. S1 specifically includes:

[0087] S11: Collect video data of group-raised pigs shot from above in natural scenes, where the video data includes group-raised pigs in three situations: uneven lighting on the pigs, severe occlusion, and no light at night;

[0088] S12: Use FFMPEG to cut the video into pictures, and use DrakLabel to label the pictures to form a video dataset; the labeling here refers to labeling as lying, standing, eating, and one of the other four categories.

[0089] S13: The video dataset is divided into training set, validation set and test set; all three datasets contain daytime and nighttime scenes, crowded and sparse pig scenes, and scenes with frequent and less pig activities.

[0090] S14: Perform Gaussian noise, Gaussian blur, brightness adjustment, saturation adjustment and hue adjustment on the images in the training set, validation set and test set, and merge the adjusted images into the corresponding data sets.

[0091] S2: As attached Figure 2 As shown, the YOLOX-S model is constructed. The YOLOX-S model consists of three parts: the basic backbone network (Backbone), the multi-scale intermediate network (Neck network), and the target detection positioning and behavior recognition network (Predictionhead); S2 specifically includes:

[0092] S21: The Focus benchmark network, CBS deep learning network structure, CSP deep learning network structure and SPP deep learning network structure are integrated in sequence to build a basic backbone network; S21 specifically includes:

[0093] S211: Input the images in the training set into the Focus benchmark network to obtain 32 convolution kernel features (320×320×32) of the input image (640×640×3), and realize the vertical and horizontal slice splicing function of the input image; specifically, during the feature extraction process of the basic backbone network on the data set, the Focus benchmark network expands the input channel four times, that is, the original image size is 640×640×3, and after the Focus structure, it becomes a feature map of 320×320×32.

[0094] S212: Input 32 convolution kernel features into CBS deep learning network structure, CSP deep learning network structure and SPP deep learning network structure to obtain rich feature information, and generate three feature maps of 20×20×512, 40×40×256, and 80×80×128. Among them, the CBS deep learning network structure consists of a convolution layer (Conv), a batch normalization layer (BN) and an activation function layer (SiLU), and the CSP deep learning network structure includes CSP1_1, CSP1_x and CSP2_x, where x represents the residual structure, which is composed of two CBS deep learning network structures fused with several residual structures (Res Unit), as shown in the attached figure. Figure 2 As mentioned above, Res Unit represents the residual structure, which allows the network to be built deeper; add represents tensor addition, which directly adds tensors without expanding the dimension. The SPP deep learning network structure uses the maximum pooling (Maxpool) method of 1×1, 5×5, 9×9, and 13×13 to perform multi-scale feature fusion. Figure 2 As shown in the figure, the three groups of CBL and CSP deep learning network structures perform more in-depth feature extraction on the feature map after the Focus operation. Among them, the CBS deep learning network structure is composed of Conv2D+BN+Leaky relu, and the CSP deep learning network structure splits the feature map into two parts, one part performs convolution operation to extract features, and the other part is directly integrated with the convolution features, respectively obtaining feature maps with feature sizes of 20×20×512 (P5), 40×40×256 (P4), and 80×80×128 (P3).

[0095] S213: To improve the accuracy of the model, the SSP module uses a 1×1 convolution to halve the number of channels for 40×40×256 (P4), and then passes through three maximum pooling layers of different sizes: 5×5, 9×9, and 13×13. Finally, a 1×1 convolution operation is performed to obtain a feature map of 20×20×512 (P5).

[0096] S22: Construct a Neck network, adopt the Feature Pyramid Networks (FPN) and Path Aggregation Networks (PAN) structures, fuse high-level and low-level features, and generate lightweight feature maps of two different scales; S22 specifically includes:

[0097] S221: Based on the feature mapping of the backbone network, FPN uses upsampling operation to fuse deep semantic features with shallow position features to enhance the semantic capabilities of multi-scale features;

[0098] S222: Based on multi-scale semantic features, PAN transmits shallow positioning information to the deep layer, enhances the positioning capability of multi-scale features, and generates three feature maps of 20×20×512, 40×40×256, and 80×80×128 through the Neck network.

[0099] S23: Build a target detection and positioning and behavior recognition network to achieve multi-target detection and classification; S23 specifically includes:

[0100] S231: Construct three decoupled head branches. First, use a 1x1 convolution to reduce the dimension of P3, P4 and P5, and then use two 3x3 convolutions in the following two parallel branches to concat the feature maps of P3, P4 and P5 in the three decoupled branches to form 20×20×9, 40×40×9 and 80×80×9 feature information respectively (where the category is 4, the foreground background discrimination is 1 and the coordinate information of the target box is 4).

[0101] S232: The three feature information in S231 are first activated by the Sigmoid function and then reshaped to obtain 400×9, 1600×9 and 6400×9 feature information respectively, and then concat fusion is performed to obtain 8400×9 prediction information, and the prediction box is associated with the target box, and label assignment is adopted to realize target detection, positioning and classification in an anchor-free manner. Sigmoid refers to the network activation function.

[0102] S233: The head category loss and target loss are calculated using BCEWithLogitsLoss and CIoU_Loss respectively. The BCEWithLogitsLoss formula is as follows:

[0103] Loss(x i ,y i )=-w i [y i log(sigmoid(x i ))+(1-y i )log(1-sigmoid(x i ))]

[0104] Among them, x i Represents the predicted value, y i represents the true value, w i is the weight, and i is the label category.

[0105] The CIoU_Loss function is as follows:

[0106]

[0107] Among them, A and B are two target boxes, IoU = |A∩B| / |A∪B|, and C represents the minimum volume (or area) surrounding A and B.

[0108] S3: Build the YOLOX-S+DeepSORT model. The YOLOX-S+DeepSORT model includes the YOLOX-S model, Kalman filtering and Hungarian matching algorithm. Figure 3 As shown. First, the Kalman filter is used to predict the position of the current object in the next video frame, and then the predicted box and the detection box are matched using the Hungarian matching algorithm, including cascade matching and IoU matching. Finally, after the matching is completed, a new set of trajectories is obtained and the parameters of the Kalman filter are updated. More specifically, step S3 is as follows:

[0109] S31: Kalman filtering predicts the target position, taking the output result of the YOLOX-S model as the input value, and using the Kalman filtering method to predict the position of the detected target in the next frame of video; S31 specifically includes:

[0110] S311: Initialize the first frame target detection frame (Detection) of the YOLOX-S model as a tracking frame (Track); wherein the detection frame is represented by trajectory information u, v, r, h, confidence and appearance information, wherein u and v correspond to the target center coordinates, r is the aspect ratio, and h is the height;

[0111] The tracking box uses the state vector x in frame k k It is expressed as: xk=(P k ,v k ),

[0112] Among them, P k and v k Represent the trajectory position and velocity respectively;

[0113] S312: Use the Hungarian matching algorithm to match the predicted box and the detected box, including cascade matching and IoU matching. In the tracking operation, predict the x of the kth frame of the i-th tracking box. i,k For the information of the next frame, we need to predict the information of the next frame including the mean y under Gaussian distribution. i,k+1 With covariance S i,k+1 , where the mean represents the target location information, and the covariance represents the uncertainty of the target location information; the Kalman filter method is used, and the prediction calculation formula is:

[0114]

[0115] Among them, F k+1 is the kinematic coefficient matrix; B k+1 is the external control matrix; uk+1 is the external control quantity; Q k+1 is the covariance matrix of external noise, F k+1 The transpose of F k+1 and The target position is predicted using a linear estimation method.

[0116] S32: Use the Hungarian matching algorithm to perform data association. To achieve target tracking that has been occluded for a long time, cascade matching and IOU matching fusion are performed in the Hungarian matching algorithm. S32 specifically includes:

[0117] S321: The tracking frames are divided into confirmed tracking frames (Confirmed_tracks) and unconfirmed tracking frames (unconfirmed_tracks). The motion matching degree and appearance matching degree are used to measure the cost matrix of whether the current detection frame matches the prediction frame. The calculation formulas of motion matching degree and appearance matching degree are:

[0118]

[0119]

[0120] Among them, d (1) (i,j) Refers to the Mahalanobis distance between the jth detection box and the i-th prediction box, d j is the trajectory information u,v,r,h,S of the current detection box i is the covariance matrix input in the prediction phase, y i Represents the mean of the input in the prediction stage; d (2) (i,j) Refers to the minimum cosine value of the appearance eigenvector, r j is the feature of the detection box, r k (i) Refers to the features of the k-th prediction box in the i-th frame, R i Represents the last 100 features of the i-th prediction box;

[0121] S322: Cascade matching, after obtaining the cost matrix based on the appearance information and the Mahalanobis distance, the Hungarian matching algorithm is used on the basis of the cost matrix to obtain the optimal solution, that is, the output is the unmatched track, the unmatched detection box, and the matched track; wherein the calculation formula of the appearance information and the Mahalanobis distance is:

[0122] c i,j =λd (1) (i,j)+(1-λ)d (2) (i,j),

[0123] Among them, λ is a fixed parameter with a value of [0,1];

[0124] S323: IoU matching, taking the unmatched detection boxes and trajectories in the cascade matching and the unconfirmed prediction boxes in the Kalman filter prediction as input, and using the IoU value as the cost matrix, Hungarian matching is performed to further screen out the matching trajectories; the IoU calculation formula is:

[0125]

[0126] Among them, A and B represent the detection box or predicted trajectory box.

[0127] S33: Kalman filter update, based on the detection detected in the k+1 frame, correct the state of the track associated with it, including the track position and speed, and update these parameters according to the Kalman filter; S33 specifically includes:

[0128] S331: First, calculate the mean error of the detection frame and the tracking frame, and then map the covariance to the detection space; the calculation formula is:

[0129] m=p-Hy,

[0130] N=HSH T +R,

[0131] Among them, p is the mean value of the detection box detected in the k+1 frame, H is called the measurement matrix, y, S are the mean and covariance matrices of the tracking box prediction stage in the kth frame; R is the noise matrix of the detector;

[0132] S332: Calculate the Kalman gain K; the calculation formula is:

[0133] K=SH T N -1 ,

[0134] S333: Update the mean and covariance according to the Kalman filter gain K; the calculation formula is:

[0135] x k+1 =y+Km,

[0136] S k+1 =(I-KH)S,

[0137] Among them, x k+1 and S k+1 The mean and covariance state matrices after the tracking box update for the k+1th frame.

[0138] S4: Construct an improved YOLOX-S+DeepSORT model. The improved YOLOX-S+DeepSORT model includes the YOLOX-S+DeepSORT model and the DeepSORT optimization algorithm. This application proposes a DeepSORT optimization algorithm that increases behavior recognition and limits the growth of target object IDs in a captive environment. S4 specifically includes:

[0139] S41: In the YOLOX-S+DeepSORT model, add the variable classes, and the variable classes=4, which represent the four categories of pigs lying, standing, eating and others; used for behavioral recognition of group-raised pig tracking;

[0140] S42: Construct a one-dimensional array with an array size of n, n=3, to store the number of targets detected by the current detector; the target number array is initially set to [n, n, n], where n is the number of pigs in the first frame. In subsequent video frames, the number of detected targets in the current video frame is inserted into the end of the array, and the number of targets stored in the head of the array is deleted. The target number array is initialized and updated as follows: Figure 4 shown.

[0141] S43: for the unmatched detection frame obtained after the Hungarian matching algorithm, if the current ID number is less than the average of the one-dimensional array, the trajectory of the detection frame is initialized, otherwise the trajectory is not initialized;

[0142] S44: For the case of false detection by the detector, no new track is created for the falsely detected target according to the change of array value; if the current target ID number remains the same in multiple consecutive frames, and then the number of targets increases in a certain frame, it is considered that the detector has a false detection, and no new ID is assigned to it. The process of creating a new track when the detector falsely detects or misses a detection is as follows Figure 5 shown.

[0143] S5: Use the validation set to evaluate the performance of the improved YOLOX-S+DeepSORT model. S5 is as follows:

[0144] S51: Select the best model from the trained models according to the validation set for testing. When reading the test video, the model will uniformly scale the image size to 640×640.

[0145] S52: Analysis of group-reared pig behavior results based on multi-target tracking on the test video set, including a comparison between the improved DeepSORT tracking model and the basic DeepSORT tracking results in natural scenes.

[0146] In the above scheme, the parameters of the detection network of YOLOX-S are set, including changing the number of network output categories from 81 to 4; the epoch is 200 times, the batch-size is 16, the IoU is 0.45, the confidence is 0.4, the optimizer is SGD, and the training batch is 64; the initial learning rate of the model is set to 0.01; the model uniformly scales the size to 640×640 when reading the training image.

[0147] In the above scheme, in the DeepSORT tracking re-identification experiment, the weight file pre-trained on the public dataset was used to retrain the network on the patent dataset to improve the pig re-identification effect, and the model was iterated 100 times. In order to extract stronger target appearance information, this paper uses 512-dimensional vectors instead of 128-dimensional vectors in the original algorithm.

[0148] Example 2

[0149] More specifically, based on Example 1, the video data set includes overhead video clips of different pig pens during the day and at night, with a resolution of 2688×1520 and a frame rate of 5 frames per second for each video segment. Due to the height and focal length of the overhead camera, other objects other than the pig sty are captured. Therefore, in the experiment, the video cropping method is used to fix the viewing angle to the pig sty to reduce the influence of the external environment.

[0150] After obtaining the video data, the video segments were edited through the FFMPEG program to complete 12 small video segments with a duration of 1 minute and a frame rate of 5 frames per second. The environments of the 12 videos are shown in Table 1. Among the 12 video segments, 4 videos were shot at night, of which 1 video had less than 10 pigs; 8 videos were shot during the day, of which 2 videos had less than 10 pigs. These video segments all contain pigs lying, eating, standing or other series of behaviors to meet the experimental requirements. The multi-target tracking and annotation tool used is DarkLabel software. After all the data annotations are completed, the data is converted into the training set, test set and validation set of YOLOv5-S at a ratio of 6:2:2. The data is converted into DeepSORT tracking data at a ratio of 7:3:1. When conducting experiments, this file needs to be converted through a python language script to meet the data format requirements of the algorithm model. The detection target training data is a COCO JSON file; the target tracking data format is a DeepSORT object image file.

[0151] Table 1 Video environment

[0152]

[0153] In the specific implementation process, the improved DeepSORT tracking algorithm is used to realize group pig behavior recognition and multi-target tracking. The software environment for training the model includes: Windows 10 operating system, Pycharm, Python3.8.8, Pytorch1.7.1, CUDA10.1 and cudnn7.6. The hardware environment includes the processor: CPU: AMD Ryzen 5 2600X; graphics card: GeForce GTX Titan X; installed memory: 64.0GB; disk: 160.0GB; system type: 64-bit operating system.

[0154] In the specific implementation process, the establishment of the tracking model based on the improved DeepSORT mainly involves three steps: data set annotation, network model training, and model verification. Figure 6 As shown in the figure, the model building mainly includes three stages: preparing the data set, building the improved DeepSORT multi-target tracking model and model optimization. First, the 12 video files are annotated and cut, and then the YOLOX-S and DeepSORT data sets are completed according to the annotated files. During the training process, the YOLOX-S and DeepSORT data sets are respectively sent to their respective training networks to obtain the detection model and tracking model. Finally, the detection model, tracking model and their combination are finally optimized.

[0155] The directory structure of the YOLOX-S dataset is as follows Figure 7 As shown in the figure, create a data folder in the root directory of YOLOX-S, which is divided into two folders: labels and images. The txt annotation files and the corresponding images are stored in these two folders. The training set, test set and validation set are further divided in these two folders. The directory structure of the DeepSORT dataset is as follows: Figure 8 As shown, create a data folder in the deep directory under the Deepsort root directory, which is divided into three folders: test, val, and train. The Reid data sets in the test and train folders are as follows Fig. 9 As shown, the left side shows the folder structure, the right side shows the image data in a specific folder, and val contains the test video file and annotation file.

[0156] In this experimental test, the YOLOX-S model was used, and the improved YOLOX-S+DeepSORT model used 7 training videos, 1 verification video and 4 test videos. The test videos are Pig02, Pig10, Pig11 and Pig15, among which the number of pigs in Pig02 is less than 10 and it is a daytime environment, the number of pigs in Pig10, Pig11 and Pig15 is more than 10, among which Pig10 and Pig15 are in a dark environment, and Pig11 is in a daytime environment. MOT16 will be used to evaluate the effect of combining the detection model with the tracking model on real-time video detection and tracking.

[0157] The results of the YOLOX-S model in the test set are shown in Table 2. There are 660 test images and a total of 8047 annotation boxes. The lie, stand, eat, and other annotation boxes account for 56.77% (4568 / 8047), 22.52% (1812 / 8047), 15.63% (1258 / 8047), and 5.08% (409 / 8047), respectively. The accuracy of each category exceeds 97%, the recall rate exceeds 98%, the average accuracy reaches 98.43%, and the average recall rate reaches 99.23%. When the IoU threshold is 0.5, the AP of each category exceeds 98.0%, and the mAP reaches 98.4%. When the IoU threshold is taken every 0.05 from 0.5 to 0.95, the mAP of each category is greater than 83.1%, and the average mAP of the four categories reaches 87.0%. When the input image is 2688×1012 in size and compressed to 640×640 at the input end, the average inference time is 0.017s, which means that 59 frames of images can be processed per second. Therefore, it can run in real time.

[0158] Table 2 Detection results of YOLOX-S model

[0159]

[0160] In the specific implementation process, the test results of the YOLOX-S+DeepSORT model are shown in Table 3. Except for Pig11, the MOTA and MOTP values ​​of the three videos are all over 95%, MT is equal to GT and the ML index is all 0, indicating that there is no lost target in the video tracking process; except for Pig02, the number of pigs in the other three videos is large, resulting in different degrees of reduction in their ID indicators compared with Pig02. From the table, Pig11 has the largest FP, indicating that the model has a high false alarm rate for detection and tracking of Pig11. The index of Pig11 is lower than that of the other three videos. The reason for this is that pigs are more active during the day, which is easy to cause overlapping occlusion. From the environmental analysis of the video, since there are fewer pigs in Pig02, it can be clearly seen that the model's detection and tracking performance is good. However, when the number of pigs increases and the overlapping occlusion is serious, the indicators in various aspects decline significantly, and the IDF1, IDP and IDR decline significantly, indicating that the model cannot track the same object in different video frames, causing the same object to be considered as another new object in different video frames, resulting in incorrect ID changes.

[0161] Table 3 Test results of YOLOX-S+DeepSORT model

[0162]

[0163] In the specific implementation process, the test results of the improved Deepsort model are shown in Table 4. The MOTA values ​​of the four videos are all over 97.1%. Compared with the situation before optimization, Pig11 has an increase of 4.5%, while MOTP is the positioning accuracy of the quantified detector, so it has almost no change in this optimization.

[0164] Table 4 Test results of the improved YOLOX-S+DeepSORT model

[0165]

[0166] In the specific implementation process, some results of the YOLOX-S+DeepSORT model are as follows: Figure 10-11 As shown; some results of the improved YOLOX-S+DeepSORT model are shown Figure 12-13 As shown. Among them, Fig.10 and Fig.12 For the same picture, different models are used for ID labeling. Fig.11 and 13 For the same picture, different models are used for ID labeling. Figure 10-13In the experiment, DeepSORT was optimized for multi-target tracking during the test phase. The pig behaviors detected in the detector were annotated as lying (lie), standing (stand), eating (eat), and other (other) categories and added to the tracker to achieve tracking effects for multiple behavior categories. Fig.10 The maximum ID identified by the YOLOX-S+DeepSORT model is 7. Fig.12 The maximum ID identified by the improved YOLOX-S+DeepSORT model is 7, which means that when the number of pigs is small, the effects of the YOLOX-S+DeepSORT model and the improved YOLOX-S+DeepSORT model are similar, and both can achieve good tracking effects.

[0167] Fig.11 The maximum ID identified by the YOLOX-S+DeepSORT model is 34. Fig.13 The maximum ID identified by the improved YOLOX-S+DeepSORT model is 17, which is quite different from the previous one, and the improved YOLOX-S+DeepSORT model is closer to the real value. This shows that when there are a large number of pigs, as the pigs move frequently, the IDs identified by the YOLOX-S+DeepSORT model increase significantly. This may be because the pigs squeeze each other when they move or eat, resulting in serious occlusion and overlap. When there are a large number of pigs, there is serious occlusion, which causes the pig IDs to change incorrectly, resulting in poor detection and tracking effects. The improved YOLOX-S+DeepSORT model can accurately track multiple targets. From the tracking effect picture, we can more intuitively see the optimization scheme for limiting ID growth, which is significantly improved compared to the detection and tracking of the YOLOX-S+Deepsort model.

[0168] It can be understood that the above embodiments only express the preferred implementation modes of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the patent scope of the present invention. It should be pointed out that, for ordinary technicians in this field, the above technical features can be freely combined without departing from the concept of the present invention, and several deformations and improvements can be made, which all belong to the protection scope of the present invention. Therefore, all equivalent changes and modifications made to the scope of the claims of the present invention should belong to the coverage of the claims of the present invention.

Claims

1. A method for identifying and tracking the behavior of group-raised pigs, characterized in that: The steps include: S1: Establish a video dataset of group-reared pigs, annotate the behaviors of the pigs in the video dataset into four categories: lying, standing, eating, and others; divide the video dataset into a training set, a validation set, and a test set; S2: Build a YOLOX-S model, which includes three parts in sequence: a basic backbone network, a multi-scale intermediate network, and a target detection and positioning and behavior recognition network; S3: constructing a YOLOX-S+DeepSORT model, wherein the YOLOX-S+DeepSORT model includes a YOLOX-S model, a Kalman filter, and a Hungarian matching algorithm; S4: constructing an improved YOLOX-S+DeepSORT model, wherein the improved YOLOX-S+DeepSORT model includes a YOLOX-S+DeepSORT model and a DeepSORT optimization algorithm; Step S2 specifically includes: S21: The Focus benchmark network, CBS deep learning network structure, CSP deep learning network structure and SPP deep learning network structure are integrated in sequence to build a basic backbone network; S22: Adopt feature pyramid and path aggregation network structure to fuse high-level and low-level features, generate feature maps of different scales, and build a multi-scale intermediate network; S23: Build target detection and positioning and behavior recognition networks to achieve multi-target detection and classification; Step S3 specifically includes: S31: Kalman filter predicts the target position. The output result of the YOLOX-S model is used as the input value, and the Kalman filter method is used to predict the position of the detected target in the next frame of video; S32: Perform data association using the Hungarian matching algorithm, and perform cascade matching and IOU matching fusion in the Hungarian matching algorithm; S33: Kalman filter update; Step S4 specifically includes: S41: In the YOLOX-S+DeepSORT model, add the variable classes, and the variable classes=4, which represent the four categories of pigs lying, standing, eating and others; S42: construct a one-dimensional array with an array size of n, n=3, to store the number of targets detected by the current detector; S43: for the unmatched detection frame obtained after the Hungarian matching algorithm, if the current ID number is less than the average of the one-dimensional array, the trajectory of the detection frame is initialized, otherwise the trajectory is not initialized; S44: In case of false detection by the detector, a new track is not created for the falsely detected target according to the change of the array value; if the current target ID number remains the same in multiple consecutive frames, and then the number of targets increases in a certain frame, it is considered that the detector has a false detection and no new ID is assigned to it.

2. The method for identifying and tracking the behavior of group-raised pigs according to claim 1, characterized in that: Step S1 specifically includes: S11: Collect video data of group-reared pigs in natural scenes, where the video data includes group-reared pigs in three situations: uneven lighting on the pigs, severe occlusion, and no light at night; S12: cutting the acquired group-raising pig video into images, and labeling the images by categories to form the video data set; S13: Divide the video dataset into training set, validation set and test set; S14: performing Gaussian noise, Gaussian blur, brightness adjustment, saturation adjustment and hue adjustment on the images in the video data set, and incorporating the adjusted images into the video data set.

3. The method for identifying and tracking the behavior of group-reared pigs according to claim 1, characterized in that: Step S21 specifically includes: S211: Input the image in the training set into the Focus benchmark network to obtain the convolution kernel features of the image, and realize the vertical and horizontal interval slicing and splicing function of the image; S212: Input the convolution kernel features into the CBS deep learning network structure, the CSP deep learning network structure and the SPP deep learning network structure to obtain feature information, and generate three feature maps at the same time; among them, the CBS deep learning network structure is composed of a convolution layer, a batch normalization layer and an activation function layer, the CSP deep learning network structure includes CSP1_1, CSP1_x and CSP2_x, where x represents the residual structure, and the SPP deep learning network structure performs multi-scale feature fusion.

4. The method for identifying and tracking the behavior of group-raised pigs according to claim 1, characterized in that: Step S23 specifically includes: S231: construct three decoupled head branches; S232: Reshape the three decoupled head branches, then fuse them, and use label assignment to associate the prediction box with the target box; S233: Head category loss and target loss are calculated using BCEWithLogitsLoss and CIoU_Loss respectively.

5. The method for identifying and tracking the behavior of group-reared pigs according to claim 1, characterized in that: Step S31 specifically includes: S311: Initialize the first frame target detection frame of the YOLOX-S model as a tracking frame; wherein the detection frame is represented by trajectory information u, v, r, h, confidence and appearance information, wherein u and v correspond to the target center coordinates, r is the aspect ratio, and h is the height; The tracking box uses the state vector x in frame k k Expressed as: x k＝ (P k ,v k ), Among them, P k and v k denote the trajectory position and velocity respectively; S312: During the tracking operation, predict x of the kth frame of the i-th tracking frame i,k The information of the next frame includes the mean y i,k+1 With covariance S i,k+1 , where the mean represents the target location information, and the covariance represents the uncertainty of the target location information; the Kalman filter method is used, and the prediction calculation formula is: Among them, F k+1 is the kinematic coefficient matrix; B k+1 is the external control matrix; u k+1 is the external control quantity; Q k+1 is the covariance matrix of external noise, F k+1 The transpose of F k+1 and The target position is predicted using a linear estimation method.

6. The method for identifying and tracking the behavior of group-reared pigs according to claim 1, characterized in that: Step S32 specifically includes: S321: The tracking frames are divided into two categories: confirmed tracking frames and unconfirmed tracking frames. The motion matching degree and appearance matching degree are used to measure the cost matrix of whether the current detection frame matches the prediction frame. The calculation formulas of the motion matching degree and the appearance matching degree are: Among them, d (1) (i,j) Refers to the Mahalanobis distance between the jth detection box and the i-th prediction box, d j is the trajectory information u,v,r,h,S of the current detection box i is the covariance matrix input in the prediction phase, y i Represents the mean of the input in the prediction stage; d (2) (i,j) Refers to the minimum cosine value of the appearance eigenvector, r j is the feature of the detection box, r k (i) Refers to the features of the k-th prediction box in the i-th frame, R i Represents the last 100 features of the i-th prediction box; S322: Cascade matching, after obtaining the cost matrix based on the appearance information and the Mahalanobis distance, the Hungarian matching algorithm is used on the basis of the cost matrix to obtain the optimal solution, that is, the output is the unmatched track, the unmatched detection box, and the matched track; wherein the calculation formula of the appearance information and the Mahalanobis distance is: c i,j =λd (1) (i,j)+(1-λ)d (2) (i,j), Among them, λ is a fixed parameter with a value of [0,1]; S323: IoU matching, taking the unmatched detection boxes and trajectories in the cascade matching and the unconfirmed prediction boxes in the Kalman filter prediction as input, and using the IoU value as the cost matrix, Hungarian matching is performed to further screen out the matching trajectories; the IoU calculation formula is: Among them, A and B represent the detection box or predicted trajectory box.

7. The method for identifying and tracking the behavior of group-reared pigs according to claim 1, characterized in that: Step S33 specifically includes: S331: First, calculate the mean error of the detection frame and the tracking frame, and then map the covariance to the detection space; the calculation formula is: m=p-Hy, N=MSH T +R, Among them, p is the mean value of the detection box detected in the k+1 frame, H is called the measurement matrix, y, S are the mean and covariance matrices of the tracking box prediction stage in the kth frame; R is the noise matrix of the detector; S332: Calculate the Kalman gain K; the calculation formula is: K=SH T N -1 , S333: Update the mean and covariance according to the Kalman filter gain K; the calculation formula is: x k+1 =y+Km, S k+1 =(I-KH)S, Among them, x k+1 and S k+1 The mean and covariance state matrices after the tracking box update for the k+1th frame.

Images