A pig growth state evaluation method based on deep learning and visual large model

Abstract

The application provides a pig growth state evaluation method based on deep learning and a visual large model SegmentAnything, relates to the technical field of deep learning and the field of the current hot visual large model, and replaces NMS in YOLOv5 with soft-NMS in view of the frequent adhesion and shielding of pigs. Improved YOLOv5 is combined with ByteTrack to track pigs, and then the tracking information of the pigs is innovatively used as a prompt to obtain a frame-by-frame ID mask of the pigs by using the visual large model SegmentAnything. Then, the pig mask is input into a pig growth and health evaluation model to perform binary mask feature extraction, the extracted features are subjected to a mean value operation after passing through a deep regression prediction network, and a pig growth state classification result is obtained, so that whether the pig grows healthily or not is determined. The application has low requirements on equipment, adopts a conventional color camera, has low requirements on computing power, and has low equipment prices, so that the scientificity of pig raising industry decision and the breeding efficiency can be improved well under the premise of low cost.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, target detection, tracking, segmentation and pig farming technology, and more specifically, to a method for evaluating the growth status of pigs based on deep learning and a large visual model. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, the livestock industry is gradually transforming towards large-scale, precision, and intelligent operations. Accurately assessing the health and growth of pigs can significantly improve breeding efficiency, reduce labor costs, and enhance productivity.

[0003] Pig health is a crucial indicator in pig farming management. In particular, issues such as thinness, malnutrition, disability, or growth deformities directly impact pig growth, production efficiency, and pork quality. Traditional manual measurement methods are not only inefficient but also prone to causing stress in pigs, impacting their health and welfare. As pig farms expand, establishing a scientific and efficient pig growth assessment method becomes increasingly important to ensure precise management of pig growth and health.

[0004] In recent years, deep learning technology has made significant progress in the field of computer vision, demonstrating strong performance in tasks such as face recognition and object detection. Subsequently, scholars applied instance segmentation in computer vision to assessing pig growth and health. While existing methods have found some application, most suffer from three major issues: First, in the complex farming environment of pig farms, instance segmentation accuracy is relatively low when pigs are occluded or stuck together; second, simple instance segmentation techniques lack correlation information between video frames and are unable to distinguish different pigs in the video stream; third, the training of existing instance segmentation models relies on a large amount of manual annotation, which is relatively inefficient and uneconomical, limiting their widespread application in actual farming scenarios.

[0005] In order to address the deficiencies in the prior art, the present invention proposes a pig growth and health assessment method based on a large visual model (SegmentAnything). The method uses the YOLOv5-SNMS model based on the improved post-processing of YOLOv5, combines it with the ByteTrack algorithm to track pigs. The pig tracking results are then used as prompts for the large visual model (SegmentAnything), and the SegmentAnything model is used to segment the pig instances to obtain pig binary masks. The method of using the pig tracking frame as a SegmentAnything prompt for instance segmentation can effectively improve the robustness of the instance segmentation model in complex scenarios such as pig occlusion and adhesion. The present invention also independently designs a pig growth and health assessment model based on binary image feature extraction and linear regression network. The model uses the pig binary mask as input, can accurately detect pig body problems (malnutrition, growth deformity), analyze the health status of pigs, optimize the breeding process, and improve production efficiency.

[0006] The present invention improves the YOLOv5 model and combines it with the ByteTrack algorithm, and uses the pig tracking results as a prompt for the large visual model (SegmentAnything), and uses the SegmentAnything model to segment pig instances. A pig growth and health assessment model with pig binary masks as input is independently designed, thereby constructing a pig growth and health assessment method based on Segment Anything. Compared with previous methods, the present invention is the first to use tracking information as a prompt and use the large visual model Segment Anything for instance segmentation. It can track and segment pigs in real time in the video stream, and can also effectively deal with complex scenes where pigs are stuck together or occluded in the pigpen. The pig mask is input into the independently designed pig growth and health assessment model, which can accurately identify pig growth abnormalities and evaluate the growth status of pigs, providing an innovative solution for group-raised pig growth and health assessment. Summary of the Invention

[0007] The present invention aims to provide a pig growth and health assessment method based on Segment Anything. This technology improves the YOLOv5 model and combines it with the ByteTrack algorithm to track pigs. The pig tracking results are then used as prompts for the large visual model (SegmentAnything), and the SegmentAnything model is used to segment the pig instances to obtain pig binary masks. This technology also independently designs a pig growth and health assessment model based on binary image feature extraction and linear regression network. The model uses the pig binary mask as input, which can accurately detect the pig's body problems and analyze the pig's health status.

[0008] A pig growth and health assessment method based on the Segment Anything model is characterized by innovatively combining target tracking, instance segmentation, and pig growth status assessment. This effectively solves the problem of monitoring the health status of individual pigs in complex farming scenarios. The method includes the following steps:

[0009] Step S1: using an online camera device to obtain group pig activity videos at different time periods and densities in a group pig house environment;

[0010] Step S2: Use video frame extraction technology to obtain and organize pig activity images. Label the pigs in the images according to the order of the video stream frames, construct a pig tracking dataset, and divide the dataset into training set, validation set, and test set;

[0011] Step S3: Design a YOLOv5-SNMS-Byte model based on the improved post-processing technology of YOLOv5 for complex scenarios such as pig occlusion and adhesion, so as to track the pigs, use the pig tracking frame and pig ID information as SAM prompts, and build a pig instance segmentation model; wherein, the pig instance segmentation model is built based on SegmentAnything, and the generated pig tracking frame information is input into the SegmentAnything model as a prompt. The SegmentAnything model outputs the corresponding pig mask in each frame according to the assigned pig ID, and performs pig mask binarization processing, and finally outputs the mask with a black fill color;

[0012] Step S4: Use the pig tracking dataset to train the YOLOv5-SNMS-Byte model to obtain the optimal weight parameters of the YOLOv5-SNMS-Byte model;

[0013] Step S5: Design a pig growth status assessment model based on binary mask feature extraction and linear regression network.

[0014] Step S6: Based on the video collected in S1, a pig growth status dataset is constructed, and the pig growth status assessment model in S5 is trained to obtain the optimal weight parameters of the pig growth status assessment model;

[0015] Step S7: Input the video into the pig instance segmentation model constructed in step S3, input the frame-by-frame and ID-by-ID masks of the pigs output by the model into the pig growth status assessment model of the optimal weight model constructed in step S6, and output the growth status classification results of each pig (normal, deformed, malnourished).

[0016] Furthermore, in step S1, a Hikvision network camera (model: DS-IPC-T13HV3-IA) was used to record real-time video. The camera resolution was set to 1920×1080 and the video frame rate was 24fps. A Python script controlled the camera to record pig activity during the following time periods: 5:00–9:00 AM, 11:00–15:00 PM, 5:00 PM–21:00 PM, and 11:00 PM–3:00 AM each day. The recorded videos were organized and saved as pig activity data videos for subsequent analysis.

[0017] Furthermore, in step S2, a pig tracking dataset is constructed by slicing the video captured in step S1 every 10 frames to obtain a set of pig detection images. The pigs in the images are labeled using LabelImg. The annotated files are named with a .txt extension, and the filenames match the image names. This results in a pig tracking dataset. The pig tracking dataset is then divided into a training set, a test set, and a validation set in a ratio of 7:2:1.

[0018] Furthermore, in step S3, an improved version of YOLOv5-Byte is used to detect and track pigs. The improved version of YOLOv5-Byte replaces the non-maximum suppression (NMS) in the original YOLOv5 with soft non-maximum suppression (soft-NMS). Soft-NMS reduces the chance of mistakenly deleting correct bounding boxes by gradually reducing the scores of overlapping boxes instead of directly deleting them. It is particularly suitable for scenes with densely packed targets or blurred boundaries. Specifically, the following steps are included:

[0019] S3-1. Input the video stream frame by frame into the feature extraction network of YOLOv5 to extract multi-scale feature maps; generate high-dimensional feature maps through several convolutional layers, pooling layers and feature fusion operations in the network;

[0020] S3-2. Based on the feature map, the network performs regression on the predefined candidate boxes, adjusting their positions and sizes to more accurately cover the target area. In this process, candidate boxes for pig detection and their corresponding classification scores are generated, representing the confidence and category probability of the target in the candidate boxes.

[0021] S3-3. When screening candidate frames for pig detection, the traditional non-maximum suppression (NMS) is replaced with Soft-NMS. The approach is as follows: for each candidate frame, sort it from high to low according to the candidate frame score; traverse each candidate frame and dynamically adjust the scores of other candidate frames; filter the candidate frames with scores below the threshold, and retain the detection frames with higher scores and better spatial distribution; finally, the candidate frame screening will retain the frame with the highest classification score to generate the final pig detection frame; the detection frame contains information such as the target category, classification probability, and the specific location coordinates of the bounding box;

[0022] Unlike traditional NMS, soft-NMS considers not only the degree of overlap of predicted boxes but also their scores. When two high-scoring boxes overlap significantly, Soft-NMS does not directly delete the box with the lower score. Instead, it attenuates its score. For each detected object i, its intersection over union (IoU) (i, j) with other objects j is calculated. This IoU is then used to calculate the attenuation exponent σ(i, j), where λ is a hyperparameter controlling the strength of the attenuation. The confidence score (score(i)) of object i is then adjusted based on the attenuation factor. All objects are then re-ranked based on the adjusted score (score'(i). This process is repeated until the confidence score falls below a certain threshold or the maximum number of detections is reached. In Soft-NMS, when two overlapping boxes are detected, the box with the highest score is retained, and the score of the other box is attenuated. If the attenuated score remains high, it is given higher priority in the next comparison, preserving more valuable information. This improves robustness in complex scenarios such as pig occlusion and adhesion. The formulas for σ(i,j) and score′(i) are defined in formula (1) and formula (2) respectively.

[0023] σ(i,j)=e -λ×IoU(i,j) (1)

[0024]

[0025] S3-4. The pig detection bounding boxes obtained in step S3-3 are passed as input to the ByteTrack algorithm to achieve cross-frame association and tracking of each target. The process is as follows: Detections are first divided into high-scoring detections (High) and low-scoring detections (Low). A Kalman filter is then used to predict the new position of each track in the Tracker within the current frame. The first association is achieved by matching high-scoring detections (High) with the 20 track sensors in the Tracker. Unmatched detections and tracks are then assigned to Dremain and Tremain, respectively. A second association is performed between low-scoring detections (Low) and unmatched tracks in Tremain. After these two association steps, any remaining unmatched detections are considered background and are therefore deleted. Unmatched tracks (Tre-remain) after the second association are retained for a predefined number of frames, typically 30, before being discarded. Finally, unmatched high-scoring detections in Dremain are initialized as new tracks. This two-layer association process not only optimizes the matching of tracks and detections, but also achieves excellent target tracking performance.

[0026] The combination of soft-NMS and ByteTrack significantly improves the robustness of detection in complex occlusion scenarios. Traditional NMS adopts a more extreme suppression strategy for adjacent frames whose intersection over union (IoU) exceeds the threshold. This can easily lead to the mistaken deletion of valid detection frames due to the dense adhesion of pigs, resulting in missed detections. Soft-NMS, on the other hand, uses a linear attenuation formula to score low-score frames based on IoU and hyperparameters instead of directly eliminating them. This not only retains the potential for effective detection of occluded targets, but also avoids excessive suppression of high-score frames. This mechanism allows partially occluded pigs to participate in track matching in the subsequent two-layer matching of ByteTrack (high-score detection - initial track association, low-score detection - secondary association of residual tracks) even if their scores are temporarily reduced, effectively alleviating the problem of ID jumps caused by occlusion. At the same time, detection frames that still maintain a high score after attenuation are given priority in the next round of matching after reordering, forming a "detection-tracking" collaborative optimization - the detection stage retains more clues of occluded targets, and the tracking stage maintains trajectory continuity through Kalman prediction and multi-level association. Ultimately, more stable target detection and ID tracking are achieved in scenarios with dense pig populations and frequent occlusions, providing better tracking information for subsequent instance segmentation based on tracking frame prompts.

[0027] S3-5. The pig tracking bounding box obtained by the ByteTrack algorithm is used as the prompt information and input into the visual large model Segment-Anything. The tracking box generated by the pig tracking algorithm is used as the input prompt. Each box is represented by the upper left corner coordinates (x1, y1) and the lower right corner coordinates (x2, y2), containing the pig's ROI and corresponding ID information. The prompt decoder converts these prompt information (x1, y1, x2, y2) and ID into an embedded vector representation and uses the Transformer module to further enhance its representation capability. At the same time, the image is input to the image encoder based on the Swin Transformer to extract local and global features; the mask decoder extracts contextual information from the embedded vector representation of the prompt information and the image features, and uses the cross-attention mechanism to combine the image features extracted by the image encoder with the prompt features output by the prompt encoder to gradually generate a target mask that matches the input prompt. The specific process is as follows:

[0028] S3-5-1. Use the tracking boxes generated by the pig tracking algorithm as input hints. Each box is represented by the upper left corner coordinates (x1, y1) and the lower right corner coordinates (x2, y2), and contains the pig's ROI and corresponding ID information. This information is input as a hint to the hint encoder of SegmentAnything. The image is input to the image encoder based on Swin Transformer;

[0029] S3-5-2. The image encoder performs hierarchical processing on the input image, extracts multi-scale features using an encoder based on Swin Transformer, and finally outputs the image multi-scale features F = {F1, F2, F3, F4} (corresponding to different resolutions respectively).

[0030] S3-5-3, prompt encoder to enhance feature representation of anchor box prompt (x1, y1, x2, y2) w = x2-x1, h = y2-y1, and use sinusoidal position encoding to map the geometric features to high-dimensional space to obtain the embedded vector representation PE of the anchor box pos =MLP(concat(center x ,center y ,w,h)). Assign a learnable embedding vector (similar to word embedding) to each unique ID, map the ID to a fixed-dimensional semantic space through the Lookup Table, and obtain the embedding vector representation of the ID. ID =EmbeddingLayer(ID). After concatenating the position encoding and the ID embedding, they are fused through a lightweight Transformer to obtain the tracking information embedding vector representation of the concatenated two: TrackingEmbed = Transformer(concat(PEpos,EmbedID)).

[0031] S3-5-4. In the mask decoder, the tracking information embedding vector TrackingEmbed is used as the query, and the image multi-scale feature F = {F1, F2, F3, F4} is used as the key. Cross attention is performed on feature maps of different resolutions, and the final mask prediction result M = CrossAttention (TrackingEmbed, F) is generated through multi-scale fusion.

[0032] A learnable gating mechanism (GatingNetwork) is introduced to distribute weights in the S3-5-4 mask decoder. This mechanism fuses the tracking embedding and image features through a multi-layer perceptron (MLP) and adaptively adjusts the fusion weights. This dynamic weight distribution mechanism adaptively adjusts the contribution ratio of the tracking embedding and image features based on the target's characteristics and context, further improving the feature fusion effect and enhancing the adaptability and accuracy of mask prediction.

[0033] Furthermore, in step S4, the YOLOv5-SNMS-Byte model is trained using the pig tracking dataset to obtain the optimal weight parameters of the YOLOv5-SNMS-Byte model. The specific process is as follows:

[0034] S4-1, the image is preprocessed through the input end, including mosaic data enhancement and adaptive image scaling;

[0035] S4-2, the preprocessed image enters the backbone network of YOLOv5-SNMS, which consists of Focus, CSP and other modules, and obtains three feature maps of different sizes through the channel adjustment module;

[0036] S4-3, inputting the three feature maps of different sizes obtained in step S4-2 into the Neck structure composed of FPN and PAN to obtain feature maps of three scales to detect small, medium and large-sized targets respectively;

[0037] S4-4, inputting the feature maps of the three scales obtained in step S4-3 into the detection head, and performing object detection on the feature maps of each scale, including predicting information such as the position, category, and confidence of the bounding box;

[0038] S4-5. Input the predicted bounding box obtained in step S4-4 into ByteTrack, perform detection box matching, and assign a detection box ID.

[0039] S4-6. When training YOLOv5-SNMS-ByteTrack, the detection part mainly evaluates the detection accuracy of the model through indicators such as mAP, Precision, Recall and F1 Score on the validation set; the tracking part evaluates the tracking accuracy of the model through indicators such as MOTA, IDF1 and the number of identity switches, and further optimizes the overall system performance by debugging the matching threshold and Kalman filter parameters.

[0040] Furthermore, the pig growth status assessment model based on the binary image feature extraction network and the linear regression network in step S5 has the pig binary mask obtained in S3 as input and the pig growth status classification result as output. The model extracts features from the pig mask generated in S3 and calculates the growth status classification result using a deep regression prediction network. OpenCV is used to analyze the binarized pig mask to extract key features including the total number of black pixels, mask boundary, minimum bounding rectangle, body size ratio, and morphological symmetry. The specific steps are as follows:

[0041] S5-1. Calculate the total number of black pixels in the pig mask to reflect the overall size A of the pig, thereby determining whether the pig is too thin;

[0042] S5-2, calculate the pig mask boundary and obtain the perimeter P reflecting the complexity of the shape. This is used to determine whether the pig has a physical disability;

[0043] S5-3, calculating the minimum circumscribed rectangle of the pig mask and calculating the aspect ratio AR, thereby determining whether the pig is too thin;

[0044] S5-4, calculating the body size ratio C of the pig mask to determine whether the pig has growth deformity;

[0045] S5-5. Calculate the deviation CD between the pig mask mass center and the theoretical symmetry axis, analyze the morphological symmetry, and thus determine whether the pig has growth deformity;

[0046] These features are then fed into a deep regression prediction network, which consists of multiple linear layers and uses the ReLU activation function to enhance model stability and convergence, while also incorporating a dropout mechanism to prevent overfitting. Finally, the network outputs the predicted value through a softmax layer, ultimately outputting the pig's growth status.

[0047] Furthermore, in step S6, the pig growth status assessment model is trained, and the data set construction and training method is as follows:

[0048] S6-1. First, prepare multiple pig images. These images include pigs with normal growth and pigs with disabilities, deformities, and malnutrition, with a 1:1 ratio. Label them according to the steps for preparing the pig tracking dataset in S2. Then, use the labeling results as prompts for Segment Anything to perform instance segmentation. Save the generated binary mask.

[0049] S6-2. Depending on whether the pigs are growing normally or not, their masks are stored in three folders: "normal", "deformed", and "malnourished". The ratio of the three masks is 1:1:1. The dataset is divided into training set, test set, and validation set in a ratio of 7:2:1.

[0050] S6-3, inputting the pig binary mask obtained in S6-1 into the binary feature network of the pig growth and health assessment model to obtain the features of the pig binary mask;

[0051] S6-4, inputting the features of the pig binary mask obtained in step S6-3 into the linear regression layer, and finally outputting classification information of whether the pig has growth abnormality;

[0052] S6-5. During the training phase, the predicted categories are first matched with the true categories to determine the attribution of positive and negative samples. Then, the weight parameters are adjusted by calculating the loss function. At the end of each iteration, the validation set is used to calculate the accuracy and average precision to continuously optimize the model parameters.

[0053] Furthermore, the linear regression part of the pig growth and health assessment model was set to have a maximum number of training iterations (epochs) of 50, an input binary mask image size of 320×320, and a batch size of 8 images per model input. The model learning rate was 0.0001.

[0054] Furthermore, during the training phase of the model, the weight parameters are adjusted by the cross entropy loss and the mean square error loss function. Where y is the true label (0 or 1), y' is the probability predicted by the model (between 0 and 1), m is the number of samples, and y i is the true value of the i-th sample, y i * is the predicted value of the i-th sample, and θ is the model parameter (including weight and bias). Its calculation formula is as shown in formula (3)-(4):

[0055] L(y,y')=-(ylog(y')+(1-y)log(1-y')) (3)

[0056]

[0057] Beneficial effects of the present invention:

[0058] 1. This paper introduces the SegmentAnything model into the field of group-housed pig instance segmentation for the first time, constructing a pig instance segmentation method based on SegmentAnything supplemented by tracking prompts. This not only improves the accuracy of pig instance segmentation, but also solves the pain point of existing methods that require a large amount of manual labeling. It provides a new solution for group-housed pig instance segmentation;

[0059] 2. The present invention independently designs a pig growth and health assessment model based on binary image feature extraction and deep regression network, which can give full play to the advantages of pig masks in pig posture analysis, accurately identify pig growth abnormalities, and evaluate the growth status of pigs, providing an innovative solution for the growth and health assessment of group-raised pigs. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is an implementation roadmap of the pig growth and health assessment system of the present invention;

[0061] Figure 2 This is a schematic diagram of a pig being photographed by a camera of the present invention;

[0062] Figure 3 This is a flow chart of the pig growth health assessment of the present invention;

[0063] Figure 4 This is a pig instance segmentation flow chart based on SegmentAnything of the present invention;

[0064] Figure 5 This is a network structure diagram of the pig growth and health model of the present invention. DETAILED DESCRIPTION

[0065] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0066] like Figure 1 As shown in the figure, the present invention provides a complete implementation route for the pig growth and health assessment system based on Segment Anything. This process covers steps such as data acquisition, target tracking, target segmentation, feature modeling, supervised learning, and model evaluation. Based on deep learning and computer vision technology, it realizes intelligent assessment of the growth status of group-raised pigs. The ordinary plane image captured by the pigpen surveillance camera in this example has a resolution of 1080p. The camera is assumed to be as shown in the figure below. Figure 2 As shown, it is set up on the side of the pig pen, 2.5m above the ground, to ensure that the pigs in the pig pen are fully photographed.

[0067] like Figure 3 As shown, the present invention provides a pig growth and health assessment method based on Segment Anything. This method improves the YOLOv5 model, combines it with the ByteTrack algorithm, and tracks pigs. The pig tracking results are then used as prompts for the large visual model (SegmentAnything), and the SegmentAnything model is used to segment the pig instances to obtain pig binary masks. The present invention also independently designs a pig growth and health assessment model based on binary image feature extraction and linear regression network. The model uses the pig binary mask as input, and can accurately detect the pig's body problems (such as thin body, malnutrition, disability and growth deformity, etc.), thereby analyzing the health status of the pig.

[0068] like Figure 3As shown in blue in the figure, the pig tracking portion of this method utilizes an improved version of the YOLOv5-Byte algorithm. The improved version of YOLOv5-Byte replaces the non-maximum suppression (NMS) in the original YOLOv5 with soft non-maximum suppression (soft-NMS). Soft-NMS considers not only the degree of overlap of predicted boxes but also their scores. When two high-scoring boxes overlap significantly, Soft-NMS does not directly delete the box with the lower score, but instead attenuates its score. For each detected object i, its intersection over union (IoU) (i, j) with other objects j is calculated. The IoU is then used to calculate the attenuation exponent σ(i, j), where λ is a hyperparameter controlling the strength of the attenuation. The confidence score (score(i)) of object i is then adjusted based on the attenuation factor. All objects are then re-ranked based on the adjusted score (score′(i). This process is repeated until the confidence score falls below a threshold or the maximum number of detections is reached. With Soft-NMS, when two overlapping boxes are detected, the box with the highest score is retained, and the score of the other box is attenuated. If the score after attenuation is still high, the box will still be given a higher priority in the next round of comparison, thereby retaining more valuable information. This improves the robustness in complex scenes such as pig occlusion and adhesion.

[0069] The improved version of YOLOv5-Byte uses the ByteTrack matching algorithm. The ByteTrack algorithm first classifies detections into high-scoring detections (High) and low-scoring detections (Low). It then uses a Kalman filter to predict the new position of each track in the Tracker within the current frame. The first association is achieved by matching high-scoring detections (High) with the 20 track sensors in the Tracker. Unmatched detections and tracks are then assigned to Dremain and Tremain, respectively. A second association is performed between low-scoring detections (Low) and unmatched tracks in Tremain. After these two association steps, any remaining unmatched detections are considered background and are removed. Unmatched tracks (Tre-remain) after the second association are retained for a predefined number of frames, typically 30, before being discarded. Finally, unmatched high-scoring detections in Dremain are initialized as new tracks. This two-layer association process not only optimizes the matching of tracks to detections but also significantly improves overall tracking performance.

[0070] The combination of soft-NMS and ByteTrack significantly improves the robustness of detection in complex occlusion scenarios. Traditional NMS adopts a more extreme suppression strategy for adjacent frames whose intersection over union (IoU) exceeds the threshold. This can easily lead to the mistaken deletion of valid detection frames due to the dense adhesion of pigs, resulting in missed detections. Soft-NMS, on the other hand, uses a linear attenuation formula to score low-score frames based on IoU and hyperparameters instead of directly eliminating them. This not only retains the potential for effective detection of occluded targets, but also avoids excessive suppression of high-score frames. This mechanism allows partially occluded pigs to participate in track matching in the subsequent two-layer matching of ByteTrack (high-score detection-primary track association, low-score detection-secondary association of residual tracks) even if their scores are temporarily reduced, effectively alleviating the problem of ID jumps caused by occlusion. At the same time, detection frames that still maintain a high score after attenuation are given priority in the next round of matching after reordering, forming a "detection-tracking" collaborative optimization - the detection stage retains more clues of occluded targets, and the tracking stage maintains trajectory continuity through Kalman prediction and multi-level association. Ultimately, more stable target detection and ID tracking are achieved in scenarios with dense pig populations and frequent occlusions, providing better tracking information for subsequent instance segmentation based on tracking frame prompts.

[0071] like Figure 3 As shown in the yellow part of the figure, the pig instance segmentation method adopts an instance segmentation method based on SegmentAnything supplemented by tracking information prompts. The specific implementation details are as follows Figure 4 shown.

[0072] First, the tracking box generated by the pig tracking algorithm is used as the input prompt. The bounding box coordinates (x1, y1, x2, y2) are normalized to a proportional value relative to the image size to eliminate resolution differences. The geometric properties of the bounding box (center point coordinates, width, height, and area) are calculated to enhance the spatial representation, and then the geometric features are mapped to a high-dimensional space through sinusoidal position encoding (or learnable position encoding). For pig IDs, a learnable embedding vector (similar to word embedding) is assigned to each ID, and the ID is mapped to a fixed-dimensional semantic space through a lookup table. Finally, the position encoding and ID embedding are concatenated and fused through a fully connected layer to obtain an embedded vector representation of the tracking prompt information.

[0073] At the same time, the image is input into the Swin Transformer-based image encoder to extract local and global features.

[0074] During the mask generation phase, the tracking embedding serves as the query, and the multi-scale feature maps (Fi) output by the image encoder serve as the key and value. Cross-attention is used to achieve feature alignment. At different image scales, cross-attention operations are performed on feature maps of different resolutions through a hierarchical interaction to generate the final mask prediction result.

[0075] In terms of weight distribution for mask generation, a learnable gating mechanism (Gating Network) is introduced to adaptively adjust the fusion weights of tracking embedding and image features, thereby optimizing the quality of the generated mask.

[0076] like Figure 3 As shown in the green part of the figure, the pig growth and health assessment model is designed based on a binary image feature extraction network and a linear regression network. The specific implementation details are as follows Figure 5 As shown in Figure 3, its input is the pig binary mask obtained in S3, and its output is the pig growth status classification result. The model extracts features from the pig mask generated in S3 and uses a deep regression prediction network to predict the pig's growth status. OpenCV is used to analyze the binarized pig mask, extracting key features such as the total number of black pixels, mask boundary, minimum bounding rectangle, body size ratio, and morphological symmetry. These features are then input into the deep regression prediction network, which consists of multiple linear layers and uses the ReLU activation function to enhance model stability and convergence. A dropout mechanism is also introduced to prevent overfitting. Finally, the predicted value output by the network passes through a softmax layer, outputting the pig's growth status classification result.

[0077] This paper presents a pig tracking dataset and a pig growth status assessment dataset based on pigpen video streams. The pig tracking dataset is a self-built dataset based on LabelImg software. It contains 980 video atlases of pig pens with a resolution of 1920×1080, different lighting conditions, and different density levels. The pig growth status assessment dataset contains 480 images of pigs in different growth conditions, which are divided into three categories: normal, deformed, and malnourished, with a ratio of 1:1:1.

[0078] The present invention uses the Pytorch1.9.0 deep learning framework and Python 3.8. The hardware environment is an Intel Core i7-13700K processor, an NVIDIA GeForce RTX 4090 24GB graphics card, and the operating system is Windows 10. After the environment is deployed, the pig tracking model is trained. During the experiment, the SGD optimizer is used with an initial learning rate of 0.0001, an SGD optimizer momentum of 0.937, a weight decay of 0.0005, 100 iterations, and a batch size of 12. After the pig tracking model is trained, the pig health assessment model is trained. During the training process, the SGD optimizer is used with an initial learning rate of 0.0001, an SGD optimizer momentum of 0.937, a weight decay of 0.0005, 50 iterations, and a batch size of 8.

[0079] The foregoing is an example of the best mode of carrying out the present invention. Any portion not described in detail herein is common knowledge within the skill of one of ordinary skill in the art. The scope of protection of the present invention is determined by the claims. Any equivalent transformation based on the technical teachings of the present invention is also within the scope of protection of the present invention.

Claims

1. A method for assessing pig growth status based on deep learning and large-scale visual models, characterized in that: Combining object tracking, instance segmentation, and pig growth status assessment to achieve individual pig health status monitoring in complex farming scenarios includes the following steps: Step S1: using an online camera device to obtain group pig activity videos at different time periods and densities in a group pig house environment; Step S2: Using video frame extraction technology, obtain and organize pig activity pictures; label the pigs in the pictures according to the order of the video stream picture frames, construct a pig tracking dataset, and divide the dataset into a training set, a validation set, and a test set; Step S3: Design a YOLOv5-SNMS-Byte model based on the improved post-processing technology of YOLOv5 for complex scenes with pig occlusion and adhesion, so as to track the pigs. Use the pig tracking frame and pig ID information as SAM prompts, build a pig instance segmentation model, and generate a binary mask that matches the input prompt. Step S4: Use the pig tracking dataset to train the YOLOv5-SNMS-Byte model to obtain the optimal weight parameters of the YOLOv5-SNMS-Byte model; Step S5: designing a pig growth status assessment model based on binary mask feature extraction and linear regression network; Step S6: Based on the video collected in S1, a pig growth status dataset is constructed, and the pig growth status assessment model in S5 is trained to obtain the optimal weight parameters of the pig growth status assessment model; Step S7: Input the video into the pig instance segmentation model constructed in step S3, input the frame-by-frame and ID-by-ID masks output by the model into the pig growth status assessment model of the optimal weight model constructed in step S6, and output the growth status classification results of each pig, including normal, deformed, and malnourished; The specific steps in step S3 include: S3-1. Input the video stream frame by frame into the YOLOv5 feature extraction network to extract multi-scale feature maps; generate high-dimensional feature maps through several convolutional layers, pooling layers and feature fusion operations in the network; S3-2. Based on the feature map, the network performs regression on the predefined candidate boxes, adjusting their positions and sizes to more accurately cover the target area. In this process, candidate boxes for pig detection and their corresponding classification scores are generated, representing the confidence and category probability of the target in the candidate boxes. S3-3. When screening candidate frames for pig detection, the traditional non-maximum suppression (NMS) is replaced with Soft-NMS. The approach is as follows: for each candidate frame, sort it from high to low according to the candidate frame score; traverse each candidate frame and dynamically adjust the scores of other candidate frames; filter the candidate frames with scores below the threshold, and retain the detection frames with higher scores and better spatial distribution; finally, the candidate frame screening will retain the frame with the highest classification score to generate the final pig detection frame; the detection frame contains information such as the target category, classification probability, and the specific location coordinates of the bounding box; S3-4. The pig detection box obtained in step S3-3 is passed as input to the ByteTrack algorithm to achieve cross-frame association and tracking of each target; the process is: first, the detection is divided into high-score detection High and low-score detection Low; then, the Kalman filter is used to predict the new position of each track in the Tracker in the current frame; the first association is achieved by matching the high-score detection High with the 20 track sensors in the Tracker; then the unmatched detections and tracks are assigned to the detection box set Dremain with higher confidence and the detection box set Tremain with lower confidence respectively; a second association is performed between the low-score detection Low and the unmatched tracks in Tremain; after these two association steps, any remaining unmatched detections are regarded as background and are therefore deleted; as for the unmatched tracks Tre-remain after the second association, they are retained for a predefined number of frames, usually 30 frames, and then discarded; finally, the unmatched high-score detections in Dremain are initialized as new tracks; S3-5. The pig tracking bounding box obtained by the ByteTrack algorithm is used as the prompt information and input into the visual large model Segment-Anything. The tracking box generated by the pig tracking algorithm is used as the input prompt. Each box is represented by the upper left corner coordinates (x1, y1) and the lower right corner coordinates (x2, y2), containing the pig's ROI and corresponding ID information. The prompt decoder converts these prompt information (x1, y1, x2, y2) and ID into embedded vector representations, and uses the Transformer module to further enhance its representation ability. At the same time, the image is input into the image encoder based on Swin Transformer to extract local and global features; the mask decoder extracts contextual information from the embedded vector representation of the prompt information and the image features, and uses the cross-attention mechanism to combine the image features extracted by the image encoder with the prompt features output by the prompt encoder to gradually generate a target mask that matches the input prompt.

2. The pig growth status assessment method based on deep learning and visual large model according to claim 1 is characterized in that: In step S1, a Hikvision DS-IPC-T13HV3-IA online camera is installed in a group pig house environment, and a python script is used to record the pig activities in four time periods of 5:00-9:00, 11:00-15:00, 17:00-21:00, and 23:00-3:00 every day in real time, and the data is compiled and saved as pig activity data videos.

3. The pig growth status assessment method based on deep learning and visual large model according to claim 1 is characterized in that: In step S2, representative pig activity data videos are screened, and a pig detection picture set is obtained using a video frame extraction script; the pigs in the picture are labeled using LabelImg, and the labeled file has a txt suffix, and the file name is consistent with the picture name, thereby obtaining a pig tracking dataset; The pig tracking dataset was divided into training set, test set and validation set in a ratio of 7:2:

1.

4. The pig growth status assessment method based on deep learning and visual large model according to claim 1 is characterized in that: In step S5, the pig growth status assessment model based on binary mask feature extraction and linear regression network has the pig binary mask obtained in S3 as input and the pig growth status classification result as output; the pig growth status assessment model extracts features from the pig mask generated in S3 and calculates the growth status classification result using a deep regression classification network; OpenCV is used to analyze the binarized pig mask to extract key features including the total number of black pixels, mask boundary, minimum enclosing rectangle, body size ratio and morphological symmetry; then, these features are input into the deep regression classification network, which contains multiple linear layers and uses a ReLU activation function to enhance the stability and convergence of the model, while introducing a Dropout mechanism to prevent overfitting; finally, the predicted value output by the network passes through a softmax layer to output the pig growth status category.

5. The pig growth status assessment method based on deep learning and visual large model according to claim 1 is characterized in that: The specific steps in step S6 include: S6-1. First, prepare multiple pig images; these images include pigs with normal growth and pigs with disabilities, deformities, and malnutrition, with a ratio of 1:

1. Label them according to the steps for preparing the pig tracking dataset in S2. Then, use the labeling results as prompts for the visual large model SegmentAnything to perform instance segmentation; save the generated binary mask; S6-2. Depending on whether the pigs are growing normally or not, their masks are stored in three folders: "normal", "deformed", and "malnourished". The ratio of the three masks is 1:1:

1. The dataset is divided into training set, test set, and validation set in a ratio of 7:2:

1. S6-3, inputting the pig binary mask obtained in S6-1 into the binary feature network of the pig growth and health assessment model to obtain the features of the pig binary mask; S6-4, inputting the features of the pig binary mask obtained in step S6-3 into the linear regression layer, and finally outputting classification information of whether the pig has growth abnormality; S6-5. During the training phase, the predicted categories are first matched with the true categories to determine the attribution of positive and negative samples. Then, the weight parameters are adjusted by calculating the loss function. At the end of each iteration, the validation set is used to calculate the accuracy and average precision to continuously optimize the model parameters.

Images