Dairy cow body condition scoring and individual identification method based on multi-task learning

Abstract

The invention discloses a dairy cow body condition scoring and individual recognition method based on multi-task learning, relates to the technical field of image recognition, and realizes automatic positioning and detection of a dairy cow back area based on improved YOLOv8 dairy cow back area detection by introducing GSConv convolution into a backbone network, introducing a VoV-GSCSP module into a neck network and modifying a loss function into WIoU. A multi-task learning model which is based on hard parameter sharing and can synchronously output dairy cow body condition scores and individual recognition is constructed by utilizing the generality that dairy cow individual recognition and body condition scoring tasks both depend on dairy cow back area features. The detection time needed by the multi-task learning model to complete the two tasks at the same time is shorter than the total time needed by a single task model to complete the two tasks respectively, and accurate and efficient monitoring of the cow individual body condition is achieved.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a method for body condition scoring and individual recognition of dairy cows based on multi-task learning. Background Art

[0002] Individual animal management and feeding are key requirements of modern animal husbandry. However, in actual breeding environments, the research on body condition scoring and individual identification methods of dairy cows based on traditional machine vision still has problems such as low model robustness, poor reliability, weak generalization, and inability to timely associate with each other. With the rapid development of computer vision and deep learning technology, algorithms based on deep learning have shown great potential in body condition scoring and individual identification of dairy cows.

[0003] As an advanced real-time target detection algorithm, YOLOv8 has been widely used in many fields for its efficient detection speed and high accuracy. However, in large-scale complex breeding environments, the YOLOv8 model has high computational complexity, long inference time, large memory usage, and is difficult to deploy in real time. Multi-task learning (MTL) uses shared information between different tasks to improve the overall performance of the model, thereby achieving better results than single-task learning. The sharing mechanism of multi-task learning includes hard parameter sharing and soft parameter sharing. The hard parameter sharing mode reduces the parameter scale and computational cost through parameter sharing, and promotes knowledge transfer and complementarity between different tasks. It is suitable for situations where the similarity between tasks is high and the underlying features can be shared. The soft parameter sharing mode allows each task to have an independent model structure and specific parameters. It is suitable for scenarios where the feature representation between tasks is quite different and each task needs to learn specific features. Both the body condition scoring and individual recognition of dairy cows rely on the biological visual features of the back and can share the underlying features. Therefore, this paper adopts a multi-task learning model based on hard parameter sharing to reduce the amount of calculation and meet the real-time requirements.

[0004] Therefore, the research on cow body condition scoring and individual identification methods based on traditional machine vision has problems such as low model robustness, poor reliability, weak generalization, and the inability to timely associate with each other. Summary of the invention

[0005] In order to solve the above technical problems, this application proposes the following technical solutions:

[0006] In a first aspect, the present application embodiment provides a multi-task learning method for cow body condition scoring and individual identification, comprising:

[0007] The 3D camera is used to collect data by taking aerial photos of the milking channel and construct the target detection dataset and multi-task classification dataset.

[0008] Improve the YOLOv8 model to improve the accuracy of model target detection;

[0009] Use the target detection dataset to train the improved YOLOv8 model, evaluate, adjust and optimize the trained model, retain the optimal weight model after training and verify it;

[0010] Construct a multi-task learning model based on hard parameter sharing, and use a multi-task classification dataset to train the multi-task learning model, evaluate, adjust and optimize the trained model, retain the optimal weight model after training and verify it, and simultaneously realize cattle individual identification and body condition score assessment;

[0011] The optimized YOLOv8 model combined with the multi-task learning model is used to classify the detected cow back area to obtain individual identity and body condition score results.

[0012] In one possible implementation, the YOLOv8 model is improved to improve the accuracy of model target detection, including:

[0013] WIoU is used as the bounding box loss function, and a dynamic non-monotonic mechanism and a reasonable gradient gain allocation strategy are used in the calculation process to reduce the impact of extreme instances on model training and improve the model's robustness, data utilization, and detection performance;

[0014] Introduce GSConv convolution in the backbone network, introduce VoV-GSCSP module in the neck network, and process the input feature map through the ordinary convolution module;

[0015] The generated feature map is then processed by depthwise separable convolution and stacked with the original feature map along the channel dimension;

[0016] Finally, a random operation is performed to generate the final output feature map, which reduces the number of network parameters while maintaining the high accuracy of the model.

[0017] In one possible implementation, WIoU is used as the bounding box loss function, and a dynamic non-monotonic mechanism and a reasonable gradient gain allocation strategy are used in the calculation process to reduce the impact of extreme instances on model training and improve the robustness, data utilization and detection performance of the model, as shown in the following formula:

[0018] L WIoU =R WIoU L IoU

[0019]

[0020] Where, L IoUis the intersection-over-union loss, x is the x-coordinate of the center point of the real box, y is the y-coordinate of the center point of the real box, xgt is the x-coordinate of the center point of the predicted box, ygt is the y-coordinate of the center point of the predicted box, Wg is the width of the minimum bounding box between the predicted box and the real box, and Hg is the height of the minimum bounding box between the predicted box and the real box.

[0021] In one possible implementation, the GSConv module uses a standard convolutional layer cv1 to map the input feature map from the c1 channel to the intermediate channel c2 / 2 to capture preliminary features; then another depth-wise separable convolutional layer cv2 is used to further transform the output of cv1 to generate a feature map with the same number of channels as the cv1 output but with different feature representations; then the output feature maps of cv1 and cv2 are concatenated along the channel dimension, and the channel order is disrupted through the Shuffle operation to form the final output feature map.

[0022] In a possible implementation manner, the implementation steps of VoVGSCSP include:

[0023] First, the input feature map is transformed through two convolutional layers cv1 and cv2, where cv1 is used to reduce the number of channels by compression ratio e, and cv2 is used to restore the number of feature map channels to the target value c2;

[0024] Then, a sequence m consisting of multiple GSBottleneck modules is used to perform deep feature extraction on the feature map transformed by cv1. Each GSBottleneck module uses GSConv convolution and shortcut connection shortcut to enhance the feature representation ability and model learning ability.

[0025] Finally, the output of sequence m is concatenated with the output of cv1 along the channel dimension, and the features are fused and output through the cv2 convolutional layer.

[0026] In a possible implementation, the evaluation indicators used to evaluate the trained model include: recall rate Recall, precision rate Precision, average precision rate mAP and F1 score F1-score, as shown in the following formula:

[0027]

[0028] In the formula, TP is the number of samples predicted as positive, FP is the number of samples predicted as positive, FN is the number of samples predicted as negative, and n is the number of categories in the data set.

[0029] In one possible implementation, a multi-task learning model based on hard parameter sharing is constructed, including:

[0030] A multi-task learning model based on hard parameter sharing is built with the ResNet network as the backbone network. When inputting the depth image and the visible light image, the corresponding feature maps are extracted. The common features are extracted through Concat feature fusion as the input for subsequent deep-level feature extraction.

[0031] Input the deep-level features into the feature extraction layer of the corresponding task to ensure that the most relevant feature information for the task can be captured;

[0032] In the feature extraction process, the skip-layer connection method is used to allow the low-level feature information to skip some convolutional layers and be added to the high-level features to form a residual connection, so as to obtain richer biological visual features of the cow's back.

[0033] In a possible implementation, the optimized YOLOv8 model is combined with a multi-task learning model to classify the detected cow back area to obtain individual identity and body condition score results, including:

[0034] Use 3D camera equipment to capture the milking channel area to obtain real-time cow back images;

[0035] The improved YOLOv8 detection model combined with the multi-task learning model is used to process real-time cow back images to obtain individual identity and body condition score results.

[0036] In the embodiment of the present application, based on the improved YOLOv8 cow back region detection, by introducing GSConv convolution in the backbone network, introducing VoV-GSCSP module in the neck network, and modifying the loss function to WIoU, the automatic positioning and detection of the cow back region is realized. Taking advantage of the commonality that both cow individual identification and body condition scoring tasks rely on the characteristics of the cow back region, a multi-task learning model based on hard parameter sharing that can synchronously output cow body condition scoring and individual identification is constructed. The detection time required for the multi-task learning model to complete the two tasks simultaneously is less than the total time required for the single-task model to complete the two tasks separately, thereby realizing accurate and efficient monitoring of the individual body condition of cows. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of a process flow of a multi-task learning method for cow body condition scoring and individual identification provided in an embodiment of the present application;

[0038] Figure 2 A schematic diagram of data collection provided for an embodiment of the present application;

[0039] Figure 3 A schematic diagram of the improved YOLOv8 model structure provided in an embodiment of the present application;

[0040] Figure 4A schematic diagram of the multi-task learning model structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The present solution is described below in conjunction with the accompanying drawings and specific implementation methods.

[0042] See also Figure 1 The multi-task learning method for cow body condition scoring and individual identification provided in this embodiment includes:

[0043] S101, collect data by taking a bird's-eye view of the milking channel with a 3D camera and complete the construction of a target detection dataset and a multi-task classification dataset.

[0044] In this implementation, a 3D camera is used to collect data by taking a bird's-eye view of the milking channel and to build a target detection dataset and a multi-task classification dataset. Figure 2 , the Microsoft Kinect Azure DK camera was selected to simultaneously capture color images and corresponding depth images, and to fully capture the multi-dimensional information of dairy cows. The camera was installed above the passage that the cows must pass through for milking, 2.7 meters above the ground, and captured the images of the cows passing by from a bird's-eye view. The annotation file of the target detection dataset saves the upper left corner position coordinates and lower right corner position coordinates of the rectangular bounding box corresponding to each cow in each picture; the multi-classification dataset contains the body condition score and individual number corresponding to each picture; and the training set and validation set are divided according to a ratio of 7.5:2.5.

[0045] S102, improve the YOLOv8 model to improve the accuracy of model target detection.

[0046] In this embodiment, the improvement of the YOLOv8 model mainly includes: using WIoU as the bounding box loss function, using a dynamic non-monotonic mechanism and a reasonable gradient gain allocation strategy in the calculation process to reduce the impact of extreme instances on model training, and effectively improve the model's robustness, data utilization, and detection performance. GSConv convolution is introduced in the backbone network, and the VoV-GSCSP module is introduced in the neck network. It processes the input feature map through an ordinary convolution module. Subsequently, the generated feature map is processed by a depth-separable convolution and stacked with the original feature map along the channel dimension. Finally, a random operation is performed to generate the final output feature map, which achieves the reduction of network parameters while maintaining the high accuracy of the model. The improved YOLOv8 model is as follows: Figure 3 shown.

[0047] Specifically, WIoU is used as the bounding box loss function, and a dynamic non-monotonic mechanism and a reasonable gradient gain allocation strategy are used in the calculation process to reduce the impact of extreme instances on model training and improve the model's robustness, data utilization, and detection performance, as shown in the following formula:

[0048] L WIoU =R WIoU L IoU

[0049]

[0050] Where, L IoU is the intersection-over-union loss, x is the x-coordinate of the center point of the real box, y is the y-coordinate of the center point of the real box, and x gt is the x coordinate of the center point of the prediction box, y gt is the y coordinate of the center point of the prediction box, Wg is the width of the minimum bounding box between the prediction box and the real box, and Hg is the height of the minimum bounding box between the prediction box and the real box.

[0051] In this embodiment, GSConv uses a standard convolutional layer cv1 to map the input feature map from the c1 channel to the intermediate channel c2 / 2 to capture preliminary features. Then, it uses another depth-wise separable convolutional layer cv2 to further transform the output of cv1 to generate a feature map with the same number of channels as the cv1 output but with different feature representations. Then, the output feature maps of cv1 and cv2 are concatenated along the channel dimension, and the channel order is disrupted through the Shuffle operation to form the final output feature map.

[0052] The implementation steps of VoVGSCSP mainly include three key parts: first, the input feature map is transformed through two convolutional layers (cv1 and cv2), where cv1 is used to reduce the number of channels through the compression ratio e, and cv2 is used to restore the number of feature map channels to the target value c2; second, the sequence m composed of multiple GSBottleneck modules is used to extract deep features from the feature map transformed by cv1, and each GSBottleneck module enhances the feature representation ability and the learning ability of the model through GSConv convolution and shortcut connection; finally, the output of sequence m is spliced ​​with the output of cv1 along the channel dimension, and the features are fused and output through the cv2 convolution layer. In this way, the VoVGSCSP module enhances the feature representation ability of the model by combining feature information of different scales, while maintaining the high efficiency of the model.

[0053] S103, using the target detection data set to train the improved YOLOv8 model, evaluate, adjust and optimize the trained model, retain the optimal weight model after training and verify it.

[0054] In this embodiment, the Adam optimizer is used for optimization, the batch size is 16, the number of iterations is 200, the learning rate is 0.0001, and the early stopping strategy is adopted. When the performance of the model on the validation set does not improve for 50 consecutive epochs, the training is automatically stopped to effectively avoid overfitting.

[0055] The evaluation indicators used to evaluate the trained model include: recall rate Recall, precision rate Precision, average precision rate mAP and F1 score F1-score, as shown in the following formula:

[0056]

[0057] In the formula, TP is the number of samples predicted as positive, FP is the number of samples predicted as positive, FN is the number of samples predicted as negative, and n is the number of categories in the data set.

[0058] S104, construct a multi-task learning model based on hard parameter sharing, and use the multi-task classification data set to train the multi-task learning model, evaluate, adjust and optimize the trained model, retain the optimal weight model after training and verify it, and simultaneously realize individual cattle identification and body condition score evaluation.

[0059] like Figure 4 As shown, in this embodiment, a multi-task learning model based on hard parameter sharing is built with the ResNet network as the basic backbone network. While inputting the depth image and the visible light image, the corresponding feature maps are extracted, and the common features are extracted by Concat feature fusion as the input for subsequent deep-level feature extraction. The deep-level features are input into the feature extraction layer of the corresponding task to ensure that the feature information most relevant to the task can be captured. The hard parameter sharing mode reduces the parameter scale and the computational cost through parameter sharing, promotes the knowledge transfer and complementarity between different tasks, and is suitable for situations where the similarity between tasks is high and the underlying features can be shared. In the feature extraction process, the low-level feature information is allowed to skip some convolutional layers and be added to the high-level features to form a residual connection by means of skip connection. This structure enables the network to learn the residual mapping between input and output, retain the low-level features while extracting high-level features, thereby obtaining richer biological visual features of the cow's back, and providing information support for subsequent body condition scoring and individual identity recognition.

[0060] S105, using the optimized YOLOv8 model combined with the multi-task learning model to classify the detected cow back area to obtain individual identity and body condition score results.

[0061] The optimal weights and designed network model are called to infer the input image to be tested, the test results are displayed, and the model is deployed on the industrial computer equipment in the dairy farm. A visualization system is designed to display the body condition score and individual number of the cow on the image.

[0062] Specifically, the back area of ​​the cow is accurately identified and marked on the RGB image through the improved YOLOv8s model. Based on the coordinate information of the detection frame, the corresponding depth area of ​​the cow's back is further located on the corresponding depth image, thereby obtaining the Depth map of the cow's back area. Subsequently, based on the RGB image and Depth map of the cow's back, the biological visual features of the cow's back are extracted. Finally, based on the common characteristics of the cow's body condition and individual identity, the ResNet34 network is used as the basic backbone network to extract key biological visual features from the RGB image and Depth map of the cow's back area and fuse them to obtain the individual identity and body condition score results.

[0063] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0064] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for body condition scoring and individual identification of dairy cows based on multi-task learning, characterized in that: include: The 3D camera is used to collect data by taking aerial photos of the milking channel and construct the target detection dataset and multi-task classification dataset. Improve the YOLOv8 model to improve the accuracy of model target detection; Use the target detection dataset to train the improved YOLOv8 model, evaluate, adjust and optimize the trained model, retain the optimal weight model after training and verify it; Construct a multi-task learning model based on hard parameter sharing, and use a multi-task classification dataset to train the multi-task learning model, evaluate, adjust and optimize the trained model, retain the optimal weight model after training and verify it, and simultaneously realize individual cattle identification and body condition score assessment; The optimized YOLOv8 model combined with the multi-task learning model is used to classify the detected cow back area to obtain individual identity and body condition score results.

2. The method for body condition scoring and individual identification of dairy cows based on multi-task learning according to claim 1, characterized in that: Improvements to the YOLOv8 model are made to improve the accuracy of model target detection, including: WIoU is used as the bounding box loss function, and a dynamic non-monotonic mechanism and a reasonable gradient gain allocation strategy are used in the calculation process to reduce the impact of extreme instances on model training and improve the model's robustness, data utilization, and detection performance; Introduce GSConv convolution in the backbone network, introduce VoV-GSCSP module in the neck network, and process the input feature map through the ordinary convolution module; The generated feature map is then processed by depthwise separable convolution and stacked with the original feature map along the channel dimension; Finally, a random operation is performed to generate the final output feature map, which reduces the number of network parameters while maintaining the high accuracy of the model.

3. The method for body condition scoring and individual identification of dairy cows based on multi-task learning according to claim 2, characterized in that: WIoU is used as the bounding box loss function, and a dynamic non-monotonic mechanism and a reasonable gradient gain allocation strategy are used in the calculation process to reduce the impact of extreme instances on model training and improve the model's robustness, data utilization, and detection performance, as shown in the following formula: L WIoU =R WIoU L IoU Where, L I o U is the intersection-over-union loss, x is the x-coordinate of the center point of the real box, y is the y-coordinate of the center point of the real box, xgt is the x-coordinate of the center point of the predicted box, ygt is the y-coordinate of the center point of the predicted box, Wg is the width of the minimum bounding box between the predicted box and the real box, and Hg is the height of the minimum bounding box between the predicted box and the real box.

4. The method for body condition scoring and individual identification of dairy cows based on multi-task learning according to claim 3, characterized in that: The GSConv module uses a standard convolutional layer cv1 to map the input feature map from the c1 channel to the intermediate channel c2 / 2 to capture preliminary features. It then uses another depth-wise separable convolutional layer cv2 to further transform the output of cv1 to generate a feature map with the same number of channels as the cv1 output but with different feature representations. The output feature maps of cv1 and cv2 are then concatenated along the channel dimension, and the channel order is disrupted through the Shuffle operation to form the final output feature map.

5. The method for body condition scoring and individual identification of dairy cows based on multi-task learning according to claim 4, characterized in that: The implementation steps of VoVGSCSP include: First, the input feature map is transformed through two convolutional layers cv1 and cv2, where cv1 is used to reduce the number of channels by compression ratio e, and cv2 is used to restore the number of feature map channels to the target value c2; Then, a sequence m consisting of multiple GSBottleneck modules is used to perform deep feature extraction on the feature map transformed by cv1. Each GSBottleneck module uses GSConv convolution and shortcut connection shortcut to enhance the feature representation ability and model learning ability. Finally, the output of sequence m is concatenated with the output of cv1 along the channel dimension, and the features are fused and output through the cv2 convolutional layer.

6. The method for body condition scoring and individual identification of dairy cows based on multi-task learning according to claim 1, characterized in that: The evaluation indicators used to evaluate the trained model include: recall rate Recall, precision rate Precision, average precision rate mAP and F1 score F1-score, as shown in the following formula: In the formula, TP is the number of samples predicted as positive, FP is the number of samples predicted as positive, FN is the number of samples predicted as negative, and n is the number of categories in the data set.

7. The method for body condition scoring and individual identification of dairy cows based on multi-task learning according to claim 1, characterized in that: Build a multi-task learning model based on hard parameter sharing, including: A multi-task learning model based on hard parameter sharing is built with the ResNet network as the backbone network. When inputting the depth image and the visible light image, the corresponding feature maps are extracted. The common features are extracted through Concat feature fusion as the input for subsequent deep-level feature extraction. Input the deep-level features into the feature extraction layer of the corresponding task to ensure that the most relevant feature information for the task can be captured; In the feature extraction process, the skip-layer connection method is used to allow the low-level feature information to skip some convolutional layers and be added to the high-level features to form a residual connection, so as to obtain richer biological visual features of the cow's back.

8. The method according to claim 1, characterized in that The optimized YOLOv8 model is combined with a multi-task learning model to classify the detected cow back area to obtain individual identity and body condition score results, including: Use 3D camera equipment to capture the milking channel area to obtain real-time cow back images; The improved YOLOv8 detection model combined with the multi-task learning model is used to process real-time cow back images to obtain individual identity and body condition score results.

Images