Intelligent monitoring method for sow reproductive behavior based on depth estimation

Abstract

The application discloses a kind of sow reproductive behavior intelligent monitoring methods based on depth estimation, belong to wisdom breeding and animal husbandry science field, which includes the following steps: based on the original sow monitoring video is preprocessed to obtain several short video clips according to frame extraction rule;Several short video clips are extracted to obtain short video representative frame, and short clip attention mask chart is obtained by processing short video representative frame based on depth estimation algorithm;Several short video clips are reorganized to obtain long monitoring video, and long monitoring time clustering table is obtained by extracting features and clustering long monitoring video;Long monitoring video summary and analysis chart are formed based on short clip attention mask chart and long monitoring time clustering table.This application can efficiently process massive monitoring video data, accurately capture the key information of sow reproductive behavior, reduce the cost of artificial monitoring, improve monitoring efficiency and accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent breeding and animal husbandry science, and particularly relates to a sow reproductive behavior intelligent monitoring method based on depth estimation. BACKGROUND

[0002] In a large-scale pig breeding farm, due to the limitation of biological safety requirements, the feeding personnel are usually restricted from entering and leaving the breeding area at will and frequently. Especially in the sow pregnancy scene, the feeding staff needs to check the behavior of the sow to confirm estrus and take artificial insemination measures. Through remote monitoring, the sow behavior can be viewed in real time, and the number of feeding staff entering and leaving the pig house can be reduced, thereby reducing the biological safety hidden danger and reducing the animal stress. However, the monitoring video content of the sow pregnancy scene has the characteristics of high similarity and strong repetition, the activity range of the sow is limited, and the main performance is forward and backward movement and slight posture adjustment, which leads to a high degree of repetition of visual patterns. This environment poses a challenge to computer vision models, especially in monitoring sow abnormal events and estrus processes.

[0003] In order to realize the intelligent monitoring of sow reproductive behavior, most of the current behavior recognition methods rely on supervised learning, and the existing patents and researches propose to recognize the climbing behavior in sow reproductive behavior, detect the lethargy behavior and vulva temperature, etc. These technologies need a large amount of manually labeled behavior data, and are difficult to apply to small sample behaviors or undefined behavior patterns. Unsupervised learning methods have potential in processing large-scale data, can automatically extract potential behavior patterns from data, and are helpful to find subtle behavior changes that may be ignored by traditional methods, but their identification explainability and reliability of the model still face challenges.

[0004] The existing technology still has the following technical problems:

[0005] 1. In the pregnancy pen environment, the sow activity is limited, which leads to a high degree of repetition of behavior patterns in video data, and the existing computer vision model is difficult to accurately distinguish the posture change of the sow. 2. The existing supervised learning method is limited by task setting and manually labeled data, and cannot flexibly detect and identify the diversification requirements such as animal behavior change, personnel activity and abnormal event. The traditional unsupervised learning method has the problem of poor explainability. 3. The long monitoring video content of the limit pen sow scene is monotonous, and manual playback, viewing and analysis are time-consuming and laborious. Therefore, the present application provides a sow reproductive behavior intelligent monitoring method based on depth estimation. SUMMARY

[0006] To solve the above technical problems, the present application provides a sow reproductive behavior intelligent monitoring method based on depth estimation to solve the problems existing in the above-mentioned prior art.

[0007] To achieve the above object, the application provides a sow reproductive behavior intelligent monitoring method based on depth estimation, comprising:

[0008] The original sow monitoring video is preprocessed based on a frame extraction rule to obtain a plurality of short video clips;

[0009] The short video representative frames are obtained by extracting frames from the plurality of short video clips, and the short clip attention mask map is obtained by processing the short video representative frames based on a depth estimation algorithm;

[0010] The long monitoring video is obtained by reorganizing the plurality of short video clips, and the long monitoring time clustering table is obtained by extracting features and clustering the long monitoring video;

[0011] The long monitoring video summary and analysis map are obtained based on the short clip attention mask map and the long monitoring time clustering table.

[0012] Optionally, the frame extraction rule comprises an image region of interest, a sampling rule, an attention sensitivity parameter, and an attention region parameter.

[0013] Optionally, the process of processing the short video representative frames based on the depth estimation algorithm to obtain the short clip attention mask map comprises:

[0014] The representative frame depth estimation matrix is obtained by processing the short video representative frames using the Video Depth Anything model;

[0015] The image change region is determined based on the representative frame depth estimation matrix to detect the depth change of different regions of the representative frame;

[0016] The image change region is filtered based on the attention sensitivity parameter and the attention region parameter to generate the short clip attention mask map.

[0017] Optionally, the process of extracting features and clustering the long monitoring video to obtain the long monitoring time clustering table comprises:

[0018] The target region of the long monitoring video is determined based on the parameters of the image region of interest;

[0019] The image features of the target region are extracted using the Resent18 network and the VGG16 network respectively, and the first feature set after dimension reduction and the second feature set after dimension reduction are obtained by dimension reduction processing the images extracted by the two networks respectively;

[0020] The first feature set and the second feature set after dimension reduction are re-encoded to obtain the encoded features;

[0021] The encoded features are clustered to obtain the long monitoring time clustering table.

[0022] Optionally, the Resent18 network comprises 4 residual modules, and each residual module comprises 2 convolution layers.

[0023] The VGG16 network comprises 13 convolution layers and 3 full-link layers.

[0024] Optionally, the FINCH model is used for clustering the encoded features.

[0025] Optionally, the process of generating a long monitoring video summary and analysis graph based on the short segment attention mask graph and the long monitoring time clustering table comprises:

[0026] The short segment attention mask graph is clustered to obtain a depth change clustering benchmark.

[0027] The long monitoring time clustering table is re-encoded based on the depth change clustering benchmark, and a transition region is smoothed to obtain a long monitoring video class correction table.

[0028] Based on the long monitoring video class correction table, the long monitoring video is reorganized, and the short segment attention mask graph generates a long monitoring video summary and analysis graph.

[0029] Compared with the prior art, the present application has the following advantages and technical effects:

[0030] The sow reproductive behavior intelligent monitoring method based on depth estimation of the present application pre-processes the original monitoring video through a frame extraction rule, generates a short video segment, extracts a short video representative frame, and generates a short segment attention mask graph using a depth estimation algorithm. Further, the short video segment is reorganized into a long monitoring video, features are extracted and clustered to generate a long monitoring time clustering table. Finally, a long monitoring video summary and analysis graph is generated by combining the short segment attention mask graph and the long monitoring time clustering table. This method can efficiently process massive monitoring video data, accurately capture key information of sow reproductive behavior, reduce manual monitoring cost, improve monitoring efficiency and accuracy, and provide strong support for sow reproductive management, which has significant practicality and innovation. BRIEF DESCRIPTION OF DRAWINGS

[0031] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application and its description together with the drawings make an aid in understanding the application. In the drawings:

[0032] Figure 1 A video analysis and summary generation flowchart of an embodiment of the present application is shown in the drawings;

[0033] Figure 2 A video segment spatial attention reinforcement dynamic detection module of an embodiment of the present application is shown in the drawings;

[0034] Figure 3 For the long monitoring static position time clustering module of the embodiment of the application;

[0035] Figure 4 For the result correction module of the embodiment of the application;

[0036] Figure 5 For the video summary generation module of the embodiment of the application;

[0037] Figure 6 For the real-time instance and result explanation of the embodiment of the application. DETAILED DESCRIPTION

[0038] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0039] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0040] Embodiment one

[0041] As Figure 1 shown, the present embodiment provides a sow reproductive behavior intelligent monitoring method based on depth estimation, the purpose of the method is to provide a depth enhanced spatiotemporal unsupervised model for analyzing long monitoring video in pregnant pigsty. The model combines depth estimation technology and unsupervised temporal action segmentation method, aiming at flexibly realizing the requirement of diversified video analysis, so as to improve the utilization efficiency and intelligent analysis level of pig farm video monitoring content.

[0042] The present application can reduce the dependence on a large amount of manually labeled data, help to discover behavior changes that have not been explored by traditional methods and research, and enhance the robustness of the system. The unsupervised video summary technology extracts key monitoring segments and reduces redundant information. This helps to reduce the workload of breeding personnel in manually playing back and analyzing video data, quickly understand the behavior state of sows, improve monitoring efficiency, save human resources, and is expected to have a profound impact in the fields of precision animal husbandry, intelligent breeding and animal welfare supervision.

[0043] The present technology takes concise sow reproductive behavior long monitoring video as the target of video summary, aiming to integrate 2D image depth estimation technology and video clustering technology to realize the screening and attention enhancement of key content in the image, such as Figure 1 as shown, the specific process is as follows:

[0044] (1) Based on the frame extraction rule, the original sow monitoring video is preprocessed to obtain a plurality of short video segments.

[0045] The raw sow monitoring video undergoes preprocessing steps through predefined task rules to initially reduce the size of the raw data. The frame extraction rules referred to in this invention include image regions of interest, sampling rules, attention sensitivity parameters, and attention region parameters.

[0046] (2) Extract frames from several short video segments to obtain representative frames of the short video, process the representative frames of the short video based on the depth estimation algorithm to obtain the attention mask map of the short segment; reassemble several short video segments to obtain a long surveillance video, and perform feature extraction and clustering on the long surveillance video to obtain a long surveillance time clustering table.

[0047] Long surveillance videos are downsampled and then sliced ​​into multiple short video segments. These multiple short video segments require two workflows:

[0048] Workflow 1 as follows Figure 2 As shown: Each short video is framed according to internally set rules to form a representative frame of the short video. Then, a depth estimation algorithm is used. In this embodiment, the Video Depth Anything model is selected to generate a depth estimation matrix of the representative frame. Then, the image change area is determined by detecting the depth changes in different regions of the representative frame. The image change areas that meet the requirements are filtered according to the set attention sensitivity parameters and attention area parameters to generate the attention mask map of the video segment.

[0049] Workflow 2, for example Figure 3 As shown: Each short video is merged into a long surveillance video, and then the target area of ​​the image is identified through the set image region of interest parameters. Resent18 and VGG16 feature extraction networks are used to extract image features respectively. The Resent18 network contains 4 residual modules, each containing 2 convolutional layers, while the VGG16 network contains 13 convolutional layers and 3 fully connected layers. The features from the two feature extraction networks are then subjected to principal component analysis to reduce the feature dimensions to 64, resulting in a first and a second dimensionality-reduced feature set. These two 64-dimensional features are then re-encoded by a 3-layer autoencoder. The autoencoder contains one fully connected layer and a ReLU activation function. Finally, an efficient parameter-free clustering algorithm, specifically the FINCH model, is selected in this embodiment to cluster different regions over long surveillance periods.

[0050] The long monitoring video summary and analysis graph is generated based on the short-fragment attention mask graph and the long monitoring time clustering table. The process includes: clustering the short-fragment attention mask graph to obtain a clustering benchmark based on depth change; mapping and recoding the class labels of the long monitoring time clustering table based on the depth change clustering benchmark, and smoothing and correcting the transition area to obtain a long monitoring video class correction table; and reorganizing the long monitoring video based on the long monitoring video class correction table, and generating the long monitoring video summary and analysis graph based on the short-fragment attention mask graph.

[0051] (3) The data processed by the two workflows is corrected by the result correction module, as shown in Figure 4 . The process includes: mapping the clustering results of different areas to the clustering results generated by the depth change to realize the labeling of the unlabeled clustering results; and smoothing and correcting the clustering results to correct and screen the clustering results according to the set attention requirements.

[0052] (4) The long monitoring video summary and analysis graph is generated based on the long monitoring video class correction table, and the long monitoring video is reorganized based on the short-fragment attention mask graph, as shown in Figure 5 . The content screening of the long monitoring video adds a "selection label" according to the long monitoring video class correction table. If the clustering results of all target areas correspond to the "no attention" type in the attention mask graph, the "selection label" is "negligible". If the clustering results of the target areas are not of the "no attention" type, the size of the corresponding attention area and the set main force parameter are used to determine whether to extract frames or full frames, and an attention semi-transparent mask is constructed for the selected image frames to increase the visualization effect of the video summary.

[0053] As a specific embodiment of the present embodiment, as shown in Figure 6 , an embodiment applied to the monitoring video of a pregnant sow in a limit stall scene is described. Taking a 24-hour long monitoring video as an example, in order to balance the time and effect of program running, the task parameter is set to extract 1 minute of video every 10 minutes for processing, and a total of 125 minutes of monitoring video is sampled from the 24-hour recording period. It is hoped that the video summary technology can select the video content of different postures of the pig and the work segment of the breeder checking the sow estrus. The attention sensitivity parameter and the attention area parameter are both set to 0.5, and the depth estimation algorithm is selected as the Video Depth Anything model. By using the method described in the present patent, 36 minutes of video summary is generated on a computer device in 10 minutes. In the long video analysis graph, the screening capability of micro-change video content, medium-change video content and strong-change video content is shown.

[0054] To verify the implementation effect of the present technology, two trained annotators labeled 4 poses (lying, kneeling, sitting, standing) and events (caregiver passing through the monitoring area, pig isolation area damage (individual escaping the designated area), and other feeding-related events) of 3 pigs, which took an average of 4.2 hours. The annotation consistency rates of the two annotators for the 3 pigs were 98.7%, 98.6%, and 91.4%, respectively. The clustering parameters of the video summary were set to the same 5 categories, and the final video summary results and 3 pig annotation mapping MOF were 96%, 93%, and 95%, respectively. Compared with the professional playback and video recording method, the technology shown in the present patent achieves the same level, reduces the time by 96%, compresses the content of the long video to 2.5%, and reduces the video storage cost by 97.5%, which has very high commercial value.

[0055] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A sow reproductive behavior intelligent monitoring method based on depth estimation, characterized in that, The method comprises the following steps: Preprocessing the original sow monitoring video based on a frame extraction rule to obtain a plurality of short video clips; Extracting frames from the plurality of short video clips to obtain short video representative frames, and processing the short video representative frames based on a depth estimation algorithm to obtain short clip attention mask maps; Reorganizing the plurality of short video clips to obtain a long monitoring video, and extracting features and clustering the long monitoring video to obtain a long monitoring time clustering table; Generating a long monitoring video summary and analysis map based on the short clip attention mask maps and the long monitoring time clustering table. 2.The sow reproductive behavior intelligent monitoring method based on depth estimation of claim 1, wherein, The frame extraction rule comprises an image region of interest, a sampling rule, an attention sensitivity parameter, and an attention region parameter. 3.The sow reproductive behavior intelligent monitoring method based on depth estimation of claim 2, wherein, The process of processing the short video representative frames based on the depth estimation algorithm to obtain short clip attention mask maps comprises: Processing the short video representative frames using a Video Depth Anything model to obtain a representative frame depth estimation matrix; Detecting depth changes of different regions of the representative frame based on the representative frame depth estimation matrix to determine image change regions; Filtering the image change regions based on the attention sensitivity parameter and the attention region parameter to generate short clip attention mask maps.

4. The sow reproductive behavior intelligent monitoring method based on depth estimation according to claim 3, characterized in that, The process of extracting features and clustering the long monitoring video to obtain a long monitoring time clustering table comprises: Determining a target region of the long monitoring video based on parameters of the image region of interest; Extracting image features of the target region using a Resent18 network and a VGG16 network, respectively, and performing dimension reduction processing on the images extracted by the two networks to obtain a first feature set after dimension reduction and a second feature set after dimension reduction; Re-encoding the first feature set and the second feature set after dimension reduction to obtain encoded features; Clustering the encoded features to obtain a long monitoring time clustering table.

5. The sow reproductive behavior intelligent monitoring method based on depth estimation according to claim 4, wherein The Resent18 network comprises 4 residual modules, and each residual module comprises 2 convolution layers; The VGG16 network comprises 13 convolution layers and 3 fully connected layers.

6. The sow reproductive behavior intelligent monitoring method based on depth estimation according to claim 4, characterized in that, The encoded features are clustered using a FINCH model.

7. The sow reproductive behavior intelligent monitoring method based on depth estimation according to claim 1, characterized in that, The process of generating a long monitoring video summary and analysis map based on the short clip attention mask maps and the long monitoring time clustering table comprises: Clustering the short clip attention mask maps to obtain a depth change clustering benchmark; Mapping and re-encoding the long monitoring time clustering table based on the depth change clustering benchmark, and smoothing and correcting the transition region to obtain a long monitoring video class correction table; Based on the long monitoring video class correction table, reorganizing the long monitoring video and the short clip attention mask maps to generate a long monitoring video summary and analysis map.

Images