A method and system for identifying ram jumping behavior based on cross-individual relationship graph reasoning

Abstract

The application discloses a ram climbing behavior recognition method based on cross-individual relationship graph reasoning, comprising the following steps: S1, constructing a sheep image dataset and a climbing behavior video dataset; S2, obtaining target boxes, category information, position information and confidence information of ram targets and ewe targets in video frames; S3, obtaining sheep skeleton key point information; S4, obtaining sheep skeleton representation which is consistent in structure and stable in time sequence; S5, generating a continuous climbing segment; S6, determining a climbed ewe, and obtaining double-individual skeleton time sequence data of the ram and the climbed ewe; S7, obtaining a discrimination result of climbing success or failure; and S8, outputting a ram climbing behavior recognition result and further outputting estrus discrimination results of corresponding ewes. After instance segmentation mask constraint key point detection is adopted, the key point detection precision is improved.

Description

Technical Field

[0001] This invention relates to the fields of intelligent monitoring in livestock farming, animal behavior recognition, and smart ranch management, and particularly to a method and system for recognizing ram mounting behavior based on cross-individual relationship graph reasoning. This method is designed for complex farming scenarios such as group-housed sheep pens. By modeling and reasoning about the spatial proximity, posture interaction, movement coordination, and temporal behavioral changes between rams and ewes, it enables the automatic detection, recognition, and analysis of ram mounting behavior. It can be applied to sheep breeding management, estrus monitoring, mating behavior recording, breeding sheep performance evaluation, and smart livestock farming systems. Background Technology

[0002] With the continuous development of large-scale and intensive farming models, sheep breeding management has a significant impact on farming efficiency, herd renewal, and production benefits. In the production of meat sheep and breeding sheep, ram mounting behavior is usually closely related to estrus and mating behavior in ewes, as well as the reproductive performance of breeding rams. It is an important behavioral basis for judging the estrus status of ewes, recording the natural mating process, and assessing the activity level of breeding rams. In traditional farming, mounting behavior mainly relies on manual inspection and experience-based judgment for recording. However, the complex environment of sheep pens, the large number of sheep, and the unpredictable timing of such behavior make manual observation susceptible to the influence of inspection frequency, observation angle, and personnel experience. This leads to problems such as missed detections, misjudgments, and discontinuous recording, making it difficult to meet the needs of modern smart animal husbandry for automated, refined, and continuous management.

[0003] In recent years, with the development of computer vision, deep learning, and intelligent monitoring technologies, animal behavior recognition methods based on video images have been gradually applied in livestock and poultry farming scenarios. Existing methods typically identify animal behaviors such as feeding, drinking, standing, lying down, and walking through target detection, target tracking, posture estimation, or action classification. These methods have achieved certain results in recognizing the behavior of individual animals. However, ram mounting behavior differs from ordinary individual actions; it is essentially an interactive behavior between rams and ewes, requiring simultaneous attention to information such as spatial proximity, body orientation, postural contact, relative positional changes, and duration. If judgment is based solely on the appearance features or partial actions of a single individual, it is easy to misidentify similar scenarios such as chasing, approaching, crowding, and rubbing as mounting behavior.

[0004] Furthermore, real-world sheepfold environments present challenges such as varying lighting, severe occlusion, similar individual appearances, dense flocking of sheep, and limited camera viewing angles. When rams and ewes engage in mounting behavior, their bodies often partially overlap or briefly overlap, and the behavior is short-lived with distinct temporal variations. Existing recognition methods based on single-frame images or short video clips struggle to fully represent the dynamic interactions between rams and ewes, and also fail to reliably distinguish genuine mounting behavior from interfering behaviors such as brief contact, walking side-by-side, or overlapping. Therefore, accurately modeling cross-individual relationships between rams and ewes in complex group rearing environments is crucial for improving the accuracy and stability of mounting behavior recognition.

[0005] Existing animal behavior recognition systems primarily focus on individual detection and action classification, lacking a relationship modeling mechanism for ram-ewe interactions, particularly lacking a unified expression and reasoning ability for cross-individual spatial, postural, and temporal relationships. Therefore, it is necessary to propose a method and system for recognizing ram mounting behavior based on cross-individual relationship graph reasoning. This method treats rams and ewes as interconnected behavioral subjects, constructs a relationship graph structure between them, and combines temporal features for reasoning and judgment. This improves the accuracy, robustness, and automation level of mounting behavior recognition in complex sheepfold environments, providing technical support for ewe estrus monitoring, mating behavior recording, reproductive performance evaluation, and smart pasture management. Summary of the Invention

[0006] This invention aims to address the shortcomings of existing sheep behavior recognition methods in detecting ram mounting behavior, namely insufficient accuracy and poor stability. Existing methods often rely on the physical characteristics, local postures, or short-term movements of individual sheep for judgment, failing to adequately represent the spatial proximity, physical contact, posture correspondence, and temporal interaction relationships between rams and ewes. This makes it easy to misclassify behaviors such as chasing, approaching, crowding, occlusion, or brief overlap as mounting behavior. Furthermore, in group-housed sheepfold environments, the high density, similarity of individuals, and frequent occlusion further increase the difficulty of recognition. Therefore, there is an urgent need for a mounting behavior recognition method and system capable of modeling and inferring cross-individual interaction relationships between rams and ewes.

[0007] To address the above problems, this invention proposes a method for recognizing ram mounting behavior based on cross-individual relationship graph reasoning, comprising the following steps:

[0008] S1. Collect monitoring video data of sheep pens, and perform frame segmentation, cleaning, filtering and enhancement processing on the monitoring video data to construct sheep image dataset and climbing behavior video dataset.

[0009] S2. Train a sheep target detection model based on the sheep image dataset, and input the video frames to be analyzed into the sheep target detection model to obtain the target bounding boxes, category information, location information and confidence information of ram targets and ewe targets in the video frames;

[0010] S3. Perform instance segmentation on the ram and ewe targets to obtain instance masks for the corresponding sheep individuals, and perform key point detection under the constraints of the instance masks to obtain key point information of the sheep skeleton. The key point information of the sheep skeleton includes key point coordinates and key point confidence. Calculate the occlusion degree of the corresponding sheep based on the integrity of the instance mask, the number of visible key points, the key point confidence, and the overlap ratio between the target box or instance mask and adjacent sheep.

[0011] S4. Adaptively select the root node based on the confidence level and occlusion degree of the key point, perform hierarchical tree diffusion along the sheep skeleton topology starting from the root node, construct a multi-level structure subgraph, and perform feature fusion on the multi-level structure subgraph to obtain a sheep skeleton representation with consistent structure and stable time sequence.

[0012] S5. The temporal features of the ram skeleton are constructed from the sequence of key points of the ram skeleton within a continuous time window. Specifically, for a continuous T-frame video, the coordinates and confidence of the key points of the ram skeleton in each frame are extracted, and the key point displacement, key point velocity, trunk tilt angle, backline angle, relative distance between forelimbs and hindlimbs, and relative positional change between the forelimbs and the ewe target are calculated between adjacent frames. The above features are spliced ​​together in chronological order to obtain the temporal features of the ram skeleton. Based on the temporal features of the ram skeleton, the relevant frames for mounting are identified, and a sliding window consistency constraint is applied to the relevant frames to generate continuous mounting segments.

[0013] S6. Within the continuous climbing segment, based on the spatial distance relationship, target box overlap relationship, key point position relationship and motion trajectory relationship between the ram target and the ewe target, determine the ewe being climbed and obtain the temporal data of the skeletal skeletons of the ram and the ewe being climbed.

[0014] S7. Based on the time series data of the dual-individual skeleton, construct a cross-individual relationship diagram, and combine the displacement change characteristics between the ram and the ewe being mounted to judge the mounting result and obtain the judgment result of successful or unsuccessful mounting.

[0015] S8. Based on the determination result of successful or unsuccessful mounting, output the ram mounting behavior identification result, and further output the corresponding ewe's estrus determination result; wherein, the ram mounting behavior identification result includes at least the mounting occurrence time, duration, corresponding ram target, corresponding ewe target, and determination result of successful or unsuccessful mounting.

[0016] Preferably, in step S1, the framing, cleaning, and filtering processes include framing the original surveillance video, removing low-quality frames, removing overexposed frames, removing ghosting frames, and filtering severely occluded frames; the enhancement processes include one or more of random deactivation, Gaussian noise enhancement, channel multiplication enhancement, horizontal flipping, vertical flipping, brightness perturbation, and scale transformation.

[0017] Preferably, in step S1, the sheep image dataset includes bounding box annotations for rams and ewes, segmentation mask annotations for individual sheep instances, and key point annotations for sheep; the mounting behavior video dataset includes category annotations for successful mounting segments, failed mounting segments, and non-mounting interaction segments, and records the correspondence between rams and the ewes being mounted.

[0018] Preferably, in step S2, the sheep target detection model is used to distinguish between ram targets and ewe targets in video frames of group-raised sheep sheds; when there are multiple ram targets or multiple ewe targets in the same video frame, the ram target and ewe target currently participating in the interaction are determined according to the target detection results, historical trajectory information and identity association results.

[0019] Preferably, in step S3, the instance segmentation is used to obtain pixel-level foreground masks for ram and ewe targets, and the pixel-level foreground masks are used to constrain the sheep target region to suppress keypoint crosstalk between adjacent sheep individuals in group occlusion scenarios.

[0020] Preferably, in step S3, the key points of the sheep skeleton include multiple key points among the head, shoulders, rump, tail root, tail tip, left forelimb, left forehoof, right forelimb, right forehoof, left hindlimb, left hindhoof, right hindlimb, and right hindhoof.

[0021] Preferably, in step S4, the root node is selected from one or more candidate key points among head key points, shoulder key points, hip key points, and trunk center key points; the selection criteria for the root node include one or more of key point confidence, occlusion degree, short-term displacement stability, and key point visibility within a time window.

[0022] Preferably, in step S4, the hierarchical tree diffusion includes: taking the root node as the 0th layer node, expanding layer by layer along the sheep skeleton topology to adjacent key points to construct a multi-level structural subgraph from local body parts to the overall skeleton structure; when performing feature fusion on the multi-level structural subgraph, attention-weighted fusion, reliability-weighted fusion, or a combination of both are used to obtain a sheep skeleton representation with consistent structure and stable temporal sequence.

[0023] Preferably, in step S5, identifying the ram skeleton-related frames based on the temporal features of the ram skeleton includes: inputting the sequence of key points of the ram skeleton in the continuous video frames into the ram skeleton frame identification model to obtain the frame-by-frame ram skeleton confidence; only when the ram skeleton confidence in the sliding window reaches a preset threshold and the number of frames determined to be ram skeletons reaches a preset number, the corresponding time interval is determined as a continuous ram skeleton segment.

[0024] Preferably, in step S6, the determination of the ewe being mounted includes: calculating the center point distance, key point center distance, target box intersection-union ratio, instance mask overlap ratio, body orientation relationship and trajectory proximity between the ram target and the candidate ewe target, and determining the ewe target that meets the preset interaction conditions as the ewe being mounted.

[0025] Preferably, in step S7, the cross-individual relationship diagram includes ram nodes, ewe nodes, part nodes, and interaction nodes; wherein, the part nodes are composed of key skeletal points or key body parts of the ram and ewe, and the interaction nodes are obtained by aggregating key points of the ram's forelimbs, key points of the chest and abdomen, or overall skeletal features, and establish cross-individual connections with key points of the ewe's rump, hips, back, or torso, so as to characterize the contact relationships and relative geometric constraints of key body parts during the process of the ram mounting the ewe.

[0026] Preferably, in step S7, the displacement change features include one or more of the following: relative displacement, relative velocity, relative orientation, key point distance change, target box overlap change, mask overlap change, and contact duration between the ram and the mounted ewe; the displacement change features are encoded as edge features or temporal auxiliary features of the cross-individual relationship graph.

[0027] Preferably, in step S7, the cross-individual relationship graph and displacement change features are input into the cross-individual spatiotemporal relationship reasoning model. Through one or more of graph neural networks, spatiotemporal graph networks, relationship reasoning networks, or attention graph reasoning networks, information aggregation, relationship reasoning, and behavior classification are performed on the ram node, the ewe node, and the interaction edges between them, and the judgment result of whether the crossing is successful or unsuccessful is output.

[0028] Preferably, in step S8, when the determination result is successful mounting, the corresponding ewe is output as being in estrus; when the determination result is unsuccessful mounting, the corresponding ewe is output as being in non-estrus or in an unconfirmed estrus state.

[0029] Preferably, in step S8, the ram mounting behavior identification results and the corresponding ewe estrus determination results are stored in the database, and can be queried, replayed, and visualized by time, sheep number, mounting results, and ewe estrus status.

[0030] This invention also discloses a ram mounting behavior recognition system based on cross-individual relationship graph reasoning, comprising:

[0031] The video acquisition and preprocessing module is used to collect monitoring video data of sheep pens and perform frame segmentation, cleaning, filtering and enhancement processing on the monitoring video data to construct sheep image datasets and climbing behavior video datasets.

[0032] The target detection module is used to detect ram and ewe targets in video frames and obtain target bounding boxes, category information, location information and confidence information.

[0033] The instance segmentation module is used to perform instance segmentation on ram and ewe targets to obtain instance masks of the corresponding individual sheep.

[0034] The key point detection module is used to detect key points of individual sheep under instance mask constraints to obtain key point information of the sheep skeleton.

[0035] The skeleton stable representation module is used to adaptively select the root node based on the confidence level and occlusion degree of key points, perform hierarchical tree diffusion along the sheep skeleton topology, construct multi-level structural subgraphs and fuse them to obtain a sheep skeleton representation that is structurally consistent and temporally stable.

[0036] The ram straddle segment detection module is used to identify straddle-related frames based on the temporal features of the ram skeleton and to generate continuous straddle segments through sliding window consistency constraints.

[0037] The ewe association module is used to determine the ewe being mounted within the continuous mounting segment based on the spatial distance relationship between the ram and the ewe, the overlap relationship of the target box, the positional relationship of the key points, and the relationship of the movement trajectory, and to obtain the temporal data of the skeletons of the ram and the ewe being mounted.

[0038] The cross-individual relationship graph construction module is used to construct a cross-individual relationship graph based on the time series data of the dual-individual skeleton, and encode the displacement change features between the ram and the mounted ewe as graph features;

[0039] The relationship graph reasoning module is used to perform information aggregation and relationship reasoning on the cross-individual relationship graph, and output the judgment result of successful or unsuccessful crawling.

[0040] The results output and visualization module is used to output the ram mounting behavior identification results and the corresponding ewe estrus identification results based on the discrimination results, and to store, query, replay and visualize the identification results.

[0041] Preferably, the target detection module, instance segmentation module, and key point detection module are connected in sequence. The instance mask output by the instance segmentation module is used to constrain the input region of the key point detection module to reduce crosstalk between individual key points in a group-based occlusion environment.

[0042] Preferably, the skeleton stable representation module includes a root node selection unit, a hierarchical tree diffusion unit, and a structural feature fusion unit; the root node selection unit is used to select the root node based on the confidence level of the key point and the degree of occlusion, the hierarchical tree diffusion unit is used to construct a multi-level structural subgraph along the sheep skeleton topology, and the structural feature fusion unit is used to fuse the features of each subgraph and generate a stable skeleton representation.

[0043] Preferably, the cross-individual relationship graph construction module includes a node construction unit, an edge construction unit, and a displacement encoding unit; the node construction unit is used to construct ram nodes, ewe nodes, location nodes, and interaction nodes; the edge construction unit is used to establish cross-individual connections between rams and ewes; and the displacement encoding unit is used to encode the relative displacement change characteristics between the ram and the ewe being mounted.

[0044] Beneficial effects of the present invention

[0045] Compared with existing methods that rely on manual inspection, individual action recognition, or simple time sequence classification, the present invention has at least the following beneficial effects.

[0046] First, this invention uses a combination of binary target detection and instance segmentation to jointly perceive individual rams and ewes in a herding scenario, achieving individual-level decoupling before keypoint inference, effectively reducing cross-individual interference in occluded environments. After employing instance segmentation mask constraints for keypoint detection, the accuracy of keypoint detection is improved.

[0047] Second, this invention performs structural prior stabilization processing on key point results through adaptive root node selection, hierarchical tree diffusion subgraph construction, and multi-level structure fusion, thereby enhancing the integrity and temporal stability of skeleton representation under occlusion conditions.

[0048] Third, this invention generates continuous crawling segments through sliding window consistency constraints and obtains a more complete crawling process by combining boundary expansion, which effectively suppresses scattered false detections in long-term monitoring videos and improves the stability of segment-level behavior localization.

[0049] Fourth, this invention introduces displacement encoding and cross-individual interaction graphs into the process of determining the success or failure of mounting, which can simultaneously model the relative displacement, contact relationship and structural coupling between rams and ewes, thereby outputting more reliable results for determining estrus in ewes.

[0050] Fifth, this invention supports result storage, playback, and visualization, which facilitates verification and identification of the evidence chain by aquaculture personnel, and has good engineering deployment value and application prospects. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating the overall process of the ram mounting behavior recognition method in a group-housed environment according to the present invention.

[0052] Figure 2 This is a flowchart of the key point detection process based on the prior of the hierarchical tree diffusion subgraph structure of the present invention.

[0053] Figure 3 This is a schematic diagram illustrating the construction of the cross-individual relationship graph of the present invention.

[0054] Figure 4 A schematic diagram illustrating the generation of crawling segments for the sliding window consistency constraint of this invention.

[0055] Figure 5 This is a visual diagram illustrating the crawling and crossing discrimination results of the present invention. Detailed Implementation

[0056] The present invention will be further described below with reference to the accompanying drawings and embodiments. The foregoing "Technical Solution of the Invention" summarizes the overall steps and core processes of the invention. The following embodiments elaborate on the specific implementation process, data source, model training method, cross-individual relationship graph construction method, discrimination rules, and result output method of steps S1 to S8. In this embodiment of the invention, a method and system for recognizing ram mounting behavior based on cross-individual relationship graph reasoning is provided. The method takes a monitoring video of a sheepfold as input, first detects and segments the rams and ewes in the video, then extracts key points under individual mask constraints, further represents them through a structural prior stable skeleton, and performs segment recognition and success / failure judgment on the actual mounting process in the long-term video, thereby outputting the ram mounting behavior recognition result. The overall process of the method is as follows: Figure 1 As shown, it mainly includes the following steps.

[0057] Example 1: Data Acquisition, Preprocessing, and Dataset Construction

[0058] Collect monitoring video data of sheep pens, and perform frame segmentation, cleaning, filtering and enhancement processing on the monitoring video data to construct sheep image dataset and climbing behavior video dataset.

[0059] The framing, cleaning, and filtering processes include framing the original surveillance video, removing low-quality frames, removing overexposed frames, removing ghosting frames, and filtering severely occluded frames.

[0060] The enhancement process includes one or more of the following: random deactivation, Gaussian noise enhancement, channel multiplication enhancement, horizontal flipping, vertical flipping, brightness perturbation, and scale transformation.

[0061] The sheep image dataset includes bounding box annotations for rams and ewes, segmentation mask annotations for individual sheep instances, and key point annotations for sheep.

[0062] The mounting behavior video dataset includes category labels for successful mounting segments, failed mounting segments, and non-mounting interaction segments, and records the correspondence between rams and ewes being mounted.

[0063] Example 2: Sheep target detection model and obtaining target detection results.

[0064] A sheep target detection model is trained based on the sheep image dataset, and the video frames to be analyzed are input into the sheep target detection model to obtain the target bounding boxes, category information, location information and confidence information of ram and ewe targets in the video frames.

[0065] The sheep target detection model is used to distinguish between rams and ewes in video frames of sheepfolds.

[0066] When multiple ram targets or multiple ewe targets exist in the same video frame, the ram target and ewe target currently participating in the interaction are determined based on the target detection results, historical trajectory information, and identity association results.

[0067] Example 3: Key Point Detection under Instance Mask Constraints

[0068] Instance segmentation is performed on the ram and ewe targets to obtain instance masks for the corresponding individual sheep. Key point detection is then performed under the constraints of the instance masks to obtain key point information of the sheep skeleton.

[0069] The key point information of the sheep skeleton includes key point coordinates and key point confidence.

[0070] Furthermore, the degree of occlusion is jointly determined by the sheep's outline completeness, keypoint visibility, and overlap with adjacent sheep. Specifically, the more complete the sheep's outline and the more visible keypoints, the lower the occlusion degree; the more severe the overlap with adjacent sheep, the higher the occlusion degree. In an optional embodiment, the occlusion degree O can be expressed as: O = Q + (1 - M) + (1 - V), where O represents the occlusion degree, Q represents the overlap between the current sheep and adjacent sheep, M represents the sheep's outline completeness, and V represents keypoint visibility. The sheep's outline completeness M is determined based on the ratio of the instance mask area to the target box area; the keypoint visibility V is determined based on the ratio of the number of effective keypoints to the preset total number of keypoints; and the overlap degree Q is determined based on the instance mask intersection-union ratio of the current sheep and adjacent sheep.

[0071] The instance segmentation is used to obtain pixel-level foreground masks for ram and ewe targets, and the pixel-level foreground masks are used to constrain the sheep target region to suppress keypoint crosstalk between adjacent sheep individuals in group occlusion scenarios.

[0072] The key points of the sheep skeleton include multiple key points among the head, shoulders, rump, tail root, tail tip, left forelimb, left forehoof, right forelimb, right forehoof, left hindlimb, left hindhoof, right hindlimb, and right hindhoof.

[0073] Example 4: Select the root node, perform hierarchical tree expansion, and obtain a stable skeleton representation.

[0074] The root node is adaptively selected based on the confidence level and occlusion level of the key points. Starting from the root node, a hierarchical tree diffusion is performed along the sheep skeleton topology to construct a multi-level structural subgraph. Feature fusion is then performed on the multi-level structural subgraph to obtain a sheep skeleton representation that is structurally consistent and temporally stable.

[0075] like Figure 2 As shown, in the root node selection process, this embodiment selects keypoints with higher stability from candidate keypoints such as the head, shoulders, hips, or torso center as root nodes. The selection of root nodes is mainly based on keypoint confidence, short-term position stability, visibility within the time window, and the degree of sheep occlusion. Specifically, the higher the keypoint confidence, the more stable its short-term position, and the easier it is to detect in consecutive frames, the more suitable the keypoint is as a root node; the higher the degree of sheep occlusion, the lower the reliability of the keypoint as a root node. In an optional implementation, the reliability score R of a candidate keypoint can be expressed as: R = C + S + VO, where C represents the keypoint confidence, S represents the keypoint position stability within the short time window, V represents the keypoint visibility within the time window, and O represents the degree of sheep occlusion. The candidate keypoint with the highest reliability score is selected as the root node for the current frame's skeleton tree expansion. During the layered tree expansion process, the root node is used as the layer 0 node, and the expansion proceeds layer by layer along the sheep skeleton topology to adjacent keypoints, forming a multi-level structural subgraph from local body parts to the overall skeleton structure. By fusing the features of subgraphs at different levels, a sheep skeleton representation with consistent structure and temporal stability is obtained, thereby reducing the impact of individual keypoint jitter, short-term missing points, or local occlusion on the overall skeleton representation.

[0076] The root node is selected from one or more candidate keypoints among the head keypoint, shoulder keypoint, hip keypoint, and trunk center keypoint.

[0077] The selection criteria for the root node include one or more of the following: key point confidence, occlusion degree, short-term displacement stability, and key point visibility within the time window.

[0078] The hierarchical tree expansion includes: taking the root node as the 0th layer node, expanding layer by layer along the sheep skeleton topology to adjacent key points, and constructing a multi-level structural subgraph from local body parts to the overall skeleton structure.

[0079] When performing feature fusion on the multi-level structural subgraph, attention-weighted fusion, reliability-weighted fusion, or a combination of both are used to obtain a sheep skeleton representation that is structurally consistent and temporally stable.

[0080] Example 5: Identify crawling-related frames and generate continuous crawling segments.

[0081] The temporal features of the ram skeleton are constructed based on the changes in key points of the ram skeleton within a continuous time window, and the climbing-related frames are identified based on the temporal features of the ram skeleton; a sliding window consistency constraint is applied to the climbing-related frames to generate continuous climbing segments.

[0082] The temporal features of the ram skeleton include the keypoint positions of the ram skeleton, keypoint confidence, keypoint position changes between adjacent frames, trunk posture changes, and relative limb position changes. In an optional embodiment, the temporal feature F of the ram skeleton can be represented as: F=[P,ΔP,A,D]; where P represents the keypoint positions of the ram skeleton in consecutive frames, ΔP represents the keypoint position changes between adjacent frames, A represents the trunk orientation, and D represents the relative distance features between the forelimbs and trunk.

[0083] like Figure 4 As shown, this embodiment uses a sliding window consistency constraint to generate continuous crawling segments. Specifically, a sliding window of a preset length is set on the video sequence, and the number of crawling-related frames and the crawling confidence within the sliding window are counted. When the number of crawling-related frames within the sliding window reaches a preset threshold and the crawling confidence meets a preset confidence threshold, the time interval corresponding to the sliding window is determined as a candidate crawling segment. Adjacent or overlapping candidate crawling segments are merged to obtain continuous crawling segments.

[0084] Specifically, the sequence of key points of the ram skeleton in consecutive video frames is input into the climbing frame recognition model to obtain the climbing confidence level frame by frame.

[0085] Only when the confidence level of the crawling segment within the sliding window reaches a preset threshold and the number of frames determined to be crawling reaches a preset number, the corresponding time interval is determined to be a continuous crawling segment.

[0086] Example 6: Identify the mounted ewe and obtain temporal data of the two individuals' skeletons.

[0087] Within the continuous climbing segment, the ewe being climbed is determined based on the spatial distance relationship, target box overlap relationship, key point position relationship, and motion trajectory relationship between the ram target and the ewe target, thus obtaining the temporal data of the skeletons of the ram and the ewe being climbed.

[0088] When determining the ewe to be mounted, this embodiment constructs candidate interaction pairs between the ram target and each candidate ewe target, and extracts interaction features such as spatial distance, keypoint distance, target box overlap, instance mask overlap, body orientation relationship, and motion trajectory similarity between the two. In an optional implementation, the matching degree S of the candidate ewe can be expressed as: S = S_d + S_o + S_a + S_t, where S_d represents the spatial proximity, S_o represents the instance mask overlap, S_a represents the body orientation matching degree, and S_t represents the motion trajectory similarity. The ewe with the highest matching degree and meeting the preset conditions is selected as the ewe to be mounted.

[0089] The determination of the ewe to be mounted includes: calculating the center point distance, key point center distance, target box intersection-union ratio, instance mask overlap ratio, body orientation relationship and trajectory proximity between the ram target and the candidate ewe target, and determining the ewe target that meets the preset interaction conditions as the ewe to be mounted.

[0090] Example 7: Construct a cross-individual relationship graph and perform crawling result discrimination.

[0091] Based on the time-series data of the two individual skeletons, a cross-individual relationship graph is constructed. By combining the displacement change characteristics between the ram and the ewe being mounted, the mounting result is judged to obtain the judgment result of successful or unsuccessful mounting.

[0092] like Figure 3 As shown, the cross-individual relationship graph can be represented as G=(V,E), where V represents the set of nodes and E represents the set of edges. The node set V includes ram nodes, ewe nodes, key part nodes, and interaction nodes; the edge set E includes skeletal structure edges within the same sheep, cross-individual interaction edges between rams and ewes, and temporal edges between consecutive frames.

[0093] In one optional implementation, the cross-individual interaction edge feature can be represented as: e=[D,O,A,T], where D represents the distance feature between the ram and the ewe, O represents the target box or instance mask overlap feature, A represents the body orientation relationship feature, and T represents the movement trajectory relationship feature. After inputting the node features and edge features into a graph neural network or spatiotemporal graph inference model, a relationship representation between the ram and the ewe is obtained, and a judgment result of successful or failed mounting is output based on the relationship representation. The cross-individual relationship graph includes ram nodes, ewe nodes, part nodes, and interaction nodes. Further, as... Figure 3As shown in the middle section, the cross-individual relationship graphs within a continuous time window are constructed into a spatial graph sequence. Based on the enhanced adjacency matrix formed by adding the intra-individual skeleton adjacency matrix Aω and the cross-individual adjacency matrix Across, graph convolution is performed on the spatial relationships of each frame. Here, Aω represents the skeletal structural connections within the same sheep, and Across represents the cross-individual connections between rams and ewes. After performing three consecutive layers of spatial graph convolution, a temporal embedding containing cross-individual spatial relationships is obtained. Figure 3 As shown on the right, the temporal embedding is sequentially input into a temporal convolutional network, a two-dimensional convolutional and batch normalization and ReLU activation layer, a global spatiotemporal pooling layer and a fully connected layer, and outputs a judgment result of whether the crawling is successful or failed.

[0094] The location nodes are composed of key skeletal points or key body parts of the ram and ewe. The interaction nodes are obtained by aggregating key points of the ram's forelimbs, key points of the chest and abdomen, or overall skeletal features, and establish cross-individual connections with key points of the ewe's buttocks, hips, back, or trunk to characterize the contact relationships and relative geometric constraints of key body parts during the process of the ram mounting the ewe.

[0095] The displacement change features include one or more of the following: relative displacement, relative velocity, relative orientation, key point distance change, target box overlap change, mask overlap change, and contact duration between the ram and the mounted ewe.

[0096] The displacement change features are encoded as edge features or temporal auxiliary features across the individual relationship graph.

[0097] The cross-individual relationship graph and displacement change features are input into the cross-individual spatiotemporal relationship reasoning model. Through one or more of graph neural networks, spatiotemporal graph networks, relationship reasoning networks, or attention graph reasoning networks, information aggregation, relationship reasoning, and behavior classification are performed on ram nodes, ewe nodes, and the interaction edges between them. The result of the judgment of successful or unsuccessful crossing is output.

[0098] Example 8: Output the recognition results and perform storage, querying, playback, and visualization.

[0099] Based on the determination results of whether mounting was successful or unsuccessful, the ram mounting behavior identification result is output, and the corresponding ewe's estrus determination result is further output.

[0100] like Figure 5 As shown, this embodiment can visualize the mounting behavior identification results to intuitively display the mounting relationship between the ram and the corresponding ewe, as well as the identification results of successful or failed mounting. In an optional embodiment, the visualization can further overlay the sheep number, mounting start time, duration, and ewe estrus identification results to facilitate playback verification by farmers.

[0101] The ram mounting behavior identification results include at least the mounting time, duration, corresponding ram target, corresponding ewe target, and the judgment result of whether the mounting was successful or failed.

[0102] When the determination result is successful mounting, the corresponding ewe is output as being in estrus; when the determination result is unsuccessful mounting, the corresponding ewe is output as being in non-estrus or in an unconfirmed estrus state.

[0103] The results of the ram mounting behavior identification and the corresponding estrus determination results of the ewes are stored in the database, and can be queried, replayed and visualized by time, sheep number, mounting result and estrus status of the ewes.

[0104] To verify the technical effectiveness of this embodiment, the manually labeled results can be used as a benchmark to evaluate target detection, key point detection, cross-segment recognition, and success / failure discrimination. Evaluation metrics include precision, recall, F1 score, segment-level average intersection-over-union ratio (tIoU), and the number of false positives.

[0105] In a set of exemplary validation data, compared with schemes based solely on single-frame target detection and rule thresholds, the introduction of instance mask constraints, occlusion degree scoring, skeleton temporal features, and cross-individual relationship graph inference improves the F1 score for crawling and crossing segment recognition from approximately 0.85 to approximately 0.92. Sliding window consistency constraints reduce short-term false detection segments by approximately 30%, and the false negative rate in occluded scenarios decreases by approximately 20%. The above data are used to illustrate the effects of the invention; actual values ​​may be adjusted depending on the training data size, shooting angle, and sheepfold environment.

[0106] Example 9: A Ram Mounting Behavior Recognition System Based on Cross-Individual Relationship Graph Reasoning

[0107] This embodiment provides a ram mounting behavior recognition system based on cross-individual relationship graph reasoning. The system is used to execute the methods described in Embodiments 1 to 8. The system includes a video acquisition and preprocessing module, a target detection module, an instance segmentation module, a key point detection module, a skeleton stable representation module, a mounting segment detection module, a ewe association module, a cross-individual relationship graph construction module, a relationship graph reasoning module, and a result output and visualization module.

[0108] The video acquisition and preprocessing module is used to collect monitoring video data of sheep pens and perform frame segmentation, cleaning, filtering and enhancement processing on the monitoring video data to construct sheep image datasets and climbing behavior video datasets.

[0109] The target detection module is used to detect ram and ewe targets in video frames and obtain target bounding boxes, category information, location information and confidence information.

[0110] The instance segmentation module is used to perform instance segmentation on ram and ewe targets to obtain instance masks for the corresponding individual sheep.

[0111] The key point detection module is used to detect key points of individual sheep under instance mask constraints to obtain key point information of the sheep skeleton.

[0112] The skeleton stable representation module is used to adaptively select the root node based on the confidence level of key points and the degree of occlusion, perform hierarchical tree diffusion along the sheep skeleton topology, construct multi-level structural subgraphs and fuse them to obtain a sheep skeleton representation that is structurally consistent and temporally stable.

[0113] The ram skeletal segment detection module is used to identify ram skeletal-related frames based on the temporal features of the ram skeleton and to generate continuous ram skeletal segments through sliding window consistency constraints.

[0114] The ewe association module is used to determine the ewe being mounted within the continuous mounting segment based on the spatial distance relationship between the ram and the ewe, the overlap relationship of the target box, the positional relationship of the key points, and the relationship of the movement trajectory, and to obtain the temporal data of the skeletons of the ram and the ewe being mounted.

[0115] The cross-individual relationship graph construction module is used to construct a cross-individual relationship graph based on the time series data of the dual-individual skeleton, and encode the displacement change features between the ram and the mounted ewe as graph features.

[0116] The relationship graph reasoning module is used to perform information aggregation and relationship reasoning on the cross-individual relationship graph, and output the judgment result of successful or unsuccessful crawling.

[0117] The results output and visualization module is used to output the ram mounting behavior identification results and the corresponding ewe estrus identification results based on the discrimination results, and to store, query, replay and visualize the identification results.

[0118] The target detection module, instance segmentation module, and key point detection module are connected in sequence. The instance mask output by the instance segmentation module is used to constrain the input area of ​​the key point detection module in order to reduce crosstalk between key points across individuals in a group-based occlusion environment.

[0119] The stable skeleton representation module includes a root node selection unit, a hierarchical tree diffusion unit, and a structural feature fusion unit. The root node selection unit is used to select the root node based on the key point confidence and occlusion degree. The hierarchical tree diffusion unit is used to construct a multi-level structural subgraph along the sheep skeleton topology. The structural feature fusion unit is used to fuse the features of each subgraph and generate a stable skeleton representation.

[0120] The cross-individual relationship graph construction module includes a node construction unit, an edge construction unit, and a displacement encoding unit. The node construction unit is used to construct ram nodes, ewe nodes, location nodes, and interaction nodes. The edge construction unit is used to establish cross-individual connections between rams and ewes. The displacement encoding unit is used to encode the relative displacement change characteristics between the ram and the ewe being mounted.

[0121] The system is used to execute the method described in any one of Embodiments 1 to 8.

[0122] Preferred Implementation

[0123] Preferably, this embodiment uses Faster R-CNN as the front-end target detection algorithm, but it can also be replaced with other target detection algorithms suitable for sheep detection, including but not limited to YOLO series models, RT-DETR series models, etc.

[0124] Preferably, this embodiment uses an instance segmentation model to extract the individual sheep mask, but it can also be replaced with other segmentation models that can achieve pixel-level separation of individuals.

[0125] Preferably, this embodiment uses RTM-Pose as the keypoint detection algorithm, but it can also be replaced with other pose estimation algorithms that can output the coordinates of keypoints in an animal skeleton.

[0126] Preferably, this embodiment uses a sliding window consistency constraint to complete the generation of crawling segments. Other temporal continuity determination strategies can also be used to suppress scattered false detection frames.

[0127] Preferably, this embodiment uses displacement encoding and cross-individual interaction graphs to determine whether the climbing is successful or not. However, other relational reasoning models that can characterize the relative relationship between two individuals and the temporal evolution law can also be used without departing from the core idea of ​​this invention.

[0128] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A method for recognizing ram mounting behavior based on cross-individual relationship graph reasoning, characterized in that, Includes the following steps: S1. Collect monitoring video data of sheep pens, and perform frame segmentation, cleaning, filtering and enhancement processing on the monitoring video data to construct sheep image dataset and climbing behavior video dataset. S2. Train a sheep target detection model based on the sheep image dataset, and input the video frames to be analyzed into the sheep target detection model to obtain the target bounding boxes, category information, location information and confidence information of ram targets and ewe targets in the video frames; S3. Perform instance segmentation on the ram and ewe targets to obtain instance masks for the corresponding sheep individuals, and perform key point detection under the constraints of the instance masks to obtain key point information of the sheep skeleton. The key point information of the sheep skeleton includes key point coordinates and key point confidence. Calculate the occlusion degree of the corresponding sheep based on the integrity of the instance mask, the number of visible key points, the key point confidence, and the overlap ratio between the target box or instance mask and adjacent sheep. S4. Adaptively select the root node based on the confidence level and occlusion degree of the key point, perform hierarchical tree diffusion along the sheep skeleton topology starting from the root node, construct a multi-level structure subgraph, and perform feature fusion on the multi-level structure subgraph to obtain a sheep skeleton representation with consistent structure and stable time sequence. S5. The temporal features of the ram skeleton are constructed from the sequence of key points of the ram skeleton within a continuous time window. Based on the temporal features of the ram skeleton, the climbing-related frames are identified, and a sliding window consistency constraint is applied to the climbing-related frames to generate continuous climbing segments. S6. Within the continuous climbing segment, based on the spatial distance relationship, target box overlap relationship, key point position relationship and motion trajectory relationship between the ram target and the ewe target, determine the ewe being climbed and obtain the temporal data of the skeletal skeletons of the ram and the ewe being climbed. S7. Based on the time series data of the dual-individual skeleton, construct a cross-individual relationship diagram, and combine the displacement change characteristics between the ram and the ewe being mounted to judge the mounting result and obtain the judgment result of successful or unsuccessful mounting. S8. Based on the determination result of successful or unsuccessful mounting, output the ram mounting behavior identification result, and further output the corresponding ewe's estrus determination result; wherein, the ram mounting behavior identification result includes at least the mounting occurrence time, duration, corresponding ram target, corresponding ewe target, and determination result of successful or unsuccessful mounting.

2. The method for recognizing ram mounting behavior based on cross-individual relationship graph reasoning according to claim 1, characterized in that, In step S1, the sheep image dataset includes bounding box annotations for rams and ewes, segmentation mask annotations for individual sheep instances, and key point annotations for sheep; the mounting behavior video dataset includes category annotations for successful mounting segments, failed mounting segments, and non-mounting interaction segments, and records the correspondence between rams and the ewes being mounted.

3. The method for recognizing ram mounting behavior based on cross-individual relationship graph reasoning according to claim 1, characterized in that, In step S3, the instance segmentation is used to obtain pixel-level foreground masks for ram and ewe targets, and the pixel-level foreground masks are used to constrain the sheep target region to suppress keypoint crosstalk between adjacent sheep individuals in group occlusion scenarios.

4. The method for recognizing ram mounting behavior based on cross-individual relationship graph reasoning according to claim 1, characterized in that, In step S3, the key points of the sheep skeleton include multiple key points among the head, shoulders, rump, tail root, tail tip, left forelimb, left forehoof, right forelimb, right forehoof, left hindlimb, left hindhoof, right hindlimb, and right hindhoof.

5. The method for recognizing ram mounting behavior based on cross-individual relationship graph reasoning according to claim 1, characterized in that, In step S4, the hierarchical tree diffusion includes: taking the root node as the 0th layer node, expanding layer by layer along the sheep skeleton topology to adjacent key points to construct a multi-level structural subgraph from local body parts to the overall skeleton structure; when performing feature fusion on the multi-level structural subgraph, attention-weighted fusion, reliability-weighted fusion, or a combination of both are used to obtain a sheep skeleton representation that is structurally consistent and temporally stable.

6. The method for recognizing ram mounting behavior based on cross-individual relationship graph reasoning according to claim 1, characterized in that, In step S5, identifying the relevant frames for crossing based on the temporal features of the ram skeleton includes: inputting the sequence of key points of the ram skeleton in the continuous video frames into the crossing frame identification model to obtain the frame-by-frame crossing confidence; only when the crossing confidence within the sliding window reaches a preset threshold and the number of frames determined to be crossing reaches a preset number, the corresponding time interval is determined as a continuous crossing segment.

7. The method for recognizing ram mounting behavior based on cross-individual relationship graph reasoning according to claim 1, characterized in that, In step S7, the cross-individual relationship graph includes ram nodes, ewe nodes, part nodes, and interaction nodes; wherein, the part nodes are composed of key skeletal points or key body parts of the ram and ewe, and the interaction nodes are obtained by aggregating key points of the ram's forelimbs, key points of the chest and abdomen, or overall skeletal features, and establish cross-individual connections with key points of the ewe's rump, hips, back, or torso to characterize the contact relationships and relative geometric constraints of key body parts during the process of the ram mounting the ewe.

8. The method for recognizing ram mounting behavior based on cross-individual relationship graph reasoning according to claim 1, characterized in that, In step S7, the displacement change features include one or more of the following: relative displacement, relative velocity, relative orientation, key point distance change, target box overlap change, mask overlap change, and contact duration between the ram and the mounted ewe; the displacement change features are encoded as edge features or temporal auxiliary features across the individual relationship graph.

9. The method for recognizing ram mounting behavior based on cross-individual relationship graph reasoning according to claim 1, characterized in that, In step S7, the cross-individual relationship graph and displacement change features are input into the cross-individual spatiotemporal relationship reasoning model. Through one or more of graph neural networks, spatiotemporal graph networks, relationship reasoning networks, or attention graph reasoning networks, information aggregation, relationship reasoning, and behavior classification are performed on the ram node, the ewe node, and the interaction edges between them, and the judgment result of whether the crossing is successful or unsuccessful is output.

10. A ram mounting behavior recognition system based on cross-individual relationship graph reasoning, characterized in that, include: The video acquisition and preprocessing module is used to collect monitoring video data of sheep pens and perform frame segmentation, cleaning, filtering and enhancement processing on the monitoring video data to construct sheep image datasets and climbing behavior video datasets. The target detection module is used to detect ram and ewe targets in video frames and obtain target bounding boxes, category information, location information and confidence information. The instance segmentation module is used to perform instance segmentation on ram and ewe targets to obtain instance masks of the corresponding individual sheep. The key point detection module is used to detect key points of individual sheep under instance mask constraints to obtain key point information of the sheep skeleton. The skeleton stable representation module is used to adaptively select the root node based on the confidence level and occlusion degree of key points, perform hierarchical tree diffusion along the sheep skeleton topology, construct multi-level structural subgraphs and fuse them to obtain a sheep skeleton representation that is structurally consistent and temporally stable. The ram straddle segment detection module is used to identify straddle-related frames based on the temporal features of the ram skeleton and to generate continuous straddle segments through sliding window consistency constraints. The ewe association module is used to determine the ewe being mounted within the continuous mounting segment based on the spatial distance relationship between the ram and the ewe, the overlap relationship of the target box, the positional relationship of the key points, and the relationship of the movement trajectory, and to obtain the temporal data of the skeletons of the ram and the ewe being mounted. The cross-individual relationship graph construction module is used to construct a cross-individual relationship graph based on the time series data of the dual-individual skeleton, and encode the displacement change features between the ram and the mounted ewe as graph features; The relationship graph reasoning module is used to perform information aggregation and relationship reasoning on the cross-individual relationship graph, and output the judgment result of successful or unsuccessful crawling. The results output and visualization module is used to output the ram mounting behavior identification results and the corresponding ewe estrus identification results based on the discrimination results, and to store, query, replay and visualize the identification results.

Images