Pet behavior analysis method based on image recognition

Abstract

This invention belongs to the field of image processing technology, specifically relating to a pet behavior analysis method based on image recognition. The method includes: Step 1: Acquiring a video frame sequence, performing superpixel segmentation on each frame to obtain a superpixel map composed of several superpixel blocks; Step 2: Using the region association map as input, constructing a Poisson-Dobernoli hybrid tracking link, and segmenting the trajectory entity set into a temporal segment set; Step 3: Using the temporal segment set as input, constructing a skeleton × object × scene hypergraph; Step 4: For the eating and drinking action segments in the initial strong recognition results of interactive behaviors, finally outputting behavior analysis results carrying the object participation type and scene type. This invention achieves unified modeling and interpretable behavior recognition of pets, objects, and scenes, possessing advantages of high robustness, high accuracy, and real-time performance.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a pet behavior analysis method based on image recognition. BACKGROUND

[0002] Currently, pet behavior analysis technology based on image recognition has attracted widespread attention in the field of smart home and animal behavior research. With the development of computer vision and deep learning technology, researchers and enterprises attempt to install camera devices in the home environment to detect and identify the activities of pets, in order to realize intelligent functions such as behavior monitoring, health warning, and companion interaction. Common systems usually use convolutional neural networks to detect targets in video frames, thereby identifying the category and posture of the pet. For example, some commercial intelligent camera products can identify whether the pet is moving, eating, or resting based on two-dimensional image sequences, and generate activity statistical curves in mobile applications. This type of solution can achieve good recognition results in static environments and single-pet scenarios, but still has obvious limitations in complex scenarios.

[0003] Most existing technologies make pet behavior judgments based on single-frame image features or short-time video features, lacking sufficient modeling of spatio-temporal relationships. In a home environment, the behavior of a pet often involves interaction with objects and functional areas in the environment, such as a feeding device, a water bowl, a toy, or a litter box. It is difficult to accurately distinguish behavior types relying solely on the appearance and posture of the pet. For example, a pet approaching a feeding device and lowering its head may be misidentified as eating, but it may actually be just sniffing or observing. Existing systems are prone to misjudgment in such interactive behaviors. In addition, most technical solutions do not consider the joint relationship between scene layout and object participation, and the system cannot understand key semantic information such as "where is the pet, what object is it interacting with, and how long does the interaction last" at the visual level, resulting in a lack of context consistency in the recognition results.

[0004] On the other hand, traditional video tracking algorithms have poor stability when dealing with fast pet movement, short-time occlusion, and posture changes. Tracking methods based on optical flow or Kalman filtering rely on continuous pixel features. When a pet's body part is occluded by furniture or produces strong reflections due to changes in lighting, the trajectory is easily broken, resulting in a lack of complete time series for behavior analysis. Although recent multi-target tracking algorithms have introduced deep feature matching, such algorithms usually rely on pre-trained models and have high computational complexity, making them unsuitable for real-time operation on embedded home devices. SUMMARY

[0005] In view of this, the main purpose of the present application is to provide a pet behavior analysis method based on image recognition, which realizes unified modeling of pets, objects, and scenes and interpretable behavior recognition, has high robustness, high precision, and real-time performance, and can maintain stable recognition output under conditions of multiple pets, complex lighting, and multiple object interference.

[0006] The technical solutions adopted by the present application are as follows:

[0007] The pet behavior analysis method based on image recognition comprises the following steps:

[0008] Step 1: Obtain a video frame sequence, perform superpixel segmentation on each frame image in the video frame sequence to obtain a superpixel graph composed of a plurality of superpixel blocks; generate a pet candidate region, an object candidate region and a scene layout on the superpixel graph, and establish a region association graph containing nodes respectively representing the pet candidate region, the object candidate region and a functional area in the scene layout, and edges reflecting the spatiotemporal adjacency relationship between the nodes;

[0009] Step 2: Taking the region association graph as input, a Poisson multi-Bernoulli mixture tracking link is constructed, and a trajectory is established for the anchor points of the pet candidate region, the object candidate region and the functional area, so as to unify the output into a trajectory entity set containing time sequence information, and the trajectory entity set is cut into a time sequence segment set;

[0010] Step 3: Taking the time sequence segment set as input, a skeleton × object × scene supergraph is constructed, wherein the supergraph contains a pet skeleton node set, an object node set and a scene node set; deterministic reasoning is performed along a preset action recognition path on the supergraph to output an initial result of strong recognition of interactive behavior;

[0011] Step 4: For the eating action segment and the drinking action segment in the initial result of strong recognition of interactive behavior, a Copula gating layer is constructed and frame-by-frame gating processing is performed to suppress false eating or false drinking behavior, and finally the behavior analysis result carrying the object participation type and the scene type is output.

[0012] Further, the generation of the pet candidate region, the object candidate region and the scene layout in step 1 comprises: obtaining a motion mask by performing inter-frame difference processing on the current frame and the adjacent previous frame, and marking the superpixels in the motion mask whose pixel coverage reaches a preset motion proportion threshold as motion blocks; region growing is performed with the motion blocks as seeds to construct a pet candidate region set; shape detection is performed on the edge graph of the current frame, and the object candidate region set containing the eating utensil / water utensil candidate region, the toy candidate region and the litter box candidate region is constructed in combination with the color saturation feature, and each object candidate region is labeled with a type label; according to the positions of the object candidate regions of various types in the object candidate region set, the eating area, the drinking area and the activity area are divided in the detected ground area as functional areas to form a scene layout.

[0013] Further, the pet candidate region set is constructed, including: taking a motion block as a seed, merging adjacent superpixels meeting the following conditions into the same connected domain: sharing a boundary length reaching a first predetermined length threshold, and a color mean difference not exceeding a first predetermined color threshold; when the number of pixels of the connected domain reaches a first predetermined area threshold, the connected domain is retained as a pet candidate region; calculating an external rectangle for each pet candidate region, and including a pet candidate region with an aspect ratio within a first predetermined range into the pet candidate region set.

[0014] Further, the object candidate region set is constructed, including: performing shape detection of circles and rectangles on the edge map of the current frame; identifying a circle with a radius within a second predetermined range and a circle arc covering angle reaching a first predetermined angle threshold as a container circle, and merging superpixels with an intersection area of the container circle occupying a container circle area reaching a first predetermined area proportion threshold into a feeder / water dispenser candidate region; identifying a connected domain with an area within a third predetermined range and a color saturation mean reaching a preset saturation threshold as a toy candidate region; identifying a rectangle connected domain with an angle point number reaching a preset angle point number threshold and an aspect ratio within a second predetermined range as a litter box candidate region.

[0015] Further, the Poisson multiple Bernoulli mixed tracking link in step two is constructed, including: between adjacent two frames, calculating the overlap ratio of the external rectangle and the centroid distance of the candidate regions of the same type, when the overlap ratio is not lower than a preset overlap ratio threshold and the centroid distance does not exceed a preset distance threshold, it is determined to be matched; for a candidate region with an associated track, if it is matched with only one candidate region in the next frame, the track is extended; if it is matched with multiple candidate regions, multiple branches of the track are generated and included in a hypothesis pool for management; the state of each track or branch is updated according to the matching state: when the number of continuously matched frames reaches a first preset frame number threshold, it is marked as a stable state; after the stable state, if one frame is not matched, it is marked as a to-be-confirmed state; if two frames are continuously not matched, the track or branch is terminated; the tracks or branches in the stable state are cut according to their continuous frame intervals to obtain a time sequence segment set, wherein each time sequence segment contains at least a second preset frame number threshold.

[0016] Further, the skeleton × object × scene hypergraph in step three is constructed, including: for each pet-related time sequence segment in the time sequence segment set, a preset number of pet body key points are extracted frame by frame to obtain a pet skeleton node set; time-overlapping pet-related time sequence segments and object-related time sequence segments are established as an object node set according to their type labels; the functional area where the centroid of the pet motion track in the pet-related time sequence segment is located is established as a scene node set; in each frame, a three-element hyperedge is generated for each pet skeleton node and the object nodes and scene nodes meeting a preset proximity condition in space to represent the high-order interaction relationship therebetween.

[0017] Further, before performing deterministic reasoning on the hypergraph, further comprising: generating a set of hyperedge attributes for the hyperedge by discretizing the ternary hyperedge, the discretizing comprising: a) discretizing spatial proximity relations of contact, proximity, and distance according to the minimum boundary distance from the skeleton node to the object candidate region; b) encoding relative orientation relations according to the centroid coordinate difference of the skeleton node relative to the object node; c) discretizing body support relations of stable support and moving support according to the position variation amplitude and displacement accumulation of the key points representing the left and right forepaws in the skeleton node within a time window; d) discretizing tail swing level relations according to the path length of the key point representing the tail tip relative to the key point representing the tail root in the skeleton node; e) marking the scene containment relation of the skeleton node with the scene node according to the centroid position of the skeleton node.

[0018] Further, performing deterministic reasoning on the hypergraph comprises a multi-round state updating process: performing a first round of updating, for the nodes related to the mouth, nose, head, and forepaws in the skeleton node set, counting the number of contact or proximity relations with the object node within a time window, thereby updating the skeleton local state of the skeleton node; performing a second round of updating, for the object node, counting the continuous appearance length of the skeleton node connected to it with the skeleton local state within a time window, and when the length reaches a third preset frame number threshold, marking the object node as a participation state; performing a third round of updating, for the scene node, counting the co-occurrence length of the skeleton node connected to it with the skeleton local state and the object node in the participation state, and when the co-occurrence length reaches a fourth preset frame number threshold, marking the scene node as an active state.

[0019] Further, constructing the Copula gating layer and performing frame-by-frame gating processing in step four comprises: generating three types of sequence labels in each frame within each feeding action segment or drinking action segment, the three types of sequence labels comprising: a) tail swing level label, graded assignment according to the path length of the tail movement; b) head-object contact duration label, graded assignment according to the contact frame number proportion of the head and the object within a time window; c) mouth shape sequence label, graded assignment as chewing sequence, swallowing sequence, or other sequence according to the opening and closing cycle number and average interval of the mouth region, and whether there is a periodic displacement of the vertical brightness band corresponding to the swallowing action; combining the three types of sequence labels into a triple, forming a Copula grid code.

[0020] Further, the frame-by-frame gating processing is performed, and the frame-by-frame gating processing further comprises: taking the Copula grid code as an index, querying a preset eating behavior pass table, a drinking behavior pass table and a general behavior inhibition table, and marking each frame as pass, inhibition or neutral; and calculating a pass frame proportion in each action segment, and confirming the action segment as a valid segment when the pass frame proportion reaches a preset pass frame proportion threshold, otherwise, the action segment is inhibited; wherein the general behavior inhibition table comprises a code satisfying the following conditions: a tail swing level indicated by a tail swing level label exceeds a preset swing level threshold, and a sequence type indicated by a mouth shape sequence label does not match a current action segment type, so as to complete the inhibition processing of false eating and false drinking behaviors by identifying the internal contradictions of the behavior signals.

[0021] With the above technical solutions, the present application has the following beneficial effects: by introducing a space-time multi-dimensional modeling strategy based on image recognition in the process of pet behavior recognition, the present application can realize unified understanding and high-precision interactive recognition among pets, objects and scenes, thereby significantly improving the stability and reliability of recognition in a complex home environment. The present application adopts a processing mode combining superpixel segmentation and region association graph, so that the pet candidate region, object candidate region and scene functional area have explicit structured representation at the image level, providing a reliable geometric basis for subsequent time sequence tracking. By introducing a Poisson multi-Bernoulli mixed tracking link, the present application establishes a dynamic balance between time sequence continuity and target existence, and can maintain trajectory continuity in the case of occlusion, light change or camera jitter, thereby realizing stable space-time tracking of pets and objects. Subsequently, by constructing a skeleton-object-scene supergraph, the present application integrates pet skeleton key points, object nodes and scene nodes into the same graph structure, and realizes logical deduction of behavior recognition through a deterministic reasoning path, thereby ensuring reviewability and explainability. The present application further designs a Copula gating layer for frame-by-frame gating analysis of behaviors such as eating and drinking, and through joint judgment of three types of behavior signals, tail swing level, contact persistence and mouth shape sequence, it can effectively distinguish between real intake actions and non-intake proximity, thereby inhibiting false eating and false drinking phenomena and improving the recognition accuracy of real pet behaviors. The structured processing mode adopted by the present application enables the system to have stronger anti-interference capability, and the system can still maintain stable recognition output under conditions such as multiple pets coexisting, vessel position changing and complex light. At the same time, the present application adopts a threshold decision mechanism that can be calculated throughout the process, which is convenient for direct deployment in different devices and application scenarios, does not rely on high-performance training models or external databases, and has feasibility and high portability in low-power environments. The present application not only provides a general algorithm basis for intelligent cameras, pet companion robots and home security systems, but also provides an interpretable, traceable and high-precision visual recognition framework for animal health monitoring, behavior data modeling and other application scenarios, which has significant technological progress and wide application prospects. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 A method flowchart of a pet behavior analysis method based on image recognition provided for an embodiment of the present application is shown in FIG. 1.

[0023] Figure 2 A pet skeleton key point detection principle diagram provided for an embodiment of the present application is shown in FIG. 2.

[0024] Figure 3 A tail swing amplitude experimental curve diagram provided for an embodiment of the present application is shown in FIG. 3.

[0025] Figure 4 A head and object contact persistence timing curve diagram provided for an embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0026] All features disclosed in this specification, and / or all steps of any methods or processes disclosed, may be combined in any combination, except combinations where at least some of the features and / or steps are mutually exclusive.

[0027] Any feature disclosed in this specification, unless stated otherwise, can be replaced by any equivalent or similar feature. That is, unless stated otherwise, each feature is one example only of a generic series of equivalent or similar features.

[0028] Embodiment 1: Reference Figure 1 A pet behavior analysis method based on image recognition, the method comprising:

[0029] Step one: Obtain a video frame sequence, perform superpixel segmentation on each frame image in the video frame sequence to obtain a superpixel graph composed of a plurality of superpixel blocks; generate a pet candidate region, an object candidate region and a scene layout on the superpixel graph, and establish a region association graph containing nodes respectively representing the pet candidate region, the object candidate region and the functional area in the scene layout, and edges reflecting the spatiotemporal adjacency relationship between the nodes.

[0030] When acquiring the video frame sequence, the camera output is set to a resolution of 1920x1080 and a frame rate of 25 frames per second to ensure subsequent image detail and temporal continuity. Each frame of image is first denoised and normalized in brightness. The denoising uses a 3x3 median filter to remove isolated noise points, and the brightness normalization aligns the main peak of the histogram to the medium brightness area by linear stretching, thereby maintaining edge and texture stability under low light and backlight conditions. Then, the current frame and the previous frame are subjected to pixel-level absolute difference, and pixels with a difference of 25 or more are marked as motion pixels to obtain a binary motion map. The binary motion map is subjected to corrosion and expansion with a radius of 1 for 1 time each to eliminate discrete noise points and connect the fine boundaries to obtain a motion mask. The inter-frame difference is selected and combined with light morphological processing because the pet motion in the home scene usually causes moderate amplitude pixel changes, and a difference threshold of 25 can cover hair shaking and body displacement while avoiding excessive response to slight light fluctuations.

[0031] After the above preprocessing is completed, superpixel segmentation is performed on each frame of image. Specifically, a simple linear iterative clustering superpixel method is used to aggregate pixels based on joint color and spatial position measurement; at a resolution of 1920x1080, the target superpixel number is set to 800 to 1200, and the average edge length of each superpixel block is about 30 to 40 pixels. This range is selected because the pet's ears, mouth, nose, and front paws and other details can be completely contained at this scale, and the object edges can be fitted by the superpixel boundaries. After segmentation, a superpixel map composed of a plurality of superpixel blocks is obtained, and the color mean value, centroid coordinates, and boundary pixel set of each superpixel block are saved.

[0032] In the process of generating pet candidate regions on the superpixel map, first, the superpixel blocks are labeled according to the motion mask: when the proportion of motion pixels in the superpixel block reaches or exceeds 50%, it is marked as a motion block. Region merging is performed starting from the motion block, with the rule being: adjacent superpixel blocks that share a boundary length reaching or exceeding 6 pixels and have a color mean difference less than or equal to 10 are merged into the same connected domain. The setting of this rule can ensure that merging only occurs when the edges are truly connected and the color is continuous, avoiding false merging of background and foreground at narrow contact points. After the connected domain is formed, the number of pixels is counted, and when the number of pixels reaches or exceeds 5000, it is registered as a pet candidate region. Calculate the bounding rectangle for each pet candidate region, and keep the region when the aspect ratio is between 0.5 and 3.0 to enter the pet candidate region set. The reason for setting the area and aspect ratio conditions is that small area noise spots and strip shadows usually cannot meet these two joint conditions, while normal size cats and dogs meet this range in common postures, thus improving the relevance of screening. In an optional implementation, the ratio of perimeter to area of the outer contour can also be calculated after region merging, and when the ratio is between 0.03 and 0.20, the region is preferentially retained. This ratio interval covers the common roughness of hair edges, which can further distinguish between flat reflections and real animal outlines.

[0033] When generating object candidate regions on the superpixel map, an edge detection method combined with geometric shape recognition is used. The current frame is converted to a hue, saturation, and brightness color space, and the brightness channel is taken for double-threshold-based gradient edge detection to obtain a refined edge map. Hough circle detection is performed in the edge map, with a radius search range of 15 to 80 pixels, and detection results with a circular arc coverage angle reaching or exceeding 180 degrees are registered as container circles. For each container circle, calculate the proportion of superpixel blocks intersecting it to the area of the container circle, and when the proportion reaches or exceeds 20%, merge these superpixel blocks and label them as feeding or water candidate regions. Then perform line detection on the edge map and perform rectangle fitting, and when the number of rectangle corner points reaches or exceeds 4 and the aspect ratio is between 1.0 and 2.5, it is registered as a rectangular candidate region. To identify toy candidate regions, calculate the mean value of each connected region in the saturation channel, and when the mean value reaches or exceeds 64 (based on 0 to 255) and the area is between 500 and 20000 pixels, the connected region is registered as a toy candidate region. The setting of the above threshold takes into account the common size and color characteristics of household items: feeding and water containers are usually circular or nearly circular with continuous edges; toys are usually made of high-saturation materials and have moderate areas; litter boxes are usually rectangular or nearly rectangular with obvious straight-line edges. In an optional implementation, candidate regions that are nearly circular but have gaps can also be closed and repaired, i.e., when the gap length between the arcs is less than or equal to 15% of the circumference, the complete circle is completed to accommodate partially obscured containers.

[0034] In generating the scene layout, the ground area is searched in the lower half of the image. The specific method is to count the texture response of each row, which can be accumulated by the absolute value of 3x3 Laplacian filter. When the accumulated value of consecutive rows remains at a low level and the width reaches or exceeds 70% of the image width, the consecutive row band is taken as the ground band. The reason for choosing low texture response as the ground band is that the floor of a home is usually a large, uniform material, continuous in the horizontal direction and with small texture fluctuations. Then, within the ground band, the functional area is divided according to the location of the object candidate area: the collective bounding rectangle of all feeding candidate areas is calculated, and each side is extended outward by 10 pixels to form a feeding area; the drinking area is formed in the same way for all water candidate areas; within the ground band, after removing the feeding area and the drinking area, the remaining connected area is classified as the activity area. This method can use the location of the object to provide a direct anchor for the functional area, aligning the functional area with the actual use location. In an optional implementation, if the camera has depth image capability, the depth histogram can be first detected for peak value, and the depth range corresponding to the peak value closest to the camera and with the widest distribution is selected as the ground range; the reason for selecting the main peak on the side close to the camera is that the common home installation position aligns the area below the camera with the ground, and the depth range corresponding to the main peak covers the passable plane, which can stabilize the positioning of the ground in complex lighting conditions; repeating the above functional area division steps within the depth range can improve the robustness in low light and reflective scenes.

[0035] In establishing the region association graph, the pet candidate area, the object candidate area, and each instance of the three functional areas in the scene layout are respectively registered as nodes. For nodes within the same frame, the edges establishing spatial relationships are established, including contact edges, adjacent edges, and containing edges. The establishment condition of the contact edge is that the intersection area of the bounding rectangles of two nodes reaches or exceeds 100 pixels; the establishment condition of the adjacent edge is that the minimum boundary distance between the bounding rectangles of two nodes is located in 1 to 15 pixels; the establishment condition of the containing edge is that the bounding rectangle of one node is completely located inside the bounding rectangle of another node and the minimum value of the boundary distance in four directions reaches or exceeds 3 pixels. The contact edge is used to represent the state of direct physical contact or high proximity, which is conducive to subsequent identification of behaviors such as eating and drinking that require contact; the adjacent edge provides a transition relationship before contact, which is conducive to forming the start criterion of the behavior; the containing edge makes the attribution of the node in the functional area clear, avoiding the loss of functional area attribution due to short-term occlusion.

[0036] To introduce the time factor, the edges of cross-frame relationship are established between adjacent two frames for the nodes of the same type. For the nodes of pet candidate regions, the ratio of the overlapping area of the bounding rectangles of adjacent two frames to the area of the bounding rectangle of the last frame is calculated. When the ratio reaches or exceeds 0.30 and the Euclidean distance of the centroids of the two frames reaches or is less than 30 pixels, the edge of cross-frame relationship is established. For the nodes of object candidate regions, the edge of cross-frame relationship is established when the ratio of the overlapping area of the bounding rectangles of two frames to the area of the bounding rectangle of the current frame reaches or exceeds 0.70 due to the relative fixation. For the nodes of functional areas in the scene layout, the edge of continuous relationship is established frame by frame when the geometric definition of the functional area remains unchanged. The edges of cross-frame relationship provide consistent association conditions for subsequent trajectory generation and time segment division, so that the spatio-temporal adjacency relationship is explicitly expressed through unified edge structure.

[0037] For the convenience of subsequent processing and review, the type of relationship and the triggering basis are recorded for each edge in the region association graph, including the intersection area value, the minimum boundary distance value, the four-way boundary distance value of the containing relationship, the cross-frame overlap ratio value and the centroid displacement value. The above values are derived from the defined geometry and pixel operation, and have repeatability. The present embodiment forms a complete path from single-frame perception to spatio-temporal association through pet candidate generation based on superpixels, relationship construction based on actual geometric conditions and scene layout with functional areas as anchor points, so that the subsequent construction of Poisson Multi-Bernoulli Mixture tracking link and skeleton-object-scene hypergraph has stable input. In an optional embodiment, after the region association graph is established, the main direction information of the edge direction histogram can be added to each node of the pet candidate region. When the main direction remains within ± 15 degrees in consecutive frames, the node is marked as stable posture to support the subsequent application scenarios that are more sensitive to the discrimination of start and end time.

[0038] Step two: taking the region association graph as input, constructing Poisson Multi-Bernoulli Mixture tracking link, establishing trajectories for pet candidate regions, object candidate regions and anchor points of functional areas respectively, so as to uniformly output a trajectory entity set containing time sequence information, and dividing the trajectory entity set into a time sequence segment set.

[0039] Specifically, taking the region association graph as input, a Poisson multi-Bernoulli mixture tracking link is constructed, and trajectories are established for pet candidate regions, object candidate regions, and anchor points of functional areas, respectively, and are uniformly output as a trajectory entity set containing time sequence information, and then a time sequence fragment set is generated according to consistent segmentation criteria. For ease of implementation, the following process takes a continuous video with a frame rate of 25 frames per second and a resolution of 1920x1080 as an example. Each node in the region association graph carries an enclosing rectangle, a centroid coordinate, a type label, and a source frame number, and the edges include contact, proximity, containment relationships in the same frame and adjacent association relationships across frames. First, the classification and timestamp synchronization of candidate observations are performed. The nodes in the same frame are divided into three groups according to the type, which are the pet candidate region group, the object candidate region group, and the anchor point group of the functional area. The pet candidate region group is used to describe the animal contour appearing in the motion mask, the object candidate region group is used to describe the feeding device, the water, the toy, and the litter box, and the anchor point group of the functional area is used to describe the spatial reference points of the feeding area, the drinking area, and the activity area. In order to ensure the temporal consistency of tracking, a frame number and an arrival order number are attached to each node. The frame number strictly increases, and the arrival order is arranged from left to right according to the horizontal position of the centroid in the same frame. This synchronization method enables the subsequent matching process to have a determined traversal order, reducing the randomness in high-density scenes.

[0040] Then a Poisson birth candidate pool is established as the source of new birth in the Poisson multi-Bernoulli mixture tracking link. For each frame, nodes that are not connected to any trajectory in the previous frame by a cross-frame relationship edge are added to the new birth candidate pool, and the motion density around the node is calculated. The method for calculating the motion density is to count the proportion of motion mask pixels in the region obtained by extending the node's enclosing rectangle outward by 10 pixels. When the proportion reaches or exceeds 40%, it is considered that there is an independent motion source in the region. This peripheral proportion method is used because when a pet enters the picture, it usually accompanies the appearance of continuous motion in the local area, and the peripheral proportion can remain stable when the background fluctuates slightly. When the number of members in the new birth candidate pool in the same frame reaches or exceeds 3, merging is performed according to the minimum distance between the centroids. Candidates with a distance of 20 pixels or less are merged into a new birth event. Each new birth event generates a new trajectory in the current frame, and the trajectory type inherits from the type label of the corresponding node. The Poisson feature is that the number of new birth events is directly driven by real-time detection results, which can quickly establish corresponding trajectories when pets suddenly enter or new objects are placed.

[0041] For existing trajectories, multi-Bernoulli existence management and data association are used to maintain the continuity of each trajectory. A state label is maintained for each active trajectory, including four states: birth, stable, pending confirmation and termination. A new trajectory is in the birth state after its generation. For the matching between adjacent two frames, geometric constraints are used for gating within the same type of nodes. The gating condition is that the overlap ratio of the current node and the bounding rectangle of the last frame of the trajectory in the last frame reaches or exceeds 0.30 and the distance between the centroids reaches or is less than 30 pixels. The setting of the gating threshold considers the maximum step of the pet under the condition of 25 frames per second and the displacement amplitude under the lens view angle. The double conditions of the overlap ratio and the distance can simultaneously constrain the deformation and displacement of the target. When a trajectory finds only one node that meets the gating condition in the next frame, the node is associated with the trajectory one-to-one and the trajectory is extended. When a trajectory finds two or more nodes that meet the gating condition in the next frame, the trajectory generates the same number of branches, each branch inherits the identification of the original trajectory and records the matching path of each branch respectively, and all branches are added to the hypothesis pool. The reason for retaining multiple branches when multiple matches occur is that indoor occlusion and mutual penetration will cause ambiguity for a short time. Retaining branches can automatically distinguish the most consistent path in subsequent frames through continuous matching.

[0042] To avoid unbounded growth of branches, branches in the hypothesis pool are ordered pruned. They are ordered by the number of consecutive matches from high to low. If there is a tie, the average of the overlap ratio of the past 5 frames is compared. The one with higher average is preferred. If there is still a tie, the variance of the centroid displacement of the past 5 frames is compared. The one with smaller variance is preferred. Each original track is kept with at most 3 active branches. The branches beyond that are terminated directly. Such pruning rule prefers to keep the paths that are stable and have better geometric consistency, while controls the computation. The whole tracking pipeline can maintain real-time performance of more than 20 frames per second on a single video. The state of a track is updated at the end of each frame. A track or branch enters stable state when the number of consecutive successful matches reaches or exceeds 5. A track in stable state enters pending state when it fails to match in the current frame. It is used to tolerate short occlusion or motion mutation. If it fails to match for 2 consecutive times, the track is terminated. The end frame number and the last bounding rectangle are recorded. This three-stage management uses fixed number as threshold. It is beneficial to keep consistent decision criteria under different lighting and different room layouts, making the engineering implementation simple and reviewable. The object candidate region group and the anchor point group of functional area are set with different matching priorities based on their motion characteristics. The feeding device and the water in the object candidate region group are usually stationary. Therefore, a more stringent geometric constraint is used in cross-frame matching. The overlap ratio of the bounding rectangle reaches or exceeds 0.70 to be associated and the centroid distance is not limited. The reason is that the slight camera jitter will cause small changes in the centroid, while the high overlap ratio is sufficient to represent the consistency of the static position. The toys and the litter box may have small displacement after user interaction. Therefore, the gating condition of the overlap ratio reaching or exceeding 0.50 and the centroid distance reaching or being less than 15 pixels is used. The anchor points of the functional area remain constant through their geometric definition. A continuous track is generated for the same area in each frame. The bounding rectangle and the centroid of the track are directly given by the geometric shape of the functional area. Such grouping strategy makes objects with different kinematic characteristics be treated differently in the same Poisson-Multinomial mixture tracking pipeline. The matching criteria and object properties are matched, thereby improving the overall robustness.

[0043] In the order of performing cross-frame data association, the pet candidate region track in stable state is processed first, then the pet candidate region track in birth state, and finally the object candidate region track and the anchor point track of functional area. The reason for this order is that the occlusion and fast action of the pet will cause greater geometric changes, and preferentially assigning the most suitable node to the stable track can reduce subsequent conflicts; the track in birth state needs to be confirmed as soon as possible, and early matching is beneficial to shorten the confirmation time; the object and the anchor point of the functional area are more stable in vision, and delayed processing will not affect the association quality. For the nodes that may be requested by multiple tracks in the same frame, the Hungarian method is used to assign between the track and the node once, and the cost is defined as the combination of the inverse order of the overlap ratio and the positive order of the centroid distance. The reason for using one-time assignment is that it can reduce conflicts and improve the consistency of matching in the sense of global optimality. After the matching of all frames is completed, the three types of tracks are uniformly summarized to form a track entity set. Each element in the track entity set contains a unique identifier, a type label, a birth frame number, an end frame number, a sequence of in-frame bounding rectangles, a sequence of centroids, and a state transition record. In order to be used for subsequent skeleton × object × scene hypergraph construction, the track of the pet candidate region in the track entity set also saves the relationship snapshot of the object candidate region and the anchor point of the functional area that it contacts or is adjacent to in each frame. The relationship snapshot is derived from the contact edge and the adjacent edge in the region association graph, and is recorded as a binary relationship list for each frame. This structure allows the temporal relationship and spatial relationship to be retained simultaneously, providing a directly usable input for action recognition path.

[0044] When the track entity set is converted into a time sequence fragment set, a unified segmentation criterion is adopted. For the track in stable state, segmentation is performed according to the state continuous interval, and each segment contains at least 15 frames. When a track transitions from stable state to pending state and terminates within the next 2 frames, the stable state end frame is taken as the end point of the fragment; when the track appears once short unmatching after the stable state and recovers matching in the next frame, the fragment is not segmented, but a short occlusion event is marked in the fragment record, and the marking of the occlusion event includes the occlusion frame number and the occlusion duration. For the object candidate region and the anchor point track of the functional area, segmentation is performed according to the time point of geometric definition change, for example, when the bounding rectangle of the feeder and the water fountain changes by more than 10% in area, it is considered that a position adjustment has occurred, and the previous frame of the position adjustment is taken as the end point of the fragment, and the next frame is taken as the starting point of the new fragment. The fragment length of at least 15 frames is adopted to ensure that each fragment covers a time range of not less than 0.6 seconds, so that the subsequent action recognition can be performed within a window containing start, duration and end information, and the influence of short jitter is reduced.

[0045] In an optional embodiment, in order to reduce the track fragmentation caused by the occlusion, track bridging can be performed within a short time window after the track termination. When the time interval of two tracks of the same type is 10 frames or less, the centroid distance between the last bounding rectangle of the end segment and the first bounding rectangle of the start segment is 40 pixels or less, and the area difference of the bounding rectangle is 30% or less, the two tracks are merged into a continuous track, and a bridge event is recorded in the track entity set. Track bridging can maintain a complete time sequence description when the pet passes through the occlusion for a short time or quickly passes through the edge of the picture, and helps the subsequent strong recognition of the interactive behavior to infer the initial result in a longer continuous window. In another optional embodiment, in order to further improve the stability of the geometric quantity, the sliding smoothing can be performed on the bounding rectangle and the centroid sequence at the track level. The median value of the sliding window with a length of 5 frames is used to replace the original value of the vertex coordinates and the centroid coordinates of the bounding rectangle of each track, and the median value is more robust than the mean value when there are abnormal points, which can reduce the influence of single-frame detection deviation on the continuity of the track. After the smoothed track is segmented, the missegmentation events caused by boundary jitter can be effectively reduced.

[0046] Step three: taking the time sequence segment set as input, constructing a skeleton x object x scene hypergraph, wherein the hypergraph includes a pet skeleton node set, an object node set and a scene node set; performing deterministic reasoning on the hypergraph along a preset action recognition path, and outputting a strong recognition initial result of the interactive behavior.

[0047] Taking the time sequence segment set as input, the pet body key points are extracted frame by frame for each time sequence segment related to the pet to generate a pet skeleton node set. The specific method is as follows: in each frame, the bounding rectangle recorded in the time sequence segment is taken, the bounding rectangle is expanded outward in each direction by 10 pixels to obtain an analysis window, the analysis window is bilinearly interpolated to 256x256 size to ensure that the ears and mouth details of small-sized pets can still be distinguished; in the analysis window, a key point response heat map is generated by using a public animal pose estimation method, the peak points are located on each heat map by 3x3 neighborhood non-maximum suppression, then the brightness centroid is calculated in the 3x3 neighborhood of the peak point to obtain a sub-pixel level coordinate, and finally the sub-pixel level coordinate is mapped back to the original frame coordinate system. Ten key points are fixedly extracted in each frame, which correspond to the head, mouth, neck, left front paw, right front paw, left rear paw, right rear paw, tail root, tail middle and tail tip respectively. The reason for fixing 10 key points is that the above-mentioned key points can cover the contact parts and supporting parts involved in the interactive behaviors such as eating, drinking and playing, while retaining the observation ability of the tail movement pattern, so that the subject action can still be stably described under the occlusion and low illumination conditions in the home scene.

[0048] Figure 2 The pet skeleton key point detection principle diagram provided by the embodiment of the application is shown in FIG. 1.Figure 2 This diagram illustrates the spatial distribution of key points on a pet's body and their interconnections. Figure 2 In this design, the pet's body outline is represented by an elliptical outer contour, with a major axis of 150 pixels and a minor axis of 80 pixels. The head area is represented by a separate circular contour with a radius of 40 pixels, located on the left front of the main body contour. This invention defines 10 key points on the pet's body, with the positions and functions of each key point described as follows: The first key point is the head key point, located at the center of the upper part of the head's circular contour, used to track the overall movement trajectory of the head; the second key point is the muzzle key point, located at the left front of the head's circular contour, which is the core contact point for determining eating and drinking behaviors; the third key point is the neck key point, located at the connection between the head and the main body, serving as a pivotal node between the head and torso. The fourth and fifth key points are the left and right forepaw key points, respectively, both located below the front of the main body, vertically 60 pixels from the ground, with a horizontal distance of 40 pixels between them. These two key points are used to determine the pet's support status. A stable support status is defined as a vertical position change of 5 pixels or less and a cumulative horizontal displacement of 15 pixels or less within a 15-frame window for both front paw key points; otherwise, a moving support status. The sixth and seventh key points are the left and right hind paw key points, located below the rear of the body, 60 pixels vertically from the ground, with a horizontal distance of 40 pixels between them. These are primarily used to assist in determining the pet's movement and posture stability. The eighth key point is the tail root key point, located at the far right of the body's outline, marking the connection between the tail and torso. The ninth key point is the tail middle key point, located 40 pixels to the right of the tail root key point. The tenth key point is the tail tip key point, located 40 pixels to the right of the tail middle key point. The tail swing level is determined by calculating the cumulative path length of the tail tip relative to the tail root within a 15-frame window. When the cumulative path length reaches or is less than 8 pixels, it is marked as level 0; when it is between 9 and 20 pixels, it is marked as level 1; and when it reaches or is greater than 21 pixels, it is marked as level 2. Figure 2 The skeletal connections between keypoints are represented by dashed lines: the head keypoint is connected to the neck keypoint; the neck keypoint is connected to the left forepaw, right forepaw, and body center; the body center is connected to the left hindaw, right hindaw, and tail root; and the tail root, tail middle, and tail tip are connected sequentially to form the tail skeletal chain. The technical parameters for keypoint extraction include: an analysis window size of 256×256 pixels, with bilinear interpolation used to normalize the circumscribed rectangular region to this size; non-maximum suppression of the heatmap response using a 3×3 neighborhood; and subpixel-level coordinates obtained by calculating the brightness centroid within the 3×3 neighborhood of the peak point.

[0049] For each time-segment related to an object, the centroid of the object's bounding rectangle in each frame is directly taken as the spatial position of the object node, and the type label is inherited. The type labels include food, water, toy and litter. For each time-segment of a functional area, the polygon boundary of the functional area in each frame is given by the geometric definition of the time-segment, and the geometric center of the polygon is taken as the spatial position of the scene node. The scene types include feeding area, drinking area and activity area. The three types of nodes are aligned on the time axis to obtain the pet skeleton node set, the object node set and the scene node set.

[0050] In each frame, a ternary hyper-edge is generated by the pet skeleton node, the object node that meets the proximity condition with the skeleton node in space, and the scene node corresponding to the functional area where the centroid of the pet's bounding rectangle is located in the frame. The proximity condition is defined as the minimum boundary distance between the skeleton node and the object node's bounding rectangle reaching or being less than 15 pixels. When there is more than one object node meeting the proximity condition, the one with the minimum distance is selected to generate the ternary hyper-edge. The minimum distance selection is used to preferentially depict the object that is more likely to have direct interaction when the toy and the food appear in the screen at the same time, thereby reducing the interference of irrelevant objects in subsequent action recognition. For the skeleton node that does not meet the proximity condition, a ternary hyper-edge is still generated with the scene node it is located in, but the object node position is set to a null placeholder, which facilitates maintaining the structural consistency on the time axis.

[0051] Each ternary hyper-edge is attached to a set of discretized attributes, all of which come from geometric and pixel computable quantities. Specifically, they include: 1. Spatial proximity relationship, which is divided into three categories of contact, proximity and distance according to the minimum boundary distance between the skeleton node and the object node's bounding rectangle, where the contact is determined as the distance reaching or being less than 3 pixels, the proximity is determined as the distance being between 4 and 15 pixels, and the distance is determined as the distance reaching or being greater than 16 pixels; 2. Relative orientation relationship, which is divided into four orientations of up, down, left and right according to the horizontal and vertical offsets of the skeleton node relative to the object node's centroid; 3. Body support relationship, which is marked as stable support when the vertical position change amplitude of the left front paw and the right front paw within a 15-frame window reaches or is less than 5 pixels and the horizontal displacement cumulative reaches or is less than 15 pixels, otherwise it is marked as mobile support; 4. Tail swing level, which is graded according to the path length of the tail tip relative to the tail base within a 15-frame window, where the length reaching or being less than 8 pixels is level 0, the length being between 9 and 20 pixels is level 1, and the length reaching or being greater than 21 pixels is level 2; 5. Scene inclusion relationship, which is marked according to whether the centroid of the pet's bounding rectangle is located in the feeding area, drinking area or activity area. The above attributes and ternary node identifiers are recorded together in the hyper-edge attribute set of the frame, and the hyper-edge attribute sets of all frames together constitute the skeleton × object × scene hypergraph.

[0052] Determination inference is performed on the skeleton × object × scene supergraph along the preset action recognition path, and the inference target is to output the strong recognition initial result of the interactive behavior on the time axis. The inference adopts two-level rules from local evidence to segment-level decision. The first level is local evidence construction. For each time sequence segment, a sliding window with a length of 15 frames and a step size of 1 frame is used on the time axis. The following computable conditions are checked in each window and a window label is generated. The window label condition of the eating path is that there is at least one object node whose type is eating utensil and the spatial proximity relationship with the mouth-nose skeleton node is contact, the relative orientation of the head skeleton node with the object node appears in the main orientation of the window, and the left front paw and the right front paw are both stable support, and the scene contains relationship is eating area. The window label condition of the drinking path is that there is at least one object node whose type is water and the spatial proximity relationship with the mouth-nose skeleton node is contact, at least one of the left front paw and the right front paw is stable support, and the scene contains relationship is drinking area. The window label condition of the playing path is that there is at least one object node whose type is toy and the spatial proximity relationship with the left front paw or the right front paw is contact or close in the window, and the cumulative frame number reaches or exceeds 9 frames, and the body support relationship is mobile support, and the scene contains relationship is activity area. The above conditions are all derived from directly computable distance, orientation and frame number statistics, and the window accumulation can filter single-frame detection noise while retaining the start and duration characteristics of the action.

[0053] The second level is segment-level decision and object participation confirmation. For the same path, when the distance between the midpoints of two windows reaches or is less than 5 frames and the window labels are consistent, the two windows are merged into a longer candidate segment. The start point of the candidate segment is the start frame of the first window, and the end point is the end frame of the last window. Then object participation confirmation is performed on the candidate segment: the continuous length of the object node appearing in contact with the mouth-nose skeleton node or the front paw skeleton node in the segment is counted, and when the length reaches or exceeds 12 frames, the object node is marked as participating, and the object type is written into the object participation field of the candidate segment. If both the eating utensil and the toy are marked as participating, the eating utensil participation is retained first and the toy participation mark is removed from the current segment, because when eating, the pet usually keeps the head stable and is in continuous contact with the utensil, which is a stronger interactive evidence, and the coexistence of the two types of participation in the same time period will cause ambiguity in the meaning of the category. The same priority strategy is adopted for the drinking path and the playing path, and the priority order is eating path first, drinking path second, and playing path last. This priority order conforms to the difference in utensil contact persistence in the family scene, and the stronger the contact persistence, the more stable the behavior, and the priority output can reduce category crossover.

[0054] Conflict resolution is performed in the presence of path conflicts. Conflict is defined as the time overlap of two different path's candidate segments within the same time segment. When conflict occurs, the ratio of contact frames within the two candidate segments is compared, the candidate segment with higher ratio of contact frames is kept, the other candidate segment is truncated into two non-overlapping segments within the overlapping interval. If the ratio of contact frames is the same, the matching degree of scene inclusion is compared, the candidate segment that is completely located in the corresponding functional area is kept. If still the same, the candidate segment with earlier start time is kept. To enhance the tolerance of short-time jitter, segment stitching is performed. For two candidate segments of the same path, when the time interval of the two reaches or is less than 5 frames and the object involvement fields of the two are the same and the adjacent two segments are consistent with the same scene type, the two are merged into one longer segment, the starting frame of the merged segment is the starting frame of the first segment, and the ending frame of the merged segment is the ending frame of the second segment. Segment stitching can eliminate the gap caused by short-time occlusion or slight departure, thereby more truly reflecting a complete interaction. After the above steps are completed, the final segments of all paths are collected in chronological order to form the strong identification initial result of the interactive behavior. Each result entry includes the following fields: behavior type label, starting frame number, ending frame number, object involvement type, scene type, and key evidence summary. The key evidence summary includes the ratio of contact frames within the segment, the main orientation of the mouth-nose skeleton node and the object node, the support state distribution of the left front paw and the right front paw, and the tail swing level distribution. The key evidence summary is saved because it can directly trace back to the geometric facts supporting the conclusion in subsequent decision-making and visualization, and it is also convenient for reviewing the adaptability of threshold selection and window length setting in engineering debugging.

[0055] In an optional embodiment, the relative orientation relationship can be extended from the four directions of up, down, left, and right to eight directions, that is, left up, right up, left down, and right down are added on the basis of the four directions, so as to obtain a more fine-grained angle description when the camera has a large overhead angle or the pet is inclined to eat. In this embodiment, the main orientation matching condition of the eating path and the drinking path is changed to the main orientation belonging to one of down, left down, and right down, which can cover the common posture of the pet approaching the container in a diagonal direction. In another optional embodiment, when the camera supports depth image acquisition, the ground plane of the functional area can be fitted once in each frame, and then the eating path and the drinking path window label are only allowed to be output when the distance between the lower edge of the pet's bounding rectangle and the fitted plane reaches or is less than 5 cm. The advantage of this is that it can exclude the false triggering caused by the visual approach of the pet to the pattern of the container at a high place of furniture, and the consistency of the depth geometry ensures that the interaction must occur in the same plane that can be reached.

[0056] Step four: For the eating action segment and the drinking action segment in the strong recognition initial result of the interactive behavior, a Copula gating layer is constructed and frame-by-frame gating processing is performed to suppress false eating or false drinking behavior, and finally the behavior analysis result carrying the object participation type and the scene type is output.

[0057] For the eating action segment and the drinking action segment in the strong recognition initial result of the interactive behavior, a Copula gating layer is constructed and frame-by-frame gating processing is performed, aiming to maintain the true eating and drinking segments while suppressing false eating or false drinking judgments caused by factors such as sniffing, approaching the vessel pattern, camera angle deviation, etc. The whole process relies on three types of calculable sequence labels, namely tail swing level label, head and object contact duration label, and mouth shape sequence label. After generating three types of labels in each frame, they are combined into a Copula grid code, and the frame-by-frame pass, suppression or neutral decision is given by looking up the table, and the confirmation and suppression are completed at the segment level. For each eating action segment or drinking action segment, first, a sliding window with a length of 15 frames and a step of 1 frame is established within its frame range. The length of 15 frames corresponds to about 0.6 seconds under the condition of 25 frames per second, which covers the complete opening and closing cycle of a chewing or the throat lifting and falling process of a swallowing, so that each window contains both instantaneous contact information and physiological action time structure. The following three types of sequence labels are calculated within the window and landed as discrete levels at each frame.

[0058] The generation of the tail swing level label is based on the tail tip key point and the tail root key point in the pet skeleton. The coordinate difference of the tail tip and the tail root is recorded for each frame, and the path formed by this difference within the window is lengthened. The cumulative method is to add the displacement length of the tail tip relative to the tail root of adjacent frames one by one to obtain the total path length within the window. If the total path length is equal to or less than 8 pixels, it is marked as level 0, if it is between 9 and 20 pixels, it is marked as level 1, and if it is equal to or greater than 21 pixels, it is marked as level 2. The path of the tail tip relative to the tail root is used as a measure because the tail swing amplitude of the pet increases significantly when it is excited, expected or interacting with people, while the tail tends to be stable or only slightly swings during real eating or real drinking; by distinguishing three amplitude levels, high amplitude swing related to emotion can be separated from eating and drinking mouth-nose contact evidence, avoiding the excitement state near the vessel being mistaken for continuous intake.

[0059] Figure 3 The tail swing amplitude experimental curve provided by the embodiment of the present application is shown in FIG. 3. Figure 3 The horizontal axis of the curve represents time, in frames, ranging from 0 frames to 150 frames, with every 25 frames corresponding to 1 second; the vertical axis represents the tail swing amplitude, in pixels, ranging from 0 pixels to 35 pixels. Figure 3Three experimental curves are shown in the figure, corresponding to tail swing amplitude time series of feeding behavior, drinking behavior and playing behavior respectively: the first curve is the feeding behavior curve, represented by a solid line, the swing amplitude in the entire time range is maintained between 3 to 7 pixels, the average value is about 5 pixels, and the fluctuation amplitude is less than 4 pixels. The curve shows that the pet keeps the tail in a relatively static state during feeding, only with slight natural swing, because feeding requires the head and body to remain stable in order to accurately contact the feeding device. The second curve is the drinking behavior curve, represented by a short dashed line (3 pixels solid line, 3 pixels blank), the swing amplitude of the curve fluctuates between 8 to 15 pixels, with an average value of about 11 pixels. The medium amplitude swing reflects the slight excitement state of the pet when drinking water, but the overall body posture remains relatively stable. The third curve is the playing behavior curve, represented by a long dashed line (5 pixels solid line, 5 pixels blank), the curve presents a large amplitude irregular fluctuation, the swing amplitude changes dramatically between 10 to 32 pixels, the peak value can reach 32 pixels, and the valley value also maintains above 10 pixels. The high amplitude and high frequency change characteristics of the curve reflect the excitement state and active body movement of the pet when playing. Figure 3 In the figure, three level thresholds are marked with horizontal dashed lines: the level 0 threshold line is located at 8 pixels, the level 1 threshold line is located at 20 pixels, and the level 2 threshold line is located at 21 pixels. According to these thresholds, the tail swing of feeding behavior is mainly in the level 0 range, the drinking behavior is in the transition range from level 0 to level 1, and the playing behavior frequently reaches the level 2 range. Experimental conditions: the data acquisition frame rate is 25 frames per second, the tail swing amplitude is obtained by calculating the displacement path length of the tail tip key point relative to the tail root key point, and the statistical window length is 15 frames, corresponding to a time span of 0.6 seconds.

[0060] The generation of the head and object contact duration mark uses the minimum boundary distance between the head and mouth-nose key points to the object candidate region circumscribed rectangle. In the window, it is counted whether each frame satisfies the distance reaching or being less than 3 pixels. The number of frames that meet the condition is divided by the window frame number to obtain the contact proportion. The contact proportion reaching or exceeding 61% is marked as level 2, located in 40% to 60% is marked as level 1, and reaching or less than 39% is marked as level 0. The lower limit of 61% is selected as level 2, because in a 15-frame window, this threshold corresponds to at least 9 frames of stable contact, which can effectively distinguish between short stays and continuous fitting; the interval of 40% to 60% is selected as level 1, which can cover the exploratory licking, short-time leaving and returning to the vessel reciprocating process, so that such behaviors are retained as secondary strong evidence in subsequent judgment instead of being directly passed.

[0061] Figure 4 The head and object contact duration time series graph is provided for the embodiments of the present application. Referring to Figure 4The horizontal axis of the graph represents time in seconds, ranging from 0 to 8 seconds, and the vertical axis represents the minimum boundary distance between the head key points (including the mouth-nose key points) and the object candidate region's bounding rectangle in pixels, ranging from 0 to 21 pixels. Figure 4 The graph contains two main experimental curves and two threshold identification lines: the first curve is the real eating behavior curve, represented by a solid line. This curve exhibits a clear two-phase feature on the time axis: in the approach phase from 0 to 1 second, the distance increases from 12 pixels to 18 pixels, then rapidly decreases to less than 1 pixel, indicating a first far away and then a rapid approach; in the sustained contact phase from 1 second to 5.5 seconds, the distance is stably maintained between 0 and 2 pixels, indicating stable contact between the mouth-nose and the eating device; in the departure phase after 5.5 seconds, the distance rapidly increases to 18 pixels. The sustained contact phase accounts for 56.25% (4.5 seconds / 8 seconds) of the total time, and the time ratio when the distance is less than or equal to 3 pixels reaches more than 61%. The second curve is the false eating behavior (sniffing behavior) curve, represented by a long dashed line (5-pixel solid line, 5-pixel blank). This curve exhibits a rapid approach followed by a rapid departure: the distance decreases from 15 pixels to less than 3 pixels within 0 to 1.2 seconds, but fails to maintain stable contact, with repeated approach and departure actions within 1.2 to 2 seconds, and then the distance continues to increase and remains above 15 pixels during 2 to 8 seconds. The contact time ratio of this curve is less than 40%, reflecting the pet's behavior of only sniffing the food without actually eating. Figure 4 The two horizontal threshold lines in the graph are: the contact threshold line, located at 3 pixels, represented by a medium dashed line (10-pixel solid line, 5-pixel blank), and the distance less than or equal to this value is determined as a contact state; the approach threshold line, located at 15 pixels, also represented by a medium dashed line, and the distance between 3 and 15 pixels is determined as an approach state; and more than 15 pixels is determined as a far away state. Figure 4 In the graph, the sustained contact area from 1 second to 5.8 seconds is marked by a dashed rectangular frame, with a width of 4.8 seconds and a height covering a distance range of 0 to 21 pixels. Within this area, more than 90% of the data points of the real eating curve are below the contact threshold, while less than 20% of the data points of the false eating curve meet this condition. The quantitative evaluation of contact duration uses a 15-frame sliding window, and within each window, the frame ratio that meets the contact condition (distance ≤ 3 pixels) is counted: a ratio of 61% or more is marked as level 2 (strong contact), a ratio between 40% and 60% is marked as level 1 (moderate contact), and a ratio of 39% or less is marked as level 0 (weak contact or no contact). The real eating behavior is mainly at level 2 in the sustained contact phase, while the false eating behavior cannot reach the level 2 standard, which constitutes an important criterion for distinguishing between true and false eating behaviors.

[0062] The mouth shape sequence label is oriented to two observable perioral patterns of chewing and swallowing. A 48x48 local region is cropped around the perioral key points in each frame. Bilateral filtering is first performed on the region to preserve edges, and then binarization based on Otsu threshold is used to separate the oral opening area from surrounding tissues, obtaining the opening area and the opening contour. Subsequently, the switching times of "open" and "close" and the average interval between adjacent "open" are counted in the window by frame sequence; when the switching times are between 3 and 8 and the average interval is between 3 and 8 frames, it is determined as a chewing sequence. In order to detect swallowing, a throat strip with a width of 8 pixels and a height of 32 pixels is taken vertically below the perioral area, the position of the longitudinal brightness projection peak in the strip is calculated for each frame, and the longitudinal displacement of the peak is tracked in the window; when a complete reciprocation of first upward and then downward or first downward and then upward occurs, the single displacement amplitude is between 2 and 4 pixels and the reciprocation duration is between 4 and 8 frames, it is determined as a swallowing sequence. If the above two conditions are not met, it is labeled as other sequences. The division of chewing and swallowing is effective because chewing shows frequent opening and closing of the mouth, and swallowing shows short-period displacement of the throat, which has stable characteristics in geometry and brightness projection, and is significantly different in time structure from the case of "only close to the vessel without intake".

[0063] After obtaining the three types of sequence labels in each frame, the tail swing level label, the head and object contact duration label, and the mouth shape sequence label are combined in order to form a Copula grid code. The grid contains 27 possible codes from the Cartesian combination of 3 tail levels, 3 contact duration levels, and 3 mouth shape sequence types. When constructing the Copula gating layer, three lookup tables are preset: eating behavior pass table, drinking behavior pass table, and general behavior inhibition table. The frame-by-frame gating process is to use the Copula grid code of the current frame as an index to obtain the gating result of the frame by looking up the table.

[0064] The eating behavior pass table includes the following semantic combinations: tail swing level is level 0, head object contact duration is level 2, and mouth shape sequence is chewing sequence; tail swing level is level 1, head object contact duration is level 2, and mouth shape sequence is chewing sequence; tail swing level is level 0, head object contact duration is level 2, and mouth shape sequence is swallowing sequence. The above three combinations cover typical eating dynamics: head stable and close to the feeder, mainly chewing in the early stage, and occasionally swallowing. The drinking behavior pass table includes the following semantic combinations: tail swing level is level 0, head object contact duration is level 2, and mouth shape sequence is swallowing sequence; tail swing level is level 1, head object contact duration is level 2, and mouth shape sequence is swallowing sequence. Drinking is mainly swallowing, and tail swing is usually weak or moderate during drinking, so swallowing with high contact duration is a strong pass signal. The general behavior inhibition table is used to identify false eating or false drinking, and includes the following semantic combinations: tail swing level is level 2, head object contact duration is level 2, and mouth shape sequence is other sequence; tail swing level is level 2, head object contact duration is level 1, and mouth shape sequence is other sequence; tail swing level is level 2, head object contact duration is level 2, and mouth shape sequence is chewing sequence, and object involvement type is water container. The first three combinations inhibit the attachment of utensils in an excited state, and the last combination avoids misjudging short-time licking or biting the edge of the water container as drinking.

[0065] Reference Figure 3 The execution rule of the frame-by-frame gating process is: mark as pass when the Copula grid code of the current frame appears in the pass table of the corresponding segment type, mark as inhibit when the Copula grid code of the current frame appears in the general behavior inhibition table, and mark as neutral in other cases. In order to maintain consistency between frame-by-frame decision and segment-level output, for each eating action segment or drinking action segment, the proportion of frames marked as pass is counted within its frame range. When the pass frame proportion reaches or exceeds 60%, the action segment is confirmed as a valid segment; when the pass frame proportion is less than 60%, the action segment is inhibited and marked as inhibited by the Copula gating layer in the result. The pass frame proportion threshold is set to 60% to ensure that most frames meet the strong evidence combination, thereby excluding the case of "short-time attachment but no intake sequence formed".

[0066] To further enhance the inhibitory effect on false eating or false drinking, a tail correlation gating unit is introduced into the Copula gating layer. The tail correlation gating unit checks the following conditions for each frame of the eating motion segment and the drinking motion segment: when the tail swing level is level 2, the head and object contact duration is level 2, and the mouth shape sequence is other sequence, the gating result of the frame is forced to be inhibited; when the current segment type is drinking, the tail swing level is level 2, and the mouth shape sequence is chewing sequence, the gating result of the frame is forced to be inhibited; when the tail swing level decreases from level 2 to level 0 or level 1 within 5 consecutive frames, the head and object contact duration remains level 2, and the mouth shape sequence changes from other sequence to swallowing sequence, the gating result of the frame and the frames after it that satisfy the condition are set to pass. The design basis of this unit is that strong tail swing usually corresponds to excitement or play, and it is easy to produce a visual "edge" illusion when approaching the vessel; once the tail swing decreases and swallowing occurs, it indicates that the behavior has transitioned from the exploration state to the intake state, and the inhibition needs to be removed in time.

[0067] To ensure that the gating is not affected by picture jitter and short-time occlusion, robustness processing is added to the calculation of the mouth shape sequence label and the head and object contact duration label. If more than 20% of the shadow blocks appear in the 48x48 local mouth shape area, a 3x3 morphological closing operation is performed on the binary result; if the minimum boundary distance between the head and the object increases by a single frame and returns to the original value in the next frame, the single frame is considered to be interpolated to the smaller value of the adjacent frames. The above processing reduces isolated abnormalities caused by hair occlusion and automatic exposure jump, making the frame-by-frame gating more stable.

[0068] After the segment-level confirmation is completed, the behavior analysis result is output. For the confirmed eating motion segment, the object participation type is written as eating utensils, and the scene type is written as eating area; for the confirmed drinking motion segment, the object participation type is written as water, and the scene type is written as drinking area; for the inhibited segment, no behavior type is output, only the segment time range and the dominant combination explanation that leads to inhibition are retained for the log, such as "tail swing level is level 2 and contact duration is level 2 and mouth shape is other sequence". Each behavior analysis result also retains a key evidence summary, including the proportion of passing frames, the frame distribution of the tail swing level, the frame distribution of the contact duration level, and the frame distribution of the mouth shape sequence, so that the trigger reason can be directly reviewed when diagnosing and reviewing.

[0069] In optional embodiments, if the camera supports a higher frame rate, such as 50 frames per second, the sliding window length can be set to 20 frames to maintain a time coverage of about 0.4 to 0.6 seconds while keeping the level division of the contact ratio unchanged, so as to capture a shorter period of swallowing at a finer time resolution; if the camera supports infrared fill light, the binarization of the mouth shape area at night is switched from a single threshold based on the Otsu threshold to an adaptive threshold, and the local block size is set to 11 pixels, so as to facilitate keeping the opening boundary clear under low light conditions; for pets with a thick tail and a color similar to the background, the measurement of the tail path length can be changed from the tail tip key point to the cumulative path between the tail middle key point and the tail root key point, so as to reduce the misclassification caused by unstable tail tip detection.

[0070] Although the specific embodiments of the present application are described above, those skilled in the art should understand that these specific embodiments are only illustrative, and those skilled in the art can make various omissions, substitutions and changes to the details of the above-mentioned methods and systems without departing from the principles and essence of the present application. For example, combining the above-mentioned method steps to perform substantially the same function in substantially the same way to achieve substantially the same result is within the scope of the present application. Therefore, the scope of the present application is only limited by the appended claims.

Claims

1. A pet behavior analysis method based on image recognition, characterized in that, The method includes: Step 1: Obtain the video frame sequence, perform superpixel segmentation on each frame in the video frame sequence to obtain a superpixel map composed of several superpixel blocks; generate pet candidate regions, object candidate regions and scene layout on the superpixel map, and establish a region association graph. This graph contains nodes that represent functional areas in the pet candidate region, object candidate region and scene layout, respectively, as well as edges that reflect the spatiotemporal adjacency relationship between nodes. Step 2: Using the region association map as input, construct a Poisson-Bernoulli hybrid tracking link, establish trajectories for the anchor points of the pet candidate region, object candidate region and functional area respectively, and output a unified set of trajectory entities containing time sequence information, and divide the trajectory entity set into a set of time sequence segments; Step 3: Using the set of time-series segments as input, construct a skeleton × object × scene hypergraph, where the hypergraph contains a set of pet skeleton nodes, a set of object nodes, and a set of scene nodes; perform deterministic inference along a preset action recognition path on the hypergraph, and output a strong initial recognition result of an interactive behavior; Step 4: For the eating and drinking action segments in the initial results of strong recognition of interactive behavior, construct a Copula gate layer and perform frame-by-frame gating processing to suppress fake eating or drinking behavior, and finally output the behavior analysis results carrying the object participation type and scene type. Step two involves constructing a PMBM tracking link, including: calculating the overlap ratio and centroid distance of the bounding rectangles for candidate regions of the same type between adjacent frames; determining a match when the overlap ratio is not lower than a preset overlap ratio threshold and the centroid distance does not exceed a preset distance threshold; for a candidate region with an existing associated trajectory, extending the trajectory if it matches only one candidate region in the next frame; generating multiple branches for the trajectory and including them in a hypothesis pool for management if it matches multiple candidate regions; updating the status of each trajectory or branch according to the matching status: marking a stable state when the number of consecutively matched frames reaches a first preset frame number threshold; marking a pending confirmation state if a frame fails to match after the stable state; terminating the trajectory or branch if two consecutive frames fail to match; and dividing the stable trajectory or branch into a set of time segments according to their consecutive frame intervals, where each time segment contains at least a second preset frame number threshold of frames.

2. The method according to claim 1, characterized in that, Step one generates pet candidate regions, object candidate regions, and scene layout, including: obtaining a motion mask by performing inter-frame difference processing on the current frame and the adjacent previous frame; marking superpixels whose pixel coverage in the motion mask reaches a preset motion coverage threshold as motion blocks; using motion blocks as seeds to perform region growth and construct a pet candidate region set; performing shape detection on the edge map of the current frame and combining it with color saturation features to construct an object candidate region set containing feeder / water container candidate regions, toy candidate regions, and potty candidate regions, and labeling each object candidate region with a type label; based on the position of each type of candidate region in the object candidate region set, dividing the detected ground area into feeding area, drinking area, and activity area as functional areas to form a scene layout.

3. The method according to claim 2, characterized in that, Constructing a pet candidate region set includes: using a motion block as a seed, merging adjacent superpixels that meet the following conditions into the same connected region: the shared boundary length reaches a first predetermined length threshold, and the color mean difference does not exceed a first predetermined color threshold; when the number of pixels in the connected region reaches a first predetermined area threshold, the connected region is retained as a pet candidate region; calculating the bounding rectangle for each pet candidate region, and including pet candidate regions with aspect ratios within a first predetermined range into the pet candidate region set.

4. The method according to claim 2, characterized in that, Constructing a set of candidate regions for objects includes: performing shape detection of circles and rectangles on the edge map of the current frame; identifying circles with radii within a second predetermined range and arc coverage angles reaching a first predetermined angle threshold as container circles, and merging superpixels whose intersection area with the container circle accounts for a percentage of the container circle's area reaching a first predetermined area percentage threshold as candidate regions for feeding / watering containers; identifying connected components with areas within a third predetermined range and average color saturation values ​​reaching a preset saturation threshold as candidate regions for toys; and identifying rectangular connected components with corner counts reaching a preset corner count threshold and aspect ratios within a second predetermined range as candidate regions for potty.

5. The method according to claim 4, characterized in that, Step 3 involves constructing a skeleton × object × scene hypergraph, including: extracting a preset number of pet body keypoints frame by frame for each pet-related temporal segment in the temporal segment set to obtain a pet skeleton node set; establishing an object node set for temporal segments that overlap with pet-related temporal segments and are related to objects, according to their type labels; establishing a scene node set for the functional area where the centroid of the pet's movement trajectory is located in the pet-related temporal segments; and generating a ternary hyperedge for each pet skeleton node and its object nodes and scene nodes whose spatial distances satisfy preset proximity conditions within each frame to represent the higher-order interaction relationships between them.

6. The method according to claim 5, characterized in that, Before performing deterministic reasoning on the hypergraph, the following steps are also included: discretizing the ternary hyperedges to generate a set of hyperedge attributes. The discretization process includes: a) discretizing the spatial proximity relationships of contact, proximity, and distance based on the minimum boundary distance from the skeleton node to the candidate object region; b) encoding the relative orientation relationship based on the difference in centroid coordinates between the skeleton node and the object node; c) discretizing the body support relationship of stable support and moving support based on the position change amplitude and cumulative displacement of the key points representing the left and right front paws in the skeleton node within the time window; d) discretizing the tail swing level relationship based on the path length between the key point representing the tail tip and the key point representing the tail root in the skeleton node; e) marking the scene inclusion relationship between the skeleton node and the scene node based on the centroid position of the skeleton node.

7. The method according to claim 6, characterized in that, Deterministic reasoning is performed on the hypergraph, which includes a multi-round state update process: The first round of update involves counting the number of times contact or proximity relationships with object nodes occur within a time window for nodes related to the mouth, head, and forepaws in the skeleton node set, thereby updating the skeleton node's local skeleton state; the second round of update involves counting the consecutive occurrence length of skeleton nodes connected to it and having local skeleton states within the time window for object nodes, and marking the object node as participating when the length reaches a third preset frame threshold; the third round of update involves counting the co-occurrence length of skeleton nodes connected to it and having local skeleton states with the object nodes in the participating state for scene nodes, and marking the scene node as active when the co-occurrence length reaches a fourth preset frame threshold.

8. The method according to claim 7, characterized in that, Step four involves constructing a Copula gating layer and performing frame-by-frame gating processing, including: generating three types of sequence markers for each frame within each eating or drinking action segment. These three types of sequence markers are: a) a tail sway level marker, which is assigned a grade based on the path length of the tail movement; b) a head-object contact duration marker, which is assigned a grade based on the proportion of frames in which the head contacts the object within the time window; and c) a mouth shape sequence marker, which is assigned a grade based on the number of opening and closing cycles and the average interval of the mouth region, as well as the presence of periodic displacement of the vertical brightness band of the larynx corresponding to the swallowing action, classifying it as a chewing sequence, swallowing sequence, or other sequences. The three types of sequence markers are then combined into triples to form a Copula grid code.

9. The method according to claim 8, characterized in that, The frame-by-frame gating process also includes: using Copula grid coding as an index, querying the preset eating behavior pass table, drinking behavior pass table, and general behavior suppression table, and marking each frame as pass, suppressed, or neutral; calculating the percentage of pass frames within each action segment, and when the percentage of pass frames reaches a preset pass frame percentage threshold, the action segment is confirmed as a valid segment; otherwise, the action segment is suppressed. The general behavior suppression table contains codes that meet the following conditions: the tail swing level indicator indicates a swing level exceeding a preset swing level threshold, and the sequence type indicated by the mouth shape sequence indicator does not match the current action segment type, thereby suppressing fake eating and fake drinking behaviors by identifying the inherent contradictions in the behavioral signals.

Images