Pet behavior recognition and emotion detection method

Abstract

The invention relates to the technical field related to pets, and discloses a pet behavior recognition and emotion detection method, which comprises the following steps: constructing a user-defined pet tag system; using an improved YOLOv11 model to carry out behavior detection and emotion reasoning on the multi-modal data of the pet; adding an attention module for pet improvement in a feature extraction network of the YOLOv11 model, adding a cross-scale feature interaction module in a neck layer, and performing multi-stage training by adopting a dynamic weight loss function and a scene adaptive data enhancement strategy; and inputting results of behavior detection and emotion reasoning and pet multi-modal data into a large language model, and generating a long text reply containing a real-time state, a reason for generating the real-time state and professional suggestions in combination with a pet professional knowledge graph. According to the method, the defects of the prior art in the aspects of comprehensive recognition, detection accuracy and reply speciality are overcome.

Description

Technical Field

[0001] This invention relates to the field of pet-related technology, specifically to a method for pet behavior recognition and emotion detection. Background Technology

[0002] As people's care for their pets continues to increase, pet behavior recognition and emotion detection have gradually become research hotspots, with significant application value in pet health monitoring, human-pet interaction optimization, and pet psychological care. Early pet behavior recognition relied heavily on single sensor data, such as accelerometers, which could only provide rough judgments on a limited number of simple behaviors and could not effectively correlate with the pet's emotional state. With the development of computer vision technology, pet behavior recognition based on single-modal images or videos has begun to emerge, capable of recognizing some relatively obvious behaviors, such as "sit" and "lie down," but it still falls short in detecting complex and subtle behaviors and emotions. Meanwhile, the application of natural language processing technology in the pet field is still relatively rudimentary, mainly focusing on simple command interactions, and it is difficult to combine pet behavior recognition results to generate professional and detailed long-form text analysis.

[0003] In recent years, multimodal technology fusion has become a trend, aiming to integrate data from multiple modalities such as images, videos, and sensors to achieve more accurate pet behavior recognition and emotion detection. However, the application of multimodal fusion in this field is still in the exploratory stage, and there is a lack of mature multimodal comprehensive analysis solutions for pet behavior and emotions. In particular, there is very little involvement in the technology of professional long text generation combined with large language models (LLM).

[0004] Existing technologies have shortcomings in terms of the comprehensiveness of pet behavior recognition, the accuracy of emotion detection, and the generation of professional long text responses by combining large language models.

[0005] (1) Existing pet behavior recognition technology based on single-modal images: This technology only uses pet image data and identifies pet behavior through traditional machine learning algorithms (such as support vector machines and random forests) or early deep learning models (such as basic convolutional neural networks). Taking the recognition of pet "standing" behavior as an example, the technical implementation process is as follows: First, extract the pet's contour features (obtained through edge detection algorithms) and limb position coordinates (based on key point detection technology to locate the torso and limb endpoints) in the image. Then, perform similarity matching between the extracted features and the preset "standing" behavior feature template (such as "the torso is upright, and the limbs are perpendicular to the ground and support the body weight"). If the matching degree reaches the preset threshold (usually above 0.7), it is determined to be "standing" behavior. However, this technology is significantly affected by external environmental factors. When the image is taken from the side or from above, it is easy to cause deviations in the extraction of limb position coordinates, and the misjudgment rate of similar behaviors such as "standing" and "squatting" exceeds 30%. At the same time, a single image cannot capture the process of the pet's movement changes. It cannot identify dynamic behaviors such as "tail wagging" and "leg rubbing", nor can it effectively infer the pet's emotional state from static images (such as not being able to distinguish between "relaxation" and "fear" from a single "tail drooping" image).

[0006] (2) Existing pet behavior recognition technology based on single video: This technology relies on the inter-frame information in the video sequence, captures the trajectory of the pet's movement changes through optical flow methods (such as the Lucas-Kanade optical flow algorithm), and extracts the temporal action features by combining 3D convolutional neural networks (3D CNN) to achieve behavior recognition. For example, when recognizing the "running" behavior of a pet, the technology calculates the displacement distance and movement speed of the pet's torso and limbs in consecutive video frames. When the displacement speed continuously exceeds 0.5m / s and the limbs show alternating swinging characteristics, it is determined to be the "running" behavior. However, this technology has obvious limitations: for subtle behaviors with small movement amplitude (such as "slight tail wagging" amplitude less than 5cm) and short duration (such as "rapid nose licking" duration less than 1s), the detection accuracy is less than 60% due to the insignificant changes in inter-frame features; and the technology only focuses on the recognition of the behavior itself, without establishing a correlation model between behavior and emotion, and cannot output the pet's emotional state; in addition, this technology does not involve the integration with large language models, and cannot combine the detection results to generate long text analysis containing professional knowledge.

[0007] (3) Existing pet-related natural language processing technologies: This technology is mainly based on pet command interaction systems using rule bases or simple semantic matching. The technical architecture includes four stages: "command input - semantic parsing - rule matching - response generation". Specifically, users input simple commands (such as "feed" or "groom") via text or voice. The system first performs semantic parsing through speech recognition (such as using the open-source PocketSphinx tool) or text segmentation (such as based on Jieba segmentation). Then, it matches the corresponding response content from the preset rule base (such as "feed → It is recommended to feed twice a day, with the amount of food each time calculated based on the pet's weight at 20g / kg) and finally generates a short text (usually no more than 50 characters) to feed back to the user. The core flaws of this technology are: First, it cannot access pet behavior recognition results, and the response content is completely disconnected from the pet's real-time status (e.g., when a user asks "Does my pet need to be fed now?", the system cannot provide an accurate response based on the pet's "just finished eating" behavior); second, the responses lack professionalism and depth, failing to cover complex needs such as pet behavior interpretation, emotion analysis, and health warnings, and unable to generate professional, detailed, and long texts about the pet's specific actions and emotions; third, its semantic understanding capabilities are limited, only able to process simple commands, and unable to effectively respond to questions that require behavioral knowledge, such as "Why does my pet keep rubbing against me?"

[0008] Disadvantages of existing technology: (1) Data labeling defects: There is no pet-specific label system. General labels (such as "action" and "expression") cannot distinguish the correspondence between pet behavior and emotion (such as "tail wagging" may correspond to "happy" or "nervous", which needs to be further subdivided by "tail wagging amplitude / frequency"), resulting in low signal-to-noise ratio of training data and poor model generalization ability.

[0009] (2) Insufficient model detection: Existing target detection models (early YOLO series, 3D CNN) have three problems: First, poor adaptability to pet breeds (e.g., the "standing" behavior characteristics of Chihuahuas and Golden Retrievers are very different, and the model is prone to missing small dog behaviors); Second, weak ability to capture dynamic behavior time sequence, with a video frame processing frame rate of <15fps, making it impossible to recognize fast actions in real time (e.g., "sudden leap"); Third, low accuracy in small target detection, with the accuracy of local feature detection such as "ear rotation" and "pupil changes" being <50%.

[0010] (3) The integration of large language models is superficial: Only the “behavior name” (such as “tail wagging”) is input into the large language model. The integration of multimodal data (video frame screenshots, behavior duration) and professional knowledge (pet behavior, veterinary common sense) is lacking, resulting in problems such as “information fragmentation” (such as only saying “pet is happy” without explaining the basis for judgment) and “generality of suggestions” (such as not giving feeding suggestions based on the pet’s age / health status). Summary of the Invention

[0011] This invention provides a method for pet behavior recognition and emotion detection, which overcomes these shortcomings by combining multimodal fusion, accurate model training and large language model.

[0012] This invention addresses the shortcomings of existing technologies in terms of comprehensiveness of recognition, accuracy of detection, and professionalism of response by constructing a multimodal data annotation system specific to pets, conducting targeted training and fine-tuning based on the YOLOv11 model, and achieving deep integration of behavior detection results with a large language model (combined with a pet behavioral knowledge graph).

[0013] To address the shortcomings of existing technologies, this invention aims to solve three core problems: Build a customized tagging system that adapts to the relationship between pet behavior and emotions, and solve the problem that general tags cannot accurately describe the correspondence between pet behavior and emotions; Optimize the pet behavior detection capabilities of the YOLOv11 model to improve the detection accuracy and real-time performance for different breeds, subtle behaviors, and dynamic temporal behaviors; Achieve deep integration of YOLOv11 model detection results with large language models, and combine multimodal data and professional knowledge graphs to generate long text responses that are professional, targeted, and coherent.

[0014] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A method for pet behavior recognition and emotion detection includes: A custom pet tagging system is constructed and combined with pet multimodal data to form a labeled dataset. The tagging system adopts a two-level hierarchical structure of behavioral tags and emotional tags. The behavioral tags include interactive, self-behavior, eating, and abnormal behaviors, while the emotional tags include positive, negative, and neutral emotions. The improved YOLOv11 model was used to perform behavior detection and emotion inference on pet multimodal data. The improvements to the YOLOv11 model include: adding an attention module for pets to the feature extraction network of the YOLOv11 model, adding a cross-scale feature interaction module to the neck layer, and using a dynamic weight loss function and scene-adaptive data augmentation strategy for multi-stage training. The results of behavior detection and emotion inference, pet multimodal data, and environmental information are input into a large language model. Combined with a pet professional knowledge graph, a long text response is generated that includes the real-time status, the reason for the real-time status, and professional advice.

[0015] In one embodiment, the pet multimodal data includes photos and videos of different breeds of pets in different scenes and under different lighting conditions; the environmental information includes feeding times and companionship times.

[0016] In one embodiment, the construction of the custom pet tagging system includes: Based on pet behavioral knowledge, initial behavioral labels, emotion labels, and correlation factor labels are defined for pet multimodal data. Behavioral labels include primary categories, secondary subcategories, and tertiary action features. Behavioral labels are defined by annotating the pet's limb movement amplitude (e.g., "tail wagging amplitude > 10cm indicates rapid tail wagging") and frequency (e.g., "paw licking > 3 times / minute indicates frequent paw licking"). Emotion labels are defined through a "behavior combination - emotion" mapping relationship. Correlation factor labels include environmental factors and physiological factors. Environmental factors are labeled through image scene recognition, while physiological factors are labeled through the time period of the behavior (e.g., "within 1 hour after feeding → after eating") or pet physiological data (e.g., "body temperature > 39℃ → abnormal physiological state"). The initial correlation between behavioral tags and emotion tags, and between behavioral tags and related factor tags, is calculated using the mutual information entropy algorithm. Relationship pairs with a correlation greater than a set threshold are retained to form the initial tag library; the threshold can be set to 0.4.

[0017] In one embodiment, the improved attention module for pets includes a channel attention branch and a spatial attention branch, with the following specific structure: Channel attention branch: A squeeze-and-excitation (SE) structure is adopted. First, the input feature map is subjected to global average pooling (squeeze operation) to obtain a 1×1×C feature vector. Then, the channel weights are output through two fully connected network layers (the first layer has C / 4 neurons and ReLU activation function; the second layer has C neurons and Sigmoid activation function). The weights are multiplied with the original feature map channel by channel to enhance the feature response of the key channels of pet behavior (such as "limb edge channel" and "facial feature channel") and suppress the interference of background channels. Spatial Attention Branch: The spatial attention structure of Convolutional Block Attention Module (CBAM) is adopted. The feature map output by the channel attention branch is subjected to channel-dimensional average pooling and max pooling, and the concatenation results in a 2×H×W feature map. The spatial weight matrix is ​​output through a 3×3 convolutional layer (with sigmoid activation function) and multiplied with the feature map pixel by pixel to highlight the features of the key body parts (face, limbs, tail) in the pet image and improve the feature extraction accuracy of the target action. The interaction between the pet-specific improved attention module and the feature extraction network: The feature maps output by the C3 module of the feature extraction network (e.g., C3-P3 corresponds to an 80×80×256 feature map, C3-P4 corresponds to a 40×40×512 feature map, and C3-P5 corresponds to a 20×20×1024 feature map) are first input into the pet-specific improved attention module. After channel and spatial attention weighting, the output enhanced feature maps flow directly to the corresponding feature layers (P3, P4, P5) of the neck layer (PAN-FPN) as input for neck layer feature fusion.

[0018] In one embodiment, adding a cross-scale feature interaction module to the neck layer specifically includes: After downsampling the large-scale feature map of the neck layer output of the YOLOv11 model, which is responsible for whole-body behavior recognition, it is interacted with the meso-scale feature map channel by channel to transmit global behavior information. After upsampling the small-scale feature map responsible for facial and limb motion recognition, it interacts with the medium-scale feature map pixel by pixel to supplement local detailed features; By using an adaptive weight fusion algorithm, the fusion ratio of feature maps at different scales is dynamically adjusted, and feature information of unoccluded areas is retained first in occluded scenes. Among them, small-scale features Figure 1 Generally, it refers to the feature map obtained by downsampling by four times, with a size of 160×160. This feature map has the smallest local receptive field, with each grid corresponding to a 4×4 area of ​​the original input image, which is suitable for detecting small targets. Mesoscale features Figure 1 Generally, it refers to the feature map obtained by downsampling by 16 times, with a size of 40×40. Its features have a certain degree of abstraction and semantic information, and are suitable for detecting medium-sized targets, with target sizes ranging from about 16×16 to 32×32 pixels. Large-scale feature maps refer to feature maps obtained by downsampling by 32 times, with a size of 20×20. They have the lowest spatial resolution but the strongest semantic information and the largest global receptive field, making them suitable for detecting larger targets, generally targets with a size of 32×32 pixels or more.

[0019] In one embodiment, the dynamic weight loss function dynamically adjusts the classification loss weight based on the number of samples of each type of behavior during training, and adjusts the localization loss weight based on the magnitude of the action.

[0020] In one embodiment, the scene adaptive data enhancement strategy enhances the set low-light scene, dynamic background scene, and occlusion scene by adopting adaptive brightness adjustment and noise suppression, background replacement and motion blur simulation, and random occlusion simulation, respectively.

[0021] In one embodiment, the multi-stage training specifically includes: Basic training phase: Input the primary behavior labels and secondary emotion labels from the labeled dataset, and train the improved YOLOv11 model with the initial learning rate to enable the YOLOv11 model to master the ability to recognize pet behaviors. Optimization and fine-tuning stage: Input labeled data containing different scenarios, and use the learning rate cosine annealing strategy to fine-tune the overall parameters of the YOLOv11 model.

[0022] Compared with the prior art, the beneficial technical effects of the present invention are: 1. Advantages of a professional tagging system: Through precise "behavior-emotion" association labeling, compared with existing general tags, it provides the model with more granular and more relevant training data, enabling the model not only to recognize "tail wagging" but also to distinguish between "fast tail wagging" and "slow tail wagging" and associate them with the corresponding emotions. This improves the accuracy of emotion detection from the data source and solves the core defect of existing technologies that "only recognize behavior but not emotions".

[0023] 2. Advantages of the improved YOLOv11 model: (1) Fine movement recognition capability: The improved attention module for pets (Pet-Attention module) enhances the extraction of small target features, improving the recognition accuracy of fine movements such as "pupil contraction" and "whisker trembling" by 8%-15%, filling the gap in the traditional YOLO model for pet fine behavior recognition.

[0024] (2) Robustness in complex scenarios: The cross-scale interaction module and the scene-adaptive data augmentation strategy improve the recognition accuracy of the model by 5%-12% and reduce the false detection rate by 7%-13% in low light, occlusion and dynamic background scenarios, meeting the needs of diverse practical application scenarios such as home and outdoors.

[0025] 3. Advantages of combining large language models: By using a "behavior-emotion" association rule base and multimodal data interaction logic, the responses generated by the large language model not only include "what the behavior is and what the emotion is," but also cover professional content such as "cause analysis, operational suggestions, and daily care." Compared with the existing technology's "simple behavior description + brief response" model, the professionalism (combining pet medical / behavioral knowledge), coherence (complete logical chain: behavior → emotion → cause → suggestion), and practicality (directly implementable care operations) of the responses are significantly improved, providing users with truly valuable pet care guidance. Attached Figure Description

[0026] Figure 1This is a system architecture diagram in an embodiment of the present invention.

[0027] Figure 2 This is a flowchart of the method in an embodiment of the present invention.

[0028] Figure 3 This is the backbone network of the YOLOV11 model before the improvement.

[0029] Figure 4 The neck network of the YOLOV11 model before improvement.

[0030] Figure 5 This is a schematic diagram of the improved YOLOv11 model.

[0031] Figure 6 This is a schematic diagram of the improved backbone network (feature extraction network). Detailed Implementation

[0032] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0033] like Figure 1 As shown, the system used in this invention mainly includes a multimodal data acquisition module, a label system management module, a model optimization and training module, a professional knowledge graph module, and a large language model interaction module.

[0034] Multimodal data acquisition module: used to acquire pet images and video data (including different scenes: indoor / outdoor; different lighting: strong light / weak light; different breeds: dogs / cats / other small mammals).

[0035] Tag system management module: Connected to the multimodal data acquisition module, it receives raw data and performs tag annotation and updates.

[0036] Model optimization and training module: It is connected to the multimodal data acquisition module and the label system management module respectively, receives multimodal data and labeled datasets, and performs behavior detection and emotion feature extraction.

[0037] Professional Knowledge Graph Module: Stores professional knowledge in pet behavior, veterinary medicine, and animal husbandry, including the "behavior-emotion-cause-suggestion" relationship.

[0038] Large Language Model Interaction Module: Connected to both the model optimization and training module and the external user terminal, it receives the behavior detection and emotion inference results output by the YOLOv11 model, and generates a response based on the pet pictures, videos, and environmental information uploaded by the user, and then sends it back to the user.

[0039] like Figure 2 As shown, the method flow of the present invention will be described below, specifically including: S1, Data Collection: Simultaneously collect pet data from different breeds (10 cats, 15 dogs) and scenarios (home, outdoor, veterinary), forming an original dataset containing 100,000 samples.

[0040] S2, Professional Hierarchical Annotation: Based on pet behavioral knowledge, pet data is annotated in multiple dimensions through a custom labeling system to generate an annotated dataset containing emotional association information.

[0041] S3, YOLOv11 Model Optimization and Training: The YOLOv11 model is structurally optimized, and an improved training strategy (dynamic weight loss function, scene-adaptive data augmentation) is adopted. The model is trained using a labeled dataset to obtain a high-precision pet behavior detection model.

[0042] S4, Multi-behavior-emotion association reasoning: After a user uploads a picture or video of their pet, the improved YOLOv11 model outputs "detailed behavior + confidence score + temporal features", which, combined with the preset "behavior-emotion" association rule base, infers the pet's emotion type and confidence score.

[0043] S5, Large Language Model Response Generation: Input the results of behavior detection and emotion inference, pet multimodal data, and environmental information into the large language model, combine it with the pet professional knowledge graph, generate professional long text responses, and provide feedback to the user.

[0044] The technical solution of this invention mainly includes three core parts: construction of a custom pet tag system, improved YOLOv11 model training and detection, and analysis of a multimodal fusion large language model.

[0045] 1. Construction of a custom pet behavior and emotion tagging system.

[0046] A hierarchical "behavior-emotion" structure is adopted, combined with a dynamic optimization mechanism, as detailed below: (1) The specific structure of the labeling system is shown in Table 1: Table 1

[0047] (2) Methods for constructing and optimizing the tag system: Basic Construction: In collaboration with 3 licensed veterinarians and 5 senior pet behavior analysts, initial labels were labeled based on 100,000 original samples. The mutual information entropy algorithm was used to calculate the behavior-emotion correlation, and strong correlation pairs with a correlation of >0.4 (such as "fast tail wagging - excitement" and "refusal to eat - anxiety") were retained to form the initial label library.

[0048] This invention constructs a label structure of "behavioral category - emotion association," covering 32 pet behaviors and corresponding emotion inference rules; the labeling mechanism of double labeling + expert arbitration ensures that the accuracy of the labeled data is ≥97%. Based on pet behavioral knowledge, it defines detailed behavioral labels (such as "rapid tail wagging" and "slow tail wagging") and establishes association rules of "action feature - emotion," solving the problem of existing labels being "coarse-grained and lacking emotional association."

[0049] 2. Improve YOLOv11 model design and training.

[0050] To address the shortcomings of traditional YOLOv11 models in pet behavior recognition, such as poor recognition of subtle movements, weak robustness in complex scenes, and insufficient ability to recognize multiple behavior associations, innovative designs are implemented from two aspects: structural optimization and training strategy improvement. (1) Model structure optimization 1) Improved feature extraction network (Backbone): The traditional YOLOv11 model's feature extraction network (CSPDarknet) is not good at extracting small target features (such as "pupil contraction" and "beard trembling") and is easily interfered with by background noise.

[0051] This invention adds a lightweight, pet-attention module to the feature extraction network, which includes channel attention and spatial attention branches. Channel attention branch: By calculating the weights of different feature channels, the feature responses of key channels of pet behavior (such as "limb edge channel" and "facial feature channel") are enhanced, while background channel interference is suppressed.

[0052] Spatial attention branch: targeting the locality of pet behavior (such as "paw movements" and "ear movements"), it highlights the features of key parts of the pet's body (face, limbs, tail) in the image through a spatial weight matrix, thereby improving the feature extraction accuracy of small target movements.

[0053] Tests showed that the accuracy of recognizing small target behaviors (such as "licking paws" and "ear turning") was improved by 8%-15% on pet image datasets containing subtle movements.

[0054] 2) Enhance the multi-scale feature fusion capability of the Neck layer: The traditional YOLOv11 model's PAN-FPN structure does not fully integrate features at different scales (such as large-scale features of "whole-body behavior" and small-scale features of "facial expressions"), resulting in decreased recognition accuracy in complex scenes (such as pets and furniture occlusion).

[0055] A cross-scale interaction (CSI) module is introduced in the Neck layer, as follows: After downsampling the large-scale feature map (responsible for whole-body behavior recognition), it interacts with the medium-scale feature map channel by channel to transmit global behavior information.

[0056] After upsampling the small-scale feature map (responsible for facial / limb subtle movement recognition), it interacts with the medium-scale feature map pixel by pixel to supplement local detailed features.

[0057] By using an adaptive weight fusion algorithm, the fusion ratio of features at different scales is dynamically adjusted, and feature information of unoccluded areas is retained first in occluded scenes.

[0058] On datasets with occluded scenarios (such as a pet hiding behind a sofa with only its head or tail showing), the accuracy of behavior recognition is improved by 5%-11%, and the false detection rate is reduced by 7%-12%.

[0059] The newly added Pet-Attention module in this invention includes channel attention and spatial attention branches to enhance the extraction of subtle pet movement features. The improved Neck layer cross-scale feature interaction (CSI) module enhances the fusion capability of behavioral features at different scales, improving robustness in complex scenarios. Addressing the "subtlety, correlation, and scene diversity" characteristics of pet behavior, the feature extraction network and Neck layer of the YOLOv11 model are specifically optimized, rather than simply using the standard model, significantly improving the accuracy of pet behavior detection.

[0060] Figure 5 In the diagram, P3 represents the small-scale feature layer in the feature pyramid, P4 represents the medium-scale feature layer in the feature pyramid, the CSI module represents the "Cross-Scale Interaction" module, which is an enhancement component for feature fusion, and the Detect layer represents the YOLO target detection head, which is the final prediction module of the model.

[0061] Figure 6 In the YOLO model, Conv represents a convolutional layer, C3k2 represents an optimized design of the feature extraction module in the YOLO series, and SPPF stands for Spatial Pyramid Pooling Fast. They are all specific expressions of the YOLO model.

[0062] (2) Improvement of training strategies 1) Design of dynamic weight loss function: The traditional YOLOv11 model uses a loss function with fixed weights, which does not take into account the "class imbalance" problem of pet behavior (such as a large number of "calm behavior" samples and a small number of "stress behavior" samples), resulting in low accuracy of the model in recognizing behaviors with few samples.

[0063] This invention designs a dynamic weight loss function (DWL) based on sample distribution: Classification loss weights: dynamically adjusted based on the number of samples for each type of behavior during training. For behaviors with fewer samples (such as "frizzy hair" or "dilated pupils"), the classification loss weight is larger (range: 1.2-1.8). For behaviors with more samples (such as "standing" or "walking"), the weight is set to 1.0 to ensure that the model pays attention to behaviors with fewer samples.

[0064] Localization loss weight: For the "difference in movement amplitude" of pet behavior, the localization loss weight is increased to 1.5 for behaviors with small movement amplitude, which strengthens the model's learning of the position of subtle movements.

[0065] Results: The accuracy of identifying stress behaviors with few samples (such as "tail clamping" and "back arching") is improved by 9%-13%, and the localization error is reduced by 8%-12%.

[0066] 2) Scene-adaptive data augmentation strategy: Traditional data augmentation does not take into account the scene characteristics of pet behavior, resulting in poor robustness of the model in specific scenarios (such as low light and dynamic backgrounds).

[0067] This invention proposes a Scene-Adaptive Augmentation (SAA) algorithm, which selects enhancement methods based on scene labels in the dataset (such as "low-light scene" and "outdoor dynamic background scene"). Low-light scene data: Adaptive brightness adjustment (brightness increase of 0.3-0.5 times) and noise suppression (Gaussian filtering) are used to enhance the data and avoid feature blurring caused by traditional enhancement methods.

[0068] Dynamic background scene data: Enhancement is achieved by using background replacement (replacing with different types of backgrounds: grass, street, living room) and motion blur simulation (simulating the blur effect when a pet moves) to improve the model's robustness to background changes.

[0069] Occlusion scene data: Random occlusion simulation (occlusion ratio 5%-20%, occlusion area: body, head, limbs) was used to enhance and train the model's feature extraction ability under partial occlusion.

[0070] On low-light scene datasets, the accuracy of behavior recognition is improved by 16%-21%; on dynamic background scene datasets, the false detection rate is reduced by 14%-18%.

[0071] (3) Multi-stage training mechanism: Phase 1 (Basic Training Phase): Input labels from the labeled dataset, train using the improved model structure and an initial learning rate (0.01), with the goal of enabling the model to master the ability to recognize basic pet behaviors. The training consists of 100 rounds, with validation every 20 rounds. When the basic behavior recognition accuracy is >85%, proceed to the next phase.

[0072] Phase 2 (Optimization and Fine-tuning Phase): Input labeled data containing complex scenes (occlusion, low light, dynamic background), and use a learning rate cosine annealing strategy (reducing from 0.001 to 0.0001) to fine-tune the overall parameters of the model. The goal is to improve the robustness of the model to complex scenes. The training is conducted for 50 epochs. The final model has an average detection accuracy (mAP) of ≥92% and an emotion judgment accuracy of ≥88% on the test set.

[0073] The Dynamic Weighted Loss (DWL) function of this invention dynamically adjusts the loss weights based on the number of behavior samples and the magnitude of the action, solving the class imbalance problem. Scene Adaptive Data Augmentation (SAA) is designed with specific augmentation methods for low-light, occluded, and dynamic background scenes. A multi-stage training mechanism learns "basic behavior - refined features - complex scene adaptation" in stages, improving the model's convergence speed and generalization ability. The training strategy is deeply matched with the characteristics of pet behavior data, avoiding the problems of poor behavior recognition with few samples and weak robustness in complex scenes caused by the "one-size-fits-all" approach of traditional training.

[0074] 3. Multimodal fusion large language model analysis module.

[0075] We employ a lightweight large language model, combined with a custom tagging system and professional knowledge graph, to achieve in-depth analysis: (1) Input feature construction: The behavioral labels and sentiment labels output by the improved YOLOv11 model are concatenated with the key features of the original multimodal data as the input sequence; (2) Knowledge fusion mechanism: The large language model associates the input sequence with the knowledge graph through an attention mechanism; (3) Output format design: Generate a long text report containing three parts: "real-time status (behavior + emotion + confidence level), possible causes (feature analysis), and professional advice (feeding / health)".

[0076] This invention is based on a rule base of "behavioral sequence - emotion type - associated knowledge" constructed from pet behavior expert knowledge; it integrates YOLOv11 model detection results, user-supplemented information, and basic pet data into the interaction logic of a large language model. It achieves an end-to-end process of "behavior recognition → emotion inference → professional response," overcoming the shortcomings of existing technologies that "only output behavior labels and lack practical guidance," thus enhancing the practical value of the solution.

[0077] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0078] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0079] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0080] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0081] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for pet behavior recognition and emotion detection, characterized in that, include: A custom pet tagging system is constructed and combined with pet multimodal data to form a labeled dataset. The tagging system adopts a two-level hierarchical structure of behavioral tags and emotional tags. The behavioral tags include interactive, self-behavior, eating, and abnormal behaviors, while the emotional tags include positive, negative, and neutral emotions. The improved YOLOv11 model was used to perform behavior detection and emotion inference on pet multimodal data. The improvements to the YOLOv11 model include: adding an attention module for pets to the feature extraction network of the YOLOv11 model, adding a cross-scale feature interaction module to the neck layer, and using a dynamic weight loss function and scene-adaptive data augmentation strategy for multi-stage training. The results of behavior detection and emotion inference, pet multimodal data, and environmental information are input into a large language model. Combined with a pet professional knowledge graph, a long text response is generated that includes the real-time status, the reason for the real-time status, and professional advice.

2. The pet behavior recognition and emotion detection method according to claim 1, characterized in that, The pet multimodal data includes photos and videos of different breeds of pets in different scenes and under different lighting conditions; the environmental information includes feeding times and times spent with pets.

3. The pet behavior recognition and emotion detection method according to claim 1, characterized in that, The construction of a custom pet tag system includes: Based on pet behavioral knowledge, initial behavioral labels, emotion labels, and correlation factor labels are defined for pet multimodal data: behavioral labels are defined by annotating the amplitude and frequency of the pet's limb movements; emotion labels are defined by the "behavior combination - emotion" mapping relationship; correlation factor labels include environmental factors and physiological factors, where environmental factors are labeled by image scene recognition, and physiological factors are labeled by the time period of behavior occurrence or pet physiological data. The initial association degree between behavioral tags and emotion tags, and between behavioral tags and association factor tags is calculated using the mutual information entropy algorithm. Association pairs with an association degree greater than a set threshold are retained to form the initial tag library.

4. The pet behavior recognition and emotion detection method according to claim 1, characterized in that, The improved attention module for pets includes a channel attention branch and a spatial attention branch, with the following specific structure: Channel attention branch: Adopting a compression and activation structure, the input feature map is first subjected to global average pooling to obtain the feature vector, and then the channel weights are output through two fully connected network layers. They are multiplied with the original feature map channel by channel to enhance the feature response of the key channels of pet behavior and suppress the interference of background channels. Spatial Attention Branch: The spatial attention structure of the convolutional block attention module is adopted. The feature map output by the channel attention branch is subjected to channel-dimensional average pooling and max pooling. The concatenated feature map is then multiplied pixel by pixel with the spatial weight matrix output by the convolutional layer. This highlights the features of the key body parts set in the pet image and improves the feature extraction accuracy of the target action. The interaction between the pet-specific improved attention module and the feature extraction network: The feature map output by the C3 module of the feature extraction network is first input into the pet-specific improved attention module. After channel and spatial attention weighting, the output enhanced feature map flows directly to the corresponding feature layer of the neck layer as the input for neck layer feature fusion.

5. The pet behavior recognition and emotion detection method according to claim 1, characterized in that, The addition of a cross-scale feature interaction module to the neck layer specifically includes: After downsampling the large-scale feature map of the neck layer output of the YOLOv11 model, which is responsible for whole-body behavior recognition, it is interacted with the meso-scale feature map channel by channel to transmit global behavior information. After upsampling the small-scale feature map responsible for facial and limb motion recognition, it interacts with the medium-scale feature map pixel by pixel to supplement local detailed features; By using an adaptive weight fusion algorithm, the fusion ratio of feature maps at different scales is dynamically adjusted, and feature information of unoccluded areas is retained first in occluded scenes. Among them, the small-scale feature map refers to the feature map obtained by downsampling by four times; the medium-scale feature map refers to the feature map obtained by downsampling by 16 times; and the large-scale feature map refers to the feature map obtained by downsampling by 32 times.

6. The pet behavior recognition and emotion detection method according to claim 1, characterized in that, The dynamic weight loss function dynamically adjusts the classification loss weight based on the number of samples of each type of behavior during training, and adjusts the localization loss weight based on the magnitude of the action.

7. The pet behavior recognition and emotion detection method according to claim 1, characterized in that, The scene adaptive data augmentation strategy employs adaptive brightness adjustment and noise suppression, background replacement and motion blur simulation, and random occlusion simulation to enhance the set low-light scene, dynamic background scene, and occlusion scene, respectively.

8. The pet behavior recognition and emotion detection method according to claim 1, characterized in that, The multi-stage training specifically includes: Basic training phase: Input the primary behavior labels and secondary emotion labels from the labeled dataset, and train the improved YOLOv11 model with the initial learning rate to enable the YOLOv11 model to master the ability to recognize pet behaviors. Optimization and fine-tuning stage: Input labeled data containing different scenarios, and use the learning rate cosine annealing strategy to fine-tune the overall parameters of the YOLOv11 model.

Images