Pet key point detection and action recognition method based on improved HRNet combined with SVM and CNN

Abstract

The application provides a pet key point detection and action recognition method based on improved HRNet and SVM and CNN, which comprises a data set construction and preprocessing step, an improved HRNet network construction and training step, a key point connection weight dynamic adjustment step and a pet action recognition step based on CNN and SVM. The application can effectively solve the problems of low key point detection accuracy and insufficient action recognition robustness of traditional methods in the occlusion scene, thereby significantly improving the accuracy and reliability of pet posture and action recognition in complex environments.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning technology, specifically to a method for pet keypoint detection and action recognition based on an improved HRNet combined with SVM and CNN, which can be widely applied to scenarios such as pet behavior analysis, intelligent monitoring, and companion animal health management. Background Technology

[0002] Pet motion recognition technology originated from general human motion recognition technology. However, due to the diverse body shapes, flexible movements, and highly personalized behavior patterns of pets (such as dogs and cats), general motion recognition models are difficult to directly adapt, prompting the technology to iterate towards pet-specific scenarios.

[0003] Early action recognition technologies primarily relied on traditional computer vision methods, combining handcrafted features (such as HOG, SIFT, and optical flow fields) with traditional machine learning algorithms like Support Vector Machines (SVM) and Hidden Markov Models (HMM) to classify actions. However, these methods are sensitive to ambient lighting and background complexity. In recent years, with the rapid development of deep learning technology, action recognition methods based on deep neural networks have become mainstream. Convolutional Neural Networks (CNNs), with their powerful image feature extraction capabilities, have overcome the limitations of traditional methods' handcrafted feature design; Recurrent Neural Networks (RNNs) and Long Short-Term Memory Networks (LSTMs) can effectively capture the temporal dependencies of actions; and models based on architectures such as 3D CNNs, dual-stream networks (spatial flow + temporal flow), and Transformers have further improved action recognition performance in complex scenes.

[0004] However, among existing pet keypoint detection methods, HRNet (High-Resolution Network), while widely used in pose estimation tasks due to its high localization accuracy and ability to maintain high-resolution feature maps throughout the inference process, has significant shortcomings in handling occluded scenes: when pet body parts are occluded or invisible, the model struggles to distinguish between valid keypoints and occluded areas, easily leading to inaccurate keypoint localization or missed detections. Furthermore, traditional methods typically use the detected keypoint sequence directly for action recognition without considering the interference of occluded keypoints on action feature extraction, resulting in poor robustness in action recognition.

[0005] Therefore, how to improve the HRNet network to enhance the accuracy of key point detection in occluded scenes, and at the same time combine SVM and CNN to achieve accurate recognition of pet movements, has become an urgent technical problem to be solved. Summary of the Invention

[0006] The purpose of this invention is to provide a pet keypoint detection and action recognition method based on an improved HRNet combined with SVM and CNN, which can effectively solve the problems of low keypoint detection accuracy and insufficient action recognition robustness of traditional methods in occluded scenes, and significantly improve the accuracy and reliability of pet posture and action recognition in complex environments.

[0007] The present invention achieves the above objectives through the following technical solutions: A pet keypoint detection and action recognition method based on an improved HRNet combined with SVM and CNN includes: Dataset Construction and Preprocessing: Collect images and videos of pets of different breeds and postures, annotate key points of the pets, and construct a pet key point dataset; preprocess the dataset and use data augmentation strategies to expand the training samples; Improved HRNet network construction: Using HRNet as the basic network architecture, a composite attention module is inserted after the multi-resolution feature fusion layer, a combined loss function is designed for model training, and the Adam optimizer is used to optimize the model parameters; Dynamic weight adjustment: Input the pet image to be detected into the trained improved HRNet network, and output the coordinate information and heatmap response value of each pet key point; construct a key point visibility scoring function and dynamically adjust the key point connection weights; Action recognition based on CNN and SVM: A two-branch classification architecture of CNN branch + SVM branch is constructed. The spatial morphological features of pet actions are extracted by the CNN branch, and the displacement features and motion dynamic features of pet actions are fused by the SVM branch. Finally, the classification output of the two branches is combined by the fusion model to obtain the pet action category.

[0008] According to the present invention, a method for pet keypoint detection and action recognition based on an improved HRNet combined with SVM and CNN is provided. When preprocessing the dataset, grayscale normalization is first performed to eliminate the influence of illumination. Then, the images are uniformly scaled to a size of 640×640, and the annotation format is standardized to (x, y, visibility), where visibility represents the keypoint visibility label. Data augmentation strategies are employed to expand the training samples, including at least: random rotation, random scaling, random cropping, brightness and contrast perturbation, and random occlusion simulation.

[0009] According to the present invention, a method for pet keypoint detection and action recognition based on an improved HRNet combined with SVM and CNN is provided. In the improved HRNet network construction step, the composite attention module consists of a GAM module and a DeformableAttention module. The GAM module is used to adjust the weights of the channel dimension and spatial dimension of the feature map to enhance the features of key regions, and the DeformableAttention module is used to dynamically capture the spatial relationship of visible pet keypoints. The combined loss function includes the MSE loss function and the cross-entropy loss function, which are expressed as: L = α×L_MSE +β×L_CE. The MSE loss function is used to supervise the pet keypoint heatmap regression to minimize the difference between the pet keypoint heatmap output by the model and the real heatmap. The cross-entropy loss function is used to supervise the pet keypoint visibility prediction to help the model learn the discriminative features of occluded areas.

[0010] According to the present invention, a method for pet keypoint detection and action recognition based on an improved HRNet combined with SVM and CNN is provided. In the dynamic weight adjustment step, the visibility scoring function calculates the visibility score S of each pet keypoint based on the heatmap response value, the spatial distance constraint between adjacent pet keypoints, and the visibility label. The visibility of the pet keypoint is determined according to the matching relationship between the visibility score S and the preset visibility threshold: when S ≥ the preset visibility threshold, the pet keypoint is determined to be a visible keypoint; when S < the preset visibility threshold, the pet keypoint is determined to be an occluded or invisible keypoint. Based on the visibility judgment results, preset weights are assigned to pet keypoint pairs with different visibility states: the connection weight between visible keypoint pairs is the first preset weight, the connection weight between keypoint pairs containing at least one occluded / invisible keypoint is the second preset weight, and the first preset weight is greater than the second preset weight. The logical expression for weight adjustment is: W ij = W v I ( S i ≥ T and S j ≥ T )+ W o I ( S i < T or S j <T ) in, W ij This represents the connection weight between the i-th pet keypoint and the j-th pet keypoint. T To preset the visibility threshold, W v As the first preset weight, W o As the second preset weight, I ( ) is an indicator function, and satisfies W v > W o >0.

[0011] According to the present invention, a method for pet keypoint detection and action recognition based on an improved HRNet combined with SVM and CNN is provided. In the action recognition step based on CNN and SVM, the processing of the pet keypoint heatmap includes the following steps: Standardize the single-frame pet key point heatmap: scale each pet key point heatmap to a uniform size of 128×128 pixels and normalize the pixel values ​​to the [0,1] range to eliminate size and brightness differences caused by different pet sizes or shooting angles; wherein, the pet key point heatmap is a probability distribution heatmap of each pet key point in a single frame image, and the heatmap value represents the probability that the corresponding position is the pet key point; The heatmaps of key points of pet action sequences are overlaid and fused: a weighted overlay strategy of maximum value and mean value is adopted; the logic of overlay and fusion is as follows: the maximum value operation retains the peak features of violent actions; the mean operation retains the overall shape of slow motion; finally, a single overlaid heatmap is generated as input data for the CNN branch.

[0012] According to the present invention, a method for pet keypoint detection and action recognition based on an improved HRNet combined with SVM and CNN is provided. In the action recognition step based on CNN and SVM, video frame preprocessing and pet keypoint coordinate cleaning are implemented according to the following steps: Video frame preprocessing steps: Grayscale conversion: Converts the input RGB video frames into grayscale images, reducing computational complexity; ROI cropping: The background area is cropped based on the detected pet bounding box, retaining only the rectangular area where the pet itself is located. The size of the rectangular area is set to 64×128 pixels. Size normalization: The cropped pet area image is uniformly scaled to a fixed size of 64×128 pixels to eliminate the resolution differences between different input images.

[0013] Key point coordinate cleaning steps: Confidence filtering: Remove invalid keypoints from all pet keypoints whose confidence scores are lower than the preset confidence score, where the confidence score reflects the reliability of keypoint detection; Missing details completion: For missing key points about the pet, the following methods are used to complete them: Temporal interpolation: linear interpolation is performed using the coordinates of the same pet keypoint in adjacent video frames; Body size constraint: The interpolation results are constrained and corrected based on the pet's body size proportion characteristics; Coordinate normalization: The coordinates of all pet key points are normalized based on the boundaries of the pet detection box and mapped to relative coordinate values ​​in the range [0,1] to eliminate the influence of different shooting distances and angles on the coordinate values.

[0014] According to the present invention, a method for pet keypoint detection and action recognition based on an improved HRNet combined with SVM and CNN is provided. In the action recognition step based on CNN and SVM, the acquisition of motion dynamic features includes: Temporal features are extracted from the continuous frame pet keypoint coordinate sequence as motion dynamic features. The motion dynamic features are used to reflect the dynamic changes of limb keypoints, including displacement features and velocity / acceleration features. The displacement feature is calculated as follows: for each key point of the torso, limbs and tail, the total displacement in consecutive frames is calculated, and the dimension of the displacement feature is the first preset dimension corresponding to the number of key points of the pet. The velocity / acceleration feature is calculated as follows: the average velocity of each pet keypoint in consecutive frames is calculated, and the velocity variance is extracted. The dimension of the velocity / acceleration feature is a second preset dimension corresponding to the number of pet keypoints.

[0015] According to the present invention, a method for pet keypoint detection and action recognition based on an improved HRNet combined with SVM and CNN is provided. This method extracts spatial morphological features of pet actions through CNN branches, including: The pet key point heatmap sequence is superimposed into a single superimposed heatmap; for occluded key points, their features are represented by a mask and the corresponding motion features are set to infinite values; a convolutional neural network is used to extract the spatial morphological features of pet movements from the single superimposed heatmap, and the spatial morphological features are used to characterize the spatial distribution of movement patterns such as limbs curled up when lying down, limbs extended when jumping, and tail wagging trajectory when wagging tail; the probability values ​​of each category of movement are output through a softmax classification layer; Specifically, the cross-entropy loss function is used to adapt to the requirements of multi-class classification tasks; the Adam optimizer is used, and the initial learning rate is set to 5×10. -4To mitigate the risk of overfitting due to the small sample size of the pet dataset, Dropout and L2 regularization layers were added to the convolutional neural network. Slight rotation and translation operations were performed on the pet keypoint heatmap to simulate the angle changes of pet movements.

[0016] According to the present invention, a method for pet keypoint detection and action recognition based on an improved HRNet combined with SVM and CNN is provided. This method fuses displacement features and motion dynamic features of pet actions through an SVM branch, including: The acquired pet motion displacement features and motion dynamic features are fused to construct a fixed-dimensional feature vector; where: The feature of each pet keypoint is the difference between the feature dimension of that keypoint and the start time of the action segment; The motion features of the pet's key points are superimposed, with the time reference of the motion features being the start time of the action segment; For key points of the pet that are occluded, their features are represented by a mask, and the corresponding motion features are set to infinite values. An SVM classifier with the RBF kernel function is used to adapt to the non-linear feature distribution of pet movements; The hyperparameters of the SVM were tuned using a grid search method. The hyperparameters included: penalty coefficient C: balancing classification error and model complexity; kernel parameter γ: controlling the radial basis function width of the RBF kernel to adapt to the feature scale of the pet dataset. The distance value output by the SVM decision function is mapped to a confidence level in the [0,1] interval using the Sigmoid function to quantify the reliability of the action classification results; By integrating the static body shape features and dynamic movement features of pets, a highly discriminative feature vector is constructed.

[0017] According to the present invention, a method for pet keypoint detection and action recognition based on an improved HRNet combined with SVM and CNN is provided. In the action recognition step based on CNN and SVM, the fusion model combines the classification results of the two branches through a weighted strategy to output the pet action category, including: Basic weights are assigned, with the basic weights of the CNN branch set as the first preset weights and the basic weights of the SVM branch set as the second preset weights. The first preset weights are set based on the advantage of the CNN branch in extracting static pose features of the pet, while the second preset weights are set based on the advantage of the SVM branch in extracting dynamic action features of the pet. The first preset weights are greater than the second preset weights. Based on the classification accuracy of the two branches under different action categories in the validation set statistics, the basic weights are dynamically adjusted: when the accuracy of the SVM branch of the target action category is higher than that of the CNN branch, the weight of the SVM branch is increased; when the accuracy of the CNN branch of the target action category is higher than that of the SVM branch, the weight of the CNN branch is increased. The CNN branch outputs the probability value of each action category; the SVM branch outputs the target category confidence score and maps this confidence score to the overall category probability, calculating a weighted probability for each action category using the following formula: P final = P CNN × w CNN + P SVM × w SVM in, P CNN The probability of this class output by the CNN branch. w CNN These are the current weights of the CNN branch; P SVM This represents the probability of that class after SVM branch mapping. w SVM The current weights of the SVM branch; The action category with the highest weighted probability is selected as the final classification result.

[0018] Therefore, compared with existing technologies, the pet keypoint detection and action recognition method based on improved HRNet combined with SVM and CNN proposed in this invention has the following beneficial effects: 1. This invention introduces a composite attention module into the HRNet network and combines it with a visibility scoring function to achieve automatic identification and accurate localization of occluded key points. This effectively solves the problem of inaccurate key point localization or missed detection in traditional HRNet when dealing with occluded scenes, significantly improves the detection accuracy of pet key points in occluded scenes, and provides more reliable basic data for action recognition.

[0019] 2. By dynamically adjusting connection weights based on key point visibility, this invention can weaken the weights of occluded or invisible parts, thereby effectively reducing the interference of occlusion on the action recognition process. This allows the model to maintain high recognition accuracy and stability when faced with complex and varied pet actions, significantly improving the robustness of action recognition.

[0020] 3. This invention achieves comprehensive capture and effective fusion of pet static posture and dynamic motion features by optimizing the heatmap overlay method, feature extraction dimensions, and fusion weights. The CNN branch focuses on extracting the spatial morphological features of pet movements, while the SVM branch integrates displacement features and dynamic motion features. This dual-branch fusion strategy significantly improves the accuracy of action classification, enabling the model to more accurately identify various pet action categories.

[0021] 4. This invention includes pet key point design, weighted heatmap overlay, and dynamic adjustment of branch weights. These designs are directly optimized for the action classification needs of common pets such as cats and dogs, exhibiting strong relevance and practicality. Therefore, the method of this invention is easily applicable in various scenarios such as pet behavior analysis, intelligent monitoring, and companion animal health management, possessing broad market prospects and social value.

[0022] 5. This invention balances the accuracy of keypoint detection with the efficiency of action recognition. By optimizing the network structure and algorithm design, this invention achieves real-time processing capabilities for pet image and video data, meeting the demands for efficient processing in various practical application scenarios.

[0023] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0024] Figure 1 This is a flowchart of an embodiment of a pet keypoint detection and action recognition method based on an improved HRNet combined with SVM and CNN according to the present invention.

[0025] Figure 2 This is a schematic diagram illustrating the steps of an embodiment of the pet keypoint detection and action recognition method based on an improved HRNet combined with SVM and CNN according to the present invention.

[0026] Figure 3 This is a flowchart illustrating the CNN branch in an embodiment of the pet keypoint detection and action recognition method based on an improved HRNet combined with SVM and CNN according to the present invention.

[0027] Figure 4 This is a flowchart illustrating the SVM branch of an embodiment of the present invention, which is based on an improved HRNet and combines SVM and CNN for pet keypoint detection and action recognition.

[0028] Figure 5 This is a flowchart illustrating the principle of the fusion model in an embodiment of the present invention, which is based on an improved HRNet and combines SVM and CNN for pet keypoint detection and action recognition. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0030] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0031] See Figures 1 to 5 This embodiment provides a method for pet keypoint detection and action recognition based on an improved HRNet combined with SVM and CNN, including: Step S1: Dataset Construction and Preprocessing: Collect images and videos of pets of different breeds and postures, annotate key points of the pets, and construct a pet key point dataset; preprocess the dataset and use data augmentation strategies to expand the training samples; Step S2: Improve HRNet network construction: Using HRNet as the basic network architecture, insert a composite attention module after the multi-resolution feature fusion layer, design a combined loss function for model training, and use the Adam optimizer to optimize model parameters. Step S3, Dynamic Weight Adjustment: Input the pet image to be detected into the trained improved HRNet network, and output the coordinate information and heatmap response value of each pet key point; construct the key point visibility scoring function and dynamically adjust the key point connection weights; Step S4: Action recognition based on CNN and SVM: Construct a two-branch classification architecture with CNN branch + SVM branch. Extract the spatial morphological features of pet actions through the CNN branch, and fuse the displacement features and motion dynamic features of pet actions through the SVM branch. Finally, combine the classification outputs of the two branches through the fusion model to obtain the pet action category.

[0032] In the dataset construction and preprocessing steps, when preprocessing the dataset, grayscale normalization is first performed to eliminate the influence of illumination, then the images are uniformly scaled to a size of 640×640, and the annotation format is standardized to (x, y,visibility), where visibility represents the visibility label of key points; Data augmentation strategies are employed to expand the training samples, including at least: random rotation (angle range -30° to 30°), random scaling (scaling ratio 0.8 to 1.2 times), random cropping, brightness and contrast perturbation, and random occlusion simulation (used to simulate scenarios where pet body parts are occluded by obstacles).

[0033] In improving the HRNet network construction process, HRNet is used as the basic network architecture, retaining its parallel multi-resolution sub-networks and feature fusion mechanism to ensure the high-resolution advantage of keypoint localization. A composite attention module is inserted after the multi-resolution feature fusion layer of HRNet. This module consists of a GAM (Generalized Attention Module) and a Deformable Attention module. The GAM module first adjusts the weights of the channel and spatial dimensions of the feature map to enhance the features of key regions. Then, the Deformable Attention module dynamically captures the spatial relationships of visible pet keypoints to improve feature extraction capabilities in occluded scenes. The combined loss function includes the MSE loss function and the cross-entropy loss function, expressed as: L = α×L_MSE +β×L_CE, where α=0.7 and β=0.3. The optimal weight coefficients are determined using a validation set. The MSE loss function is used to supervise the pet keypoint heatmap regression to minimize the difference between the model's output pet keypoint heatmap and the real heatmap. The cross-entropy loss function is used to supervise the pet keypoint visibility prediction to assist the model in learning the discriminative features of occluded regions. The Adam optimizer is used for model training, with an initial learning rate set to 1e- 4 The loss function is reduced to 0.5 every 10 epochs, and training is iterated for 50-80 epochs until the loss function converges.

[0034] In the dynamic weight adjustment step, the visibility scoring function calculates the visibility score S for each pet keypoint based on the heatmap response value, the spatial distance constraint between adjacent pet keypoints, and the visibility label. The visibility of the pet keypoint is determined according to the matching relationship between the visibility score S and the preset visibility threshold: when S ≥ the preset visibility threshold, the pet keypoint is determined to be a visible keypoint; when S < the preset visibility threshold, the pet keypoint is determined to be an occluded or invisible keypoint. For example, in this embodiment, the preset visibility threshold is 0.6. When S ≥ 0.6, it is determined to be a visible keypoint; when S < 0.6, it is determined to be an occluded or invisible keypoint. Dynamically adjust key point connection weights: Based on the visibility judgment results, assign preset weights to pet key point pairs with different visibility states: the connection weight between visible key point pairs is the first preset weight, the connection weight between key point pairs containing at least one occluded / invisible key point is the second preset weight, and the first preset weight is greater than the second preset weight. The logical expression for weight adjustment is: W ij = W v I ( S i ≥ T and S j ≥ T )+ W o I ( S i < T or S j < T ) in, W ij This represents the connection weight between the i-th pet keypoint and the j-th pet keypoint. T To preset the visibility threshold, W v As the first preset weight, W o As the second preset weight, I ( ) is an indicator function, and satisfies W v > W o >0.

[0035] For example, based on the visibility assessment results, the connection weight between visible keypoints is set to 0.9, while the connection weight between occluded or invisible keypoints is weakened to 0.2, reducing the interference of occluded parts on subsequent action recognition; the weight adjustment formula is: W ij =0.9 I ( S i ≥0.6 and S j ≥0.6)+0.2 I ( S i <0.6 orS j <0.6).

[0036] In the action recognition step based on CNN and SVM, for the pet action classification scenario, a two-branch classification architecture of "CNN branch (spatial features) + SVM branch (dynamic and static fusion features) + result fusion" is constructed.

[0037] In this embodiment, the processing of the pet key point heatmap includes the following steps: Standardize the single-frame pet key point heatmap: scale each pet key point heatmap to a uniform size of 128×128 pixels and normalize the pixel values ​​to the [0,1] range to eliminate size and brightness differences caused by different pet sizes or shooting angles; the pet key point heatmap is a probability distribution heatmap of each pet key point in a single frame image. The heatmap value represents the probability that the corresponding position is a key point of that pet. The number of key points is set to 16~20 according to the pet type (e.g., 17 for cats, 19 for dogs). The larger the value, the higher the probability that the position is the corresponding key point; The pet action sequence (e.g., 20 consecutive frames) is overlaid and fused using a weighted overlay strategy of maximum and mean values, where the maximum value has a weight of 0.7 and the mean value has a weight of 0.3. The logic of the overlay and fusion is as follows: the maximum value operation preserves the peak features of violent actions (e.g., jumping, tail whipping); the mean operation preserves the overall shape of slow-motion actions (e.g., licking fur); finally, a single overlaid heatmap is generated as input data for the CNN branch.

[0038] In the action recognition step based on CNN and SVM, video frame preprocessing and pet keypoint coordinate cleaning are performed according to the following steps: Video frame preprocessing steps: Grayscale conversion: Converts the input RGB video frames into grayscale images, reducing computational complexity; ROI cropping: The background area is cropped based on the detected pet bounding box, retaining only the rectangular area where the pet itself is located. The size of the rectangular area is set to 64×128 pixels. Size normalization: The cropped pet area image is uniformly scaled to a fixed size of 64×128 pixels to eliminate the resolution differences between different input images.

[0039] Key point coordinate cleaning steps: Confidence filtering: Remove invalid keypoints from all pet keypoints whose confidence scores are lower than the preset confidence score. The confidence score reflects the reliability of keypoint detection. For example, keypoints with a confidence score <0.4 are removed (pet fur occlusion can easily lead to low keypoint confidence). Missing details completion: For key pet details (such as the tail) that are missing due to occlusion or other reasons, the following methods can be used to complete them: Temporal interpolation: linear interpolation is performed using the coordinates of the same pet keypoint in adjacent video frames; Body size constraint: The interpolation results are constrained and corrected by combining the pet's body size proportion characteristics (such as the standard distance ratio from the dorsal node to the tail node); Coordinate normalization: The coordinates of all pet key points are normalized based on the boundaries of the pet detection box and mapped to relative coordinate values ​​in the range [0,1] to eliminate the influence of different shooting distances and angles on the coordinate values.

[0040] In the action recognition process based on CNN and SVM, the acquisition of motion dynamic features includes: Temporal features are extracted from the continuous frame pet keypoint coordinate sequence as motion dynamic features. The motion dynamic features are used to reflect the dynamic changes of limb keypoints, including displacement features and velocity / acceleration features. Taking 15 pet key points as an example, as shown in Table 1, the displacement feature is calculated as follows: for each key point of the torso (head / back), limbs and tail, calculate its total displacement in consecutive frames (using Euclidean distance metric). The dimension of the displacement feature is the first preset dimension corresponding to the number of pet key points, and the dimension of the displacement feature is 15. The speed / acceleration feature is calculated as follows: the average speed of each pet keypoint in consecutive frames is calculated, and the speed variance is extracted (the speed variance is used to characterize the intensity of the action, such as distinguishing the difference between jumping and licking). The dimension of the speed / acceleration feature is the second preset dimension corresponding to the number of pet keypoints, and the dimension of the speed / acceleration feature is 30.

[0041] Table 1: Pet Motion Feature Extraction (15 key points as an example)

[0042] In the action recognition step based on CNN and SVM, spatial morphological features of pet actions are extracted through CNN branches, including: The pet key point heatmap sequence is superimposed into a single superimposed heatmap; for occluded key points, their features are represented by a mask and the corresponding motion features are set to infinite values; a convolutional neural network is used to extract the spatial morphological features of pet movements from the single superimposed heatmap, and the spatial morphological features are used to characterize the spatial distribution of movement patterns such as limbs curled up when lying down, limbs extended when jumping, and tail wagging trajectory when wagging tail; the probability values ​​of each category of movement are output through a softmax classification layer; Specifically, the cross-entropy loss function is used to adapt to the requirements of multi-class classification tasks; the Adam optimizer is used, and the initial learning rate is set to 5×10. -4To mitigate the risk of overfitting due to the small sample size of the pet dataset, a Dropout layer (with a dropout probability of 0.3) and an L2 regularization layer (with a weight decay coefficient of 1×10⁻⁶) were added to the convolutional neural network. -4 ); Slightly rotate (angle range ±10°) and translate (pixel range ±5) the pet's key point heatmap to simulate the angle change scene of the pet's movement.

[0043] In the action recognition step based on CNN and SVM, the displacement features and motion dynamic features of pet actions are fused through SVM branches, including: The acquired pet motion displacement features are fused with motion dynamic features to construct a fixed-dimensional feature vector, which is then input into an SVM for classification. This compensates for the shortcomings of CNNs in capturing temporal motion information. The feature of each keypoint is the difference between the feature dimension of that node and the start time of the action segment (capturing temporal changes). The motion features of this key point are superimposed, and the time reference of the motion features is the start time of the action segment; For occluded keypoints, their features are represented by a mask, and the corresponding motion features are set to infinite values ​​(marking the occlusion state).

[0044] An SVM classifier with the RBF kernel function is used to adapt to the non-linear feature distribution of pet movements (pet movements are diverse and linear models are difficult to capture complex patterns).

[0045] The hyperparameters of the SVM were tuned using a grid search method. The hyperparameters included the penalty coefficient C (with a value ranging from 0.1). 50): Balancing classification error with model complexity; kernel parameter γ (range 0.0001) 0.1): Controls the radial basis function width of the RBF kernel to adapt to the feature scale of the pet dataset.

[0046] The distance value output by the SVM decision function is mapped to a confidence level in the [0,1] interval using the Sigmoid function to quantify the reliability of the action classification results (solving the problem that the SVM output has no probabilistic meaning).

[0047] By integrating the static body features of pet movements (such as limb length and torso proportion) with "dynamic motion features" (such as temporal changes of key points), a highly discriminative feature vector is constructed to improve the adaptability to the non-linear distribution of pet movements (making up for the limitation of single features in distinguishing complex movements).

[0048] In the action recognition step based on CNN and SVM, the fusion model combines the classification results of the two branches through a weighted strategy to output pet action categories, adapting to the action recognition of common pets such as cats and dogs (such as running, jumping, lying down, licking fur, wagging tail, etc.), including: Basic weights are assigned, with the basic weights of the CNN branch set as the first preset weights and the basic weights of the SVM branch set as the second preset weights, for example, 0.65 for CNN and 0.35 for SVM. The first preset weights are set based on the CNN branch's advantage in extracting static pet posture features (such as spatial layout features of lying down and sitting), while the second preset weights are set based on the SVM branch's advantage in extracting dynamic pet action features (such as temporal change features of tail wagging and jumping). The first preset weights are greater than the second preset weights. Based on the classification accuracy of the two branches under different action categories in the validation set statistics, the basic weights are dynamically adjusted. For example, when the accuracy of the SVM branch for the target action category (such as "tail wagging") is higher than that of the CNN branch, the weight of the SVM branch is increased, such as increasing the SVM weight to 0.6; when the accuracy of the CNN branch for the target action category (such as "lying down") is higher than that of the SVM branch, the weight of the CNN branch is increased, such as increasing the CNN weight to 0.7.

[0049] The CNN branch outputs the probability value of each action category (e.g., the probability of "jump" is 0.92); the SVM branch outputs the target category confidence score (e.g., the confidence score of "jump" is 0.90), and maps this confidence score to the probability of all categories (the probability of the target category "jump" is 0.90, and the remaining 0.10 is evenly distributed among the other categories). Calculate the weighted probability for each action category using the following formula: P final = P CNN × w CNN + P SVM × w SVM in, P CNN The probability of this class output by the CNN branch. w CNN These are the current weights of the CNN branch; P SVM This represents the probability of that class after SVM branch mapping. w SVM The current weights of the SVM branch; The action category with the highest weighted probability is selected as the final classification result.

[0050] In this embodiment, an exception handling mechanism is also provided, including: Confidence threshold determination: Set a minimum weighted probability threshold, such as 0.7. When the final weighted probability is lower than this threshold, output the unknown action category to avoid misclassification of rare pet actions; Category difference correction: When the action categories output by the CNN branch and the SVM branch differ significantly (e.g., CNN judges "lying down" while SVM judges "jumping"), the branch result with higher accuracy in that action category in the validation set is given priority.

[0051] In practical applications, the key point detection for pets includes the following steps: Dataset Construction: Collect 1000 images of pets of different breeds and in different poses, and annotate the key skeletal points of the pets (16 key points in total, including the head, neck, torso, and limbs) to construct a pet keypoint dataset.

[0052] Preprocessing: The dataset is grayscale normalized, and the images are uniformly scaled to 640×640 pixels. The standardized annotation format is (x, y, visibility).

[0053] Data augmentation: Random rotation (-30°~30°), scaling (0.8~1.2x), cropping, brightness and contrast perturbation, and random occlusion simulation were used to expand the training samples to 5000 images.

[0054] Improved HRNet network training: Based on HRNet as the basic network architecture, a composite attention module is inserted, and a combined loss function is used for training. The initial learning rate is set to 1e- 4 The loss function is reduced to 0.5 every 10 epochs, and training is iterated for 60 epochs until the loss function converges.

[0055] Keypoint detection: Input the pet image to be detected into the trained improved HRNet network, output the coordinate information and heatmap response value of each keypoint, calculate the visibility score S of each keypoint, determine whether the keypoint is occluded based on the S value, and dynamically adjust the keypoint connection weights.

[0056] In practical applications, pet motion recognition includes the following steps: Dataset construction: Collect 500 videos of pet actions, each video is about 5 seconds long, and label the pet action categories (sitting, lying down, running, jumping, licking fur, pouncing and biting, etc.).

[0057] Preprocessing and key point detection: The video frames are preprocessed by grayscale conversion, ROI cropping, and size normalization. An improved HRNet network is used for key point detection to obtain the key point coordinate sequence.

[0058] Motion feature extraction: Extract displacement and velocity / acceleration features from the keypoint coordinate sequence to construct a motion feature vector.

[0059] Heatmap overlay and fusion: For each video segment, 20 frames of images are selected to generate a heatmap sequence of key pet nodes. The maximum value + mean value are weighted and overlaid (weight 7:3) to generate a single overlaid heatmap.

[0060] CNN Branch Training and Classification: The superimposed heatmap is input into the CNN branch, and it is trained using the cross-entropy loss function and Adam optimizer (learning rate 0.0005). Dropout layer and L2 regularization are added to output the probability value of each action category.

[0061] SVM Branch Training and Classification: Input the motion feature vector into the SVM branch, use the RBF kernel function, optimize the hyperparameters (C=10, γ=0.01) through grid search, output the target class confidence and map it to the probability of all classes.

[0062] Fusion classification: The classification results of CNN and SVM branches are fused using a weighted voting method. The basic weight of CNN is 0.65 and the basic weight of SVM is 0.35. The weights are dynamically adjusted according to the accuracy of the validation set to output the final action category.

[0063] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0064] The above embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of protection of the present invention. Any non-substantial changes and substitutions made by those skilled in the art based on the present invention shall fall within the scope of protection claimed by the present invention.

Claims

1. A method for pet keypoint detection and action recognition based on an improved HRNet combined with SVM and CNN, characterized in that, include: Dataset construction and preprocessing: Collect images and videos of pets of different breeds and postures, annotate key points of the pets, and construct a pet key point dataset; The dataset is preprocessed, and data augmentation strategies are used to augment the training samples. Improved HRNet network construction: Using HRNet as the basic network architecture, a composite attention module is inserted after the multi-resolution feature fusion layer, a combined loss function is designed for model training, and the Adam optimizer is used to optimize the model parameters; Dynamic weight adjustment: Input the pet image to be detected into the trained improved HRNet network, and output the coordinate information and heatmap response value of each pet key point; construct a key point visibility scoring function and dynamically adjust the key point connection weights; Action recognition based on CNN and SVM: A two-branch classification architecture of CNN branch + SVM branch is constructed. The spatial morphological features of pet actions are extracted by the CNN branch, and the displacement features and motion dynamic features of pet actions are fused by the SVM branch. Finally, the classification output of the two branches is combined by the fusion model to obtain the pet action category.

2. The method according to claim 1, characterized in that: When preprocessing the dataset, grayscale normalization is first performed to eliminate the influence of illumination. Then, the images are uniformly scaled to a size of 640×640, and the annotation format is standardized to (x, y, visibility), where visibility represents the visibility label of key points. Data augmentation strategies are employed to expand the training samples, including at least: random rotation, random scaling, random cropping, brightness and contrast perturbation, and random occlusion simulation.

3. The method according to claim 1, characterized in that: In the improved HRNet network construction steps, the composite attention module consists of a GAM module and a Deformable Attention module. The GAM module is used to adjust the weights of the channel dimension and spatial dimension of the feature map to enhance the features of key regions, while the Deformable Attention module is used to dynamically capture the spatial relationship of visible pet key points. The combined loss function includes the MSE loss function and the cross-entropy loss function, which are expressed as: L = α×L_MSE + β×L_CE. The MSE loss function is used to supervise the pet keypoint heatmap regression to minimize the difference between the pet keypoint heatmap output by the model and the real heatmap. The cross-entropy loss function is used to supervise the pet keypoint visibility prediction to help the model learn the discriminative features of occluded areas.

4. The method according to claim 1, characterized in that: In the dynamic weight adjustment step, the visibility scoring function calculates the visibility score S for each pet keypoint based on the heatmap response value, the spatial distance constraint between adjacent pet keypoints, and the visibility label. The visibility of the pet keypoint is determined according to the matching relationship between the visibility score S and the preset visibility threshold: when S ≥ the preset visibility threshold, the pet keypoint is determined to be a visible keypoint; when S < the preset visibility threshold, the pet keypoint is determined to be an occluded or invisible keypoint. Based on the visibility judgment results, preset weights are assigned to pet keypoint pairs with different visibility states: the connection weight between visible keypoint pairs is the first preset weight, the connection weight between keypoint pairs containing at least one occluded / invisible keypoint is the second preset weight, and the first preset weight is greater than the second preset weight. The logical expression for weight adjustment is: W ij = W v I ( S i ≥ T and S j ≥ T )+ W o I ( S i < T or S j < T ) in, W ij This represents the connection weight between the i-th pet keypoint and the j-th pet keypoint. T To preset the visibility threshold, W v As the first preset weight, W o As the second preset weight, I ( ) is an indicator function, and satisfies W v > W o >0.

5. The method according to claim 1, characterized in that, In the action recognition process based on CNN and SVM, the processing of the pet keypoint heatmap includes the following steps: Standardize the single-frame pet key point heatmap: scale each pet key point heatmap to a uniform size of 128×128 pixels and normalize the pixel values ​​to the [0,1] range to eliminate size and brightness differences caused by different pet sizes or shooting angles; wherein, the pet key point heatmap is a probability distribution heatmap of each pet key point in a single frame image, and the heatmap value represents the probability that the corresponding position is the pet key point; The heatmaps of key points of pet action sequences are overlaid and fused: a weighted overlay strategy of maximum value and mean value is adopted; the logic of overlay and fusion is as follows: the maximum value operation retains the peak features of violent actions; the mean operation retains the overall shape of slow motion; finally, a single overlaid heatmap is generated as input data for the CNN branch.

6. The method according to claim 1, characterized in that, In the action recognition step based on CNN and SVM, video frame preprocessing and pet keypoint coordinate cleaning are performed according to the following steps: Video frame preprocessing steps: Grayscale conversion: Converts the input RGB video frames into grayscale images, reducing computational complexity; ROI cropping: The background area is cropped based on the detected pet bounding box, retaining only the rectangular area where the pet itself is located. The size of the rectangular area is set to 64×128 pixels. Size normalization: The cropped pet area image is uniformly scaled to a fixed size of 64×128 pixels to eliminate the resolution differences between different input images; Key point coordinate cleaning steps: Confidence filtering: Remove invalid keypoints from all pet keypoints whose confidence scores are lower than the preset confidence score, where the confidence score reflects the reliability of keypoint detection; Missing details completion: For missing key points about the pet, the following methods are used to complete them: Temporal interpolation: linear interpolation is performed using the coordinates of the same pet keypoint in adjacent video frames; Body size constraint: The interpolation results are constrained and corrected based on the pet's body size proportion characteristics; Coordinate normalization: The coordinates of all pet key points are normalized based on the boundaries of the pet detection box and mapped to relative coordinate values ​​in the range [0,1] to eliminate the influence of different shooting distances and angles on the coordinate values.

7. The method according to claim 1, characterized in that, In the action recognition process based on CNN and SVM, the acquisition of motion dynamic features includes: Temporal features are extracted from the continuous frame pet keypoint coordinate sequence as motion dynamic features. The motion dynamic features are used to reflect the dynamic changes of limb keypoints, including displacement features and velocity / acceleration features. The displacement feature is calculated as follows: for each key point of the torso, limbs and tail, the total displacement in consecutive frames is calculated, and the dimension of the displacement feature is the first preset dimension corresponding to the number of key points of the pet. The velocity / acceleration feature is calculated as follows: the average velocity of each pet keypoint in consecutive frames is calculated, and the velocity variance is extracted. The dimension of the velocity / acceleration feature is a second preset dimension corresponding to the number of pet keypoints.

8. The method according to claim 1, characterized in that, Spatial morphological features of pet movements are extracted using CNN branches, including: The pet key point heatmap sequence is superimposed into a single superimposed heatmap; for occluded key points, their features are represented by a mask and the corresponding motion features are set to infinite values; a convolutional neural network is used to extract the spatial morphological features of pet movements from the single superimposed heatmap, and the spatial morphological features are used to characterize the spatial distribution of movement patterns such as limbs curled up when lying down, limbs extended when jumping, and tail wagging trajectory when wagging tail; the probability values ​​of each category of movement are output through a softmax classification layer; Specifically, the cross-entropy loss function is adopted to adapt to the requirements of multi-class classification tasks; the Adam optimizer is used, and the initial learning rate is set to 5×10. -4 To mitigate the risk of overfitting due to the small sample size of the pet dataset, Dropout and L2 regularization layers were added to the convolutional neural network. Slight rotation and translation operations were performed on the pet keypoint heatmap to simulate the angle changes of pet movements.

9. The method according to any one of claims 1 to 8, characterized in that, By fusing the displacement features and motion dynamic features of pet actions through SVM branches, including: The acquired pet motion displacement features and motion dynamic features are fused to construct a fixed-dimensional feature vector; where: The feature of each pet keypoint is the difference between the feature dimension of that keypoint and the start time of the action segment; The motion features of the pet's key points are superimposed, with the time reference of the motion features being the start time of the action segment; For key points of the pet that are occluded, their features are represented by a mask, and the corresponding motion features are set to infinite values. An SVM classifier with the RBF kernel function is used to adapt to the non-linear feature distribution of pet movements; The hyperparameters of the SVM were tuned using a grid search method. The hyperparameters included: penalty coefficient C: balancing classification error and model complexity; kernel parameter γ: controlling the radial basis function width of the RBF kernel to adapt to the feature scale of the pet dataset. The distance value output by the SVM decision function is mapped to a confidence level in the [0,1] interval using the Sigmoid function to quantify the reliability of the action classification results; By integrating the static body shape features and dynamic movement features of pets, a highly discriminative feature vector is constructed.

10. The method according to any one of claims 1 to 8, characterized in that, In the action recognition step based on CNN and SVM, the fusion model combines the classification results of the two branches through a weighted strategy to output the pet action category, including: Basic weights are assigned, with the basic weights of the CNN branch set as the first preset weights and the basic weights of the SVM branch set as the second preset weights. The first preset weights are set based on the advantage of the CNN branch in extracting static pose features of the pet, while the second preset weights are set based on the advantage of the SVM branch in extracting dynamic action features of the pet. The first preset weights are greater than the second preset weights. Based on the classification accuracy of the two branches under different action categories in the validation set statistics, the basic weights are dynamically adjusted: when the accuracy of the SVM branch of the target action category is higher than that of the CNN branch, the weight of the SVM branch is increased; when the accuracy of the CNN branch of the target action category is higher than that of the SVM branch, the weight of the CNN branch is increased. The CNN branch outputs the probability value of each action category; the SVM branch outputs the target category confidence score and maps this confidence score to the overall category probability, calculating a weighted probability for each action category using the following formula: P final = P CNN × w CNN + P SVM × w SVM in, P CNN The probability of this class output by the CNN branch. w CNN These are the current weights of the CNN branch; P SVM This represents the probability of that class after SVM branch mapping. w SVM The current weights of the SVM branch; The action category with the highest weighted probability is selected as the final classification result.

Images