Pet highlight clip extraction method and system

Abstract

The present application relates to a kind of pet highlight extraction method and system, belong to pose recognition and evaluation technical field, the method includes: obtaining candidate segment and the absolute time stamp corresponding to each frame in candidate segment;Obtain pet image frame, and the pet image frame that satisfies identification confidence threshold is marked as candidate frame;Obtain effective segment;Obtain the static score corresponding to each effective segment;Obtain the dynamic score of each effective segment;Obtain the motion score corresponding to each effective segment, and obtain the highlight of pet.The method and system provided in the present application, by the way of static score and dynamic score combination, can accurately measure the attraction and interestingness of effective segment, improve the understanding and evaluation ability of pet behavior;In addition, pet highlight extraction system has the ability of accurately identifying pet, evaluating pet behavior and providing intelligent monitoring function, provides comprehensive, efficient and competitive advantage solution for pet owner.

Description

Technical Field

[0001] This invention relates to the field of posture recognition and evaluation technology, and in particular to a method and system for extracting memorable moments from pets. Background Technology

[0002] With the rapid development of artificial intelligence and the Internet of Things (IoT) technologies, pet monitoring has emerged as a new application scenario in intelligent home supervision. Against this backdrop, pet-owning families have a need to record memorable moments of their pets. Pet monitoring technology can help pet owners monitor their pets' condition at any time, promptly detect abnormalities, and take effective measures. Furthermore, this technology can save memorable moments of pets, making it convenient for pet owners to review, share, and cherish. However, current pet monitoring devices on the market primarily provide real-time video, and existing technologies mainly focus on accurately identifying the pet and its position in the frame, lacking methods for recognizing and scoring memorable moments. At the same time, existing technologies require pet owners to manually select and record memorable moments, increasing human interference and workload. In addition, existing technologies mainly focus on real-time observation of the pet's situation, but have limited monitoring of the pet's specific behaviors and postures. Summary of the Invention

[0003] The present invention aims to provide a method and system for extracting exciting moments from pets, in order to overcome the shortcomings of the existing technology. The technical problem to be solved by the present invention is achieved through the following technical solution.

[0004] The method for extracting exciting moments from pets provided by this invention includes: Multiple video clips of pet actions are captured by a camera and used as candidate clips. The candidate clips are then decoded into a frame sequence, and the absolute timestamp of each frame in the frame sequence is obtained. The pre-trained pet recognition model is used to identify pets in each frame of the frame sequence, obtain pet image frames, record the pet names and their corresponding recognition confidence scores in the pet image frames, and mark the pet image frames that meet the recognition confidence threshold as candidate frames. Adjacent candidate frames are matched to form candidate segments. Based on the absolute timestamps corresponding to adjacent candidate frames, the time interval between adjacent candidate frames is obtained. The length of candidate segments that meet the time interval threshold is judged, and the valid segments are obtained according to the judgment result. By assigning corresponding weights to each frame in the valid segments, the confidence score corresponding to the valid segments is obtained. By performing posture scoring on each frame in the valid segments and averaging the posture scores of each frame, the pet posture score corresponding to the valid segments is obtained. By performing a weighted sum of the confidence score and the pet posture score, the static score corresponding to each valid segment is obtained. By extracting motion feature vectors from all valid segments and inputting the extracted motion feature vectors into the trained motion estimation and scoring model, the dynamic scores of all valid segments are obtained, and the average of the dynamic scores of all valid segments is calculated to obtain the dynamic score of each valid segment. The static and dynamic scores are weighted and summed to obtain the motion score for each valid segment. Valid segments with motion scores higher than the motion score threshold are designated as the pet's highlight segments.

[0005] In the above scheme, adjacent candidate frames are matched to form candidate segments. Based on the absolute timestamps corresponding to adjacent candidate frames, the time interval between adjacent candidate frames is obtained. The length of candidate segments that meet the time interval threshold is judged, and the valid segments are obtained according to the judgment result. Subtract the absolute timestamps of adjacent candidate frames in the candidate segment to obtain the time interval between adjacent candidate frames in the candidate segment; When the time interval between adjacent candidate frames in a candidate segment is less than the time interval threshold, it is determined whether its length is within the length threshold range. Candidate segments that are within the length threshold and whose time interval between adjacent candidate frames is less than the time interval threshold will be considered as valid segments.

[0006] In the above scheme, the confidence score corresponding to the valid segment is obtained through the confidence score calculation formula, whereby the confidence score calculation formula is: , of which S confidence The confidence score is given by n, where n is the number of frames in the valid segment, and w is the number of frames in the valid segment. t c represents the weight corresponding to the t-th frame in the effective segment. t Let be the recognition confidence score corresponding to the t-th frame in the valid segment.

[0007] In the above scheme, the extraction of motion feature vectors for all valid segments includes: Match the corresponding pixels of two adjacent frames in all valid segments, obtain the image matrix corresponding to the two adjacent frames in each valid segment based on the corresponding pixels of the two adjacent frames, calculate the difference between the image matrices corresponding to the two adjacent frames in each valid segment, and use the difference between the image matrices corresponding to the two adjacent frames as the motion difference matrix of the pet between the two adjacent frames. The optical flow vector is calculated based on the position of a pixel in the frame to be analyzed in the effective segment and the position of that pixel in the previous frame adjacent to the frame to be analyzed. The pet's movement direction, pet's movement speed and pet's movement acceleration features are obtained through the optical flow vector. The obtained motion difference matrix, pet movement direction, pet movement speed, and pet movement acceleration features are combined to construct the motion feature vector corresponding to each effective segment. The motion feature vectors corresponding to all effective segments are then combined to form a motion feature vector sequence.

[0008] In the above scheme, before inputting the extracted motion feature vectors into the trained motion estimation scoring model, the extracted motion feature vectors are dimensionality reduced, the Pearson correlation coefficient between the dimensionality-reduced motion feature vectors is calculated, and the correlation of the dimensionality-reduced motion feature vectors is screened using the Pearson correlation coefficient.

[0009] In the above scheme, the confidence score and the pet posture score are weighted and summed using a static scoring formula to obtain the static score for each valid segment. The static scoring formula is: S static =αS confidence +(1-α)S posture , of which S static For static scoring, S confidence For the confidence score, S posture Rate the pet's posture, where α is the static rating coefficient.

[0010] In the above scheme, the static score and dynamic score are weighted and summed using the motion score calculation formula to obtain the motion score corresponding to each valid segment. The motion score calculation formula is: FinalScore = βS Dynamic +(1-β)S static Where FinalScore is the athletic score, S Dynamic For dynamic scoring, S static β is the static score, and β is the motion score coefficient.

[0011] The pet highlight extraction system provided by this invention extracts pet highlights using the pet highlight extraction method described above, including: The candidate segment acquisition module is used to acquire multiple pet action video segments through the camera and use them as candidate segments, and to decode the candidate segments into a frame sequence and obtain the absolute timestamp corresponding to each frame in the frame sequence. The candidate frame acquisition module is used to identify pets in each frame of the frame sequence using a pre-trained pet recognition model, acquire pet image frames, record the pet names and their corresponding recognition confidence scores in the pet image frames, and mark pet image frames that meet the recognition confidence threshold as candidate frames. The effective segment acquisition module is used to match adjacent candidate frames to form candidate segments, obtain the time interval between adjacent candidate frames based on the absolute timestamps corresponding to the adjacent candidate frames, and judge the length of candidate segments that meet the time interval threshold. The effective segment is obtained according to the judgment result. The static score acquisition module is used to obtain the confidence score corresponding to the effective segment by setting corresponding weights for each frame in the effective segment, to obtain the pet posture score corresponding to the effective segment by performing posture score on each frame in the effective segment and averaging the posture scores of each frame, and to obtain the static score corresponding to each effective segment by performing a weighted sum of the confidence score and the pet posture score. The dynamic scoring acquisition module is used to extract motion feature vectors from all valid segments, input the extracted motion feature vectors into the trained motion estimation scoring model, obtain the dynamic scores of all valid segments, and calculate the average of the dynamic scores of all valid segments to obtain the dynamic score of each valid segment. The highlight segment acquisition module is used to perform a weighted summation of static and dynamic scores to obtain the motion score corresponding to each valid segment, and to select valid segments with motion scores higher than the motion score threshold as the pet's highlight segments.

[0012] In the above solution, the camera has a built-in video recognition unit, an SDK unit, and an ffmpeg unit. The video recognition unit is used to detect pet action video clips; the SDK unit is used to send the detected pet action video clips to the cloud network; and the ffmpeg unit is used to decode the pet action video clips received from the cloud network into a frame sequence and obtain the absolute timestamp corresponding to each frame in the frame sequence.

[0013] In the above scheme, a trained pose evaluation model scores the pose of each frame in the effective segment. The pose evaluation model includes a data acquisition unit, a data processing unit, a feature extraction unit, a classifier unit, and an evaluation unit. The data acquisition unit is used to acquire pet image data; the data processing unit is used to preprocess the acquired pet image data, including pose annotation and data augmentation; the feature extraction unit is used to extract features from the preprocessed pet image data; the classifier unit is used to predict the pet's pose information based on the extracted features; and the evaluation unit is used to obtain a score for the pet's pose based on the predicted pet pose information and output the score result.

[0014] The embodiments of the present invention have the following advantages: The pet highlight extraction method and system provided in this invention process multiple pet action video clips to obtain valid clips. Static and dynamic scores are then obtained for each valid clip. These scores are weighted and summed to obtain a motion score for each valid clip. Clips with motion scores exceeding a certain threshold are designated as highlight clips. This combination of static and dynamic scoring accurately measures the attractiveness and entertainment value of valid clips, improving the understanding and evaluation of pet behavior. Furthermore, the pet highlight extraction system accurately identifies pets, evaluates pet behavior, and provides intelligent monitoring capabilities, offering pet owners a comprehensive, efficient, and competitive solution. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the steps of a method for extracting exciting moments from pets according to the present invention; Figure 2 This is a schematic diagram of the components of a pet highlight extraction system according to the present invention; Figure 3 This is a schematic diagram of the camera assembly of the present invention; Figure 4 This is a schematic diagram of the composition of the attitude evaluation model of the present invention. Detailed Implementation

[0016] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0017] like Figure 1 As shown, this invention provides a method for extracting exciting moments from pets, including: Step S1: Capture multiple video clips of pet actions using a camera, use them as candidate clips, decode the candidate clips into a frame sequence, and obtain the absolute timestamp corresponding to each frame in the frame sequence.

[0018] Specifically, the pet action video clips are video streams in h.264 or h.265 format. The camera has a built-in ffmpeg unit. The ffmpeg unit can decode the pet action video clips it receives into a frame sequence, and combine the absolute time and offset of the pet action video clips to determine the absolute timestamp corresponding to each frame in the frame sequence.

[0019] Step S2: Use a pre-trained pet recognition model to identify pets in each frame of the frame sequence, obtain pet image frames, record the pet names and their corresponding recognition confidence scores in the pet image frames, and mark the pet image frames that meet the recognition confidence threshold as candidate frames.

[0020] Specifically, the pre-trained pet recognition model can batch process each frame of data in the frame sequence in batch_size. After processing each frame of data, the pre-trained pet recognition model can return information such as the position of the pet box, the pet name, and the recognition confidence in each frame.

[0021] Step S3: Match adjacent candidate frames to form candidate segments. Based on the absolute timestamps corresponding to adjacent candidate frames, obtain the time interval between adjacent candidate frames. Then, determine the length of candidate segments that meet the time interval threshold and obtain valid segments based on the determination results.

[0022] Specifically, step S3 includes: Subtract the absolute timestamps of adjacent candidate frames in the candidate segment to obtain the time interval between adjacent candidate frames in the candidate segment; When the time interval between adjacent candidate frames in a candidate segment is less than the time interval threshold, it is determined whether its length is within the length threshold range. Candidate segments that are within the length threshold and whose time interval between adjacent candidate frames is less than the time interval threshold will be considered as valid segments.

[0023] Step S4: By setting corresponding weights for each frame in the valid segments, obtain the confidence score corresponding to the valid segments. By performing pose scoring on each frame in the valid segments and averaging the pose scores of each frame, obtain the pet pose score corresponding to the valid segments. By performing a weighted summation of the confidence score and the pet pose score, obtain the static score corresponding to each valid segment.

[0024] Specifically, each frame in the effective segment is assigned a corresponding weight according to a Gaussian distribution, where the weight is obtained using the following formula: , where x is the sequence number of the frame in the effective segment, μ is the mean of the Gaussian distribution, representing the sequence number of the center frame in the effective segment, and σ is the standard deviation of the Gaussian distribution, which controls the distribution of weights on the time axis. The closer the frame in the effective segment is to the center frame, the greater the weight; the farther the frame is from the center frame, the smaller the weight. By adjusting the values ​​of the mean and standard deviation, the decay rate and distribution shape of the weights on the time axis can be controlled.

[0025] Specifically, the confidence score for the valid segment is obtained through the confidence score calculation formula, whereby the confidence score calculation formula is: , where S confidence The confidence score is given by n, where n is the number of frames in the valid segment, and w is the number of frames in the valid segment. t c represents the weight corresponding to the t-th frame in the effective segment. t Let be the recognition confidence score corresponding to the t-th frame in the valid segment.

[0026] Specifically, the pose evaluation model is trained to score the pose of each frame in the effective segment. Pet image data including multiple poses of the pet are collected to train the pose evaluation model. These multiple poses of the pet include running, jumping, frontal, side, standing, crawling, sleeping, eating, etc. The fine-tuning method is used during training to fine-tune the network structure of the pose evaluation model to adapt to the pet pose recognition task. After training, each frame in the effective segment is input into the trained pose evaluation model to obtain the pose score of each frame in the effective segment. Finally, the pose scores of all frames in the effective segment are averaged to obtain the pet pose score corresponding to the effective segment.

[0027] Specifically, the static score for each valid segment is obtained by weighted summation of the confidence score and the pet posture score using a static scoring calculation formula. The static scoring calculation formula is: S static =αS confidence +(1-α)S posture , where S static For static scoring, S confidence For the confidence score, S posture Rate the pet's posture; α is the static rating coefficient, which can be adjusted according to the actual situation.

[0028] Step S5: Extract motion feature vectors from all valid segments and input the extracted motion feature vectors into the trained motion estimation scoring model to obtain the dynamic scores of all valid segments. Then, calculate the average of the dynamic scores of all valid segments to obtain the dynamic score of each valid segment.

[0029] Before inputting the extracted motion feature vectors into the trained motion estimation scoring model, the extracted motion feature vectors are dimensionality reduced. The Pearson correlation coefficient between the dimensionality-reduced motion feature vectors is calculated, and the correlation between the dimensionality-reduced motion feature vectors is screened using the Pearson correlation coefficient. Specifically, the Pearson correlation coefficient is used to represent the correlation between the dimensionality-reduced motion feature vectors. When the correlation is higher than the correlation threshold, one motion feature vector is retained from all the motion feature vectors corresponding to that correlation, and the rest of the motion feature vectors are removed.

[0030] Specifically, the motion feature vector extraction for all valid segments includes: Match the corresponding pixels of two adjacent frames in all valid segments. Based on the corresponding pixels of two adjacent frames, obtain the image matrix corresponding to two adjacent frames in each valid segment. Calculate the difference between the image matrices corresponding to two adjacent frames in each valid segment. Use the difference between the image matrices corresponding to two adjacent frames as the motion difference matrix of the pet between two adjacent frames. Match the corresponding pixels of two adjacent frames in the valid segment using SIFT, corner points, etc. Denote the corresponding pixels of two adjacent frames in the valid segment as (x, y). Denote the image matrix corresponding to the frame at the current time t in the valid segment as I(x, y, t). Denote the image matrix corresponding to the frame at the previous time t-1 adjacent to the frame at the current time t as I(x, y, t-1). The calculation formula for the motion difference matrix D(x, y, t) between two adjacent frames is as follows: D(x, y, t) = I(x, y, t) - I(x, y, t-1). The optical flow vector is calculated based on the position of a pixel in the frame to be analyzed in the effective segment and the position of that pixel in the previous frame adjacent to the frame to be analyzed. The pet's movement direction, pet's movement speed and pet's movement acceleration features are obtained through the optical flow vector. If a pixel in the frame to be analyzed within the effective segment is at position (x, y), and the pixel in the preceding frame adjacent to the frame to be analyzed is at position (x', y'), then based on the assumption of constant brightness, the following optical flow equation can be obtained: I_x×u + I_y×v = -I_t, where I_x and I_y represent the horizontal and vertical gradients of the pixel at (x, y) in the frame to be analyzed, respectively, I_t represents the grayscale or brightness change between the frame to be analyzed and the preceding frame adjacent to the frame to be analyzed, and (u, v) is the optical flow vector, representing the displacement of the pixel between the two frames. Solving this formula using the least squares method yields the optical flow vector (u, v) corresponding to each pixel. Specifically, the direction of the pet's movement is obtained through the optical flow vector using the following formula: angle = arctan2(u,v), where angle is the direction of the pet's movement and (u,v) is the optical flow vector; Specifically, the pet's movement speed is obtained through the magnitude of the optical flow vector, as shown in the following formula: V = sqrt(u2 +v2); Specifically, to calculate the pet's motion acceleration, analysis needs to be performed on consecutive time frames. Assuming the optical flow vector corresponding to the frame at time t is (u_t, v_t), and the optical flow vector corresponding to the adjacent frame at time t+1 is (u_t+1, v_t+1), the motion acceleration can be obtained by calculating the difference between the optical flow vectors of the frames at time t and t+1. a = (u_t+1 - u_t, v_t+1 - v_t) Where a represents the pet's motion acceleration vector, and u_t+1 - u_t and v_t+1 - v_t represent the differences in the optical flow vector in the horizontal and vertical directions, respectively.

[0031] The obtained motion difference matrix, pet movement direction, pet movement speed, and pet movement acceleration features are combined to construct the motion feature vector corresponding to each effective segment. The motion feature vectors corresponding to all effective segments are then combined to form a motion feature vector sequence.

[0032] Specifically, the motion estimation scoring model is built based on deep learning algorithms, which mainly include structures such as convolutional neural networks and recurrent neural networks. It is trained using the backpropagation algorithm. During the testing phase, the motion feature vector of each input is forward propagated to obtain a real number output, which represents the motion state score corresponding to the effective segment.

[0033] Step S6: Weight the static score and the dynamic score to obtain the motion score corresponding to each valid segment, and select the valid segments with motion scores higher than the motion score threshold as the pet's highlight segments.

[0034] Specifically, the static score and dynamic score are weighted and summed using the motion score calculation formula to obtain the motion score for each valid segment. The motion score calculation formula is: FinalScore = βS Dynamic +(1-β)S static Where FinalScore is the athletic score, S Dynamic For dynamic scoring, S static β is the static score, and β is the coefficient of the motion score.

[0035] like Figure 2 As shown, this invention provides a pet highlight extraction system, which uses the pet highlight extraction method described above to extract pet highlights, including: The candidate segment acquisition module is used to acquire multiple pet action video segments through the camera and use them as candidate segments, and to decode the candidate segments into a frame sequence and obtain the absolute timestamp corresponding to each frame in the frame sequence. The candidate frame acquisition module is used to identify pets in each frame of the frame sequence using a pre-trained pet recognition model, acquire pet image frames, record the pet names and their corresponding recognition confidence scores in the pet image frames, and mark pet image frames that meet the recognition confidence threshold as candidate frames. The effective segment acquisition module is used to match adjacent candidate frames to form candidate segments, obtain the time interval between adjacent candidate frames based on the absolute timestamps corresponding to the adjacent candidate frames, and judge the length of candidate segments that meet the time interval threshold. The effective segment is obtained according to the judgment result. The static score acquisition module is used to obtain the confidence score corresponding to the effective segment by setting corresponding weights for each frame in the effective segment, to obtain the pet posture score corresponding to the effective segment by performing posture score on each frame in the effective segment and averaging the posture scores of each frame, and to obtain the static score corresponding to each effective segment by performing a weighted sum of the confidence score and the pet posture score. The dynamic scoring acquisition module is used to extract motion feature vectors from all valid segments, input the extracted motion feature vectors into the trained motion estimation scoring model, obtain the dynamic scores of all valid segments, and calculate the average of the dynamic scores of all valid segments to obtain the dynamic score of each valid segment. The highlight segment acquisition module is used to perform a weighted summation of static and dynamic scores to obtain the motion score corresponding to each valid segment, and to select valid segments with motion scores higher than the motion score threshold as the pet's highlight segments.

[0036] Specifically, the final output of the pet highlight extraction system includes the start time, end time, and motion score of each valid segment, and can filter out valid segments with motion scores higher than the motion score threshold as the pet's highlight segments.

[0037] like Figure 3 As shown, the camera has a built-in video recognition unit, SDK unit, and ffmpeg unit. The video recognition unit is used to detect pet action video clips; the SDK unit is used to send the detected pet action video clips to the cloud network; and the ffmpeg unit is used to decode the pet action video clips received from the cloud network into a frame sequence and obtain the absolute timestamp corresponding to each frame in the frame sequence.

[0038] like Figure 4 As shown, a trained pose evaluation model scores the pose of each frame in a valid segment. The pose evaluation model includes a data acquisition unit, a data processing unit, a feature extraction unit, a classifier unit, and an evaluation unit. The data acquisition unit is used to acquire pet image data; the data processing unit is used to preprocess the acquired pet image data, including pose annotation and data augmentation; the feature extraction unit is used to extract features from the preprocessed pet image data; the classifier unit is used to predict the pet's pose information based on the extracted features; and the evaluation unit is used to obtain a score for the pet's pose based on the predicted pet pose information and output the score result.

[0039] It should be noted that the above detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0040] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0041] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0042] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.

[0043] For ease of description, spatial relative terms such as "above," "on top of," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation beyond the orientation of the device as described in the figures. For example, if the device in the figures were inverted, a device described as "above" or "on top of" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways, such as rotated 90 degrees or in other orientations, and the spatial relative descriptions used herein will be interpreted accordingly.

[0044] In the detailed description above, reference has been made to the accompanying drawings, which form part of this document. In the drawings, similar symbols typically identify similar parts unless the context otherwise indicates otherwise. The illustrated embodiments described in the detailed specification, drawings, and claims are not intended to be limiting. Other embodiments may be used and other changes may be made without departing from the spirit or scope of the subject matter presented herein.

[0045] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for extracting exciting moments from pets, characterized in that, The method includes: Multiple video clips of pet actions are captured by a camera and used as candidate clips. The candidate clips are then decoded into a frame sequence, and the absolute timestamp of each frame in the frame sequence is obtained. The pre-trained pet recognition model is used to identify pets in each frame of the frame sequence, obtain pet image frames, record the pet names and their corresponding recognition confidence scores in the pet image frames, and mark the pet image frames that meet the recognition confidence threshold as candidate frames. Based on the absolute timestamps corresponding to adjacent candidate frames, the time interval between adjacent candidate frames is obtained, and the length of candidate segments that meet the time interval threshold is judged. Valid segments are obtained based on the judgment result. By setting corresponding weights for each frame in the effective segment, the confidence score corresponding to the effective segment is obtained based on the recognition confidence and weight corresponding to each frame in the effective segment. The pet posture score corresponding to the effective segment is obtained by performing posture scoring on each frame in the effective segment and averaging the posture scores of each frame. The static score corresponding to each effective segment is obtained by weighted summation of the confidence score and the pet posture score. By extracting motion feature vectors from all valid segments and inputting the extracted motion feature vectors into the trained motion estimation and scoring model, the dynamic scores of all valid segments are obtained, and the average of the dynamic scores of all valid segments is calculated to obtain the dynamic score of each valid segment. The specific steps for extracting motion feature vectors from all valid segments include: matching corresponding pixels between two adjacent frames in all valid segments; obtaining the image matrix corresponding to each of the two adjacent frames in each valid segment based on the corresponding pixels; calculating the difference between the image matrices corresponding to each of the two adjacent frames in each valid segment; using the difference between the image matrices corresponding to each of the two adjacent frames as the motion difference matrix of the pet between the two adjacent frames; calculating the optical flow vector based on the position of a pixel in the frame to be analyzed in the valid segment and the position of that pixel in the previous frame adjacent to the frame to be analyzed; and obtaining the pet's motion direction, pet's motion speed, and pet's motion acceleration features through the optical flow vector. The obtained motion difference matrix, pet movement direction, pet movement speed and pet movement acceleration features are combined to construct the motion feature vector corresponding to each effective segment, and the motion feature vectors corresponding to all effective segments are combined to form a motion feature vector sequence; Furthermore, before inputting the extracted motion feature vectors into the trained motion estimation scoring model, the extracted motion feature vectors are dimensionality reduced, the Pearson correlation coefficient between the dimensionality-reduced motion feature vectors is calculated, and the correlation of the dimensionality-reduced motion feature vectors is screened using the Pearson correlation coefficient. The static and dynamic scores are weighted and summed to obtain the motion score for each valid segment. Valid segments with motion scores higher than the motion score threshold are designated as the pet's highlight segments.

2. The method for extracting exciting pet clips according to claim 1, characterized in that, Based on the absolute timestamps corresponding to adjacent candidate frames, the time interval between adjacent candidate frames is obtained, and the length of candidate segments that meet the time interval threshold is judged. Valid segments are then obtained based on the judgment result, including: Subtract the absolute timestamps of adjacent candidate frames in the candidate segment to obtain the time interval between adjacent candidate frames in the candidate segment; When the time interval between adjacent candidate frames in a candidate segment is less than the time interval threshold, it is determined whether its length is within the length threshold range. Candidate segments that are within the length threshold and whose time interval between adjacent candidate frames is less than the time interval threshold will be considered as valid segments.

3. The method for extracting exciting pet clips according to claim 1, characterized in that, The confidence score for the valid segment is obtained using the confidence score calculation formula, which is as follows: , among which, S confidence The confidence score is given by n, where n is the number of frames in the valid segment, and w is the number of frames in the valid segment. t c represents the weight corresponding to the t-th frame in the effective segment. t Let be the recognition confidence score corresponding to the t-th frame in the valid segment.

4. The method for extracting exciting pet clips according to claim 1, characterized in that, The static score for each valid segment is obtained by weighted summation of the confidence score and the pet posture score using a static scoring formula. The static scoring formula is as follows: , among which, S static For static scoring, S confidence For the confidence score, S posture Rate the pet's posture, where α is the static rating coefficient.

5. The method for extracting exciting pet clips according to claim 1, characterized in that, The motion score is obtained by weighted summation of static and dynamic scores using the motion score calculation formula, which is as follows: Where FinalScore is the athletic score, S Dynamic For dynamic scoring, S static β is the static score, and β is the coefficient of the motion score.

6. A pet highlight extraction system, comprising extracting pet highlight clips using the pet highlight extraction method as described in any one of claims 1-5, characterized in that, The system includes: The candidate segment acquisition module is used to acquire multiple pet action video segments through the camera and use them as candidate segments, and to decode the candidate segments into a frame sequence and obtain the absolute timestamp corresponding to each frame in the frame sequence. The candidate frame acquisition module is used to identify pets in each frame of the frame sequence using a pre-trained pet recognition model, acquire pet image frames, record the pet names and their corresponding recognition confidence scores in the pet image frames, and mark pet image frames that meet the recognition confidence threshold as candidate frames. The effective segment acquisition module is used to obtain the time interval between adjacent candidate frames based on the absolute timestamps corresponding to adjacent candidate frames, and to judge the length of candidate segments that meet the time interval threshold, and to obtain the effective segments based on the judgment result. The static score acquisition module is used to set corresponding weights for each frame in the effective segment, obtain the confidence score corresponding to the effective segment based on the recognition confidence and weight corresponding to each frame in the effective segment, obtain the pet posture score corresponding to the effective segment by performing posture scoring on each frame in the effective segment and averaging the posture scores of each frame, and obtain the static score corresponding to each effective segment by performing a weighted sum of the confidence score and the pet posture score. The dynamic scoring acquisition module is used to extract motion feature vectors from all valid segments, input the extracted motion feature vectors into the trained motion estimation scoring model, obtain the dynamic scores of all valid segments, and calculate the average of the dynamic scores of all valid segments to obtain the dynamic score of each valid segment. The highlight segment acquisition module is used to perform a weighted summation of static and dynamic scores to obtain the motion score corresponding to each valid segment, and to select valid segments with motion scores higher than the motion score threshold as the pet's highlight segments.

7. The pet highlight extraction system according to claim 6, characterized in that, The camera has a built-in video recognition unit, SDK unit, and ffmpeg unit. The video recognition unit is used to detect pet action video clips; the SDK unit is used to send the detected pet action video clips to the cloud network; and the ffmpeg unit is used to decode the pet action video clips received from the cloud network into a frame sequence and obtain the absolute timestamp corresponding to each frame in the frame sequence.

8. The pet highlight extraction system according to claim 6, characterized in that, A trained pose evaluation model scores the pose of each frame in a valid segment. The model includes a data acquisition unit, a data processing unit, a feature extraction unit, a classifier unit, and an evaluation unit. The data acquisition unit collects pet image data; the data processing unit performs preprocessing on the collected pet image data, including pose annotation and data augmentation; the feature extraction unit extracts features from the preprocessed pet image data; the classifier unit predicts the pet's pose information based on the extracted features; and the evaluation unit obtains a pose score for the pet based on the predicted pose information and outputs the score result.

Images