Pet behavior prediction method, device, equipment and storage medium

Abstract

This disclosure provides a method, apparatus, device, and storage medium for predicting pet behavior. The method includes: acquiring pet type, image data, audio data, and text data describing pet behavior; extracting features from the image data, audio data, and text data respectively, and performing multimodal feature fusion to obtain the pet's fused features; based on the pet type and fused features, using a pre-constructed large-scale pet behavior prediction model to predict the pet's behavior, obtaining the pet behavior prediction result. This disclosure, in predicting pet behavior, not only introduces visual modal data but also simultaneously fuses auditory features from the audio modality with the contextual semantics of the owner's text description. It also eliminates behavioral ambiguity from single visual data through a cross-modal dynamic weighting mechanism and captures emotional cues in the sound signal and implicit states in the text description, thereby comprehensively constructing a multidimensional cognitive model of pet behavior and significantly improving prediction accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for predicting pet behavior. Background Technology

[0002] In recent years, the rapid development of artificial intelligence technology, especially the breakthroughs in large-scale modeling, has provided a significant opportunity for intelligent prediction of pet behavior. The powerful pattern recognition and complex association learning capabilities exhibited by large-scale models offer new hope for overcoming the bottlenecks in traditional pet behavior understanding. However, existing pet behavior prediction methods largely rely on single visual modality data sources such as images and videos, resulting in relatively poor accuracy. Therefore, improving the accuracy of pet behavior prediction has become a pressing technical problem for those skilled in the art. Summary of the Invention

[0003] In view of this, the present disclosure provides a method, apparatus, device and storage medium for predicting pet behavior, which can improve the accuracy of pet behavior prediction.

[0004] According to a first aspect of this disclosure, a method for predicting pet behavior is provided, comprising:

[0005] Acquire the pet's type, image data, audio data, and text data describing the pet's behavior;

[0006] Feature extraction is performed on the image data, audio data, and text data respectively, and multimodal feature fusion is performed to obtain the fused features of the pet;

[0007] Based on the pet's type and fusion features, a pre-built large-scale pet behavior prediction model is used to predict the pet's behavior, resulting in a prediction of the pet's behavior.

[0008] In one possible implementation, when performing feature extraction and multimodal feature fusion on the image data, audio data, and text data respectively to obtain the fused features of the pet, a cross-modal attention mechanism is used.

[0009] In one possible implementation, when performing feature extraction and multimodal feature fusion on the image data, audio data, and text data based on a cross-modal attention mechanism to obtain the fused features of the pet, the process includes:

[0010] Feature extraction is performed on the image data, the audio data, and the text data respectively to obtain image spatial feature vector, audio temporal feature vector, and text semantic feature vector;

[0011] Based on the image spatial feature vector, the audio temporal feature vector, and the text semantic feature vector, the initial attention score for each modality is calculated;

[0012] Based on the current time, historical behavioral characteristics within a set time period, and the text data, the initial attention scores for each modality are adjusted to obtain the final attention scores for each modality.

[0013] Based on the final attention scores of each modality, the image spatial feature vector, the audio temporal feature vector, and the text semantic feature vector are weighted and summed to obtain the fusion features of the pet.

[0014] In one possible implementation, when adjusting the initial attention score of each modality based on the current time, historical behavioral features within a set time period, and the text data to obtain the final attention score of each modality, the process includes:

[0015] Based on the current time, historical behavioral characteristics within a set time period, and the text data, calculate the dynamic adjustment weights for each modality;

[0016] Based on the dynamic adjustment weights of each modality, the initial attention scores of each modality are adjusted to obtain the final attention scores of each modality.

[0017] In one possible implementation, the construction of the large-scale model for predicting pet behavior includes:

[0018] Fine-tuning training data is constructed based on the pet's type and historical behavior data;

[0019] Using the fine-tuned training data, the pre-trained initial pet behavior prediction model is fine-tuned to obtain the pet behavior prediction model.

[0020] In one possible implementation, the large-scale pet behavior prediction model includes a controller, a multi-layer Transformer, and a multi-task output head, wherein the controller selectively activates the multi-layer Transformer based on the pet's type and the fused features.

[0021] In one possible implementation, when the controller selectively activates the multi-layer Transformer based on the pet's type and the fusion feature, it includes:

[0022] Calculate the information entropy value of the fused feature and the feature vector corresponding to the type;

[0023] Based on the information entropy and the feature vector, calculate the activation vector of the multi-layer Transformer;

[0024] Based on the activation vector, the multi-layer Transformer is selectively activated.

[0025] According to a second aspect of this disclosure, a device for predicting pet behavior is provided, comprising:

[0026] A multimodal data acquisition module is used to acquire pet type, image data, audio data, and text data describing the pet's behavior;

[0027] The multimodal data fusion module is used to extract features from the image data, the audio data, and the text data respectively, and perform multimodal feature fusion to obtain the fused features of the pet;

[0028] The behavior prediction module is used to predict the behavior of the pet based on the pet's type and the fusion features, using a pre-built large-scale pet behavior prediction model, to obtain the pet's behavior prediction result.

[0029] According to a third aspect of this disclosure, a pet behavior prediction device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to perform the method described in the first aspect of this disclosure.

[0030] According to a fourth aspect of this disclosure, a non-volatile computer-readable storage medium is provided that stores computer program instructions thereon, wherein the computer program instructions, when executed by a processor, implement the method described in the first aspect of this disclosure.

[0031] This disclosure provides a method, apparatus, device, and storage medium for predicting pet behavior. The method includes: acquiring pet type, image data, audio data, and text data describing pet behavior; extracting features from the image data, audio data, and text data respectively, and performing multimodal feature fusion to obtain the pet's fused features; and predicting the pet's behavior using a pre-constructed large-scale pet behavior prediction model based on the pet type and fused features to obtain the pet behavior prediction result. This disclosure, in predicting pet behavior, not only introduces visual modal data but also simultaneously fuses auditory features from the audio modality with the contextual semantics of the owner's text description. It also eliminates behavioral ambiguity from single visual data through a cross-modal dynamic weighting mechanism and captures emotional cues in the sound signal and implicit states in the text description, thereby comprehensively constructing a multidimensional cognitive model of pet behavior and significantly improving prediction accuracy.

[0032] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0033] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0034] Figure 1 A flowchart illustrating a method for predicting pet behavior according to an embodiment of the present disclosure is shown;

[0035] Figure 2 A schematic block diagram of a pet behavior prediction device according to an embodiment of the present disclosure is shown;

[0036] Figure 3 A schematic block diagram of a pet behavior prediction device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0037] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0038] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0039] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0040] <Method Implementation>

[0041] Figure 1 A flowchart illustrating a method for predicting pet behavior according to an embodiment of this disclosure is shown. Figure 1 As shown, the method includes steps S1100-S1300.

[0042] S1100 acquires pet type, image data, audio data, and text data describing pet behavior. The pet type refers to the specific breed of pet, such as dog or cat. Image data consists of images or videos of the pet's daily activities captured by a camera device. Audio data consists of pet sound data captured by an audio acquisition device, such as barking or purring. Text data consists of descriptions of the pet's behavior by the pet owner, such as "poor appetite today," or "doesn't like playing ball and eats less."

[0043] Specifically, the system implementing the method disclosed herein is equipped with a data acquisition interface. Users can upload image and audio data of pets collected within the same time period through this data acquisition interface. At the same time, users can input the pet type and a descriptive text of the pet's behavior within the time period in the data acquisition interface. After the data input is completed, clicking "confirm" will upload the multi-source data to the system. In this way, the system can obtain the pet type, image data, audio data, and text data describing the pet's behavior.

[0044] In one possible implementation, after acquiring image and audio data, a sliding window of a preset size (e.g., 5 seconds) is applied to calculate the inter-frame pixel difference in the image data and the inter-frame short-time energy change in the audio data. If the inter-frame pixel difference and / or inter-frame short-time energy change exceed a set threshold, an abnormal event (e.g., abnormal stillness or sudden barking) is flagged in the current frame. For the current frame flagged for the abnormal event, the behavior within the current frame is identified, and the start and end frames of the behavior are located. The image, audio, and text data corresponding to the start and end frames are then prioritized for predicting abnormal pet behavior, thereby improving the timeliness and accuracy of abnormal pet behavior prediction. If no abnormality is detected, pet behavior prediction is performed based on the complete collected data.

[0045] S1200 extracts features from image data, audio data, and text data respectively, and performs multimodal feature fusion to obtain the fused features of the pet.

[0046] In one possible implementation, when extracting features from image data, audio data, and text data respectively and performing multimodal feature fusion to obtain the fused features of the pet, a cross-modal attention mechanism is used.

[0047] In this feasible approach, when extracting features from image data, audio data, and text data based on a cross-modal attention mechanism and performing multimodal feature fusion to obtain the fused features of the pet, the following steps may be included:

[0048] First, feature extraction is performed on image data, audio data, and text data respectively to obtain image spatial feature vectors, audio temporal feature vectors, and text semantic feature vectors.

[0049] Specifically, when the image data is a specific image input by the user, the image is first preprocessed by Gaussian denoising, pixel normalization, and size adjustment. Then, a pre-trained ResNet-50 model (pre-trained on the ImageNet dataset) is used to extract spatial features from the preprocessed image to obtain the image spatial feature vector.

[0050] It should be noted that when the image data is video, the image feature vector extraction operation described above will first be performed on each image frame in the video to obtain the image feature vector corresponding to each frame. Then, the image spatial feature vectors of each frame will be combined in frame order to form the temporal spatial feature sequence of the video (i.e., the image spatial feature vectors corresponding to the video), which will be used as the input for subsequent modeling. The image spatial feature vectors of each frame are aggregated through inter-frame average pooling.

[0051] For audio data, preprocessing operations such as pre-emphasis, framing, and endpoint detection are performed first. Then, a pre-trained bidirectional LSTM model (pre-trained on the LibriSpeech dataset) is used to extract MFCC features from the preprocessed audio data, obtaining the corresponding audio temporal feature vector. The pre-emphasis coefficient can be set according to specific needs; for example, it can be set to 0.97. The frame length and frame shift used for framing can also be set according to specific needs; for example, the frame length can be set to 25ms and the frame shift to 10ms.

[0052] For text data, preprocessing operations such as word segmentation are performed first; then, the BERT model is used to extract semantic features from the preprocessed text, thereby obtaining the text semantic feature vector corresponding to the text data.

[0053] Second, based on image spatial feature vectors, audio temporal feature vectors, and text semantic feature vectors, the initial attention score for each modality is calculated. Specifically, based on image spatial feature vectors, audio temporal feature vectors, and text semantic feature vectors, the attention score for each modality can be calculated using existing cross-modal attention mechanisms. The attention score for each modality calculated using existing cross-modal attention mechanisms is the initial attention score for each modality.

[0054] Third, based on the current time, historical behavioral characteristics within a set time period, and text data, the initial attention scores for each modality are adjusted to obtain the final attention scores for each modality. This may specifically include the following steps:

[0055] First, based on the current time, historical behavioral characteristics within a set time period, and text data, the CAAF algorithm is used to calculate the dynamic adjustment weights of each modality. The current time is the time when pet behavior prediction is initiated. The historical behavioral characteristics within the set time period are the statistical characteristics of various preset abnormal behaviors of the pet within that time period. These statistical characteristics can include at least one of frequency features, duration features, and behavior distribution features. Frequency features can be the number of times each preset abnormal behavior occurs (e.g., "scratching at the door" occurs 22 times in 30 days), the average daily total number of abnormal behaviors, etc.; duration features can be the average duration of each abnormal behavior (e.g., average barking duration), the number of abnormal behaviors exceeding a threshold (e.g., >5 minutes), etc.; behavior distribution features can be the percentage of abnormal behaviors corresponding to each modality (e.g., audio abnormalities account for 80%, image abnormalities account for 20%, text abnormalities account for 0%), the time distribution percentage of abnormal behaviors (e.g., 70% during the day, 30% at night), etc. The set time period can be set according to specific needs; for example, it can be set to only 30 days. The text data is the text data input by the user in step S1100 to describe the pet's current behavior.

[0056] Specifically, when calculating the dynamic adjustment weights of each modality based on historical behavioral features and text data within the current time and a set time period, the historical behavioral features and text data within the current time and the set time period are first converted into a unified vector form. Then, based on the vectors corresponding to the historical behavioral features and text data within the current time and the set time period, the dynamic adjustment weights of each modality are calculated. The specific calculation formula is shown below:

[0057] w = sigmoid(MLP([t, h, s]))

[0058] In the formula, t is the vector corresponding to the current time, h is the vector corresponding to the historical behavioral features within the set time period, and s is the vector corresponding to the text data. The concatenated vector [t, h, s] of t, h, and s is input into a two-layer perceptron MLP (hidden layer dimension 256). The MLP will output an initial adjustment weight vector [score_image, score_audio, score_text] including three floating-point values, where score_image, score_audio, and score_text represent the initial adjustment weights of the image, audio, and text modalities, respectively. The initial adjustment weight vector is input into the sigmoid function for normalization, which yields the dynamic adjustment weight vector w, which includes the dynamic adjustment weights of the image, audio, and text modalities, w = [w_img, w_aud, w_txt], where w_img, w_aud, and w_txt represent the dynamic adjustment weights of the image, audio, and text modalities, respectively, and w_img + w_aud + w_txt = 1.

[0059] In one possible implementation, when converting the current time into a vector form, the current time can be divided into hour intervals (such as 0–6, 6–12, 12–18, 18–24) and then one-hot encoded to obtain the vector representation corresponding to the current time.

[0060] In an embodiment where historical behavioral features within a defined time period include frequency-based features, duration-based features, and behavioral distribution-based features, the process of converting historical behavioral features into vector form may include the following steps:

[0061] First, obtain frequency-based features: the number of occurrences of various preset abnormal behaviors within a set time period and the average daily total number of abnormal behaviors. Then, concatenate the values ​​of the number of occurrences of various preset abnormal behaviors and the average daily total number of abnormal behaviors within the set time period in sequence to obtain an initial frequency-based feature vector. Normalizing each data point in the initial frequency-based feature vector yields the final frequency-based feature vector. For example, preset behavior types include scratching at doors, barking, and destroying objects. Analyzing the pet's behavior profile over the past 30 days, the number of occurrences of each preset abnormal behavior over the past 30 days is as follows: scratching at doors 22 times, barking 15 times, and destroying objects 3 times, with an average daily total of 2.1 abnormal behaviors. The resulting initial frequency-based feature vector is [22, 15, 3, 2.1]. Normalizing the initial frequency-based feature vector yields the final frequency-based feature vector. Here, preset abnormal behaviors refer to all abnormal behaviors that the system pre-defines as possible for the pet.

[0062] Next, duration-related features are obtained: the average duration of each abnormal behavior occurring within a set time period and the percentage of times exceeding a threshold (e.g., 5 minutes). The average duration of each abnormal behavior and the percentage of times exceeding the threshold (e.g., 5 minutes) within the set time period are concatenated sequentially to obtain an initial duration-related feature vector. Normalizing the data in the initial duration-related feature vector yields the final duration-related feature vector. For example, if the average duration of each abnormal behavior occurring in the past 30 days is 3.2 minutes and the percentage of times exceeding the threshold (e.g., 5 minutes) is 25%, then the initial duration-related feature vector is [3.2, 0.25]. Normalizing the initial duration-related feature vector yields the final duration-related feature vector.

[0063] Next, the behavioral distribution features are obtained: the percentage of abnormal behavior categories in each modality and the percentage of time periods are set within a given time period; then, the percentages of abnormal behavior categories in each modality and the percentages of time periods are concatenated sequentially to obtain an initial behavioral distribution feature vector. Normalizing the data in the initial behavioral distribution feature vector yields the final behavioral distribution feature vector. For example, if the percentages of abnormal behavior in each modality over the past 30 days are 80% for audio, 20% for image, and 0% for text, and the percentages of time periods for abnormal behavior are 70% during the day and 30% at night, then the initial behavioral distribution feature vector would be [0.8, 0.2, 0, 0.7, 0.3]. Normalizing the initial behavioral distribution feature vector yields the final behavioral distribution feature vector. Here, the percentage of abnormal behavior in each modality represents the proportion of abnormal behavior in different modalities, and the percentage of time periods for abnormal behavior represents the proportion of abnormal behavior occurring in different time periods.

[0064] Finally, the final frequency-class feature vector, the final duration-class feature vector, and the final behavior distribution-class feature vector are concatenated, and normalized (to avoid different feature scales) and dimensionality reduction (e.g., PCA, MLP) are used to obtain a vector representation h corresponding to a fixed-dimensional historical behavior feature.

[0065] Secondly, based on the dynamically adjusted weights for each modality, the initial attention scores for each modality are adjusted to obtain the final attention scores for each modality. The calculation formulas for the final attention scores for each modality are as follows:

[0066] w_img'=A_img×w_img

[0067] w_aud'=A_aud×w_aud

[0068] w_txt'= A_txt×w_txt

[0069] In the formula, A_img, A_aud, and A_txt represent the initial attention scores for the image, audio, and text modalities, respectively; w_img, w_aud, and w_txt represent the dynamically adjusted weights for the image, audio, and text modalities, respectively; and w_img', w_aud', and w_txt' represent the final attention scores for the image, audio, and text modalities, respectively.

[0070] Fourth, based on the final attention scores of each modality, the image spatial feature vector, audio temporal feature vector, and text semantic feature vector are weighted and summed to obtain the pet's fused features. The formula for calculating the fused features is shown below:

[0071] F = w_img'× F_img + w_aud'×F_aud + w_txt'×F_txt

[0072] In the formula, F represents the fusion feature of the pet, F_img is the image spatial feature vector, F_aud is the audio temporal feature vector, F_txt is the text semantic feature vector, and w_img', w_aud', and w_txt' represent the final attention scores for the image, audio, and text modalities, respectively.

[0073] In this embodiment, a CAAF algorithm module is introduced into the existing cross-modal attention mechanism. This module can calculate the dynamically adjusted weights of each modality based on contextual information such as the current time, historical behavioral features within a set time period, and text data. This dynamically adjusted weights then dynamically adjust the initial attention scores of each modality calculated by the original cross-modal attention mechanism, improving the relevance of the final fused features by approximately 10%. In other words, the CAAF algorithm module can combine the three contextual semantics of the current time, historical behavioral features within a set time period, and text data to dynamically adjust the weights corresponding to the image / audio / text modalities, thereby improving the accuracy and personalized response capability of the final model in judging abnormal behavior.

[0074] In one possible implementation, the CAAF algorithm module adjusts the modal weights according to the following principles:

[0075] 1. Text-Driven Principle: This principle states that when user text data explicitly describes a specific behavior, the weight of the text and the modality associated with that behavior is increased. For example, if the text mentions a sound-related behavior (such as "barking incessantly"), the text modality weight increases to 0.45, the audio modality weight increases to 0.40, and the image modality weight decreases to 0.15; if the text mentions a visual behavior (such as "biting its tail"), the text modality weight increases to 0.45, the image modality weight increases to 0.40, and the audio modality weight decreases to 0.15.

[0076] 2. Temporal Context-Driven Principle: When the current time falls within a period of high historical incidence of abnormal behavior, the weight of non-text modalities associated with the high incidence of abnormal behavior is increased, while the weight of text modalities is decreased. For example, if nighttime hours (such as 8:00 PM to 6:00 AM) are a high-incidence period for dog barking, then the weight of the audio modality is increased to 0.55, the weight of the image modality is increased to 0.30, and the weight of the text modality is decreased to 0.15.

[0077] 3. Historical Behavior Dependence-Driven Principle: When a certain modality anomaly occurs frequently in historical behavior, the weight of that modality is increased. For example, if audio anomalies (such as door-scratching sounds) account for ≥75% within 30 days, the weight of the audio modality is increased to 0.60, the weight of the image modality is reduced to 0.25, and the weight of the text modality remains at 0.15.

[0078] In order for the CAAF algorithm module to adjust the modality weights according to the above principles, training data needs to be constructed according to the above principles, and the CAAF algorithm module needs to be trained using the constructed training data to obtain a CAAF algorithm module with the above functions.

[0079] In one possible implementation, the training data for the CAAF module can include the following steps: First, collect pet behavior samples in multiple modalities, including video, audio, and text, and extract the corresponding current time features, historical behavior feature vectors, and text feature vectors as described above; then, automatically generate ideal dynamic adjustment weights for each modality according to preset rules (such as text-driven principles, time context-driven principles, and historical behavior dependency-driven principles) to form training samples. The collected sample data needs to cover different seasons, time periods, pet personality differences, and text description differences.

[0080] After constructing the training data for the CAAF module, a multi-layer neural network can be trained using supervised learning methods based on this data. This network takes a vector [t, h, s] (a concatenation of current time features, historical behavior feature vectors, and text feature vectors) as input and automatically adjusts the weights dynamically for each data modality. The CAAF module includes an input layer, intermediate layers, and an output layer. The functions of each layer are as follows:

[0081] Input layer: Current time features + historical behavior feature vectors + text feature vectors

[0082] Intermediate layer: Multilayer Perceptron (MLP) or Transformer Encoder

[0083] Output layer: 3D vector, processed by softmax, representing the dynamically adjusted weights for the three modalities of image / audio / text.

[0084] The loss function, optimization objective, and deployment method for training the CAAF module are as follows:

[0085] Loss function: Mean squared error (MSE) or cross-entropy is used to fit the ideal weights.

[0086] Optimization goal: To make the dynamic weights output by the model as close as possible to the label weights.

[0087] Deployment: The trained CAAF model outputs three weights during inference, which are multiplied by the initial score of the original cross-modal attention to obtain the final attention score.

[0088] After calculating the pet's fusion features, S1300 can be executed. Based on the pet's type and fusion features, a pre-built large-scale pet behavior prediction model is used to predict the pet's behavior and obtain the pet's behavior prediction results.

[0089] First, it should be noted that the Pet-Specific Large Model (PSLM) adopts a Dynamic Layer Selection Transformer (DLST) architecture, which includes a controller, multiple Transformers, and a multi-task output head.

[0090] In one possible implementation, the controller in the large pet behavior prediction model selectively activates a multi-layer Transformer based on the pet's type and fused features. The specific activation steps are as follows:

[0091] First, calculate the information entropy value of the fused features and the feature vector corresponding to the pet type. Specifically, convert the pet type into the corresponding feature vector, for example, [1, 0] represents a dog, [0, 1] represents a cat, and calculate the information entropy value of the pet fused features using the following formula:

[0092]

[0093] In the formula, H represents the fusion feature F H Information entropy value, p i It is the dimension value of the i-th normalized dimension in the fused feature vector F, where n is the dimension value of the fused feature vector F. H The total number of dimensions included.

[0094] Second, based on the information entropy value of the fused features and the feature vector corresponding to the pet type, the activation vector of the multi-layer Transformer is calculated. The specific calculation formula is shown below:

[0095] L = floor(MLP([H, c, E]) *p) + q

[0096] In the formula, L is the activation vector of the multi-layer Transformer, H is the information entropy value of the fused features, c is the feature vector corresponding to the pet type, E represents the information entropy of the type distribution of historical abnormal behavior within a set time period (the value of E is pre-calculated and stored in the system and continuously updated), q is the number of layers in the Transformer that are fixedly activated, p is the number of layers in the Transformer that can be selectively activated, and q + p = the total number of layers in the Transformer. The total number of layers in the Transformer, as well as the number of fixed and selectively activated layers, can be set according to the specific scenario.

[0097] In one possible implementation, calculating the entropy of the type distribution information of historical abnormal behaviors within a set time period can include the following steps: First, analyze the pet's specific behavior profile to determine the types of abnormal behaviors that occurred during the set time period, and calculate the percentage of each type of abnormal behavior (e.g., biting (40%), barking (30%), attacking cats (30%)). The vector composed of the percentages of each type of abnormal behavior is used as the type distribution information of historical abnormal behaviors within the set time period. Then, calculate the entropy value of the type distribution information of historical abnormal behaviors within the set time period. This calculated entropy value is the type distribution information entropy E of historical abnormal behaviors within the set time period. After calculating E, the E value is stored in the system, so that the pre-calculated E value can be automatically retrieved when calculating L. The set time period can be set according to specific needs; for example, it can be set to the most recent 30 days. It should be noted that the pet's specific behavior profile is constantly updated. After the system detects an update to the pet's specific behavior profile, it will recalculate the E value in the above manner to ensure continuous updating of the E value.

[0098] In one possible implementation, the formula for calculating the entropy E of the type distribution information of historical anomalous behavior is as follows:

[0099]

[0100] In the formula, E represents the entropy of the distribution information of historical anomalous behaviors. p k Information F is the distribution information of historical anomalous behavior types. E The normalized percentage of the k-th abnormal behavior, where m is the distribution information of the types of historical abnormal behaviors in F. E The total number of abnormal behaviors included.

[0101] It should be noted that when calculating the activation vector L, the higher the input entropy (e.g., a significant increase in H and / or E), the more Transformer layers L can be activated. To achieve this, when constructing the above-mentioned calculation model for the activation vector L, it is necessary to construct corresponding training data and use the constructed training data to perform supervised training on the above-mentioned calculation model, thereby obtaining a calculation model for the activation vector L with the aforementioned functions.

[0102] In one possible implementation, the training data for constructing the computational model of activation vector L may include the following steps: First, obtain multiple sample data, each of which includes the information entropy value of the fused features, the feature vector corresponding to the pet type, and the information entropy of the type distribution of historical abnormal behaviors; then, for each sample data, label the optimal activation vector L by manual annotation, thereby obtaining the training data for training the computational model of activation vector L.

[0103] After constructing the training data for the computational model of the activation vector L, the computational model of the activation vector L is trained using the training data to obtain the final computational model of the activation vector L.

[0104] Furthermore, the more complex the distribution of historical abnormal behaviors within a given time period (including at least two types of abnormal behaviors), the larger the corresponding E value; conversely, the simpler the distribution of historical abnormal behaviors within a given time period (including only one type of abnormal behavior), the smaller the corresponding E value. For example, if the distribution of a pet's historical abnormal behaviors within a given time period is: biting (40%), barking (30%), and attacking cats (30%), then the information entropy calculated for this complex distribution of historical abnormal behaviors is 1.57. On the other hand, if the distribution of a pet's historical abnormal behaviors within a given time period is "biting" (100%), then the information entropy calculated for this simple distribution of historical abnormal behaviors is 0.

[0105] In a preferred embodiment, the total number of layers in the Transformer can be set to 12, the number of layers q that are fixedly activated can be set to 4, and the number of layers p that are selected for activation can be set to 8. The formula for calculating the activation vector L of the multi-layer Transformer is as follows:

[0106] L = floor(MLP([H, c, E]) *8) + 4

[0107] Third, based on the activation vector, selective activation is performed on the multi-layer Transformer.

[0108] In this implementation, the controller can determine the number of Transformer activation layers based on the information entropy value of the input fused features, the pet species (distinguishing between dogs and cats through embedding vectors), and the information entropy of the distribution of pet historical abnormal behaviors over a pre-calculated time period. The smaller the input entropy value, the fewer the activation layers (e.g., low entropy input, 4 activation layers); the larger the input entropy value, the more activation layers (e.g., high entropy input, 12 activation layers). This design can reduce computation by approximately 35% while reducing inference latency by 25%.

[0109] When building a large-scale pet behavior prediction model, first load the basic model with the aforementioned Transformer architecture featuring dynamic layer selection; then, pre-train the basic model using public datasets (such as the Kaggle pet behavior video library, containing 100,000 labeled samples) and user-uploaded labeled data (covering multi-source data of various pets such as dogs and cats, as well as behavior prediction results for multi-source data labels) to obtain the initial large-scale pet behavior prediction model.

[0110] In one possible implementation, to improve the accuracy of the large pet behavior model in predicting the behavior of a specific pet, a personalized adaptive mechanism for the large model is introduced. That is, after obtaining the initial large pet behavior prediction model, the following steps are also included:

[0111] First, fine-tuning training data is constructed based on the pet's type and historical behavioral data. Specifically, a unique behavioral profile is created for each pet and stored in the cloud. This profile records the pet's type and multiple historical behavioral data points. Each historical behavioral data point includes collected multimodal data and behavioral predictions based on that data. The most recent 100 or so historical behavioral data points, combined with the pet's type, can be extracted from the pet's unique behavioral profile and used as fine-tuning training data for that pet.

[0112] Second, fine-tuning training data is used to fine-tune the pre-trained initial large-scale pet behavior prediction model to obtain the large-scale pet behavior prediction model.

[0113] In this feasible approach, by using pet-specific behavior profiles and online learning algorithms (i.e., introducing pet-specific behavior profiles and building fine-tuning training data based on pet-specific behavior profiles to fine-tune the large model), the large pet behavior prediction model can dynamically adapt to the unique behavior patterns of each pet, reducing generalization prediction errors and improving the F1 score of personalized predictions by about 20%.

[0114] In this feasible approach, a self-supervised pre-training strategy (i.e., modeling based on masked data and training for 100 epochs) is first used to train an initial large-scale pet behavior prediction model. Then, the initial large-scale pet behavior prediction model is supervisedly fine-tuned using fine-tuning training data (using cross-entropy loss, learning rate 1e-5, fine-tuning for 20 epochs) to obtain the final large-scale pet behavior prediction model that matches a specific pet, thereby improving the model's generalization ability to sparse data.

[0115] Furthermore, for each pet behavior prediction, the multi-source data and corresponding behavior prediction results at the time of prediction will be collected and stored as historical behavior data in the pet's exclusive behavior file. After collecting a set amount of new historical behavior data, the pet behavior prediction model will be fine-tuned and updated in a timely manner based on the set amount of new historical behavior data.

[0116] After training the large-scale prediction model, the pet's type and fused features are input into the model. The model then automatically predicts the pet's behavior, providing the prediction results. These predictions include information on the pet's next behavior type (including various possible behavior types and their corresponding confidence scores), emotional state information (including various possible emotions and their corresponding confidence scores), and explanatory descriptions of the behavior type and emotional state.

[0117] After the large model outputs pet behavior prediction results, the behavior type and emotional state information will be displayed in a visual format. For example, bar charts will be used to show various possible behavior types and their corresponding confidence scores, and bar charts will also be used to show various possible emotional states and their corresponding confidence scores. Simultaneously, explanatory descriptions of the pet's behavior type and emotional state will be presented in text form. For example, "The pet is currently predicted to 'pac repeatedly.' Considering the decrease in daily interaction time and the increase in nighttime activity frequency over the past week, it is inferred that the pet may have mild anxiety, with a confidence level of 0.78." Through visual output and explanatory descriptions, users can intuitively understand the pet behavior prediction results.

[0118] Furthermore, after displaying the behavioral prediction results, the system also includes receiving user feedback (provided via a system interface with options for "accurate," "inaccurate," and optional text descriptions). Feedback data is stored in the cloud and used for continuous model optimization. For example, the model is updated monthly in batches using the latest collected historical behavioral data and user feedback, prioritizing high-frequency feedback samples. This feedback mechanism further optimizes model performance, and user satisfaction surveys show that over 90% of users find the prediction results useful.

[0119] The following is a further explanation of the disclosed solution using a specific example. Specifically, it includes the following steps:

[0120] First, users can upload a 30-second pet video (MP4 format, 1080p, including barking and walking animations), a photo of the pet sleeping (JPEG format, 2000x1500 pixels), and a text description ("Did not like playing ball today, and ate less") via the mobile app (version 2.0, supports iOS 14+ and Android 10+).

[0121] Second, the system verifies the data format and quality (video frame rate ≥ 24fps, audio signal-to-noise ratio ≥ 20dB, text length ≤ 200 characters) to ensure the data is suitable for subsequent processing.

[0122] Third, feature extraction and fusion:

[0123] Spatial features of each video frame are extracted using ResNet-50 (pre-trained on ImageNet) to generate a 2048-dimensional feature vector, which is then aggregated between frames using average pooling.

[0124] The audio was processed using a bidirectional LSTM (trained on the LibriSpeech dataset) to extract 13-dimensional MFCC features (frame length 25ms, frame shift 10ms) and generate a 512-dimensional temporal feature vector.

[0125] The text is segmented and vectorized using BERT (a pre-trained Chinese model), generating a 768-dimensional semantic feature vector.

[0126] A 1024-dimensional fused feature vector is generated by fusing three features through a cross-modal attention mechanism (8-head attention, trained on a multimodal sentiment dataset).

[0127] Third, the 1024-dimensional fused feature vector can be input into the pre-built pet behavior prediction model (PSLM, based on a 12-layer Transformer with 120 million parameters) to automatically predict pet behavior. The prediction results are "fatigue" (confidence 0.88), "anxiety" (0.09), and "hunger" (0.03).

[0128] The APP interface displays the prediction results: the bar chart shows the probability distribution of "fatigue (0.85)", "anxiety (0.09)" and "hunger (0.03)", and the text description is "Based on the recent low barking frequency and the owner's description of 'not liking to play ball and eating less', it is predicted that the dog may be in a state of fatigue. It is recommended to reduce stimulating activities and increase rest time".

[0129] Users confirm the prediction result by clicking the "Accurate" button in the app and add a note such as "It was indeed very quiet today." The feedback data is then uploaded to the cloud database. This statement indicates that the user acknowledges and supports the observation that "the dog is tired," representing a cross-validation between the model's output and real-life conditions.

[0130] This disclosure provides a method for predicting pet behavior, including: acquiring pet type, image data, audio data, and text data describing pet behavior; extracting features from the image data, audio data, and text data respectively, and performing multimodal feature fusion to obtain the pet's fused features; based on the pet type and fused features, using a pre-constructed large-scale pet behavior prediction model to predict the pet's behavior, obtaining the pet behavior prediction result. In predicting pet behavior, this disclosure not only introduces visual modal data but also simultaneously fuses auditory features from the audio modality with the contextual semantics of the owner's text description. It also eliminates behavioral ambiguity from single visual data (e.g., distinguishing between excited tail wagging and anxious tail wagging) through a cross-modal dynamic weighting mechanism, and captures emotional cues in sound signals (e.g., whimpering indicating pain) and implicit states in text descriptions (e.g., "loss of appetite" suggesting illness), thereby comprehensively constructing a multidimensional cognitive model of pet behavior and significantly improving prediction accuracy. Experiments show that the behavior classification accuracy can reach over 90%, an improvement of approximately 15% compared to single-modal methods.

[0131] <Device Embodiment>

[0132] Figure 2 A schematic block diagram of a pet behavior prediction device according to an embodiment of the present disclosure is shown. Figure 2 As shown, the device 100 includes:

[0133] The multimodal data acquisition module 110 is used to acquire the pet's type, image data, audio data, and text data describing the pet's behavior;

[0134] The multimodal data fusion module 120 is used to extract features from the image data, the audio data, and the text data respectively, and perform multimodal feature fusion to obtain the fused features of the pet;

[0135] The behavior prediction module 130 is used to predict the behavior of the pet based on the pet's type and the fusion features, using a pre-built large-scale pet behavior prediction model, to obtain the pet's behavior prediction result.

[0136] <Equipment Example>

[0137] Figure 3 A schematic block diagram of a pet behavior prediction device according to an embodiment of the present disclosure is shown. Figure 3As shown, the pet behavior prediction device 200 includes a processor 210 and a memory 220 for storing executable instructions of the processor 210. The processor 210 is configured to implement any of the pet behavior prediction methods described above when executing the executable instructions.

[0138] It should be noted here that the number of processors 210 can be one or more. Furthermore, the pet behavior prediction device 200 of this embodiment may also include an input device 230 and an output device 240. The processors 210, memory 220, input device 230, and output device 240 can be connected via a bus or other means, which are not specifically limited here.

[0139] The memory 220, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and various modules, such as the program or module corresponding to the pet behavior prediction method of this disclosure embodiment. The processor 210 executes various functional applications and data processing of the pet behavior prediction device 200 by running the software program or module stored in the memory 220.

[0140] Input device 230 can be used to receive input digital numbers or signals. These signals may include key signals related to user settings and function control of the device / terminal / server. Output device 240 may include a display device such as a screen.

[0141] <Storage Medium Examples>

[0142] According to a fourth aspect of this disclosure, a non-volatile computer-readable storage medium is also provided, having stored thereon computer program instructions that, when executed by processor 210, implement the method for predicting pet behavior as described above.

[0143] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for predicting pet behavior, characterized in that, include: Acquire the pet's type, image data, audio data, and text data describing the pet's behavior; Based on a cross-modal attention mechanism, feature extraction is performed on the image data, audio data, and text data respectively, and multimodal feature fusion is performed to obtain the fused features of the pet. Specifically, feature extraction is performed on the image data, audio data, and text data respectively to obtain image spatial feature vector, audio temporal feature vector, and text semantic feature vector. Based on the image spatial feature vector, the audio temporal feature vector, and the text semantic feature vector, the initial attention score for each modality is calculated; Based on the current time, historical behavior characteristics within a set time period, and the text data, the dynamic adjustment weights of each modality are calculated. The current time is the time when pet behavior prediction is started, and the historical behavior characteristics within the set time period are the statistical characteristics of various preset abnormal behaviors of the pet within the set time period. The statistical characteristics of abnormal behaviors include at least one of frequency features, duration features, and behavior distribution features. Based on the dynamic adjustment weights of each modality, the initial attention scores of each modality are adjusted to obtain the final attention scores of each modality; Based on the final attention scores of each modality, the image spatial feature vector, the audio temporal feature vector, and the text semantic feature vector are weighted and summed to obtain the fusion features of the pet; Based on the pet's type and fusion features, a pre-built large-scale pet behavior prediction model is used to predict the pet's behavior, resulting in a prediction of the pet's behavior. The large-scale pet behavior prediction model includes a controller, a multi-layer Transformer, and a multi-task output head. The controller selectively activates the multi-layer Transformer based on the pet's type and the fusion features.

2. The method according to claim 1, characterized in that, The construction of the large-scale model for predicting pet behavior includes: Fine-tuning training data is constructed based on the pet's type and historical behavior data; Using the fine-tuned training data, the pre-trained initial pet behavior prediction model is fine-tuned to obtain the pet behavior prediction model.

3. The method according to claim 1, characterized in that, When the controller selectively activates the multi-layer Transformer based on the pet's type and the fusion feature, it includes: Calculate the information entropy value of the fused feature and the feature vector corresponding to the type; Based on the information entropy value and the feature vector, calculate the activation vector of the multi-layer Transformer; Based on the activation vector, the multi-layer Transformer is selectively activated.

4. A device for predicting pet behavior, characterized in that, include: A multimodal data acquisition module is used to acquire pet type, image data, audio data, and text data describing the pet's behavior; The multimodal data fusion module is used to extract features from the image data, audio data, and text data based on a cross-modal attention mechanism and perform multimodal feature fusion to obtain the fused features of the pet. Specifically, it includes: extracting features from the image data, audio data, and text data to obtain image spatial feature vectors, audio temporal feature vectors, and text semantic feature vectors. Based on the image spatial feature vector, the audio temporal feature vector, and the text semantic feature vector, the initial attention score for each modality is calculated; Based on the current time, historical behavior characteristics within a set time period, and the text data, the dynamic adjustment weights of each modality are calculated. The current time is the time when pet behavior prediction is started, and the historical behavior characteristics within the set time period are the statistical characteristics of various preset abnormal behaviors of the pet within the set time period. The statistical characteristics of abnormal behaviors include at least one of frequency features, duration features, and behavior distribution features. Based on the dynamic adjustment weights of each modality, the initial attention scores of each modality are adjusted to obtain the final attention scores of each modality; Based on the final attention scores of each modality, the image spatial feature vector, the audio temporal feature vector, and the text semantic feature vector are weighted and summed to obtain the fusion features of the pet; The behavior prediction module is used to predict the behavior of the pet based on the pet's type and the fusion features, using a pre-built large-scale pet behavior prediction model, to obtain the pet's behavior prediction result; The large-scale pet behavior prediction model includes a controller, a multi-layer Transformer, and a multi-task output head. The controller selectively activates the multi-layer Transformer based on the pet's type and the fusion features.

5. A device for predicting pet behavior, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 3 when executing the executable instructions.

6. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 3.

Images