Pet behavior prediction method and device, equipment and storage medium

Abstract

The invention provides a pet behavior prediction method and apparatus, a device and a storage medium. The method comprises the steps of obtaining a pet type, image data, audio data and text data describing pet behaviors; performing feature extraction and multi-modal feature fusion on the image data, the audio data and the text data to obtain fusion features of the pet; and based on the type of the pet and the fusion feature, predicting the behavior of the pet by adopting a pre-constructed pet behavior prediction large model to obtain a behavior prediction result of the pet. When pet behaviors are predicted, visual modal data are introduced, auditory features of an audio modal and situational semantics described by an owner text are synchronously fused, behavior ambiguity of single visual data is eliminated through a cross-modal dynamic weighting mechanism, emotion clues in sound signals and recessive states described by the text are captured, and the pet behaviors are predicted. Therefore, a multi-dimensional cognitive model of pet behaviors is comprehensively constructed, and the prediction accuracy is remarkably improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a pet behavior prediction method, apparatus, device, and storage medium. Background Art

[0002] The rapid development of artificial intelligence (AI), particularly breakthroughs in large-scale modeling, has provided a significant opportunity for intelligent pet behavior prediction in recent years. The powerful pattern recognition and complex association learning capabilities demonstrated by these large models offer new hope for overcoming the bottlenecks in traditional pet behavior understanding. However, existing pet behavior prediction methods often rely on a single visual modality data source, such as images or videos, resulting in poor accuracy. Therefore, improving the accuracy of pet behavior prediction has become a pressing technical challenge for those skilled in the art. Summary of the Invention

[0003] In view of this, the present disclosure proposes a pet behavior prediction method, apparatus, device and storage medium, which can improve the accuracy of pet behavior prediction.

[0004] According to a first aspect of the present disclosure, a method for predicting pet behavior is provided, comprising: Obtaining a pet's type, image data, audio data, and text data describing the pet's behavior; Performing feature extraction on the image data, the audio data, and the text data respectively and performing multimodal feature fusion to obtain fused features of the pet; Based on the type of the pet and the fusion features, a pre-built pet behavior prediction model is used to predict the behavior of the pet to obtain a behavior prediction result of the pet.

[0005] In one possible implementation, when feature extraction is performed on the image data, the audio data, and the text data respectively and multimodal feature fusion is performed to obtain the fused features of the pet, it is implemented based on a cross-modal attention mechanism.

[0006] In one possible implementation, when extracting features from the image data, the audio data, and the text data based on a cross-modal attention mechanism and performing multimodal feature fusion to obtain the fused features of the pet, the method includes: Performing feature extraction on the image data, the audio data, and the text data respectively to obtain an image spatial feature vector, an audio temporal feature vector, and a text semantic feature vector; Calculating an initial attention score for each modality based on the image spatial feature vector, the audio temporal feature vector, and the text semantic feature vector; Adjusting the initial attention score of each modality based on the current time, historical behavior characteristics within a set time period, and the text data to obtain a final attention score for each modality; Based on the final attention score of each modality, the image space feature vector, the audio time series feature vector, and the text semantic feature vector are weightedly summed to obtain the fusion feature of the pet.

[0007] In one possible implementation, adjusting the initial attention score of each modality based on the current time, historical behavior characteristics within a set time period, and the text data to obtain the final attention score of each modality includes: Calculate the dynamic adjustment weight of each modality based on the current time, historical behavior characteristics within a set time period, and the text data; Based on the dynamic adjustment weight of each modality, the initial attention score of each modality is adjusted to obtain the final attention score of each modality.

[0008] In one possible implementation, when constructing the pet behavior prediction model, the following steps are included: Constructing fine-tuning training data based on the type of the pet and historical behavior data; The fine-tuning training data is used to perform fine-tuning training on the pre-trained initial pet behavior prediction large model to obtain the pet behavior prediction large model.

[0009] In one possible implementation, the pet behavior prediction model includes a controller, a multi-layer Transformer, and a multi-task output head, wherein the controller selectively activates the multi-layer Transformer based on the type of the pet and the fusion features.

[0010] In a possible implementation, when the controller selectively activates the multi-layer Transformer based on the type of the pet and the fusion feature, the controller includes: Calculating the information entropy value of the fusion feature and the feature vector corresponding to the type; Calculating an activation vector of the multi-layer Transformer based on the information entropy and the feature vector; Based on the activation vector, the multi-layer Transformer is selectively activated.

[0011] According to a second aspect of the present disclosure, there is provided a device for predicting pet behavior, comprising: A multimodal data acquisition module for acquiring the pet's type, image data, audio data, and text data describing the pet's behavior; A multimodal data fusion module, configured to extract features from the image data, the audio data, and the text data, and perform multimodal feature fusion to obtain fused features of the pet; The behavior prediction module is used to predict the behavior of the pet based on the type of the pet and the fusion features, using a pre-built pet behavior prediction model to obtain the pet behavior prediction result.

[0012] According to a third aspect of the present disclosure, a device for predicting pet behavior is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the method described in the first aspect of the present disclosure.

[0013] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the method described in the first aspect of the present disclosure.

[0014] The present disclosure provides a pet behavior prediction method, apparatus, device, and storage medium. The method comprises: obtaining the pet's type, image data, audio data, and text data describing the pet's behavior; performing feature extraction on the image data, audio data, and text data, respectively, and performing multimodal feature fusion to obtain the pet's fused features; and using a pre-built pet behavior prediction model to predict the pet's behavior based on the pet's type and fused features, obtaining a pet behavior prediction result. When predicting pet behavior, the present disclosure not only introduces visual modality data, but also synchronously fuses the auditory features of the audio modality with the contextual semantics of the owner's text description. A cross-modal dynamic weighting mechanism is used to eliminate behavioral ambiguity in single visual data, and to capture emotional cues in sound signals and the implicit state of text descriptions, thereby comprehensively constructing a multidimensional cognitive model of pet behavior and significantly improving prediction accuracy.

[0015] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0017] Figure 1 A flowchart showing a method for predicting pet behavior according to an embodiment of the present disclosure; Figure 2 A schematic block diagram illustrating a device for predicting pet behavior according to an embodiment of the present disclosure is shown; Figure 3A schematic block diagram illustrating a device for predicting pet behavior according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0018] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0019] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0020] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0021] <Method Example> Figure 1 Flowchart showing a method for predicting pet behavior according to an embodiment of the present disclosure. Figure 1 As shown, the method includes steps S1100-S1300.

[0022] S1100: Obtain the pet type, image data, audio data, and text data describing the pet's behavior. The pet type refers to the specific type of pet, such as a dog or cat. Image data refers to images or videos of the pet's daily activities captured by a camera. Audio data refers to sound data of the pet, such as barking and snoring, captured by an audio capture device. Text data refers to the pet owner's description of the pet's behavior, such as "poor appetite today," "not interested in playing ball, reduced food intake," etc.

[0023] Specifically, a data collection interface is provided in the system for executing the method disclosed herein, through which the user can upload the image data and audio data of the pet collected within the same time period, and at the same time enter the pet type and the user's description text of the pet's behavior within the time period in the data collection interface. After the data input is completed, click confirm to upload the input multi-source data to the system. In this way, the system can obtain the pet's type, image data, audio data and text data describing the pet's behavior.

[0024] In one possible implementation, after acquiring the image data and audio data, a sliding window of a preset size (e.g., a window size of 5 seconds) is applied to calculate the inter-frame pixel difference in the image data and the inter-frame short-time energy change in the audio data. If the inter-frame pixel difference and / or inter-frame short-time energy change exceeds a set threshold, the current frame is marked as having an abnormal event (e.g., abnormal stillness or sudden barking). For the current frame marked as an abnormal event, the behavior in the current frame is identified, and the start and end frames of the behavior are found. The image data, audio data, and text data corresponding to the start and end frames are used as the focus for predicting abnormal pet behavior, thereby improving the timeliness and accuracy of abnormal pet behavior prediction. If no abnormality is detected, pet behavior prediction is performed based on the complete collected data.

[0025] S1200: Extract features from the image data, audio data, and text data respectively and perform multimodal feature fusion to obtain fused features of the pet.

[0026] In one possible implementation, when extracting features from image data, audio data, and text data respectively and performing multimodal feature fusion to obtain the fused features of the pet, it is implemented based on a cross-modal attention mechanism.

[0027] In this implementation, when extracting features from image data, audio data, and text data based on a cross-modal attention mechanism and performing multimodal feature fusion to obtain a fused feature of a pet, the following steps may be included: First, feature extraction is performed on image data, audio data, and text data respectively to obtain image spatial feature vectors, audio temporal feature vectors, and text semantic feature vectors.

[0028] Specifically, when the image data is a specific image input by the user, the image is first preprocessed by Gaussian denoising, pixel normalization, size adjustment and other operations; then, the pre-trained ResNet-50 model (pre-trained on the ImageNet dataset) is used to extract spatial features of the pre-processed image to obtain the image spatial feature vector of the image.

[0029] It should be noted that when the image data is a video, the aforementioned image feature vector extraction operation is first performed on each frame in the video to obtain the image feature vector corresponding to each frame. Then, the image spatial feature vectors of each frame are combined in frame order to form a temporal spatial feature sequence of the video (i.e., the image spatial feature vectors corresponding to the video), which serves as input for subsequent modeling. The image spatial feature vectors of each frame are aggregated using inter-frame average pooling.

[0030] The audio data is preprocessed through pre-emphasis, framing, and endpoint detection. A pre-trained bidirectional LSTM model (pre-trained on the LibriSpeech dataset) is then used to extract MFCC features from the pre-processed audio data, generating the corresponding audio time series feature vector. The pre-emphasis coefficient can be set based on specific needs; for example, 0.97 can be used. The frame length and frame shift used for framing can also be set based on specific needs; for example, the frame length can be set to 25ms and the frame shift to 10ms.

[0031] For text data, we first perform preprocessing operations such as word segmentation on the text data; then, we use the BERT model to extract semantic features from the preprocessed text to obtain the text semantic feature vector corresponding to the text data.

[0032] Second, based on the image spatial feature vector, audio temporal feature vector, and text semantic feature vector, the initial attention score of each modality is calculated. Specifically, based on the image spatial feature vector, audio temporal feature vector, and text semantic feature vector, the attention score of each modality can be calculated using the existing cross-modal attention mechanism. The attention score of each modality calculated using the existing cross-modal attention mechanism is the initial attention score of each modality.

[0033] Third, based on the current time, historical behavior characteristics within a set time period, and text data, the initial attention score of each modality is adjusted to obtain the final attention score of each modality. This can specifically include the following steps: First, the CAAF algorithm is used to calculate the dynamically adjusted weights of each modality based on the current time, historical behavior characteristics within a set time period, and text data. The current time is the time when pet behavior prediction is initiated. The historical behavior characteristics within the set time period are the statistical characteristics of various preset abnormal behaviors of the pet during the set time period. Statistical characteristics of abnormal behaviors may include at least one of frequency, duration, and behavior distribution. Frequency characteristics may include the number of occurrences of each preset abnormal behavior (e.g., "door scratching" 22 times in 30 days) or the average number of abnormal behaviors per day. Duration characteristics may include the average duration of each abnormal behavior (e.g., average barking duration) or the number of abnormal behaviors exceeding a threshold (e.g., >5 minutes). Behavior distribution characteristics may include the percentage of abnormal behaviors corresponding to each modality (e.g., 80% for audio, 20% for image, and 0% for text), the time distribution of abnormal behaviors, and so on. The set time period can be set according to specific needs; for example, the set time period can be set to only 30 days. The text data refers to the text data entered by the user in step S1100 to describe the pet's current behavior.

[0034] Specifically, when calculating the dynamic adjustment weight of each modality based on the current time, historical behavior characteristics within a set time period, and text data, the current time, historical behavior characteristics within a set time period, and text data are first converted into a unified vector form. Then, based on the vector corresponding to the current time, historical behavior characteristics within a set time period, and text data, the dynamic adjustment weight of each modality is calculated. The specific calculation formula is as follows: w = sigmoid(MLP([t, h, s])) Where t is the vector corresponding to the current time, h is the vector corresponding to the historical behavior characteristics within the set time period, and s is the vector corresponding to the text data. t, h, and s are concatenated into the vector [t, h, s] and input into a two-layer perceptron MLP (hidden layer dimension 256). The MLP outputs an initial adjustment weight vector consisting of three floating-point values: [score_image, score_audio, score_text]. Score_image, score_audio, and score_text represent the initial adjustment weights for the image, audio, and text modalities, respectively. The initial adjustment weight vector is input into the sigmoid function for normalization to obtain the dynamic adjustment weight vector w for the image, audio, and text modalities, w = [w_img, w_aud, w_txt], where w_img, w_aud, and w_txt represent the dynamic adjustment weights for the image, audio, and text modalities, respectively, and w_img + w_aud + w_txt = 1.

[0035] In one possible implementation, when converting the current time into a vector form, the current time may be divided into hour intervals (such as 0–6, 6–12, 12–18, 18–24) and then one-hot encoded to obtain a vector representation corresponding to the current time.

[0036] In an embodiment where the historical behavior features within a set time period include frequency features, duration features, and behavior distribution features, converting the historical behavior features into a vector form may include the following steps: First, frequency features are obtained: the number of occurrences of each type of preset abnormal behavior within a set time period and the average number of abnormal behaviors per day. Then, the numerical values ​​of the number of occurrences of each type of preset abnormal behavior and the average number of abnormal behaviors per day within the set time period are sequentially concatenated to obtain an initial frequency feature vector. The data in the initial frequency feature vector is normalized to obtain the final frequency feature vector. For example, the preset behavior types include door scratching, barking, and property destruction. By analyzing the pet's behavior profile for the past 30 days, the number of occurrences of each type of preset abnormal behavior is: door scratching 22 times, barking 15 times, and property destruction 3 times, with an average total of 2.1 abnormal behaviors per day. The resulting initial frequency feature vector is [22, 15, 3, 2.1]. After normalizing the initial frequency feature vector, the final frequency feature vector is obtained. The preset abnormal behaviors are all the abnormal behaviors that the pet can potentially experience, as defined in the system.

[0037] Next, duration features are obtained: the average duration of each abnormal behavior occurring within a set time period and the percentage of times it exceeds a threshold (e.g., 5 minutes). These average durations and percentages of times it exceeds the threshold (e.g., 5 minutes) are sequentially concatenated to form an initial duration feature vector. The data in this initial duration feature vector is then normalized to create a final duration feature vector. For example, if the average duration of each abnormal behavior occurring within the past 30 days is 3.2 minutes and the percentage of times it exceeds the threshold (e.g., 5 minutes) is 25%, then the initial duration feature vector is [3.2, 0.25]. Normalizing this initial duration feature vector yields the final duration feature vector.

[0038] Next, we obtain behavior distribution features: the proportion of abnormal behavior categories and the time period distribution for each modality within a set time period. These proportions are then sequentially concatenated to generate an initial behavior distribution feature vector. The data in this initial behavior distribution feature vector is then normalized to obtain the final behavior distribution feature vector. For example, if the proportion of abnormal behavior for each modality over the past 30 days is 80% for audio, 20% for image, and 0% for text, and the time period distribution of abnormal behavior is 70% during the day and 30% at night, the resulting initial behavior distribution feature vector is [0.8, 0.2, 0, 0.7, 0.3]. Normalizing this initial behavior distribution feature vector yields the final behavior distribution feature vector. The proportion of abnormal behavior for each modality represents the ratio of the number of abnormal behaviors in each modality, and the time period distribution represents the proportion of abnormal behaviors occurring in different time periods.

[0039] Finally, the final frequency feature vector, the final duration feature vector, and the final behavior distribution feature vector are concatenated, and normalized (to avoid different feature scales) and dimensionality reduction processing (such as PCA and MLP) are performed to obtain a vector representation h corresponding to the historical behavior feature of a fixed dimension.

[0040] Secondly, based on the dynamic adjustment weight of each modality, the initial attention score of each modality is adjusted to obtain the final attention score of each modality. The calculation formula of the final attention score of each modality is as follows: w_img'=A_img×w_img w_aud'=A_aud×w_aud w_txt'= A_txt×w_txt Where A_img, A_aud, and A_txt represent the initial attention scores of the image, audio, and text modalities, respectively; w_img, w_aud, and w_txt represent the dynamically adjusted weights of the image, audio, and text modalities, respectively; and w_img', w_aud', and w_txt' represent the final attention scores of the image, audio, and text modalities, respectively.

[0041] Fourth, based on the final attention scores of each modality, the image spatial feature vector, audio temporal feature vector, and text semantic feature vector are weighted and summed to obtain the pet's fused features. The calculation formula for the fused features is as follows: F = w_img'× F_img + w_aud'×F_aud + w_txt'×F_txt Where F is the fusion feature of the pet, F_img is the image space feature vector, F_aud is the audio temporal feature vector, F_txt is the text semantic feature vector, and w_img', w_aud', and w_txt' represent the final attention scores of the image, audio, and text modalities, respectively.

[0042] In this embodiment, a CAAF algorithm module is actually introduced into the original cross-modal attention mechanism. This module can calculate the dynamic adjustment weight of each modality based on contextual information such as the current time, historical behavior characteristics within a set time period, and text data. Then, the initial attention score of each modality calculated by the original cross-modal attention mechanism is dynamically adjusted through the dynamic adjustment weight of each modality, so that the correlation of the final fusion feature is increased by about 10%. That is, the CAAF algorithm module can combine the three contextual semantics of the current time, historical behavior characteristics within a set time period, and text data to dynamically adjust the weights corresponding to the three modalities of image / audio / text, thereby improving the accuracy of the final model's judgment of abnormal behavior and personalized response capabilities.

[0043] In one possible implementation, the CAAF algorithm module adjusts the weights of each modality as follows: 1. Text-driven principle: When user text data clearly describes a specific behavior, the weight of the text and the modalities related to that behavior are increased. For example, if the text mentions a sound-related behavior (such as "screaming non-stop"), the text modal weight is increased to 0.45, the audio modal weight is increased to 0.40, and the image modal weight is reduced to 0.15. If the text mentions a visual behavior (such as "biting the tail"), the text modal weight is increased to 0.45, the image modal weight is increased to 0.40, and the audio modal weight is reduced to 0.15.

[0044] 2. Temporal context-driven principle: When the current time is a period of historically high incidence of abnormal behavior, the weight of non-text modalities related to these behaviors is increased, while the weight of text modalities is decreased. For example, during the nighttime hours (e.g., 8:00 PM to 6:00 AM), when dog barking is most frequent, the weight of the audio modal is increased to 0.55, the weight of the image modal is increased to 0.30, and the weight of the text modal is decreased to 0.15.

[0045] 3. Historical behavior dependency-driven principle: When a modality appears abnormally frequently in historical behavior, the weight of that modality is increased. For example, if audio anomalies (such as door scratching) account for ≥ 75% of the total number of incidents within 30 days, the audio modality weight is increased to 0.60, the image weight is reduced to 0.25, and the text weight remains at 0.15.

[0046] In order to enable the CAAF algorithm module to adjust the weights of each modality according to the above principles, it is necessary to construct training data according to the above principles, and train the CAAF algorithm module with the constructed training data to obtain a CAAF algorithm module with the above functions.

[0047] In one possible implementation, building training data for the CAAF module can include the following steps: First, collect pet behavior samples from multiple modalities, including video, audio, and text. See above for extracting the corresponding current time features, historical behavior feature vectors, and text feature vectors. Then, based on preset rules (such as text-driven principles, temporal context-driven principles, and historical behavior dependency-driven principles), automatically generate ideal dynamic adjustment weights for each modality to form training samples. The collected sample data should cover different seasons, time periods, pet personality differences, and text description variations.

[0048] After constructing the training data for the CAAF module, a supervised learning method can be used to train a multi-layer neural network based on this training data. After inputting the vector [t, h, s] consisting of the current time feature, the historical behavior feature vector, and the text feature vector, the neural network automatically adjusts the weights of each modality to achieve the desired dynamics. The CAAF module consists of an input layer, an intermediate layer, and an output layer. The functions of each layer are as follows: Input layer: current time feature + historical behavior feature vector + text feature vector Middle layer: Multilayer Perceptron (MLP) or Transformer Encoder Output layer: 3D vector, after softmax, representing the dynamically adjusted weights of the three modalities of image / audio / text.

[0049] When training the CAAF module, the loss function, optimization target, and deployment method are as follows: Loss function: Mean squared error (MSE) or cross entropy is used to fit the ideal weights.

[0050] Optimization goal: Make the dynamic weight of the model output as close as possible to the label weight.

[0051] Deployment: The trained CAAF model outputs three weights during inference, which are multiplied by the initial score of the original cross-modal attention to obtain the final attention score.

[0052] After calculating the fusion features of the pet, S1300 can be executed to predict the pet's behavior based on the pet's type and fusion features using a pre-built pet behavior prediction model to obtain the pet's behavior prediction results.

[0053] First of all, it should be noted that the Pet-Specific Large Model (PSLM) adopts the Dynamic Layer Selection Transformer (DLST) architecture, which specifically includes a controller, a multi-layer Transformer, and a multi-task output head.

[0054] In one possible implementation, the controller in the large pet behavior prediction model selectively activates the multi-layer Transformer based on the pet type and fusion features. The specific activation steps are as follows: First, calculate the information entropy of the fused features and the feature vector corresponding to the pet type. Specifically, convert the pet type into the corresponding feature vector, for example, [1, 0] represents a dog and [0, 1] represents a cat, and use the following formula to calculate the information entropy of the pet fused features: Where H is the fusion feature F H The information entropy value of p i is the dimension value of the normalized i-th dimension in the fused feature vector F, and n is the dimension value of the fused feature vector F H The total number of dimensions included in .

[0055] Second, based on the information entropy value of the fused features and the feature vector corresponding to the pet type, the activation vector of the multi-layer Transformer is calculated. The specific calculation formula is as follows: L = floor(MLP([H, c, E]) *p) + q Where L is the activation vector of the multi-layer Transformer, H is the information entropy value of the fused features, c is the feature vector corresponding to the pet type, and E represents the information entropy of the type distribution of historical abnormal behaviors within a set time period. The E value is pre-calculated and stored in the system and continuously updated. q is the number of fixedly activated layers in the Transformer, and p is the number of selectively activated layers in the Transformer. q + p = the total number of layers in the Transformer. The total number of layers in the Transformer, as well as the number of fixedly activated layers and selectively activated layers, can be set according to the specific scenario.

[0056] In one possible implementation, calculating the type distribution information entropy of historical abnormal behaviors within a set time period may include the following steps: First, analyzing the pet's unique behavior profile to determine the types of abnormal behaviors exhibited by the pet within the set time period, and calculating the percentage of each type of abnormal behavior (e.g., biting (40%), barking (30%), and attacking cats (30%)). A vector consisting of the percentages of each type of abnormal behavior is used as the type distribution information of historical abnormal behaviors within the set time period. Next, the entropy value of the type distribution information of historical abnormal behaviors within the set time period is calculated. This calculated entropy value is the type distribution information entropy E of historical abnormal behaviors within the set time period. After calculating E, the E value is stored in the system so that the pre-calculated E value can be automatically retrieved when calculating L. The set time period can be set according to specific needs, for example, the last 30 days. It should be noted that the pet's unique behavior profile is continuously updated. Upon detecting an update to the pet's unique behavior profile, the system will recalculate the E value in the aforementioned manner to ensure continuous updating of the E value.

[0057] In a possible implementation, the calculation formula for the type distribution information entropy E of historical abnormal behaviors is as follows: Where E is the information entropy of the type distribution of historical abnormal behaviors, p k is the type distribution information of historical abnormal behavior F E The normalized proportion of the kth abnormal behavior in the F E The total number of abnormal behaviors included in .

[0058] It should be noted that when calculating the activation vector L, the higher the input entropy value (for example, a significant increase in H or E), the more Transformer layers that can be activated by the calculated L. To achieve this goal, when constructing the above-mentioned calculation model for the activation vector L, it is necessary to construct corresponding training data and use the constructed training data to perform supervised training on the above-mentioned calculation model, thereby obtaining a calculation model for the activation vector L with the above-mentioned functions.

[0059] In one possible implementation, when constructing the training data for the calculation model of the activation vector L, the following steps can be included: first, multiple sample data are obtained, each sample data includes the information entropy value of the fusion feature, the feature vector corresponding to the pet type, and the type distribution information entropy of historical abnormal behaviors; then, for each sample data, the best activation vector L is annotated by manual annotation, so as to obtain the training data for training the calculation model of the activation vector L.

[0060] After completing the construction of the training data for the calculation model of the training activation vector L, the training data is used to train the calculation model of the activation vector L to obtain the final calculation model of the activation vector L.

[0061] Furthermore, the more complex the distribution of historical abnormal behavior types within a set time period (including at least two abnormal behaviors), the larger the corresponding E-value; conversely, the simpler the distribution of historical abnormal behavior types within a set time period (including only one abnormal behavior), the smaller the corresponding E-value. For example, if the distribution of historical abnormal behavior types for a pet within a set time period is: biting (40%), barking (30%), and attacking cats (30%), the information entropy calculated for this complex historical abnormal behavior type distribution is 1.57. For another example, if the distribution of historical abnormal behavior types for a pet within a set time period is "biting" (100%), the information entropy calculated for this simple historical abnormal behavior type distribution is 0.

[0062] In a preferred embodiment, the total number of layers in the Transformer can be set to 12, the number of fixed activated layers q is set to 4, and the number of selected activated layers p is set to 8. The calculation formula of the activation vector L of the multi-layer Transformer is as follows: L = floor(MLP([H, c, E]) *8) + 4 Third, based on the activation vector, the multi-layer Transformer is selectively activated.

[0063] In this implementation, the controller can determine the number of Transformer activation layers based on the information entropy value of the input fusion features, the pet type (distinguishing between dogs and cats through embedded vectors), and the pre-calculated information entropy of the type distribution of historical abnormal behaviors of the pet within a set time period. The smaller the input entropy value, the fewer the number of activated layers (for example, low entropy input, 4 layers are activated); the larger the input entropy value, the more the number of activated layers (for example, high entropy input, 12 layers are activated). This design can reduce the amount of computation by about 35%, while reducing the inference delay by 25%.

[0064] When building a large pet behavior prediction model, first load the above-mentioned basic model with the dynamic layer selection Transformer architecture; then, use public datasets (such as the Kaggle pet behavior video library, which contains 100,000 labeled samples) and user-uploaded labeled data (covering multi-source data of various pets such as dogs and cats, as well as behavior prediction results based on the multi-source data) to pre-train the basic model to obtain the initial pet behavior prediction model.

[0065] In one possible implementation, in order to improve the accuracy of the pet behavior prediction model for a specific pet, a personalized adaptive mechanism of the large model is introduced. That is, after obtaining the initial pet behavior prediction large model, the following steps are also included: First, fine-tuning training data is constructed based on the pet's type and historical behavior data. Specifically, a unique behavior profile is created for each pet and stored in the cloud. This profile records the pet's type and multiple historical behavior data items. Each historical behavior data item includes the collected multimodal data and the behavior prediction results based on the multimodal data. At least the most recently stored 100 historical behavior data items in the pet's unique behavior profile can be extracted and combined with the pet's type as the fine-tuning training data for that pet.

[0066] Second, fine-tuning training data is used to fine-tune the pre-trained initial pet behavior prediction model to obtain the pet behavior prediction model.

[0067] In this feasible method, through the pet-specific behavior profile and online learning algorithm (i.e., introducing the pet's exclusive behavior profile and constructing fine-tuning training data based on the pet's exclusive behavior profile to fine-tune the large model), the pet behavior prediction large model can dynamically adapt to the unique behavior pattern of each pet, reduce the generalization prediction error, and improve the F1 score of personalized prediction by about 20%.

[0068] In this feasible method, a self-supervised pre-training strategy (i.e., modeling based on masked data and training for 100 epochs) is first used to train an initial pet behavior prediction model. Then, the fine-tuning training data is used to perform supervised fine-tuning on the initial pet behavior prediction model (using cross-entropy loss, learning rate 1e-5, and fine-tuning for 20 epochs) to obtain the final pet behavior prediction model that matches a specific pet, so as to improve the model's generalization ability for sparse data.

[0069] Furthermore, each time a pet behavior prediction is made, multi-source data from the prediction and the corresponding behavior prediction results will be collected as historical behavior data and stored in the pet's exclusive behavior file. After each new historical behavior data of the set data is collected, the pet behavior prediction model will be promptly fine-tuned and updated based on the set number of new historical behavior data.

[0070] After the prediction model is trained, the pet's type and fused features are input into the pet behavior prediction model. The model then automatically predicts the pet's behavior, generating a prediction result. This prediction includes information about the pet's next behavior type (including various possible behavior types and their corresponding confidence scores), emotional state information (including various possible emotions and their corresponding confidence scores), and explanatory descriptions of the behavior type and emotional state.

[0071] After the large model outputs the pet's behavior prediction results, it displays the behavior type and emotional state information in a visual format. For example, a bar chart will be used to display various possible behavior types and their corresponding confidence scores, and a bar chart will be used to display various possible emotional states and their corresponding confidence scores. At the same time, an explanatory description of the pet's behavior type and emotional state will be displayed in text form. For example, "The pet is currently predicted to be 'pacing repeatedly.' Combined with the decrease in average daily interaction time and increased frequency of nighttime activity over the past week, it is speculated that it may be mildly anxious, with a confidence level of 0.78." Through visual output and explanatory descriptions, users can intuitively understand the pet's behavior prediction results.

[0072] Furthermore, after displaying the behavior prediction results, the system also receives user feedback (provided via the system interface, indicating "accurate," "inaccurate," and optional textual explanations). This feedback data is stored in the cloud and used to continuously optimize the model. For example, the model is updated monthly in batches using newly collected historical behavior data and user feedback, prioritizing high-frequency feedback samples. This feedback mechanism further optimizes model performance, and user satisfaction surveys show that over 90% of users find the prediction results useful.

[0073] The following is a detailed example to further illustrate the solution of the present disclosure. Specifically, the following steps are included: First, users upload a 30-second pet video (MP4 format, 1080p, including dog barking and walking animation), a photo of the pet sleeping (JPEG format, 2000x1500 pixels) and a text description ("I don't like to play with the ball today, and my appetite has decreased") through the mobile app (version 2.0, supporting iOS 14+ and Android 10+).

[0074] Second, the system verifies the data format and quality (video frame rate ≥ 24fps, audio signal-to-noise ratio ≥ 20dB, text length ≤ 200 words) to ensure that the data is suitable for subsequent processing.

[0075] Third, feature extraction and fusion: ResNet-50 (pre-trained on ImageNet) is used to extract spatial features of each video frame, generating a 2048-dimensional feature vector, which is then aggregated across frames using average pooling. Use a bidirectional LSTM (trained on the LibriSpeech dataset) to process the extracted 13-dimensional MFCC features (frame length 25ms, frame shift 10ms) on the audio to generate a 512-dimensional time series feature vector; Use BERT (a Chinese pre-trained model) to segment and vectorize the text, generating a 768-dimensional semantic feature vector; The three features are fused through a cross-modal attention mechanism (8-head attention, trained on a multimodal sentiment dataset) to generate a 1024-dimensional fused feature vector.

[0076] Third, the 1024-dimensional fused feature vector is input into the pre-built pet behavior prediction model (PSLM, based on a 12-layer Transformer, with 120 million parameters) to automatically predict pet behavior. The prediction results are "tired" (confidence 0.88), "anxious" (0.09), and "hungry" (0.03).

[0077] The app interface displays the prediction results: a bar chart shows the probability distribution of "tiredness (0.85)", "anxiety (0.09)", and "hunger (0.03)", and the text description is "Based on the recent low barking frequency and the owner's description of 'not liking to play with the ball and reduced appetite', it is predicted that the dog may be in a state of fatigue. It is recommended to reduce stimulating activities and increase rest time."

[0078] The user confirms the prediction by clicking the "Accurate" button on the app and adds a note, "It's indeed very quiet today." This feedback is then uploaded to the cloud database. This statement indicates that the user acknowledges and supports their observation that the dog is tired, cross-validating the model's output with real-life conditions.

[0079] This disclosure provides a pet behavior prediction method, comprising: obtaining a pet's type, image data, audio data, and text data describing the pet's behavior; performing feature extraction on the image data, audio data, and text data, and performing multimodal feature fusion to obtain a fused pet feature; and using the pet's type and fused features, using a pre-built large-scale pet behavior prediction model to predict the pet's behavior, thereby obtaining a pet behavior prediction result. This method not only incorporates visual modality data but also simultaneously integrates auditory features from the audio modality with the contextual semantics of the owner's text description. A cross-modal dynamic weighting mechanism eliminates behavioral ambiguity in visual data alone (e.g., distinguishing between excited and anxious tail wagging), captures emotional cues in acoustic signals (e.g., whimpering indicates pain) and implicit states in text descriptions (e.g., "loss of appetite" indicates illness), thereby comprehensively constructing a multidimensional cognitive model of pet behavior and significantly improving prediction accuracy. Experiments have shown that the behavior classification accuracy can reach over 90%, an improvement of approximately 15% compared to single-modality methods.

[0080] <Device Example> Figure 2 FIG. 1 is a schematic block diagram of a device for predicting pet behavior according to an embodiment of the present disclosure. Figure 2 As shown, the device 100 includes: The multimodal data acquisition module 110 is used to obtain the type of pet, image data, audio data, and text data describing the pet's behavior; A multimodal data fusion module 120 is configured to extract features from the image data, the audio data, and the text data, and perform multimodal feature fusion to obtain fused features of the pet; The behavior prediction module 130 is used to predict the behavior of the pet based on the type of the pet and the fusion features using a pre-built pet behavior prediction model to obtain a behavior prediction result of the pet.

[0081] <Equipment Example> Figure 3 FIG. 1 is a schematic block diagram of a device for predicting pet behavior according to an embodiment of the present disclosure. Figure 3 As shown, the pet behavior prediction device 200 includes a processor 210 and a memory 220 for storing executable instructions of the processor 210. The processor 210 is configured to implement any of the aforementioned pet behavior prediction methods when executing the executable instructions.

[0082] It should be noted that there may be one or more processors 210. Furthermore, the pet behavior prediction device 200 according to the disclosed embodiment may also include an input device 230 and an output device 240. The processor 210, memory 220, input device 230, and output device 240 may be connected via a bus or other means, which are not specifically limited herein.

[0083] Memory 220, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and various modules, such as the programs or modules corresponding to the pet behavior prediction method according to the embodiments of the present disclosure. Processor 210 executes the software programs or modules stored in memory 220 to perform various functional applications and data processing of pet behavior prediction device 200.

[0084] The input device 230 may be used to receive input numbers or signals. The signals may be key signals related to user settings and function control of the device / terminal / server. The output device 240 may include a display device such as a display screen.

[0085] <Storage Medium Embodiment> According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium is further provided, on which computer program instructions are stored. When the computer program instructions are executed by the processor 210, any of the above-mentioned pet behavior prediction methods is implemented.

[0086] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technical improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for predicting pet behavior, characterized in that: include: Obtaining a pet's type, image data, audio data, and text data describing the pet's behavior; Performing feature extraction on the image data, the audio data, and the text data respectively and performing multimodal feature fusion to obtain fused features of the pet; Based on the type of the pet and the fusion features, the behavior of the pet is predicted using a pre-built pet behavior prediction model to obtain a behavior prediction result of the pet; The pet behavior prediction model includes a controller, a multi-layer Transformer and a multi-task output head, wherein the controller selectively activates the multi-layer Transformer based on the type of the pet and the fusion features.

2. The method according to claim 1, characterized in that When feature extraction is performed on the image data, the audio data, and the text data respectively and multimodal feature fusion is performed to obtain the fusion feature of the pet, it is achieved based on a cross-modal attention mechanism.

3. The method according to claim 2, characterized in that When extracting features from the image data, the audio data, and the text data respectively and performing multimodal feature fusion based on a cross-modal attention mechanism to obtain fused features of the pet, the method includes: Performing feature extraction on the image data, the audio data, and the text data respectively to obtain an image spatial feature vector, an audio temporal feature vector, and a text semantic feature vector; Calculating an initial attention score for each modality based on the image spatial feature vector, the audio temporal feature vector, and the text semantic feature vector; Adjusting the initial attention score of each modality based on the current time, historical behavior characteristics within a set time period, and the text data to obtain a final attention score for each modality; Based on the final attention score of each modality, the image space feature vector, the audio time series feature vector, and the text semantic feature vector are weightedly summed to obtain the fusion feature of the pet.

4. The method according to claim 3, characterized in that Adjusting the initial attention score of each modality based on the current time, historical behavior characteristics within a set time period, and the text data to obtain the final attention score of each modality includes: Calculate the dynamic adjustment weight of each modality based on the current time, historical behavior characteristics within a set time period, and the text data; Based on the dynamic adjustment weight of each modality, the initial attention score of each modality is adjusted to obtain the final attention score of each modality.

5. The method according to claim 1, wherein When constructing the pet behavior prediction model, it includes: Constructing fine-tuning training data based on the type of the pet and historical behavior data; The fine-tuning training data is used to perform fine-tuning training on the pre-trained initial pet behavior prediction large model to obtain the pet behavior prediction large model.

6. The method according to claim 1, characterized in that When the controller selectively activates the multi-layer Transformer based on the type of the pet and the fusion feature, it includes: Calculating the information entropy value of the fusion feature and the feature vector corresponding to the type; Calculating an activation vector of the multi-layer Transformer based on the information entropy and the feature vector; Based on the activation vector, the multi-layer Transformer is selectively activated.

7. A device for predicting pet behavior, characterized in that: include: A multimodal data acquisition module for acquiring the pet's type, image data, audio data, and text data describing the pet's behavior; A multimodal data fusion module, configured to extract features from the image data, the audio data, and the text data, and perform multimodal feature fusion to obtain fused features of the pet; A behavior prediction module, configured to predict the behavior of the pet based on the type of the pet and the fusion features using a pre-built pet behavior prediction model to obtain a behavior prediction result of the pet; The pet behavior prediction model includes a controller, a multi-layer Transformer and a multi-task output head, wherein the controller selectively activates the multi-layer Transformer based on the type of the pet and the fusion features.

8. A device for predicting pet behavior, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method according to any one of claims 1 to 6 when executing the executable instructions.

9. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images