Pet pacifying method

Abstract

The invention discloses a pet pacifying method, and relates to the technical field of emotion recognition, and the pet pacifying method comprises the following steps: obtaining pet behavior data, and inputting the pet behavior data into a pet emotion recognition model to recognize the current emotion type of a pet; according to the current emotion type of the pet, determining a pacifying strategy corresponding to the current emotion type; and pacifying the pet based on the pacifying strategy. According to the method, the emotion of the pet is recognized from multiple dimensions, the accuracy of pet emotion recognition is improved, whether the pet is in an anger, happy or calm state is judged, corresponding strategies are selected for pacifying according to different emotional states of the pet, and the anxiety, tension and other emotions of the pet can be relieved more effectively.

Description

Technical Field

[0001] The present application relates to the technical field of emotion recognition, and in particular to a method for soothing a pet. Background Art

[0002] With the progress of society, people's material living standards are constantly improving, and more and more families are starting to keep pets. Pets play an important role in life, not only providing emotional companionship, but also helping people relieve stress and loneliness. Interaction with pets can bring a sense of happiness, improve mental health, and even encourage owners to participate in more social activities and improve their quality of life.

[0003] However, pets' emotions are often easily affected by the environment, health status and owners. When pets' emotions fluctuate, they may show anxiety, aggression or restlessness, which will cause distress to the owners and even increase emotional stress.

[0004] Experienced pet owners can accurately judge the pet's emotions through the tone, volume, frequency, etc. of the pet's voice, and thus provide effective soothing methods, but this ability is not possessed by every pet owner. Therefore, how to accurately grasp the pet's emotions in order to better soothe and interact has become a problem that pet owners want to solve. Summary of the invention

[0005] The main purpose of this application is to provide a pet soothing method, which aims to solve the problem of identifying the emotions of pets and soothing them.

[0006] To achieve the above objectives, the present application proposes a pet soothing method, the method comprising:

[0007] Obtain pet behavior data, and input the pet behavior data into a pet emotion recognition model to identify the pet's current emotion type;

[0008] According to the current emotion type of the pet, determining a soothing strategy corresponding to the current emotion type;

[0009] The pet is soothed based on the soothing strategy.

[0010] In one embodiment, the pet soother includes a speaker and a vibrator, and in the step of determining a soothing strategy corresponding to the pet emotion recognition result according to the pet emotion recognition result:

[0011] The soothing strategy includes playing a soothing sentence generated according to the pet owner's timbre and controlling the vibrator to operate in a corresponding vibration mode according to the pet's current emotional type.

[0012] In one embodiment, the pet emotion recognition model is established by the following steps:

[0013] Acquire a first pet behavior data set, wherein the first pet data set includes pet sound data, action data, expression data, and environmental information;

[0014] labeling the sound data, the action data, and the expression data in the first pet behavior data set with emotion tags to obtain a second pet behavior data set;

[0015] Performing data cleaning and data enhancement on the second pet behavior dataset to obtain a third pet behavior dataset;

[0016] Use animal behavior datasets to pre-train a general neural network to obtain a pre-trained model;

[0017] The pre-trained model is fine-tuned using the third pet behavior dataset to obtain a pet emotion recognition model.

[0018] In one embodiment, after the step of labeling the sound data, the action data, and the expression data in the first pet behavior data set with emotion tags to obtain the second pet behavior data set, the step further includes:

[0019] A behavior classifier is constructed to extract pet behavior features, identify pet emotions, and classify pet emotions.

[0020] In one embodiment, the steps of constructing a behavior classifier, extracting pet behavior features through the behavior classifier, identifying pet emotions, and classifying pet emotions include:

[0021] Use convolutional neural networks to extract audio features, facial expressions, and action features that represent emotions;

[0022] Using long short-term memory networks to process the time series characteristics of audio and / or video;

[0023] Construct a fusion layer to fuse the pet audio features, expression features, and action features to obtain fusion features;

[0024] A behavior classifier is constructed according to the fusion features.

[0025] In one embodiment, the step of extracting audio features, expression features, and action features representing emotions using a convolutional neural network includes:

[0026] Perform short-time Fourier transform on the audio signal to obtain a spectrum diagram;

[0027] Convert the frequency axis in the spectrogram to the Mel frequency scale to obtain a Mel spectrogram;

[0028] Apply cepstral analysis to the mel-frequency spectrogram and generate mel-frequency cepstral coefficients through discrete cosine transform;

[0029] The audio features representing emotions in the Mel-frequency spectrogram or Mel-frequency cepstral coefficients are extracted through convolutional neural networks.

[0030] In one embodiment, the step of extracting audio features, expression features, and action features representing emotions using a convolutional neural network includes:

[0031] Extract the pet's facial area from the video or image to obtain the pet's expression image;

[0032] Resize and normalize pet expression images;

[0033] The features of the pet's facial area in the pet expression image are extracted layer by layer through a convolutional neural network to obtain the expression features.

[0034] In one embodiment, the step of extracting audio features, expression features, and action features representing emotions using a convolutional neural network includes:

[0035] Extract image features of each frame in the pet video through convolutional neural network;

[0036] The image features are input into the long short-term memory network to process the time series to obtain the pet action features;

[0037] or,

[0038] A 3D convolutional neural network is used to perform convolution operations between multiple frames to extract motion features at different time points to obtain pet motion features.

[0039] In one embodiment, the step of using the third pet behavior dataset to fine-tune the pre-trained model to obtain a pet emotion recognition model includes:

[0040] Inputting the behavior data in the third pet behavior data set into a pre-training model, wherein the pre-training model outputs a prediction result according to the input behavior data;

[0041] Use the loss function to calculate the error between the pre-trained model output and the true label to get the loss value;

[0042] According to the loss value, a back propagation algorithm is used to calculate the gradient of each parameter in the pre-trained model, wherein the parameters of the pre-trained model include convolution kernel weights, fully connected layer weights, and fully connected layer biases;

[0043] Using an optimization algorithm to adjust the parameters of the model according to the gradient, and / or adjusting the hyperparameters, the step of inputting the behavior data in the third pet behavior data set into the pre-trained model, wherein the pre-trained model outputs a prediction result according to the input behavior data, is performed until the loss value reaches a preset value;

[0044] Among them, hyperparameters include learning rate, batch size, number of network layers, convolution kernel size, and regularization coefficient.

[0045] In one embodiment, the step of using the third pet behavior dataset to fine-tune the pre-trained model to obtain a pet emotion recognition model includes:

[0046] A multi-task learning algorithm is used to build a multi-task model, which is used to recognize the pet's emotions, the pet's actions, and the pet's environment.

[0047] In one embodiment, the step of using the third pet behavior dataset to fine-tune the pre-trained model to obtain a pet emotion recognition model includes:

[0048] Regularization methods are used to control and constrain the parameters or structure of the model to prevent the model from overfitting on the training data.

[0049] The technical solution proposed in this application has at least the following technical effects:

[0050] The present invention obtains pet behavior data, including body movements, sounds, expressions, etc. The pet behavior data is input into a pet emotion recognition model to identify the pet's current emotion type. And according to the pet's current emotion type, a soothing strategy corresponding to the current emotion type is determined. The pet is soothed based on the soothing strategy, and the pet's emotions are identified from multiple dimensions to improve the accuracy of pet emotion recognition, so as to judge whether the pet is in an angry, happy, or calm state. According to the different emotional states of the pet, the corresponding strategy is selected for soothing, which can more effectively relieve the pet's anxiety, tension and other emotions. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0053] Figure 1 A flowchart diagram of an embodiment of a pet soothing method of the present application;

[0054] Figure 2 A flowchart diagram of an embodiment of a method for establishing a pet emotion recognition model in the present application;

[0055] Figure 3 A flowchart diagram of another embodiment of the method for establishing a pet emotion recognition model of the present application;

[0056] Figure 4 A flowchart diagram of another embodiment of the method for establishing a pet emotion recognition model of the present application is provided;

[0057] Figure 5 A flowchart diagram of another embodiment of the method for establishing a pet emotion recognition model of the present application is provided;

[0058] Figure 6 A flowchart diagram of another embodiment of the method for establishing a pet emotion recognition model of the present application is provided;

[0059] Figure 7 A flowchart diagram of an embodiment of a method for establishing a pet emotion recognition model in the present application;

[0060] Figure 8 A flowchart diagram of another embodiment of the method for establishing a pet emotion recognition model of the present application is provided;

[0061] Fig. 9 A flowchart diagram of another embodiment of the method for establishing a pet emotion recognition model of the present application is provided;

[0062] Fig.10 A flowchart diagram is provided for another embodiment of the method for establishing a pet emotion recognition model of the present application.

[0063] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0064] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0065] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0066] With the progress of society, people's material living standards are constantly improving, and more and more families are starting to keep pets. Pets play an important role in life, not only providing emotional companionship, but also helping people relieve stress and loneliness. Interaction with pets can bring a sense of happiness, improve mental health, and even encourage owners to participate in more social activities and improve their quality of life.

[0067] However, pets' emotions are often easily affected by the environment, health status and owners. When pets' emotions fluctuate, they may show anxiety, aggression or restlessness, which will cause distress to the owners and even increase emotional stress.

[0068] Experienced pet owners can accurately judge the pet's emotions through the tone, volume, frequency, etc. of the pet's voice, and thus provide effective soothing methods, but this ability is not possessed by every pet owner. Therefore, how to accurately grasp the pet's emotions in order to better soothe and interact has become a problem that pet owners want to solve.

[0069] In order to solve the above problems, the present invention proposes a pet soothing method, which aims to identify the emotions of the pet and soothe it.

[0070] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of a pet soothing method of the present application. The pet soothing method proposed in the present application includes steps S100 to S300, wherein:

[0071] Step S100, obtaining pet behavior data, and inputting the pet behavior data into a pet emotion recognition model to identify the pet's current emotion type;

[0072] Step S200, determining a soothing strategy corresponding to the current emotion type of the pet according to the current emotion type of the pet;

[0073] Step S300, soothing the pet based on the soothing strategy.

[0074] In this embodiment, the pet behavior data includes the pet's audio, pet's image, and pet's heart rate data. The pet's voice can be collected through a microphone, the pet's expression and pet's movements can be obtained through a camera, and the pet's heart rate can be collected through a heartbeat sensor.

[0075] After obtaining the pet behavior data, the pet behavior data is input into the pet emotion recognition model, and the pet emotion recognition model is used to identify the pet's current emotion type, helping the pet owner to identify the pet's emotions based on the pet's behavior, thereby providing effective soothing methods or improving the quality of interaction between the pet owner and the pet, thereby improving the quality of the pet's companionship.

[0076] In daily life, pets' emotions are affected by the environment, physical condition, and the behavior of their owners. If a pet feels anxious, afraid, or stressed, it may exhibit aggressive, destructive behavior, or health problems such as decreased appetite and insomnia. When a pet is sick, anxious, or behaves abnormally, the owner may also feel worried or anxious. Pets that are overly dependent on their owners may show separation anxiety when their owners are away, affecting the pet's quality of life. In these cases, if pet owners can adopt appropriate soothing strategies to soothe their pets, it can help the pets maintain emotional stability and continue to enjoy the positive effects of pets, but this requires rich experience to support it.

[0077] To this end, in one embodiment, the pet soother includes a speaker and a vibrator, and in the step of determining a soothing strategy corresponding to the pet emotion recognition result according to the pet emotion recognition result:

[0078] The soothing strategy includes playing a soothing sentence generated according to the pet owner's timbre and controlling the vibrator to operate in a corresponding vibration mode according to the pet's current emotional type.

[0079] In this embodiment, the pet soother infers the input pet behavior data through a pet emotion recognition model to identify the pet's current emotion, and then generates a corresponding soothing strategy based on the pet's emotional state to help the pet relieve negative emotions.

[0080] Specifically, the pet soother includes a speaker and a vibrator. When a pet shows anxiety, fear or other abnormal emotions, the application will first collect the pet's voice through a microphone, collect pictures or videos of the pet's expression and movements through a camera, and collect behavioral data such as the pet's heart rate through a heartbeat sensor, and input these behavioral data into a pet emotion recognition model. The pet emotion recognition model will infer the input pet behavior data, identify the pet's current emotional state, and combine previous emotional data to determine the current emotional type. Then, the pet soother will generate a gentle sentence based on the pet owner's timbre, play it through a speaker, and imitate the owner's tone to soothe the pet. At the same time, the vibrator works in a specific vibration mode according to the pet's current emotional type. For example, a slow vibration mode may help the pet relax when anxious, while a faster vibration mode may be suitable for use when the pet needs stimulation or interaction.

[0081] In some embodiments, the pet soother further includes an AI model. The AI ​​model is used to further enhance the functionality of the pet soother by generating personalized soothing statements and dynamically adjusting the soothing strategy.

[0082] Based on the pet's emotion recognition results, the AI ​​model can generate soothing sentences that match the current emotional state. For example, when a pet shows anxiety, AI can generate soothing and calming sentences, using simple and gentle language to comfort it.

[0083] AI not only generates sentences with appropriate content, but also plays them by imitating the pet owner's voice through audio synthesis technology. Using the owner's voice helps pets feel familiar voices when they are away from their owners, thus alleviating separation anxiety.

[0084] Through sensors or cameras, the system can monitor the pet's response to soothing statements in real time, including the pet's behavioral changes, physiological data (such as heart rate, breathing rate, etc.) and emotional fluctuations. In this way, it can be determined whether the current soothing strategy is effective.

[0085] Based on the pet's real-time response, the AI ​​model can dynamically adjust the content and playback method of the soothing statement. If the pet's emotions are not effectively relieved, the system will generate a new, more appropriate soothing statement and play it again. The frequency of adjustment and the content of the statement can be gradually optimized according to the pet's emotional changes, forming a highly personalized soothing strategy.

[0086] The AI ​​model continuously optimizes the generation process of soothing sentences through machine learning. The pet soothing device will record the pet's emotional changes and the effectiveness of the soothing strategy. By analyzing this data, the accuracy and soothing effect of future sentence generation will be gradually improved, forming an adaptable and personalized pet soothing device.

[0087] In this way, the soother not only helps pets relieve separation anxiety when their owners are away, but also soothes and calms pets when they are overly nervous or stressed. At the same time, the combination of the pet owner's timbre and vibration pattern simulates an actual soothing scenario, bringing more realistic emotional support to pets.

[0088] Through this AI-based smart soother, the combination of the pet owner's voice and personalized sentences can not only provide immediate emotional support, but also meet the pet's emotional needs more accurately over time, thereby significantly improving the pet's quality of life and ensuring its emotional health and behavioral stability.

[0089] In one embodiment, see Figure 2 , Figure 2 A flowchart of an embodiment of a method for establishing a pet emotion recognition model is provided for this application. The pet emotion recognition model is established by the following steps:

[0090] Step S110, obtaining a first pet behavior data set, wherein the first pet data set includes pet sound data, action data, expression data and environmental information;

[0091] Step S120, labeling the sound data, the action data, and the expression data in the first pet behavior data set with emotion labels to obtain a second pet behavior data set;

[0092] Step S130, performing data cleaning and data enhancement on the second pet behavior dataset to obtain a third pet behavior dataset;

[0093] Step S140, pre-training a general neural network using an animal behavior dataset to obtain a pre-trained model;

[0094] Step S150: fine-tune the pre-trained model using the third pet behavior dataset to obtain a pet emotion recognition model.

[0095] In this embodiment, a large amount of pet behavior data is first collected to form a first pet behavior data set. The data set includes multimodal data of the pet. The first pet behavior data set includes:

[0096] Sound data: such as pets’ calls, barks, or other sounds;

[0097] Motion data: Capture your pet’s body movements through sensors or cameras;

[0098] Expression data: Obtain pet’s facial expression changes through visual analysis;

[0099] Text data: Owners or pet caregivers record their pets’ daily behaviors and related events;

[0100] Environmental information: including external factors such as temperature, light, noise, etc., which may affect the pet's mood.

[0101] Secondly, the collected sound, action, and expression data are annotated with emotional labels, and the pet's behavior is associated with its corresponding emotions (such as happiness, anxiety, fear, relaxation, etc.) to form a second pet behavior data set with emotional labels.

[0102] Next, the second pet behavior dataset is cleaned and enhanced. Data cleaning includes removing invalid, duplicate or abnormal data to ensure the quality of the dataset. Data enhancement is to generate new training samples by performing different forms of transformations on the training data, such as rotation, flipping, cropping, adding noise, etc. This not only expands the size of the dataset, but also allows the model to be trained on more diverse data, prevents overfitting, increases the generalization ability of the model, and obtains a richer third pet behavior dataset.

[0103] In some embodiments, during data cleaning and data enhancement of the second pet behavior data set, different weights may be assigned to the categories of multimodal data to balance the amount of each category of data. For example, a 40% weight may be assigned to audio data, a 30% weight may be assigned to video data, a 10% weight may be assigned to text data, and a 20% weight may be assigned to image data.

[0104] Then, we use the animal behavior dataset to pre-train the general neural network and obtain a basic model that can extract animal behavior features. The animal behavior dataset includes animal audio and video data. At this stage, the model learns how to extract useful features from input data such as sound, movement, and expression.

[0105] In this embodiment, the general neural network can be Wav2Vec or Transformer. Wav2Vec is a deep learning model for speech recognition, originally proposed by Facebook AI Research. It is a method based on self-supervised learning and is mainly used for pre-training of speech signals. Wav2Vec learns acoustic features by predicting missing parts in speech segments. This method allows the model to learn useful representations from a large amount of unlabeled audio data. Wav2Vec can be regarded as a feature extractor, whose output can be further used as input for downstream tasks (such as speech recognition, sentiment analysis, etc.).

[0106] Transformer is an architecture that has made a major breakthrough in the field of natural language processing (NLP). It first appeared in the paper "Attention is All You Need". Unlike traditional RNN or LSTM, Transformer uses the self-attention mechanism to process sequence data in parallel, greatly improving training efficiency. Transformer has become the basis of many NLP tasks, including machine translation, text summarization, question-answering systems, etc.

[0107] Finally, the pre-trained general neural network is combined with the third pet behavior dataset, and the model parameters are fine-tuned to adapt it to the emotion recognition task of a specific pet. In this way, a deep learning model specifically for pet emotion recognition is obtained, which can automatically judge the emotional state of the pet based on the input pet behavior data.

[0108] In one embodiment, see Figure 3 , Figure 3 A flowchart diagram of another embodiment of the method for establishing a pet emotion recognition model of the present application is provided. Step S120, after the step of labeling the sound data, the action data and the expression data in the first pet behavior data set with an emotion tag to obtain a second pet behavior data set, further includes:

[0109] Step S160, constructing a behavior classifier, extracting pet behavior features through the behavior classifier, identifying pet emotions, and classifying pet emotions.

[0110] In this embodiment, the behavior classifier is a deep learning-based model for automatically analyzing and understanding the behavior data of pets. The classifier can accurately extract the behavior characteristics by learning the complex relationship between the pet's behavior and its emotional state based on the multimodal data (such as sound, movement, expression, etc.) extracted in the previous step.

[0111] The primary function of a behavior classifier is to extract features from the pet's behavior data. These features may include the pet's movement patterns, vocalization patterns, facial expression changes, etc. By extracting these features, the classifier can convert complex behaviors into numerical or vector representations that are easy to analyze.

[0112] Motion feature extraction: By analyzing the pet’s body posture, walking trajectory, jumping or running behaviors, the classifier can identify the type of specific actions (such as active, stationary, chasing, etc.).

[0113] Sound feature extraction: Extract features such as pitch, volume, and frequency changes from a pet’s barking, calling, or other sound signals to identify emotional information in the sound.

[0114] Expression feature extraction: Through visual recognition technology, the changes in pet's facial expressions are extracted, especially the movements in the ears, eyes and mouth, to judge the pet's emotions.

[0115] Once the behavioral features are extracted, the behavioral classifier will use these features to classify the pet's emotions. The classifier is usually based on pre-defined emotion categories (such as happiness, anxiety, fear, excitement, etc.) and uses machine learning algorithms (such as convolutional neural networks CNN, long short-term memory networks LSTM, etc.) to predict these categories.

[0116] The classification process can be divided into several steps:

[0117] Feature input: The extracted features are used as input and fed into the classifier.

[0118] Model inference: The classifier processes the input data through a neural network or other machine learning algorithm to infer the category of the current emotion.

[0119] Emotion classification output: The model finally outputs the emotional state of the pet and gives its classification result. For example, the classifier may output "The pet's current emotion is anxious, with a confidence level of 85%."

[0120] In practical applications, behavioral classifiers can be adaptively adjusted by continuously learning and updating data. For example, the classifier can gradually optimize its classification accuracy based on the pet's historical behavioral data and its owner's feedback, ensuring that it can more personalized the emotion recognition for each pet.

[0121] The constructed behavior classifier can be applied to the pet emotion monitoring system in real time. When a pet exhibits a specific behavior, the system can automatically identify the pet's emotional state through the classifier to help the owner or automated equipment respond in time. Based on the classification results, the owner can adopt corresponding soothing strategies or behavioral interventions to improve the pet's emotional state.

[0122] In one embodiment, see Figure 4 , Figure 4 A flowchart diagram of another embodiment of the method for establishing a pet emotion recognition model of the present application is provided. Step S160, constructing a behavior classifier, extracting pet behavior features through the behavior classifier, identifying pet emotions, and classifying pet emotions includes:

[0123] Step S161, using a convolutional neural network to extract audio features, expression features, and action features representing emotions;

[0124] Step S162, using a long short-term memory network to process the time series characteristics of the audio and / or video;

[0125] Step S163, constructing a fusion layer, fusing the pet audio features, expression features and action features to obtain fusion features;

[0126] Step S164: construct a behavior classifier based on the fusion features.

[0127] In this embodiment, a convolutional neural network (CNN) is used to extract various behavioral features of the pet, including audio features, expression features, and action features.

[0128] By performing time-frequency analysis on pet sound signals, CNN can extract emotion-related features such as pitch, tone, amplitude changes, etc. from the spectrogram.

[0129] CNN is used to analyze images of pets’ faces and extract the displacement or state change features of facial organs such as ears, eyes, and mouth. These features help to infer the pet’s emotional state.

[0130] By analyzing the pet's movement trajectory, posture changes, etc., CNN can extract representative behavioral features from videos or other sensor data, such as jumping, walking, and stillness.

[0131] Since the pet's sound data and video data are time series signals, in this embodiment, a long short-term memory network (LSTM) is used to process the long-term dependency of the sound data. For example, LSTM can identify the continuous change pattern of the sound, thereby determining whether there are sound features related to emotions, such as continuous barking may indicate anxiety.

[0132] The pet’s movements and expressions are also time series data, and LSTM can capture the dynamic changes of these behaviors. For example, LSTM can identify the gradual transition of a pet from calm to excited, combining data from multiple time steps to determine emotional changes.

[0133] LSTM can well process the long-term dependency information in these sequence data, thereby accurately capturing the temporal dynamic characteristics of emotional changes.

[0134] Through the fusion layer, the audio features, expression features, and action features extracted from CNN will be integrated into a comprehensive feature vector. The fusion process can use splicing, weighted summation, or more complex fusion strategies to ensure that the information between different modalities can complement each other and form a more accurate emotional feature representation.

[0135] By combining features from different sensory channels, the system can more fully understand the pet’s emotions. For example, a pet’s facial expression may indicate that it is slightly nervous, while combining sound and movement features can further confirm whether this emotion is anxiety or excitement.

[0136] The classifier can use a fully connected layer or other common classification methods to input the fused features into the model and finally output the pet's emotion category. Based on the weights obtained during the training process, the classifier can accurately classify the pet's emotions, such as "anxious", "happy", "angry", etc.

[0137] After inputting the pet behavior data, the classifier will output the corresponding classification results based on the fusion features and pre-defined emotion labels, and use the confidence indicator to indicate the model's certainty about the classification results.

[0138] This embodiment combines a convolutional neural network (CNN) with a long short-term memory network (LSTM) to extract and process the audio, expression and action features of the pet, and integrates the multimodal data through a fusion layer. Finally, the pet's emotions are accurately classified through a behavioral classifier. This method can not only capture static behavioral characteristics, but also process dynamic changes in time series, thereby more accurately identifying the emotional state of the pet. This multimodal fusion emotion recognition system can significantly improve the accuracy and reliability of pet emotion analysis.

[0139] In one embodiment, see Figure 5 , Figure 5 A flowchart diagram of another embodiment of the method for establishing a pet emotion recognition model of the present application is provided. Step S161, the step of using a convolutional neural network to extract audio features, expression features, and action features representing emotions includes:

[0140] Step S1611, performing short-time Fourier transform on the audio signal to obtain a spectrum diagram;

[0141] Step S1612, converting the frequency axis in the spectrogram into a Mel frequency scale to obtain a Mel spectrogram;

[0142] Step S1613, applying cepstrum analysis to the mel-frequency spectrum to generate mel-frequency cepstrum coefficients by discrete cosine transform;

[0143] Step S1614, extracting audio features representing emotions from the Mel-frequency spectrogram or the Mel-frequency cepstral coefficients through a convolutional neural network.

[0144] In this embodiment, first, the pet sound signal is converted into a spectrogram by short-time Fourier transform (STFT). STFT can convert the audio signal from the time domain to the frequency domain by windowing the time series signal, thereby capturing the frequency changes of the sound at different time points.

[0145] The spectrogram can represent the frequency components of an audio signal within a specific time period, which helps capture the time-frequency characteristics of the sound, and in particular can reveal the dynamic fluctuations in sound frequency when a pet's emotions change.

[0146] Then, the frequency axis of the spectrogram obtained in step S1611 is converted to a Mel frequency scale, thereby obtaining a Mel spectrogram. The Mel frequency scale is a frequency scale that is more consistent with human auditory perception, and it represents the frequency in a nonlinear manner to better reflect the perceived difference between low-frequency sounds and high-frequency sounds.

[0147] Mel-spectrogram is better able to capture emotion-related audio features because it highlights the areas in the sound signal that are sensitive to sounds by humans or animals, and is particularly suitable for processing audio data of pet emotions.

[0148] Next, cepstral analysis is applied to the mel-spectrogram to generate mel-frequency cepstral coefficients (MFCCs) through discrete cosine transform (DCT). MFCCs are widely used features in audio processing and can effectively capture the spectral envelope information of sound.

[0149] MFCCs extracts complex spectrogram information into a set of compact feature vectors, which contain the main spectral information of the sound and can better represent the emotional characteristics of the sound. MFCCs have been widely used in speech recognition and sentiment analysis, and have high emotion recognition.

[0150] Finally, a convolutional neural network (CNN) is used to extract emotion-related audio features from the mel-spectrogram or mel-frequency cepstral coefficients (MFCCs). CNN is able to capture local features and spatial dependencies in the input data through its convolutional layers, thereby automatically identifying audio patterns related to emotions. A convolutional neural network (CNN) consists of convolutional layers, pooling layers, and fully connected layers, where:

[0151] Convolutional layer: The convolutional layer of a CNN performs a convolution operation on the input spectrogram or MFCCs to identify specific frequency patterns and time series changes in the audio signal that are related to emotions.

[0152] Pooling layer: Through the pooling operation, the feature map is further reduced, the key emotional features in the audio signal are extracted, and redundant information is removed.

[0153] Fully connected layer: Finally, the features extracted by the convolutional neural network are integrated into a feature vector through the fully connected layer for subsequent emotion classification or behavior analysis.

[0154] In this way, convolutional neural networks can automatically extract high-order features representing emotions in audio signals, which play an important role in subsequent pet emotion recognition.

[0155] In one embodiment, see Figure 6 , Figure 6 A flowchart diagram of another embodiment of the method for establishing a pet emotion recognition model of the present application is provided. Step S161, the step of using a convolutional neural network to extract audio features, expression features, and action features representing emotions comprises:

[0156] Step S1615, extracting the pet's facial region from the video or image to obtain a pet's expression image;

[0157] Step S1616, adjusting the size of the pet expression image and performing normalization processing;

[0158] Step S1617, extracting features of the pet's facial area in the pet's expression image layer by layer through a convolutional neural network to obtain expression features.

[0159] In step S1615: extracting the pet's facial region from the video or image to obtain the pet's expression image, the convolutional neural network identifies and extracts the pet's facial region from the video stream or static image. Using an object detection algorithm (such as a Haar feature cascade classifier or a deep learning facial detection model), the pet's face can be accurately identified and the region can be extracted as a separate image.

[0160] By extracting the pet's facial regions, the system is able to focus on the most expressive visual information, such as changes in the eyes, ears, and mouth. Small changes in these areas can often reflect the pet's emotional state.

[0161] In step S1616: resizing and normalizing the pet expression image, the extracted pet expression image is resized to a uniform size in this step to meet the input requirements of the convolutional neural network. This usually involves scaling the image to a fixed size (e.g., 224x224 pixels or 128x128 pixels).

[0162] Normalization is to normalize the image pixel values ​​and scale the pixel values ​​to between 0 and 1 (or between -1 and 1), thereby reducing the deviation of the value range and improving the convergence speed and stability of the model. Normalization can ensure that the network training is more efficient while avoiding the abnormal influence of specific pixel values.

[0163] In step S1617: the features of the facial area in the pet expression image are extracted layer by layer through the convolutional neural network to obtain the expression features. The normalized pet expression image is input into the convolutional neural network (CNN) to extract the expression features of the pet's facial area in the image layer by layer.

[0164] The convolution layer scans the pet's facial expression image by applying a convolution kernel (filter) to capture local features in the image. These features may include subtle muscle changes on the pet's face, the opening and closing of the eyes, the position of the ears, and the shape of the mouth. The convolution layer can effectively extract these subtle emotional clues.

[0165] After the convolution operation, nonlinearity is introduced through activation functions (such as ReLU), which enables the network to capture the complex nonlinear relationships in the image and further enhance the expressiveness of emotional features.

[0166] The pooling layer is used to reduce the size of the feature map, retain the most important features, and reduce data redundancy. The most common pooling method is max pooling, which can retain the local maximum features in the image and thus extract key emotional features, such as ears pricked up when nervous or eyes half-closed when relaxed.

[0167] After the convolutional and pooling layers, the feature maps are flattened and fed into the fully connected layer, which integrates these extracted expression features into a high-dimensional feature vector. This feature vector summarizes the emotional characteristics of the pet's face and is used for subsequent emotion classification and behavior analysis.

[0168] Through this layer-by-layer feature extraction, the convolutional neural network can automatically extract deep emotional information from expression images and form a feature vector with emotion distinguishing ability.

[0169] In one embodiment, see Figure 7 , Figure 7 A flowchart of an embodiment of a method for establishing a pet emotion recognition model of the present application is provided. Step S161, the step of using a convolutional neural network to extract audio features, expression features, and action features representing emotions comprises:

[0170] Step S1618, extracting image features of each frame in the pet video through a convolutional neural network, inputting the image features into a long short-term memory network to process the time series to obtain pet action features;

[0171] or,

[0172] Step S1619, applying a 3D convolutional neural network to perform convolution operations between multiple frames, extracting motion features at different time points to obtain pet motion features.

[0173] In this embodiment, based on a convolutional neural network (CNN) and its combination with time series processing, two different methods can be used to extract pet motion features.

[0174] Step S1618: The image features of each video frame are extracted through a convolutional neural network, and the time series is processed through a long short-term memory network (LSTM) to obtain pet action features. The video is decomposed into multiple frames, and each frame is input into the convolutional neural network as a separate image. CNN extracts image features related to the pet's action from each frame, which include the pet's posture, limb movement, dynamic changes of the head or tail, etc. at each moment.

[0175] Among them, the convolution layer: performs convolution operation on each frame of the image to extract spatial features in the image, such as the pet's posture and local motion features.

[0176] Pooling layer: Reduce the dimension of the feature map and retain the most important action-related features.

[0177] The extracted image features of each frame are used as time series data and input into the long short-term memory network (LSTM). LSTM is good at processing sequence data and can capture the temporal dependencies between video frames, thereby extracting the overall movement characteristics of the pet.

[0178] LSTM can capture the continuity and dynamic changes of actions through the memory and forget mechanism, and recognize the process of pets changing from one action to another. For example, LSTM can learn the action patterns of pets such as running, jumping or slow movement from multiple consecutive frames.

[0179] Through the time series processed by LSTM, the system can extract complete action features, reflecting the pet's movement pattern and its emotion-related dynamic characteristics in the entire video.

[0180] Step S1619: Apply 3D convolutional neural network (3D-CNN) to perform convolution operation between multiple frames, extract motion features at different time points, and obtain pet action features. 3D convolutional neural network (3D-CNN) is used to directly perform convolution operation between multiple consecutive video frames. Different from traditional 2D CNN, 3D CNN can perform convolution in both time and space dimensions to capture motion information between video frames.

[0181] 3D CNN can extract the motion features of pets at different time points by performing three-dimensional convolution operations on video frame sequences. This method can not only capture the spatial features in each frame (such as the posture of the pet at a specific moment), but also capture the dynamic changes between consecutive frames (such as the process of the pet moving from one position to another).

[0182] 3D convolution can simultaneously take into account the pet's temporal motion changes and spatial posture changes, integrating multi-dimensional information of time and space, which enables this method to more efficiently extract the pet's complex motion patterns, such as fast chasing or slow crawling.

[0183] The motion features extracted through 3D convolution operations can reflect the pet's motion changes and dynamic characteristics throughout the video, providing strong motion information support for emotion recognition.

[0184] In one embodiment, see Figure 8 , Figure 8 A flowchart diagram of another embodiment of the method for establishing a pet emotion recognition model of the present application is provided. Step S150, the step of using the third pet behavior data set to fine-tune the pre-trained model to obtain a pet emotion recognition model comprises:

[0185] Step S151, inputting the behavior data in the third pet behavior data set into a pre-training model, and the pre-training model outputs a prediction result according to the input behavior data.

[0186] In this step, the behavior data (such as audio, action, expression and other multimodal data) in the third pet behavior dataset is input into the pre-trained model. The pre-trained model has been pre-trained with other general animal behavior datasets and has certain feature extraction and emotion recognition capabilities.

[0187] Input behavioral data: These data enter the model after preprocessing (such as normalization, data enhancement, etc.), and the model outputs prediction results based on the input data, such as the pet’s current emotional state (such as anxiety, happiness, fear, etc.).

[0188] Step S152, using the loss function to calculate the error between the pre-trained model output and the true label to obtain the loss value.

[0189] In this step, the prediction results output by the model are compared with the true emotion labels predefined in the third pet behavior dataset, and the error between the two is calculated using the loss function to obtain the loss value.

[0190] Loss function: Common loss functions include cross entropy loss (for classification tasks) or mean squared error (for regression tasks). The loss function can quantify the difference between the model prediction and the true emotion label. The smaller the error, the more accurate the model's prediction.

[0191] Loss value: The loss value is an indicator of the current performance of the model, reflecting the degree of deviation between the model output and the true label. A higher loss value indicates that the performance of the model needs to be further optimized.

[0192] Step S153, according to the loss value, using the back propagation algorithm to calculate the gradient of each parameter in the pre-trained model, wherein the parameters of the pre-trained model include the convolution kernel weight, the weight of the fully connected layer, and the bias of the fully connected layer.

[0193] In this step, the back propagation algorithm is used to calculate the gradient of the parameters of each layer in the model. The parameters of the pre-trained model include the convolution kernel weights in the convolutional neural network, the weights of the fully connected layers, and the bias of the fully connected layers.

[0194] Backpropagation: This is an algorithm used to train neural networks. By backpropagating the loss value to each layer of the network, the gradient of the model parameters relative to the loss value is calculated. These gradients represent the direction and magnitude of the change in the loss value with the parameters, guiding the adjustment direction of the model parameters.

[0195] Gradient calculation: For the convolution kernel weights, fully connected layer weights and biases, their gradients relative to the loss value are calculated. These gradients are the basis for the optimization algorithm to adjust the model parameters.

[0196] Step S154, using an optimization algorithm to adjust the parameters of the model according to the gradient, and / or adjust the hyperparameters, and executing the step of inputting the behavior data in the third pet behavior data set into the pre-trained model, and the pre-trained model outputting a prediction result according to the input behavior data, until the loss value reaches a preset value;

[0197] Among them, hyperparameters include learning rate, batch size, number of network layers, convolution kernel size, and regularization coefficient.

[0198] In this step, an optimization algorithm (such as Adam, SGD, etc.) is used to adjust the parameters of the model according to the calculated gradients, aiming to minimize the loss value.

[0199] Parameter adjustment: Based on the gradient value of each parameter, the optimization algorithm updates the weights and biases of the model so that the model output is closer to the true label.

[0200] Adjust hyperparameters: During model fine-tuning, you can also adjust some hyperparameters, including:

[0201] Learning rate: controls the step size of model parameter updates. A higher learning rate can speed up training but may lead to unstable convergence; a lower learning rate may slow down model convergence.

[0202] Batch size: refers to the number of data samples used each time the model parameters are updated. A larger batch size helps the model to be trained more stably, but requires more computing resources.

[0203] Number of network layers: The depth of the model. Too deep may lead to overfitting, while too shallow may cause the model to fail to learn enough features.

[0204] Convolution kernel size: The size of the convolution kernel affects the refinement of feature extraction.

[0205] Regularization coefficient: A hyperparameter used to prevent model overfitting and adjust the generalization ability of the model.

[0206] Repeat the input and adjustment process: After the parameters are adjusted, continue to input behavioral data into the model, recalculate the loss value and gradient, and adjust the parameters. This process is iterated continuously until the loss value reaches the preset threshold or the model converges.

[0207] In this embodiment, the data in the third pet behavior data set is input into the pre-trained model, the loss value between the prediction result and the true label is calculated, the parameter gradient is calculated using the back propagation algorithm, and the model parameters are adjusted by the optimization algorithm, and finally the pet emotion recognition model is fine-tuned. In this process, the adjustment of hyperparameters can further optimize the performance of the model and ensure that the model can accurately and stably identify the emotional state of the pet in practical applications.

[0208] In one embodiment, see Fig. 9 , Fig. 9 A flowchart diagram of another embodiment of the method for establishing a pet emotion recognition model of the present application is provided. Step S150, the step of using the third pet behavior data set to fine-tune the pre-trained model to obtain a pet emotion recognition model comprises:

[0209] Step S155, using a multi-task learning algorithm to build a multi-task model, wherein the multi-task model is used to recognize the pet's emotions, the pet's actions, and the pet's environment.

[0210] Task learning is a machine learning technique that allows the model to learn multiple related tasks simultaneously, share information, and improve the generalization ability of the model.

[0211] In this embodiment, the multi-task learning algorithm is used to build a model that can handle multiple tasks simultaneously. These tasks include:

[0212] Pet emotion recognition: Identify the pet’s current emotional state, such as happiness, anxiety, anger, relaxation, etc.

[0213] Pet action recognition: Identify pet actions in videos or images, such as running, standing still, jumping, eating, etc.

[0214] Environmental recognition: Identify the pet’s current environment, such as indoors, outdoors, noisy, quiet, etc., because environmental changes have a great impact on the pet’s emotions and behaviors.

[0215] This embodiment achieves multi-dimensional recognition of pet emotions, actions, and environments by building a multi-task model using a multi-task learning algorithm. Through the shared feature layer and the task-specific layer, the model can effectively extract emotion, action, and environment-related features from the pet's behavior data, and optimize the performance of each task at the same time. This method not only improves the accuracy of emotion recognition, but also provides more comprehensive behavior analysis and environmental perception, helping pet owners better understand their pets' behavior and emotional state.

[0216] In one embodiment, see Fig.10 , Fig.10 A flowchart diagram of another embodiment of the method for establishing a pet emotion recognition model of the present application is provided. Step S150, using the third pet behavior data set to fine-tune the pre-trained model to obtain a pet emotion recognition model comprises:

[0217] Step S156, using a regularization method to control and constrain the parameters or structure of the model to prevent the model from overfitting on the training data.

[0218] In the process of training the pet emotion recognition model, the model may perform well on the training data but poorly on the new data, i.e., overfitting. To prevent this from happening, the regularization method is used in this embodiment to control and constrain the complexity of the model, thereby improving the generalization ability of the model.

[0219] In this embodiment, L1 regularization, L2 regularization, and Dropout regularization may be used to prevent the model from overfitting on the training data.

[0220] L1 regularization constrains the absolute value of model parameters and adds a penalty term for the absolute value of parameters in the loss function, which can force some weights to approach zero, thereby achieving parameter sparsity. It is suitable for feature selection and reducing model complexity. In the pet emotion recognition model, it can remove unimportant features, thereby reducing the risk of overfitting.

[0221] L2 regularization constrains the square of the model parameters and adds a penalty term of the square of the parameters to the loss function. L2 regularization limits the size of the weight value so that the model weight does not become too large, thereby preventing over-reliance on specific features. This smooth weight distribution helps the model perform better on new data and improves its generalization ability.

[0222] Dropout is a structural regularization method that randomly discards a certain proportion of neurons during training to prevent the model from relying on specific neurons or features. In each training iteration, some neurons are randomly removed so that the model can learn more robust features. It can effectively prevent the model from overfitting, especially in deep neural networks. For the pet emotion recognition model, Dropout can prevent the model from relying too much on a certain type of input feature when extracting audio, expression or action features, thereby improving the generalization of the model.

[0223] The technical solution proposed in this application has at least the following technical effects:

[0224] The present invention obtains pet behavior data, including body movements, sounds, expressions, etc. The pet behavior data is input into a pet emotion recognition model to identify the pet's current emotion type. And according to the pet's current emotion type, a soothing strategy corresponding to the current emotion type is determined. The pet is soothed based on the soothing strategy, and the pet's emotions are identified from multiple dimensions to improve the accuracy of pet emotion recognition, so as to judge whether the pet is in an angry, happy, or calm state. According to the different emotional states of the pet, the corresponding strategy is selected for soothing, which can more effectively relieve the pet's anxiety, tension and other emotions.

Claims

1. A pet soothing method, applied to a pet soothing device, characterized in that: The method includes: Obtain pet behavior data, and input the pet behavior data into a pet emotion recognition model to identify the pet's current emotion type; According to the current emotion type of the pet, determining a soothing strategy corresponding to the current emotion type; The pet is soothed based on the soothing strategy.

2. The method according to claim 1, characterized in that The pet soother includes a speaker and a vibrator, and in the step of determining a soothing strategy corresponding to the pet emotion recognition result according to the pet emotion recognition result: The soothing strategy includes playing a soothing sentence generated according to the pet owner's timbre and controlling the vibrator to operate in a corresponding vibration mode according to the pet's current emotional type.

3. The method according to claim 1, characterized in that The pet emotion recognition model is established by the following steps: Acquire a first pet behavior data set, wherein the first pet data set includes pet sound data, action data, expression data, and environmental information; labeling the sound data, the action data, and the expression data in the first pet behavior data set with emotion tags to obtain a second pet behavior data set; Performing data cleaning and data enhancement on the second pet behavior dataset to obtain a third pet behavior dataset; Use animal behavior datasets to pre-train a general neural network to obtain a pre-trained model; The pre-trained model is fine-tuned using the third pet behavior dataset to obtain a pet emotion recognition model.

4. The method according to claim 3, characterized in that After the step of labeling the sound data, the action data and the expression data in the first pet behavior data set with emotion tags to obtain the second pet behavior data set, the following step further comprises: A behavior classifier is constructed to extract pet behavior features, identify pet emotions, and classify pet emotions.

5. The method according to claim 4, characterized in that The steps of constructing a behavior classifier, extracting pet behavior features through the behavior classifier, identifying pet emotions, and classifying pet emotions include: Use convolutional neural networks to extract audio features, facial expressions, and action features that represent emotions; Using long short-term memory networks to process the time series characteristics of audio and / or video; Construct a fusion layer to fuse the pet audio features, expression features, and action features to obtain fusion features; A behavior classifier is constructed according to the fusion features.

6. The method according to claim 5, characterized in that The step of using a convolutional neural network to extract audio features, expression features, and action features representing emotions includes: Perform short-time Fourier transform on the audio signal to obtain a spectrum diagram; Convert the frequency axis in the spectrogram to the Mel frequency scale to obtain a Mel spectrogram; Apply cepstral analysis to the mel-frequency spectrogram and generate mel-frequency cepstral coefficients through discrete cosine transform; The audio features representing emotions in the Mel-frequency spectrogram or Mel-frequency cepstral coefficients are extracted through convolutional neural networks.

7. The method according to claim 5, characterized in that The step of using a convolutional neural network to extract audio features, expression features, and action features representing emotions includes: Extract the pet's facial area from the video or image to obtain the pet's expression image; Resize and normalize pet expression images; The features of the pet's facial area in the pet expression image are extracted layer by layer through a convolutional neural network to obtain the expression features.

8. The method according to claim 5, characterized in that The step of using a convolutional neural network to extract audio features, expression features, and action features representing emotions includes: The image features of each frame in the pet video are extracted through a convolutional neural network, and the image features are input into a long short-term memory network to process the time series to obtain the pet action features; or, A 3D convolutional neural network is used to perform convolution operations between multiple frames to extract motion features at different time points to obtain pet motion features.

9. The method according to claim 3, characterized in that The step of using the third pet behavior data set to fine-tune the pre-trained model to obtain a pet emotion recognition model comprises: Inputting the behavior data in the third pet behavior data set into a pre-training model, wherein the pre-training model outputs a prediction result according to the input behavior data; Use the loss function to calculate the error between the pre-trained model output and the true label to get the loss value; According to the loss value, a back propagation algorithm is used to calculate the gradient of each parameter in the pre-trained model, wherein the parameters of the pre-trained model include convolution kernel weights, fully connected layer weights, and fully connected layer biases; Using an optimization algorithm to adjust the parameters of the model according to the gradient, and / or adjusting the hyperparameters, the step of inputting the behavior data in the third pet behavior data set into the pre-trained model, wherein the pre-trained model outputs a prediction result according to the input behavior data, is performed until the loss value reaches a preset value; Among them, hyperparameters include learning rate, batch size, number of network layers, convolution kernel size, and regularization coefficient.

10. The method according to claim 3, characterized in that The step of using the third pet behavior data set to fine-tune the pre-trained model to obtain a pet emotion recognition model comprises: A multi-task learning algorithm is used to build a multi-task model, wherein the multi-task model is used to recognize the pet's emotions, the pet's actions, and the pet's environment; Regularization methods are used to control and constrain the parameters or structure of the model to prevent the model from overfitting on the training data.

Images