Pet emotion recognition method and device based on motion sensing

Abstract

The invention provides a pet emotion recognition method and device based on motion sensing, and the method comprises the following steps: 1, enabling motion sensors disposed at four limbs of a pet to obtain motion sensing signals; 2, performing Kalman filtering processing on the motion sensing signal, and calculating an amplitude change rate of the motion sensing signal; 3, respectively fusing the amplitude change rate signals obtained by the four motion sensors, and calculating second-order signals of the amplitude change rate signals; step 4, performing feature extraction on the second-order signal, wherein the extracted features comprise time domain features and frequency domain features; and step 5, judging the features of the feature extraction module, selecting the most suitable features to identify the limb movement of the pet, and identifying the emotion of the pet according to the limb movement. The method has the beneficial effects that the emotion of the pet is judged according to the motion state of the pet, and the method has the advantages of convenience in use, low cost, no environmental interference, high accuracy and the like.

Description

Technical Field

[0001] The present invention relates to the field of pet information technology, and in particular to a pet emotion recognition method and device based on motion sensing. Background Art

[0002] With the development of society, more and more people have pets to relieve loneliness. Pets can enrich human life and promote people's physical and mental health. The emotions of pets are affected by various factors. Experienced pet owners can accurately judge the emotions of pets by the tone and volume of their voices and the movements of their pets, so as to provide effective comfort methods. However, this ability is not possessed by every pet owner. Most pet owners often cannot effectively identify the emotions expressed by their pets, resulting in communication barriers, causing the pets to be out of control, and the pets may bite family members or others. In addition, in the process of raising pets, it is impossible for pet owners to accompany their pets all the time. From work to get off work, it is impossible to observe the emotional state of pets in real time. Therefore, how to accurately identify the emotions of pets is one of the important issues in pet management.

[0003] At present, pet emotion recognition methods are mainly based on video or audio data analysis, obtaining pet action data through cameras or obtaining pet sound data through microphones. However, obtaining video data requires installing cameras in fixed places. When the pet leaves the area, it can no longer be recognized. Audio data is often easily disturbed by surrounding noise, resulting in inaccurate recognition and low accuracy. Summary of the invention

[0004] In view of the current difficulties in pet emotion recognition, the present invention proposes a pet emotion recognition method based on motion sensing. The method mainly obtains the pet's body movement status in real time through motion sensors, and analyzes the pet's emotions according to its movement status. The method has the advantages of easy use, low cost, high accuracy, etc., and is achieved through the following technical solution.

[0005] A pet emotion recognition method based on motion sensing includes the following steps: Step 1: Place motion sensors on the pet's limbs to obtain motion sensing signals; Step 2: Process the motion sensor signal through Kalman filtering to calculate the amplitude change rate of the motion sensor signal; Step 3: The amplitude change rate signals obtained by the above four motion sensors are fused with each other to calculate the second-order signals; Step 4: Extract features of the second-order signal, where the extracted features include time domain features and frequency domain features; Step 5: Judge the features of the feature extraction module, select the most appropriate features to identify the pet's body movements, and identify the pet's emotions based on the body movements.

[0006] A pet emotion recognition device comprises a motion acquisition module, a signal preprocessing module, a signal fusion module, a feature extraction module, a feature selection module and an emotion recognition module, wherein: The signal preprocessing module mainly implements Kalman filtering on the acquired motion sensor signal, and on this basis, calculates the amplitude change rate of the motion sensor signal; The motion acquisition module mainly acquires the motion sensing signals of the pet's limbs through motion sensors; The signal fusion module is mainly used to fuse the amplitude change rate signals obtained by the above four motion sensors and calculate their second-order signals; The feature extraction module mainly extracts features of the second-order signal. The extracted features include time domain features and frequency domain features. The feature selection module mainly judges the features of the feature extraction module and selects the most appropriate features to identify the pet's body movements; The pet emotion recognition module mainly makes judgments based on the features obtained by the feature selection module, thereby realizing pet emotion recognition.

[0007] The beneficial effects of the present invention are: judging the mood of a pet according to the movement state of the pet, and having the advantages of being easy to use, low cost, not being affected by the environment, and high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 4 is a flow chart of a pet emotion recognition method in an embodiment of the present invention.

[0009] Figure 2 It is a Kalman filter processing flow chart in an embodiment of the present invention.

[0010] Figure 3 It is a flowchart of extracting time domain features from a second-order signal in an embodiment of the present invention.

[0011] Figure 4 It is a flow chart of extracting frequency domain features from a second-order signal in an embodiment of the present invention. DETAILED DESCRIPTION

[0012] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0013] A pet emotion recognition device comprises a motion acquisition module, a signal preprocessing module, a signal fusion module, a feature extraction module, a feature selection module and an emotion recognition module.

[0014] The signal preprocessing module mainly implements Kalman filtering on the acquired motion sensor signal, and on this basis, calculates the amplitude change rate of the motion sensor signal.

[0015] The motion acquisition module mainly acquires the motion sensing signals of the pet's limbs through motion sensors.

[0016] The signal fusion module is mainly used to fuse the amplitude change rate signals obtained by the above four motion sensors respectively and calculate their second-order signals.

[0017] The feature extraction module is mainly used to extract features of the second-order signal. The extracted features include time domain features and frequency domain features.

[0018] The feature selection module is mainly used to judge the features of the feature extraction module and select the most appropriate features to identify the pet's body movements.

[0019] The pet emotion recognition module mainly makes judgments based on the features obtained by the feature selection module, thereby realizing pet emotion recognition.

[0020] like Figure 1 As shown, a pet emotion recognition method based on motion sensing includes the following steps: S1, motion sensors are configured on the pet's limbs to obtain motion sensing signals; S2, processing the motion sensor signal through Kalman filtering, and calculating the amplitude change rate of the motion sensor signal; S3, fusing the amplitude change rate signals obtained by the above four motion sensors respectively, and calculating the second-order signals thereof; S4, extracting features from the second-order signal, where the extracted features include time domain features and frequency domain features; S5. Judge the features of the feature extraction module, select the most appropriate features to identify the pet's body movements, and identify the pet's emotions based on the body movements.

[0021] The above steps are described below according to an embodiment of the present invention.

[0022] In step S1, a motion sensor needs to be configured. The motion sensor can be integrated into a flexible webbing and worn on the limbs of the dog, or integrated into a pet shoe, etc. The motion sensor is a device integrated with a gyroscope, and has the function of detecting motion and posture.

[0023] In this patent, the pet needs to wear at least one motion sensor. In order to better measure the pet's limb movements, a motion sensor is worn on each of the pet's limbs in this patent, which are named motion sensor 1, motion sensor 2, motion sensor 3, and motion sensor 4. After the pet's limbs are equipped with motion sensors, the motion sensing signals of the limbs in the three directions of X, Y, and Z can be obtained.

[0024] When the sampling frequency of the motion sensor is F sp Hz, within 60 seconds (it can also be set to other time lengths, such as 120 seconds, 300 seconds, etc., here the explanation is based on the time length of 60 seconds), you can get F in the X, Y, and Z directions respectively sp × 60 data. After data collection is completed, the data is sent to the signal preprocessing module for processing.

[0025] like Figure 2 As shown, in step S2, the Kalman filter processes the motion sensor signal to obtain the amplitude change rate of the motion sensor signal.

[0026] S21, taking the sensing signals of the motion sensor in the X, Y and Z directions as inputs respectively, and calculating the pre-estimated optimal value equation according to the previous optimal value of the Kalman filter model.

[0027] S22, calculating the mean square error of the pre-estimated optimal value according to the pre-estimated optimal value equation.

[0028] S23, calculating the Kalman gain matrix according to the mean square error of the pre-estimated optimal value.

[0029] S24, updating the optimal value equation according to the obtained Kalman gain matrix to obtain the estimated optimal value at the current moment. By repeating the above steps, Kalman filtering of the motion sensor signal is finally achieved.

[0030] Assume that after Kalman filtering, the motion sensor signal of motion sensor 1 in the X direction is {x 1,1 , x 1,2 ,x 1,3 ,..., x 1,n}, n represents the total number of elements in the set, n= F sp ×60. The amplitude change rate is obtained by subtracting the previous element value from the next element value, and dividing the difference by the previous element value. Therefore, the amplitude change rate of the motion sensor 1 in the X direction can be expressed as the following set: {χ 1,1 , χ 1,2 , χ 1,3 ,... 1,n-1}, where χ1,1=(x 1,2 -x 1,1 ) / x1,1 , χ 1,2 =(x 1,3 -x 1,2 ) / x 1,2 , the calculation method of the remaining elements is similar to the inference. According to the above calculation method, the amplitude change rate of the motion sensor 1 in the X, Y, and Z directions can be expressed as a matrix S1, as shown below:

[0031] where {γ 1,1 , γ 1,2 , γ 1,3 ,..., γ 1,n-1} represents the amplitude change rate set of motion sensor 1 in the Y direction; {ζ 1,1 , ζ 1,2 , ζ 1,3 , ..., ζ 1,n-1} represents the set of amplitude change rates of motion sensor 1 in the Z direction.

[0032] Similarly, the amplitude change rates of motion sensor 2, motion sensor 3, and motion sensor 4 in the X, Y, and Z directions can be represented by matrices S2, S3, and S4.

[0033]

[0034]

[0035] In step S3, the amplitude change rate signals obtained by the four motion sensors in step S2 are fused with each other to calculate the second-order signals.

[0036] The amplitude change rate signals S1, S2, S3, and S4 obtained by the above four motion sensors are fused with each other to calculate their second-order signals. After the fusion of S1 and S2, the second-order signal obtained is expressed as follows: In step S3, the amplitude change rate signals obtained by the four motion sensors in step S2 are fused with each other to calculate the second-order signals.

[0037] The amplitude change rate signals S1, S2, S3, and S4 obtained by the above four motion sensors are fused with each other to calculate their second-order signals. After the fusion of S1 and S2, the second-order signal obtained is expressed as follows: , the output result is is a (n-1)×(n-1) matrix, For the matrix The transposed matrix of Represents the cross product operation, which can be expressed as: in , Other results are calculated similarly.

[0038] Similarly, the fused second-order signal S1 and S3 13 , the second-order signal S after the fusion of S1 and S4 14 , the second-order signal S after the fusion of S2 and S3 23 , the second-order signal S after the fusion of S2 and S4 24 , the second-order signal S after the fusion of S3 and S4 34 It can be expressed as follows:

[0039]

[0040]

[0041] like Figure 3 In step S4, the second-order signal S 12 , S 13 , S 14 , S 23 , S 24 , S 34 Perform feature extraction, the extracted features include time domain features and frequency domain features. Time domain features are mainly extracted through turning points and trend segments. The specific process is as follows: S401: For the second-order signal S 12 For each row of , determine the turning point of the row. First, calculate the standard deviation of all elements in the row and use the standard deviation as the threshold for judging fluctuations.

[0042] S402: For each element value of the row, subtract it from the previous element value of the row to obtain the difference. If the difference is less than the threshold value set in step 1, the element value is an invalid fluctuation value. If the difference is not less than the threshold value set in step 1, the element value is a valid fluctuation value. When it is a valid fluctuation value, if the difference is greater than 0, it indicates that the signal is in a rising state. If the difference is not greater than 0, it indicates that the signal is in a falling state. When the signal changes from a rising state to a falling state, or from a falling state to a rising state, the element point is marked as a turning point. According to this method, all turning points of the elements in the row are obtained, and their positions are recorded.

[0043] S403: Calculate the position distances between different turning points, and the difference between the position distances is defined as the trend segment.

[0044] S404: Different weights are assigned according to the length of the trend segment. If the trend segment is in an upward state, the weight is 0 to 1, and the longer the trend segment is, the greater the weight is. If the trend segment is in a downward state, the weight is -1 to 0, and the longer the trend segment is, the smaller the weight is.

[0045] S405: Based on the result obtained in S404, a comprehensive evaluation is performed on the trend change to obtain a final change trend value.

[0046] According to S401 to S405, the second-order signal S is calculated respectively. 12 The final change trend value of each row and column.

[0047] Repeat the above steps to calculate other second-order signals such as S 12 , S 13 , S 14 , S 23 , S 24 , S 34 The final change trend value of each row and column.

[0048] like Figure 4 As shown, in terms of frequency domain features, the spectral features are mainly extracted, and the specific process is as follows: S411: For the second-order signal S 12 Each line of is framed and windowed. The frame signal is obtained by dividing the segment signal into several frames of length L. At the same time, an overlapping area (usually 50% of the frame length) needs to be set between adjacent frames to prevent the loss of continuous dynamic change information in the signal.

[0049] S412: In order to increase the continuity of the left and right ends of the frame signal and reduce the "spectral leakage" caused by frame truncation, each frame signal is multiplied by a window function to obtain a windowed signal. The window function can be a rectangular window, a Hanning window, a Hamming window, etc.

[0050] S413: Perform fast Fourier transform on the windowed signal to obtain its spectrum, and further modulate the spectrum to obtain the amplitude spectrum of the signal. In the specified frequency analysis range, use N filter groups to perform bandpass filtering on the amplitude spectrum to finally obtain its amplitude spectrum eigenvalue.

[0051] S414: According to S411 to S413, the second-order signal S is calculated respectively. 12 The amplitude spectrum eigenvalues ​​of each row and column of .

[0052] According to S411 to S414, other second-order signals such as S are calculated. 12 , S 13 , S 14 , S 23 , S 24 , S 34The amplitude spectrum eigenvalues ​​of each row and column of .

[0053] In an embodiment of the present invention, based on a feature selection method, the time domain features and frequency domain features extracted by the feature extraction module are analyzed, and if the features meet the following two conditions, they are identified as valid features.

[0054] This feature ranks in the top 50% of all features in terms of feature importance; This statistical feature is selected simultaneously by three different feature selection algorithms, including the univariate feature selection algorithm (UFS), the recursive feature elimination algorithm (RFE), and the L1-based feature selection algorithm (L1-FS). That is, this feature is the intersection of the above three feature selection algorithms.

[0055] The algorithms used for analyzing time domain features and frequency domain features are: one or more machine learning methods such as gradient reinforcement tree, random forest, support vector machine, naive Bayes, etc. The input is the characteristics of the sensor signal obtained by the motion sensor, and the output is the pet's current emotional state and intensity.

[0056] According to the pet's movement state, which is equivalent to human body language, it corresponds to the pet's emotions.

[0057] Taking pet dogs as an example, dogs can express their emotions in many ways, including common emotions.

[0058] When it is happy, it may wag its tail, jump, make bright barking sounds, and run to its owner to lick him.

[0059] When acting coquettishly, they may rub against people, lick their owner's hands, and give their owners a begging look.

[0060] When being friendly, the dog may show its belly, strike poses, shake its body, sit on the ground with its two front paws propped up, or lie flat with its belly exposed.

[0061] Anxiety may manifest itself by hunching over and lowering the ears.

[0062] Threats may be expressed by barking and leaning back.

[0063] Stress relief can be achieved by shaking the body.

[0064] These movements are easily recognized by motion sensors and generate gesture data.

[0065] Pet cats also have a wide variety of emotional expressions, such as: Arched back and hair standing up: This is a sign that the cat senses danger and is about to attack. It may be afraid or angry.

[0066] Approaching you with its tail standing upright: This means the cat is being coquettish with you or wants food.

[0067] Standing or sitting with ears erect: At this time, the cat's expression will be particularly alert. It may be observing and thinking about something, and it is vigilant and curious about its surroundings.

[0068] Lying down to sleep: When a cat sleeps, it lies down with its belly on and sometimes sleeps with its arms and legs spread out. This means that it is no longer wary of you and is very relaxed and happy.

[0069] In addition, cats may also express their emotions through other actions, including ears flat, body arched, tail straight up, pupils shrunk into a straight line, ears back, etc. It is necessary to judge and understand the emotional state of cats in combination with specific situations, and each cat may express emotions differently.

[0070] The output emotional states include five types: happiness, mania, fear, sadness, and surprise. The type of pet and the pet's actions are matched to the five emotional states. The emotional state of the pet is first identified based on the actions.

[0071] It is then divided into levels from 1 to 10 based on the intensity of the action. The higher the level, the stronger the emotional state.

[0072] Based on the output results of the pet emotion recognition module, pet owners can promptly learn about the emotional state of their pets and take targeted actions and behaviors based on the emotional state to maintain the health of their pets.

[0073] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A pet emotion recognition method based on motion sensing, comprising the following steps: Step 1: Place motion sensors on the pet's limbs to obtain motion sensing signals; Step 2: Perform Kalman filtering on the motion sensor signal to calculate the amplitude change rate of the motion sensor signal; Step 3: The amplitude change rate signals obtained by the above four motion sensors are fused with each other to calculate the second-order signals; Step 4: Extract features of the second-order signal, where the extracted features include time domain features and frequency domain features; Step 5: Judge the features of the feature extraction module, select the most appropriate features to identify the pet's body movements, and identify the pet's emotions based on the body movements.

2. The pet emotion recognition method according to claim 1, characterized in that: In step 1, after 1-4 motion sensors are attached to the pet's limbs, motion sensing signals of each motion sensor along the limbs in the three directions of X, Y, and Z are obtained.

3. The pet emotion recognition method according to claim 1, characterized in that: In step 2, the steps of Kalman filtering the motion sensor signal are: First, the sensor signals of the motion sensor in the X, Y, and Z directions are input into the Kalman filter model respectively, and the estimated optimal value equation is calculated according to the previous optimal value of the Kalman filter model; Secondly, according to the pre-estimated optimal value equation, the mean square error of the pre-estimated optimal value is calculated; Again, the Kalman gain matrix is ​​calculated based on the mean square error of the pre-estimated optimal value; Finally, the optimal value equation is updated according to the obtained Kalman gain matrix to obtain the estimated optimal value at the current moment, and the above steps are repeated to finally realize the Kalman filtering of the motion sensor signal.

4. The pet emotion recognition method according to claim 3, characterized in that: The calculation method for obtaining the amplitude change rate of the motion sensor signal is: subtract the previous element value from the next element value in the motion sensor signal set, and divide the difference by the previous element value.

5. The pet emotion recognition method according to claim 1, characterized in that: In step 4, the time domain features are extracted through turning points and trend segments, which specifically includes the following steps: Step 1: Calculate the standard deviation of all elements in each row of the second-order signal and use the standard deviation as the threshold for determining fluctuations; Step 2: Subtract the value of an element in each row from the value of the previous element in the row, compare it with the threshold to obtain the rising and falling states of the signal, mark the element points of the rising and falling states of the signal as turning points, obtain all the turning points of the elements in the row, and record their locations; Step 3: Calculate the position distances between different turning points, and the difference between the position distances is defined as the trend segment; Step 4: Assign different weights according to the length of the trend segment; Step 5: Based on the results obtained in step 4, conduct a comprehensive evaluation of the trend change to obtain the final change trend value.

6. The pet emotion recognition method according to claim 1, characterized in that: The steps to extract spectral features in the frequency domain are: Step 1: Framing each line of the second-order signal, dividing the signal into several frames of length L, and obtaining a frame signal; Step 2: Multiply each frame signal by the window function to obtain the windowed signal; Step 3: Perform fast Fourier transform on the windowed signal to obtain its spectrum, and further modulo the spectrum to obtain the amplitude spectrum of the signal.

7. The pet emotion recognition method according to any one of claims 5 or 6, characterized in that: The time domain features and frequency domain features extracted by the feature extraction module are analyzed. If the features meet the following two conditions, they are considered valid features: a. The feature ranks in the top 50% in terms of importance among all features; b. The statistical feature is selected simultaneously by three different feature selection algorithms, including univariate feature selection algorithm, recursive feature elimination algorithm, and L1-based feature selection algorithm (L1-FS).

8. A pet emotion recognition device, characterized in that: The method for identifying pet emotions according to any one of claims 1 to 9 is configured in a wearable device on the pet's limbs, wherein the wearable device comprises a motion acquisition module, a signal preprocessing module, a signal fusion module, a feature extraction module, a feature selection module, and an emotion recognition module, wherein: The signal preprocessing module implements Kalman filtering on the acquired motion sensor signal, and on this basis, calculates the amplitude change rate of the motion sensor signal; A motion acquisition module acquires motion sensing signals of the pet's limbs through motion sensors; A signal fusion module is used to fuse the amplitude change rate signals obtained by the above four motion sensors respectively and calculate the second-order signals; A feature extraction module is used to extract features of the second-order signal respectively, and the extracted features include time domain features and frequency domain features; The feature selection module mainly judges the features of the feature extraction module and selects the most appropriate features to identify the pet's body movements; The pet emotion recognition module mainly makes judgments based on the features obtained by the feature selection module, thereby realizing pet emotion recognition.

Images