Animal emotion recognition method and device based on millimeter waves

Abstract

The invention provides an animal emotion recognition method and device based on millimeter waves. One specific implementation mode of the method comprises the steps that a millimeter wave signal sequence collected by millimeter wave radar equipment for a target animal is acquired; generating a three-dimensional point cloud sequence based on the millimeter wave signal sequence, and generating behavior feature sequence information based on the three-dimensional point cloud sequence; on the basis of the millimeter wave signal sequence, physiological feature information of the target animal is determined, and the physiological feature information comprises feature information of heart rate variability and the heart rate; and inputting the behavior feature sequence information and the physiological feature information into a pre-trained animal emotion recognition model, performing emotion recognition by the animal emotion recognition model, and outputting emotion information of the target animal.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of animal health monitoring, and in particular to a millimeter wave-based animal emotion recognition method and device. BACKGROUND

[0002] With the continuous development of society and the change of people's living state, more and more people spend considerable time with their favorite animals (e.g., pets). With the increasing attention of people to animals, animal mental health problems have gradually been taken seriously. Studies have shown that long-term anxiety, fear or depression not only affects the behavior of animals (such as excessive licking, increased aggressiveness, etc.), but also can induce physiological problems such as immune system disorders and digestive system diseases. However, animals cannot express their emotional state through language, and the traditional method of relying on observation by veterinarians or subjective judgment by owners has a lag and a risk of misjudgment. The current mainstream animal emotion monitoring technology mainly relies on wearable devices (such as smart collars) to collect activity or heart rate variability data through accelerometers, heart rate sensors, etc., but these methods require animals to always wear smart devices, and the discomfort caused by wearing smart devices hinders their widespread use in daily activities. Therefore, there is an urgent need for an emotion recognition method that is more suitable for animals. SUMMARY

[0003] In view of this, the embodiments of the present application provide a millimeter wave-based animal emotion recognition method and device to eliminate or improve one or more defects in the prior art.

[0004] According to a first aspect, a millimeter wave-based animal emotion recognition method is provided, wherein the method comprises: acquiring a millimeter wave signal sequence collected by a millimeter wave radar device for a target animal; generating a three-dimensional point cloud sequence based on the above-mentioned millimeter wave signal sequence, and generating behavior feature sequence information based on the above-mentioned three-dimensional point cloud sequence; determining physiological feature information of the target animal based on the above-mentioned millimeter wave signal sequence, wherein the physiological feature information includes heart rate variability feature information and heart rate; inputting the behavior feature sequence information and the physiological feature information into a pre-trained animal emotion recognition model, and outputting emotion information of the target animal by emotion recognition of the animal emotion recognition model.

[0005] According to a second aspect, a millimeter wave based animal emotion recognition device is provided, comprising: an acquisition unit configured to acquire a millimeter wave signal sequence collected by a millimeter wave radar device for a target animal; a generation unit configured to generate a three-dimensional point cloud sequence based on the millimeter wave signal sequence, and generate behavior feature sequence information based on the three-dimensional point cloud sequence; a determination unit configured to determine physiological feature information of the target animal based on the millimeter wave signal sequence, wherein the physiological feature information comprises heart rate variability feature information and heart rate; and a recognition unit configured to input the behavior feature sequence information and the physiological feature information into a pre-trained animal emotion recognition model, and output emotion information of the target animal by emotion recognition performed by the animal emotion recognition model.

[0006] According to a third aspect, an electronic device is provided, comprising a processor, a memory, and a computer program / instruction stored on the memory, wherein the processor is configured to execute the computer program / instruction, and the electronic device implements the steps of the method according to any one of the first aspect when the computer program / instruction is executed.

[0007] According to a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program, when executed in a computer, causes the computer to perform the method according to any one of the implementations of the first aspect.

[0008] The millimeter wave based animal emotion recognition method and device provided by the embodiments of the present disclosure can first acquire a millimeter wave signal sequence collected by a millimeter wave radar device for a target animal. Then, a three-dimensional point cloud sequence is generated based on the millimeter wave signal sequence, and behavior feature sequence information is generated based on the three-dimensional point cloud sequence. Then, physiological feature information of the target animal is determined based on the millimeter wave signal sequence, and the physiological feature information comprises heart rate variability feature information and heart rate. Finally, the behavior feature sequence information and the physiological feature information are input into a pre-trained animal emotion recognition model, and emotion information of the target animal is output by emotion recognition performed by the animal emotion recognition model. Thus, the pet emotion is monitored without sensing based on the millimeter wave signal collected by the millimeter wave radar device, and the pet does not need to wear any device, which is more suitable for application in animal emotion recognition. In addition, the animal emotion recognition is performed by combining the behavior feature sequence information and the physiological feature information, so that the emotion recognition result is more accurate.

[0009] Additional advantages, objects, and features of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following or can be learned by practice of the application. The objects and other advantages of the application can be realized and attained by the structure particularly pointed out in the description and claims hereof as well as the appended drawings.

[0010] Those skilled in the art will understand that the objects and advantages of the application can be realized and attained by means of the subject-matter as described in the following detailed description and as illustrated in the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the detailed description serve to explain the principles of the application.

[0012] Figure 1 A flow chart of a method for millimeter wave-based animal emotion recognition is shown according to an embodiment; Figure 2 A schematic diagram of an example of a model structure of an animal emotion recognition model is shown; Figure 3 A schematic diagram of an application scenario to which embodiments of the present specification can be applied is shown; Figure 4 A schematic block diagram of a millimeter wave-based animal emotion recognition device according to an embodiment is shown. DETAILED DESCRIPTION

[0013] To make the objects, technical solutions and advantages of the application clearer, the application will be further described in detail below with reference to the embodiments and drawings. Here, the illustrative embodiments of the application and their descriptions are used to explain the application, but are not intended to limit the application.

[0014] It should also be noted that, in order to avoid obscuring the application due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the application are shown in the drawings, and other details not closely related to the application are omitted.

[0015] It should be emphasized that the term "comprises / comprising" as used herein means the presence of the stated features, elements, steps or components, but does not preclude the presence or addition of one or more other features, elements, steps or components.

[0016] It should also be noted that, unless otherwise specified, the term "connected" as used herein can not only mean direct connection, but also indirect connection in the presence of an intermediate.

[0017] It should be understood that the ordinal numbers "first", "second", etc. mentioned in the present specification are only used to distinguish a plurality of objects (such as components, steps, parameters, etc.) of the same category or different categories, and do not represent priority, importance or sequence relationship between the objects, nor constitute a limitation on the technical features.

[0018] As mentioned earlier, there is an urgent need for an emotion recognition method that is more suitable for animals at this stage.

[0019] Existing long-term emotion monitoring solutions include questionnaire filling and wearable device measurement. Among them, the questionnaire filling method is for subjects to record their stress levels by filling out a carefully designed questionnaire regularly, and further assess the subjects' recent emotional state by stress. However, this method is designed for humans and obviously cannot be applied to animals. The wearable device measurement method mainly relies on wearable devices to collect animal activity or heart rate variability data. However, this method has some obvious limitations, for example, the rejection behavior of animals (such as scratching, biting, etc.) to external devices such as collars may cause data distortion. For example, different breeds of animals have large physiological baseline differences (such as the normal heart rate range of a Chihuahua is 2 times that of a golden retriever), and the device needs to be calibrated frequently. For example, contact-type sensor measurement is easily affected by individual differences such as hair thickness and skin wrinkles.

[0020] Therefore, the embodiments of the present specification provide a millimeter wave-based animal emotion recognition method, which can realize non-invasive monitoring of pet emotions based on millimeter wave signals collected by a millimeter wave radar device, without the need for pets to wear any device, and is more suitable for application in animal emotion recognition.

[0021] Please refer to Figure 1 , Figure 1 A flowchart of a millimeter wave-based animal emotion recognition method according to an embodiment is shown. It can be understood that the method can be executed by any device, equipment, platform, device cluster with computing and processing capabilities. As Figure 1 shown, the millimeter wave-based animal emotion recognition method can include the following steps 101 to 104, specifically: Step 101, obtaining a millimeter wave signal sequence collected by a millimeter wave radar device for a target animal.

[0022] In this embodiment, the millimeter wave signal sequence for the target animal can be collected by a millimeter wave radar device. For example, a FMCW (Frequency-Modulated Continuous Wave) radar with a working frequency of 60 GHz (gigahertz) can be deployed in a position (such as a corner in a home environment) that can cover the main activity area of the animal. The FMCW radar can continuously emit millimeter wave signals to its monitoring area and receive reflected echoes generated by animal activities.

[0023] It can be understood that before formal data collection using the millimeter wave radar device, radar calibration can be performed. For example, a 0.1 m² corner reflector can be placed 1 m in front of the radar, and the ratio of the theoretical RCS (Radar Cross Section) to the measured RCS is recorded as the system gain correction coefficient G_cal. Then, multiply each frame of point cloud intensity by G_cal to complete the amplitude calibration. Spatial calibration uses a calibration frame with a known size (30 cm x 30 cm x 30 cm), extracts the three-dimensional coordinates of its corner points, and performs ICP (Iterative Closest Point) registration with the radar point cloud to obtain the rotation matrix R_cal and the translation vector t_cal. All high-confidence point clouds are uniformly transformed to the real-world coordinate system according to the following formula, i.e., the calibration is completed.

[0024] wherein P_radar represents the point cloud coordinates output by the radar, and P_world represents the coordinates in the real-world coordinate system.

[0025] In this example, based on the millimeter wave radar device, it has all-weather adaptability and environmental robustness, is not affected by environmental light changes and animal activity types, can effectively cover various behavior patterns from static to high-intensity motion, and is especially suitable for long-term and continuous monitoring in a home environment. While achieving high-precision emotion perception, it can strictly follow the privacy protection principle. The millimeter wave radar only collects coarse-grained point cloud and motion trajectory information and does not involve any optical image data, fundamentally eliminating the risk of personal privacy leakage. This non-intrusive monitoring method not only provides a reliable way for animal owners to understand the psychological state of animals in real time, but also provides data support for animal health management, while ensuring the comfort and privacy safety of the home environment.

[0026] Step 102, generating a three-dimensional point cloud sequence based on the millimeter wave signal sequence, and generating behavior feature sequence information based on the three-dimensional point cloud sequence.

[0027] In this embodiment, each frame of millimeter wave signal can be converted into a three-dimensional point cloud data, where the three-dimensional point cloud data can include three-dimensional coordinates (x, y, z), radial velocity v, SNR (Signal-to-Noise Ratio), etc., so that a three-dimensional point cloud sequence can be obtained. Then, based on the three-dimensional point cloud sequence, behavior feature sequence information can be generated. For example, clustering analysis (for example, clustering analysis using Euclidean clustering) can be performed on each frame of three-dimensional point cloud, and a plurality of points corresponding to the target animal are identified from the point cloud after clustering analysis. Then, statistical analysis can be performed on the plurality of points corresponding to the target animal in each frame of three-dimensional point cloud, for example, the center of gravity coordinates, coordinate quantile, average speed, speed standard deviation, average signal-to-noise ratio, etc. are calculated, and the statistical analysis results are taken as point cloud features. In some implementations, the point cloud features corresponding to each frame of three-dimensional point cloud can be taken as the behavior features corresponding to the frame of three-dimensional point cloud, so that the behavior feature sequence information corresponding to the three-dimensional point cloud sequence can be obtained.

[0028] In some examples, the behavior feature sequence information can include a timestamp, a point cloud feature, and a trajectory. And the above step 102 can include the following steps 1021 to 1026, in particular: Step 1021, based on the millimeter wave signal sequence, a three-dimensional point cloud sequence with a timestamp is generated.

[0029] In this example, by analyzing the time difference and frequency change of the millimeter wave radar device signal, a three-dimensional point cloud sequence describing the spatial position of the target animal can be generated, and each point cloud data frame is stamped with an accurate timestamp. For example, each frame of signal of the FMCW radar device can be converted into a frame of three-dimensional point cloud wherein, may represent the three-dimensional space coordinates, v may represent the radial velocity, and SNR may represent the signal-to-noise ratio. Here, to suppress ground clutter and static multipath, the points in the three-dimensional point cloud with |v|<0.03 m / s and SNR<15 dB can be first removed, and for the remaining points, the detection statistic ζ output by the CFAR (Constant False Alarm Rate) detector built-in the radar can be used to normalize ζ to [0, 1] as the posterior confidence weight w_cloud, and the points below 0.4 are also removed.

[0030] Step 1022, clustering analysis is performed on each frame of three-dimensional point cloud in the three-dimensional point cloud sequence, and a target point set corresponding to the target animal in each frame of three-dimensional point cloud is obtained.

[0031] In this example, clustering analysis can be performed on each frame of the three-dimensional point cloud sequence, for example, spatial clustering can be performed using the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm, for example, the field radius ε = 6 cm (half of the typical width of the torso of animals such as dogs and cats), and the minimum number of points MinPts = 3. Then, the cluster with the most points and the most concentrated space in the clustering analysis can be identified as the "animal main cluster", and the redundant points outside the cluster and the false targets generated by the multipath effect can be removed. In this example, the points in the "animal main cluster" can be used as the target point set corresponding to the target animal.

[0032] In step 1023, statistical analysis is performed on the target point set of each frame of the three-dimensional point cloud to obtain the point cloud features corresponding to each frame of the three-dimensional point cloud.

[0033] In this example, statistical analysis can be performed on the target point set of each frame of the three-dimensional point cloud (i.e., the points in the animal main cluster) to obtain the point cloud features corresponding to each frame of the three-dimensional point cloud.

[0034] For example, for the t-th frame of the three-dimensional point cloud, assuming that the target point set in the t-th frame of the three-dimensional point cloud includes N1 points, and the data structure of each point Pi is Then, various types of features such as position and distribution features, speed and dynamic features, shape and orientation features, signal quality and density features can be calculated by statistical analysis, as the point cloud features corresponding to each frame of the three-dimensional point cloud. Specifically: 1), position and distribution features.

[0035] As an example, the position and distribution features can include but are not limited to the centroid, the range, the coordinate quantile, and the three-dimensional variance var_3d. The centroid can be used to calculate the centroid of the animal main cluster using the median, which is more robust to outliers. The range can represent the physical size of the cluster in three dimensions, and can estimate the pet body shape. The coordinate quantile can describe the spatial distribution of the point cloud, avoiding the influence of extreme points. The three-dimensional variance var_3d can be the average squared distance of all points in the animal main cluster to the centroid, which can represent the "compactness" of the cluster.

[0036] As an example, the calculation formulas of the centroid, the range, the coordinate quantile, and the three-dimensional variance var_3d can be as follows: .

[0037] wherein, may represent the 25th percentile (lower quartile) and 75th percentile (upper quartile) of the data x.

[0038] 2), velocity and dynamic features.

[0039] As an example, the velocity and dynamic features can include, but are not limited to, mean velocity mean_v, velocity standard deviation std_v, velocity quantile, high speed point ratio high_speed_ratio, etc., wherein the mean velocity mean_v can represent the overall radial velocity of the cluster. The velocity standard deviation std_v can represent the consistency of the velocity, and a large value indicates that different parts of the body are not synchronized. The velocity quantile can more comprehensively describe the velocity distribution. The high speed point ratio high_speed_ratio can measure the fast moving part.

[0040] For example, the calculation formulas of the mean velocity mean_v, the velocity standard deviation std_v, the velocity quantile, and the high speed point ratio high_speed_ratio can be as follows: .

[0041] 3), shape and orientation features.

[0042] As an example, to determine the shape and orientation features, first, principal component analysis can be performed, and all points of each frame of the animal body cluster are composed into a matrix, and PCA (Principal Component Analysis) is performed on the matrix to obtain eigenvalues ​) and the first principal component PC1 = [a, b, c] (approximate animal orientation). Then, the linearity linearity and the planarity planarity are calculated according to the eigenvalues, where a high linearity linearity can represent that the cluster is long strip-shaped, and a high planarity planarity can represent that the cluster is flat. For example, the calculation formulas of the linearity linearity and the planarity planarity can be as follows: .

[0043] 4), signal quality and density features.

[0044] As an example, the signal quality and density features can include but are not limited to the mean signal-to-noise ratio mean_snr, the signal-to-noise ratio standard deviation std_snr, and the point cloud density density, where the mean signal-to-noise ratio mean_snr can represent the overall signal quality of the point cloud cluster. The signal-to-noise ratio standard deviation std_snr can represent the consistency of the signal-to-noise ratio. The point cloud density density can represent the number of points in a unit volume, reflecting the surface reflection characteristics and the detection effect. For example, the calculation formulas of the mean signal-to-noise ratio mean_snr, the signal-to-noise ratio standard deviation std_snr, and the point cloud density density can be as follows: .

[0045] Step 1024, weighted center calculation is performed on the target point set in each frame of three-dimensional point cloud to obtain the weighted center of each target point set.

[0046] In the present example, the weighted center calculation is performed on the target point set in each frame of three-dimensional point cloud to obtain the weighted center of each target point set.

[0047] For example, the instantaneous center of gravity can be calculated by weighting all points (assuming including n points) in the animal body cluster according to the SNR: , .

[0048] wherein, can represent the SNR weight value of the i-th point in the k-th frame, is the current frame center of gravity, or the animal observation position value of the current frame, and the reflection signal is ensured to have a higher weight in positioning by weighting the signal-to-noise ratio.

[0049] At step 1025, a trajectory coordinate corresponding to each frame of the three-dimensional point cloud is generated based on a weighted center of a target point set in each frame of the three-dimensional point cloud and a preset tracking algorithm.

[0050] In the present example, one-step tracking of "SNR weighted center + a-b-g filter" can be adopted. Specifically, the SNR weighted center of the animal subject cluster after DBSCAN clustering of each frame of point cloud is directly taken as an observation value, and an a-b-g filter (for example, a=0.45, b=0.25, and g=0.05) is adopted to estimate and predict the state of the animal, and the state vector is nine-dimensional: wherein v can represent velocity, and a can represent acceleration, Both can be calculated from the position coordinates of the previous and subsequent two frames, the nearest neighbor correlation is performed with the 25 cm threshold of the predicted value of the previous frame and the current observation, and the state is updated, if there is no match for 5 consecutive frames, the old trajectory is sealed, and a new trajectory ID (identification) is opened. After generating a complete trajectory, the Savitzky-Golay filter is used for smoothing processing to remove high-frequency jitter and retain the true motion trend. The point cloud coordinates can finally be rotated and translated by the radar external parameters R_cal and t_cal (obtained from the radar calibration step) to output the continuous trajectory in the ground coordinate system .

[0051] At step 1026, the behavior feature of each frame of the three-dimensional point cloud is determined based on the point cloud feature and the trajectory coordinate of each frame of the three-dimensional point cloud, and the behavior feature sequence information of the three-dimensional point cloud sequence is formed based on the behavior feature of each frame of the three-dimensional point cloud.

[0052] In the present example, the behavior feature of each frame of the three-dimensional point cloud can be determined based on the point cloud feature and the trajectory coordinate of each frame of the three-dimensional point cloud, and the behavior feature sequence information of the three-dimensional point cloud sequence can be formed based on the behavior feature of each frame of the three-dimensional point cloud.

[0053] For example, assuming that the input time window is 30 seconds, for a radar with a frame rate of 200 Hz (i.e., 200 frames per second), the time step is 6000 data points. First, the statistical features (such as position, velocity, shape, signal-to-noise ratio, etc.) of the "animal subject cluster" point cloud identified in each frame are calculated to obtain a point cloud feature sequence with a dimension of [6000, D_pointcloud], and in the present example, D_pointcloud= 27. Subsequently, the point cloud feature sequence is spliced with the trajectory coordinates (x'_k, y'_k, z'_k) aligned at the frame level within the same time window in the feature dimension to form a behavior feature sequence with a dimension of [6000, 30]. This sequence completely describes the macro motion and micro posture change of the pet within 30 seconds.

[0054] ​​Step 103, determining the physiological feature information of the target animal based on the sequence of millimeter wave signals.

[0055] In this embodiment, the physiological feature information of the target animal can be determined based on the sequence of millimeter wave signals collected by the millimeter wave radar device. For example, the heartbeat signal can be determined from the sequence of millimeter wave signals, and then the physiological feature information of the target animal is determined through the heartbeat signal. Here, the physiological feature information can include heart rate (HR), and can also include the feature information of heart rate variability (HRV). Here, the heart rate variability can refer to the change of the difference between successive heart rate periods. The feature information of the heart rate variability can include time domain features and / or frequency domain features. It can be understood that since each frame of millimeter wave signals in the sequence of millimeter wave signals has a time stamp, the behavior feature sequence information and the physiological feature information generated based on the sequence of millimeter wave signals can be aligned in time.

[0056] In some examples, the above step 103 can include the following steps 1031 and 1032, specifically: Step 1031, determining the heartbeat signal from the sequence of millimeter wave signals.

[0057] In this example, it is assumed that a FMCW radar with a working frequency of 60 GHz, a frame rate fs of 200 Hz, and a sweep bandwidth of 4 GHz is used for signal collection. The radar periodically transmits millimeter wave signals and receives echoes reflected from the surface of the animal's chest and abdomen. The raw data collected by each receiving antenna is an intermediate frequency signal after mixing and low-pass filtering processing, and then digitized by an ADC (Analog-to-Digital Converter) at a sampling rate of 5-20 Msps (Megasamples per second).

[0058] The signal processing flow can include the following: Distance dimension processing: the IF (Intermediate Frequency) signal corresponding to each Chirp signal (linear frequency modulation signal) in each frame is subjected to distance dimension fast Fourier transform. The distance dimension fast Fourier transform can convert the signal from the time domain to the frequency domain to obtain the distance spectrum. Since the frequency of the IF signal is proportional to the target distance, the distance spectrum directly shows the intensity and phase information of the reflection signal on different distance units.

[0059] Target selection and tracking: based on the distance spectrum, the target distance unit corresponding to the animal's chest and abdominal region is automatically located in each frame by a peak search algorithm. This region usually produces the strongest reflection signal peak due to its significant physiological activity.

[0060] Phase extraction and unwrapping: for the complex signal after Range-FFT of the selected target range cell (where t represents the slow time), whose phase is extracted. Due to the periodicity of the function, the extracted phase is confined in the interval , therefore, phase unwrapping is needed to recover the continuous phase variation. In this example, a classical unwrapping algorithm based on phase difference is adopted: where If , then ; if , then ; otherwise .

[0061] Finally, the unwrapped continuous phase , which contains the tiny displacement information caused by the chest micro-motion, can be used for the subsequent extraction of vital signs such as respiration and heartbeat.

[0062] In some examples, the above step 1031 can include the following steps 1) and step 2), specifically: Step 1), singular spectrum analysis is performed on each frame of the pre-processed millimeter wave signal sequence, and noise in each frame of the millimeter wave signal is removed, wherein the pre-processing can include phase unwrapping.

[0063] Step 2), the variational mode decomposition is performed on each frame of the millimeter wave signal after noise removal, and the heartbeat signal is determined based on the decomposition result.

[0064] In this example, first, singular spectrum analysis (SSA) can be used for coarse separation to remove noise and part of the respiratory harmonic, and to retain most of the heartbeat energy. Then, variational mode decomposition (VMD) is used for fine decomposition to adaptively separate the heartbeat component in the residual. Specifically: First step: singular spectrum analysis.

[0065] 1. Embedding: the unwrapped phase signal (length N2) is denoted as , which is mapped to a trajectory matrix with where represents the window length, which is recommended to be slightly longer than one respiratory period (e.g., 3-4 seconds, corresponding to 600-800 sampling points), Trajectory matrix may be expressed as follows: .

[0066] 2. Singular value decomposition: Perform singular value decomposition on the trajectory matrix , and get the singular values (located in the diagonal matrix ) in descending order. Each singular value corresponds to an empirical orthogonal function, which represents a specific component pattern in the signal.

[0067] 3. Grouping: The grouping strategy can include two strategies of direct removal and conservative reservation. Direct removal can refer to grouping the first r components and directly reconstructing the respiratory signal , and the residual signal contains heartbeats and noise. Conservative reservation only removes the extremely high-order components that are obviously noise (for example, the part after the singular value spectrum curve becomes flat), and reconstructs the first r components into a denoised mixed signal , which is used for the next step of VMD. This strategy is more conservative and can retain more heartbeat energy.

[0068] 4. Diagonal averaging: Reconstruct the time series signal with a length of N2 from the grouped sub-matrix.

[0069] Second step: Variational mode decomposition.

[0070] Take the output of SSA processing or as the input of VMD.

[0071] First, parameter setting. In this example, the mode number K is set to 4 or 5, the penalty parameter is set to 2000, and the fidelity is set to 0.1.

[0072] Then, the parameters and the input signal are sent into the VMD algorithm to get K intrinsic mode functions .

[0073] Finally, the spectrum of each is calculated, and according to prior knowledge (the heart rate range is usually 0.8Hz-3.0Hz), find the whose dominant frequency falls within this interval. If the dominant frequencies of multiple IMFs are within this range, select the one with the highest energy or the most similar heartbeat waveform form as the heartbeat signal y(t).

[0074] In other examples, the above step 1031 can include the following steps (1) to (3), specifically: Step (1), after preprocessing each frame of the millimeter wave signal sequence, a low-pass filter with a preset cutoff frequency is used to filter each frame of the millimeter wave signal to obtain a breathing signal in each frame of the millimeter wave signal, wherein the preprocessing includes phase unwrapping.

[0075] Step (2), for each frame of the millimeter wave signal, the frame of the millimeter wave signal is subtracted from the breathing signal in the frame of the millimeter wave signal to obtain a residual signal.

[0076] Step (3), the residual signal of each frame of the millimeter wave signal is processed by using a central difference filter to obtain a heartbeat signal.

[0077] In this example, after obtaining continuous phase change information through phase unwrapping, a signal containing animal chest micro-motion information can be obtained. The signal is actually a mixed signal modulated by breathing and heartbeat motion. Among them, the chest displacement amplitude caused by breathing is large and the frequency is low (usually 0.1-0.3 Hz), while the displacement amplitude caused by heartbeat is small and the frequency is high (usually 0.8-3.0 Hz). In order to separate the weak heartbeat component from such a mixed phase signal, the following steps can be used for extraction: First, suppress the breathing signal. Specifically, first, a low-pass filter with a cutoff frequency fc (for example, 0.5 Hz) can be used to filter the unwrapped phase signal . The purpose of this step is to preliminarily retain the low-frequency breathing signal . Then, subtract the filtered breathing signal from the phase signal to obtain a residual signal that has preliminarily removed most of the breathing components. This method can more directly retain the energy in the frequency band of the heartbeat.

[0078] Second, enhance the heartbeat signal. Specifically, in order to further enhance the high-frequency micro-motion characteristics caused by heartbeat pulsation in the residual signal , a seven-point central difference filter can be used to process it. As a strong high-pass filter, this filter can effectively amplify the high-frequency components of the signal (corresponding to the acceleration change caused by the heartbeat), and its discrete time operation is defined by the following formula: where y(t) is the enhanced output signal, is the sampling interval of the slow time axis, .

[0079] Step 1032, based on the heartbeat signal, determining the feature information of the heart rate variability and the heart rate of the target animal.

[0080] In the present example, after the heartbeat signal is determined, the heart rate of the target animal can be determined, and the characteristic information of the heart rate variability can be determined.

[0081] In some examples, the above step 1032 can include the following steps (one) to step (three), in particular: Step (one), using an R-peak detection algorithm to detect R-peak of the heartbeat signal to obtain an R-peak index list, wherein the R-peak index list is used to represent the position of each R-peak in the heartbeat signal.

[0082] Step (two), according to the position of each R-peak, the time interval between each two adjacent R-peak is calculated to obtain an RR interval sequence.

[0083] Step (three), based on the RR interval sequence, the heart rate, the time domain feature of the heart rate variability and the frequency domain feature of the heart rate variability are calculated.

[0084] In the present example, it is assumed that for a heartbeat signal y(t) with length N3 and sampling interval , an R-peak detection algorithm based on adaptive threshold sliding window can be used to detect R-peak and further calculate heart rate and heart rate variability. The implementation of the R-peak detection algorithm can be as follows: 1. Initialize window and threshold: take the initial segment of the signal as the initialization window , the length of which is , usually corresponding to 1.5 to 2 seconds of data (for example, for 200 Hz frame rate, take =300 to 400 sampling points). Take the absolute value of the data w(n) in , to get |w(n)|. Calculate the initial signal threshold: , wherein represents a gain factor, and the recommended value is between 1.2 and 1.5. Calculate the initial noise threshold: , wherein represents a gain factor, and the recommended value is between 0.5 and 0.8.

[0085] 2. Initialize parameters: set the sliding window length (for example, 3-5 seconds, corresponding to 600-1000 sampling points), the sliding step (for example 25% of the length of the sliding window). Set the irregular period (for example, 200 ms, corresponding to 40 sampling points). Set the valid peak interval range (for example, [60, 300]). Initialize the valid R-peak index list .

[0086] 3. Cycle processing: in the current window , find all local maximum points with amplitude greater than noise threshold as candidate peaks, whose position is and amplitude is . For each candidate peak, if is within the time of the last confirmed R-peak, skip this candidate peak. Otherwise, calculate the peak interval with the last confirmed R-peak , if PP is within the valid peak interval range and amplitude is greater than signal threshold, confirm it as a valid R-peak, add its position to the R-peak index list ; if PP is too short, reject the candidate, if PP is too long (e.g. > 1.5 times average PP interval), indicating that a missed detection may have occurred, enable backtracking search. After processing the current window, slide the window forward , update the window index k = k + 1, update the signal threshold and noise threshold, start processing the next window, until the entire signal is processed.

[0087] 4. Backtracking search: use the noise threshold as the candidate R-peak threshold in the interval, re-search the local maximum, select the one with the largest amplitude from the candidate points, confirm it as a valid R-peak even if it does not reach the signal threshold.

[0088] 5. Adaptive threshold update strategy: wherein 1 can represent a forgetting factor (e.g. 0.8), may represent a scaling factor (e.g. 0.6), is the amplitude set of all confirmed valid R-peaks in the current window.

[0089] wherein may represent a forgetting factor (e.g. 0.8), may represent a scaling factor (e.g. 1.2), is the amplitude set of all rejected candidate peaks in the current window.

[0090] After that, the characteristic information of heart rate and heart rate variability is calculated.

[0091] For the R-peak index list obtained by the R-peak detection algorithm, first calculate all continuous RR intervals (unit: seconds) according to the R-peak position: , The obtained RR interval sequence . Calculate heart rate based on RR interval mean value: .

[0092] Time domain features of heart rate variability and calculation method: .

[0093] Wherein, N is the total number of RR intervals, is the mean value of RR interval. The above-mentioned time domain features of heart rate variability include SDNN (Standard Deviation of Normal-to-Normal intervals, standard deviation of normal sinus rhythm interval), RMSSD (Root Mean Square of Successive Differences, root mean square of adjacent NN interval difference), NN50 (Number of pairs of adjacent NN intervals differing by more than 50 ms, number of adjacent NN interval difference greater than 50 milliseconds) and (Proportion of NN50 divided by total number of NN intervals, percentage of NN50 number to all NN interval number).

[0094] Frequency domain features of heart rate variability and calculation method: First, the RR interval sequence needs to be converted into a stationary signal that can be used for spectral analysis. For example, commonly used methods include resampling (e.g., using a frequency of 4 Hz), interpolating the non-uniform RR interval sequence into a uniform time sequence. Then, the power spectral density of the signal is estimated using the fast Fourier transform or the autoregressive model. Core frequency domain features are obtained by analyzing the frequency spectral density: total power TP, low frequency power LF, high frequency power HF, very low frequency power VLF, low / high frequency power ratio LF / HF, normalized LF power LFnu, and normalized HF power HFnu.

[0095] In step 104, the behavior feature sequence information and the physiological feature information are input into a pre-trained animal emotion recognition model, and the animal emotion recognition model performs emotion recognition and outputs the emotion information of the target animal.

[0096] In this embodiment, the animal emotion recognition model can be pre-trained, which can be used to represent the corresponding relationship between the behavior feature sequence information, the physiological feature information, and the animal emotion information. For example, the animal emotion recognition model can be various machine learning models for classification, such as convolutional neural networks, decision tree models, support vector machines, etc. Based on this, the behavior feature sequence information and the physiological feature information can be input into the animal emotion recognition model, and the animal emotion recognition model outputs the emotion information of the target animal. For example, the probability information of the calm, anxious, excited, fearful, and aggressive emotional states can be output.

[0097] In some examples, the above-mentioned animal emotion recognition model can include a dimensionality increasing module, a global field of view module, a local field of view module, and a data imbalance processing module. Based on this, the animal emotion recognition model can perform the following steps 1) to 5) on the behavior feature sequence information and the physiological feature information, specifically: Step 1), the dimensionality increasing module increases the dimensionality of the input behavior feature sequence information, outputs the dimensionality-increased behavior feature sequence information, and inputs the dimensionality-increased behavior feature sequence information into the global field of view module.

[0098] Step 2), the global field of view module performs global analysis on the dimensionality-increased behavior feature sequence information based on the self-attention mechanism to obtain global features, and inputs the global features into the local field of view module.

[0099] In step 3), the local field module performs local analysis on the global features to obtain local features.

[0100] In step 4), the global features, the local features, and the physiological feature information are fused to obtain fused features.

[0101] In step 4), the data imbalance processing module outputs the emotional information of the target animal based on the fused features.

[0102] In the present example, the dimension increasing module, the global field module, the local field module, and the data imbalance processing module each include multiple network layers. As shown in Figure 2 Figure 2 FIG. 1 shows a schematic diagram of one example of the model structure of the animal emotion recognition model.

[0103] In the example shown in Figure 2 The dimension increasing module can sequentially include a linear layer, a one-dimensional convolutional layer, a one-dimensional convolutional layer, a pooling layer, a random inactivation layer, and the like. The dimension increasing module can map the input behavior feature sequence information to a higher-dimensional feature space through full connection of the linear layer, one-dimensional convolutional operation, and the like, aiming to amplify the differences between different behavior patterns and lay a foundation for subsequent fine recognition.

[0104] The global field module can sequentially include a position encoding layer, a multi-head self-attention layer, a stacking layer and normalization, a feedforward network, a stacking layer and normalization, and the like. The multi-head self-attention layer, the stacking layer and normalization, the feedforward network, the stacking layer and normalization can be repeated 2 times. Thus, the global field module can perform global analysis on the entire behavior feature sequence information based on the self-attention mechanism, and is good at capturing those sparse but key “class displacement reaction” patterns from long-time activities, thereby macroscopically grasping the behavior rhythm changes caused by emotions. The global field module can output global features. Here, the “class displacement reaction” can refer to a short, repetitive, and unconscious behavior segment exhibited by an animal in a specific emotional state, such as circling in place, frequent licking of the paw when anxious, or rapid short-distance U-turn running when excited. These behaviors will exhibit unique spatiotemporal patterns on the trajectory data. The “global field module” and “local field module” designed here are exactly to capture these microscopic, fixed-length patterns from macroscopic activity sequences and encode them into feature vectors for classification.

[0105] ​The local field of view module can sequentially include an adaptive window segmentation layer, a multi-path one-dimensional convolution network (for example, a parallel 3-path one-dimensional convolution network) layer, a global maximum pooling layer (or a global average pooling layer), a weighted voting fusion layer, and the like. The adaptive window segmentation layer can divide the data into multiple local regions for local analysis. The weighted voting fusion layer can make a weighted vote on all window outputs (where the weight = window and global attention score correlation), and output local features. The local field of view module, as an effective supplement to global analysis, can use a multi-scale sliding window strategy to perform local fine scanning on the sequence. The local field of view module can adaptively verify and confirm the internal structure of the indefinite length "class replacement reaction" segment, and improve the accuracy of feature extraction. The local field of view module can output local features.

[0106] The global features, local features, and physiological feature information can be fused to obtain fused features.

[0107] The data imbalance processing module can output the emotional information of the target animal based on the fused features. For example, the data imbalance processing module can directly output the emotional information based on the fused features.

[0108] In some examples, in order to make the model classification more accurate, the data imbalance processing module outputs the emotional information of the target animal based on the fused features, which can specifically include: first, correcting the fused features based on a pre-stored emotional label prior distribution, wherein the emotional label prior distribution is obtained from a preset training set. Then, output the emotional information of the target animal based on the corrected fused features.

[0109] In this example, the data imbalance processing module can include a class weight layer based on a prior distribution, a learnable gating and feature re-weighting layer, a fully connected layer, and a Softmax function layer. The data imbalance processing module can use the emotional label prior distribution obtained from the training set to correct the fused features, forcing the model to pay equal attention to rare emotional categories (such as anxiety, excitement, etc.) during the training process, thereby optimizing the average recognition performance of the model on each category. Finally, the corrected fused features are sent to the fully connected layer and the Softmax function layer, which outputs the precise probability prediction of the pet's emotional state of calm, anxiety, excitement, etc. by comprehensively analyzing the pet's behavior details and physiological state.

[0110] For example, first, the label prior distribution P_prior(c) in the preset training set can be counted, where c represents the label, c = calm / anxiety / excitement…, and P_prior(c) can be specifically: , Then, the balance weight w_c is calculated, wherein γ2 represents a smoothing factor, which can be valued according to actual needs (for example, γ2 = 0.3), and the role of γ2 is to prevent overfitting caused by too large weight.

[0111] Then, w c is applied to the fusion feature F through a learnable gating mechanism to correct the fusion feature F, and a corrected fusion feature F' is obtained, specifically: - Gate = Sigmoid( Linear(F) ) ⊙w_c; - F' = Gate ⊙ F.

[0112] Finally, the corrected fusion feature F' is input into a fully connected layer and a Softmax function layer, and an emotion probability distribution is output, which is emotion information. In this example, by introducing a global-local collaborative attention mechanism and a feature reweighting strategy based on a prior distribution, the technical difficulties of accurately identifying a short and sparse "placebo effect" from animal free activity data and efficiently fusing it with physiological signals at the semantic level are effectively solved.

[0113] In practice, when training the above animal emotion recognition model, a supervised training method can be used. To ensure the reliability of the labels in the training data, multiple trainers with IAABC (International Association of Animal Behavior Consultants) or CCPDT (Certification Council for Professional Dog Trainers) certification and at least five years of experience in evaluating animal behavior can be invited to participate in emotion labeling. For example, during the process of millimeter wave data synchronous collection, the trainer induces different emotional states of the pet through specific interaction methods, and the trainer wears a low-delay Bluetooth button (delay 8ms). When the typical characteristics of the target emotion are recognized, such as ear position change, tail posture adjustment, pupil contraction, or lip tension, the trainer triggers the marker and simultaneously speaks the emotion category, which is automatically transcribed into an emotion label by a high-sensitivity sound pickup device.

[0114] On this basis, an iterative labeling strategy can be used to improve sample quality. Specifically, first, an initial network is trained based on the labeled true value data, and then unlabelled samples are predicted to filter out high-uncertainty samples with low model judgment confidence, which are then subjected to a second round of fine labeling by the trainer. This process can be repeated multiple times to efficiently build a large-scale, reliable training sample set.

[0115] Please continue to see Figure 3 , Figure 3 A schematic diagram showing an application scenario to which embodiments of the present specification can be applied is shown. In Figure 3In the illustrated application scenario, the goal is to perform non-contact emotion monitoring on a pet cat. Specifically, a 60 GHz FMCW radar transmits a (TX) signal to the area where the cat is located and receives (RX) reflected echoes generated by the cat's activities, forming a millimeter-wave signal. On one hand, this millimeter-wave signal can be used for point cloud generation and trajectory tracking, obtaining timestamps, trajectories, and point cloud features. On the other hand, it can also be used for chest cavity micro-motion extraction, singular spectrum analysis, variational mode decomposition, R-peak detection, and further extraction of heart rate and heart rate variability features. Then, the timestamps, trajectory, point cloud features, and heart rate and heart rate variability features are input into a pre-trained animal emotion recognition model. In this example, the animal emotion recognition model includes a dimensionality enhancement module, a global vision module, a local vision module, and a data imbalance processing module. The animal emotion recognition model outputs the pet cat's emotional information, such as calmness, anxiety, excitement, etc.

[0116] Reviewing the above process, in the embodiments of this specification, pet emotion monitoring can be achieved non-intrusively based on millimeter-wave signals collected by millimeter-wave radar equipment, without requiring the pet to wear any devices, making it more suitable for animal emotion recognition. Furthermore, by combining behavioral feature sequence information and physiological feature information for animal emotion recognition, the emotion recognition results can be made more accurate.

[0117] According to another embodiment, a millimeter-wave-based animal emotion recognition device is provided. This millimeter-wave-based animal emotion recognition device can be deployed in any device, platform, or cluster of devices with computing and processing capabilities.

[0118] Figure 4 A schematic block diagram of a millimeter-wave-based animal emotion recognition device according to one embodiment is shown. Figure 4 As shown, the millimeter-wave-based animal emotion recognition device 400 includes: an acquisition unit 401 configured to acquire a millimeter-wave signal sequence collected by a millimeter-wave radar device for a target animal; a generation unit 402 configured to generate a three-dimensional point cloud sequence based on the millimeter-wave signal sequence, and generate behavioral feature sequence information based on the three-dimensional point cloud sequence; a determination unit 403 configured to determine the physiological feature information of the target animal based on the millimeter-wave signal sequence, wherein the physiological feature information includes heart rate variability feature information and heart rate; and a recognition unit 404 configured to input the behavioral feature sequence information and the physiological feature information into a pre-trained animal emotion recognition model, and have the animal emotion recognition model perform emotion recognition and output the emotion information of the target animal.

[0119] According to another aspect, an embodiment also provides a computer readable storage medium having stored thereon a computer program, which, when executed in a computer, causes the computer to perform Figure 1 the method described.

[0120] According to another aspect, an embodiment also provides an electronic device comprising a memory and a processor, wherein the memory has stored thereon executable code that, when executed in the processor, realizes Figure 1 the method described.

[0121] Those of ordinary skill in the art will appreciate that the various illustrative components, systems and methods described in connection with the embodiments disclosed herein can be implemented as hardware, software, firmware or any combination thereof. The various illustrative components, systems and methods described in connection with the embodiments disclosed herein can be implemented in hardware, software, firmware or any combination thereof. The particular implementation as hardware, software, firmware or any combination thereof is dependent upon the specific application and design constraints imposed on the particular implementation. Skilled persons can employ different methods to implement the described functions for each particular application, but such implementation should not be considered to be beyond the scope of the present application. When implemented in hardware, the hardware can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), suitable firmware, a plug-in, a functional card, etc. When implemented in software, the elements of the present application are the program or code segments to perform the necessary tasks. The program or code segments can be stored in a machine readable medium, or transmitted by a carrier wave over a transmission medium or a communication link.

[0122] It is to be understood that the application is not limited to the particular configurations and processes described herein and shown in the drawings, which are provided by way of example only. Detailed descriptions of known methods are omitted so as not to obscure the description of the present application. In the above embodiments, several specific steps are described and illustrated in order to provide a thorough disclosure of the application. However, the skilled artisan will recognize that the method process of the present application can be implemented with fewer than all of the described steps, with additional steps not mentioned, or with steps in a different order than that shown.

[0123] In the present application, features described and / or illustrated with respect to one embodiment can be used in the same or a similar way in one or more other embodiments, and / or combined with or substituted for features of other embodiments.

[0124] The above description merely provides examples of the present application and is not intended to limit the application. Based on the application concept and principles disclosed in the present application, those skilled in the art can make various modifications and variations to the present application without departing from the scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall fall within the scope of the present application.

Claims

1. A millimeter-wave-based method for animal emotion recognition, characterized in that, The method includes: Acquire millimeter-wave signal sequences collected by millimeter-wave radar equipment targeting the animal; A three-dimensional point cloud sequence is generated based on the millimeter-wave signal sequence, and behavioral feature sequence information is generated based on the three-dimensional point cloud sequence. Based on the millimeter-wave signal sequence, the physiological characteristics of the target animal are determined, wherein the physiological characteristics include heart rate variability and heart rate. The behavioral feature sequence information and the physiological feature information are input into a pre-trained animal emotion recognition model, which then performs emotion recognition and outputs the emotion information of the target animal.

2. The method according to claim 1, characterized in that, The animal emotion recognition model includes a dimensionality enhancement module, a global perspective module, a local perspective module, and a data imbalance processing module; and the animal emotion recognition model performs the following processing on the behavioral feature sequence information and the physiological feature information: The dimension-upgrading module upgrades the input behavioral feature sequence information, outputs the upgraded behavioral feature sequence information, and inputs the upgraded behavioral feature sequence information into the global vision module. The global vision module performs global analysis on the upgraded behavioral feature sequence information based on the self-attention mechanism to obtain global features, and then inputs the global features into the local vision module. The local view module performs local analysis on the global features to obtain local features; The global features, local features, and physiological features are fused to obtain fused features; The data imbalance processing module outputs the emotional information of the target animal based on the fusion features.

3. The method according to claim 2, characterized in that, The data imbalance processing module outputs the target animal's emotional information based on the fusion features, including: The data imbalance processing module corrects the fusion features based on a pre-stored prior distribution of emotion labels, wherein the prior distribution of emotion labels is statistically obtained from a preset training set. Based on the corrected fusion features, the emotional information of the target animal is output.

4. The method according to claim 1, characterized in that, The behavioral feature sequence information includes timestamps, point cloud features, and trajectories; And, the step of generating a three-dimensional point cloud sequence based on the millimeter-wave signal sequence, and generating behavioral feature sequence information based on the three-dimensional point cloud sequence, includes: Based on the millimeter-wave signal sequence, a three-dimensional point cloud sequence with timestamps is generated; Cluster analysis is performed on each frame of the three-dimensional point cloud sequence to obtain the set of target points corresponding to the target animal in each frame of the three-dimensional point cloud; Statistical analysis was performed on the target point set of each frame of the 3D point cloud to obtain the point cloud features corresponding to each frame of the 3D point cloud. The weighted center of each target point set in the 3D point cloud of each frame is calculated. Based on the weighted center of the target point set in each frame of the 3D point cloud and the preset tracking algorithm, the trajectory coordinates corresponding to each frame of the 3D point cloud are generated. Based on the point cloud features and trajectory coordinates of each frame of 3D point cloud, the behavioral features of each frame of 3D point cloud are determined, and the behavioral features of each frame of 3D point cloud form the behavioral feature sequence information of the 3D point cloud sequence.

5. The method according to claim 1, characterized in that, The determination of the physiological characteristics of the target animal based on the millimeter-wave signal sequence includes: Determine the heartbeat signal from the millimeter-wave signal sequence; Based on the heartbeat signal, the characteristic information of the heart rate variability and the heart rate of the target animal are determined.

6. The method according to claim 5, characterized in that, Determining the heartbeat signal from the millimeter-wave signal sequence includes: After preprocessing each frame of the millimeter-wave signal sequence, singular spectrum analysis is performed to remove noise from each frame of the millimeter-wave signal. The preprocessing includes phase unwinding. Variational mode decomposition was performed on each frame of the noise-removed millimeter-wave signal, and the heartbeat signal was determined based on the decomposition results.

7. The method according to claim 5, characterized in that, Determining the heartbeat signal from the millimeter-wave signal sequence includes: After preprocessing each frame of millimeter-wave signal in the millimeter-wave signal sequence, a low-pass filter with a preset cutoff frequency is used to filter each frame of millimeter-wave signal to obtain the breathing signal in each frame of millimeter-wave signal. The preprocessing includes phase unwinding. For each frame of millimeter wave signal, the breathing signal in that frame of millimeter wave signal is subtracted from the millimeter wave signal in that frame to obtain the residual signal; The residual signals of each frame of millimeter-wave signal are processed using a center differential filter to obtain the heartbeat signal.

8. The method according to claim 5, characterized in that, The determination of the characteristic information of the heart rate variability and heart rate of the target animal based on the heartbeat signal includes: The heartbeat signal is subjected to R-peak detection algorithm to obtain R-peak index list, wherein the R-peak index list is used to characterize the position of each R-peak in the heartbeat signal; Based on the position of each R peak, the time interval between any two adjacent R peaks is calculated to obtain the RR interval sequence; Based on the RR interval sequence, the temporal characteristics of heart rate and heart rate variability and the frequency characteristics of heart rate variability are calculated.

9. An electronic device comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that, The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the electronic device implements the steps of the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 8.

Images