Method and system for revealing animal behaviors in zoo

Abstract

The invention relates to the technical field of zoo management, in particular to a method and system for revealing animal behaviors in a zoo. According to the animal behavior identification method and system, the relationship with emotions and behaviors is established through video data, audio data, physiological data and environment data, multi-modal fusion is carried out through video features, audio features, physiological features and environment data, and animal behaviors are identified based on multi-modal fusion features by adopting a multi-modal Transform model; according to the invention, analysis of behaviors of animals in the zoo is realized, and a decision basis is provided for animal management of the zoo.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of zoo management, and particularly relates to a method and system for revealing animal behavior in a zoo. BACKGROUND

[0002] In the field of animal behavior research, animal behavior refers to the visible actions or reactions exhibited by animals in response to environmental stimuli that are beneficial to their survival and reproduction. Animal behaviors such as running, jumping, and minor changes in action can reflect the state of the animal and may affect the subsequent behavior of the animal or other animals. Research on animal behavior helps to understand the overall state of the animal and meet the animal welfare requirements, therefore, the identification of animal behavior can assess the state of the animal, facilitating the evaluation and management of animals in the zoo. SUMMARY

[0003] In view of the above problems of the prior art, the present application aims to provide a method and system for revealing animal behavior in a zoo to identify the behavior of animals in the zoo and facilitate the analysis of the state of the animals.

[0004] To solve the above problems, the present application adopts the following technical solutions:

[0005] In one aspect, the present application provides a method for revealing animal behavior in a zoo, comprising:

[0006] Collecting video data, audio data, physiological data, and environmental data of animals in the zoo.

[0007] Establishing a behavior classification system based on the behavior of the animals, the behavior classification system including behavior states and postures, and constructing a posture model based on the relationship between the postures of the animals and the key points of the skeletons of the animals.

[0008] Analyzing the sound spectrogram based on a CNN or a Transformer model, establishing a sound model, and analyzing the relationship between sound, emotion, and behavior through the sound model.

[0009] Establishing a physiological parameter feature model under different behavior states, and identifying the relationship between physiological parameters and emotion and behavior.

[0010] Extracting the sequence of key points of the skeleton in the video data through an LSTM or a GRU network, obtaining posture data of the animals based on the posture model, and obtaining motion features and facial features of the animals in the video data as video features; extracting the call type and sound parameters of the animals in the audio data as audio features; and extracting the heart rate pattern, body temperature change, and breathing pattern in the physiological data as physiological features.

[0011] Video features, audio features, physiological features and environmental data of the animal at the same time or time period are acquired, feature extraction is performed, and multi-modal fusion is performed.

[0012] A multi-modal Transformer model is adopted to recognize the behavior of the animal based on the multi-modal fusion features.

[0013] As an implementable manner, the behavior classification system comprises:

[0014] Motor behavior: walking, running, jumping, swimming and climbing posture.

[0015] Resting behavior: standing, lying and sleeping posture.

[0016] Feeding behavior: foraging, hunting, drinking and chewing posture.

[0017] Social behavior: close, sniff, groom, play and fight posture.

[0018] Marking behavior: urine marking, claw marking and rubbing marking posture.

[0019] Self-maintenance: licking, scratching and stretching posture.

[0020] Exploratory behavior: patrol, sniff and observe posture.

[0021] Abnormal behavior: stereotyped behavior, self-injury and excessive vigilance posture.

[0022] As an implementable manner, the structure of the LSTM or GRU network comprises:

[0023] Input layer: for processing 60 frames x 17 key points x 2 (x, y coordinates) sequence data.

[0024] Spatial feature extraction layer: for processing the spatial relationship between the skeleton points by adopting a graph convolution network (GCN).

[0025] Time sequence feature extraction layer: double-layer bidirectional LSTM / GRU network, hidden unit number 256 and 128.

[0026] Attention mechanism: attention is applied to the time step to highlight the key action frame.

[0027] Output module: full connection layer followed by a Softmax classifier.

[0028] As an implementable manner, the posture data of the animal obtained based on the posture model comprises:

[0029] Based on the extracted skeleton key point sequence, the included angle of the key point is calculated :

[0030]

[0031] wherein, , is the dot product of two direction vectors of any two key points, is the dot product of two direction vectors, is the length of two direction vectors.

[0032] According to the calculated angle of the key points Construct a pose feature vector :

[0033]

[0034] wherein, N is the number of angles of the key points, d is the key distance of the key points corresponding to the angle, and M is the number of key distances.

[0035] Based on the pose feature vector Calculate the similarity through the pose model, and match the corresponding pose.

[0036] As an implementable manner, the feature extraction and multi-modal fusion include:

[0037] A feature selection algorithm is used to screen a feature combination from the video features, audio features and physiological features of the animal at the same time or time period.

[0038] Based on the screened feature combination, multi-modal fusion is performed based on weighted voting and adaptive weight adjustment using environmental data.

[0039] The performance weight of each modality is calculated using the historical recognition accuracy of each modality, and the final weight of each modality is obtained through the performance weight of each modality, so as to optimize the weight of each modality in multi-modal fusion.

[0040] As an implementable manner, it further includes:

[0041] Based on the multi-modal fusion feature, the emotional state of the animal is analyzed:

[0042]

[0043] wherein, is the weight of the feature, is the normalized value of the feature, and N is the total number of features.

[0044] As an implementable manner, it further includes predicting the behavior of the animal based on the multi-modal fusion feature, including:

[0045] Based on the multi-modal fusion feature, a time series is constructed.

[0046] The LSTM model is used to process the time series, and the LSTM model comprises:

[0047]

[0048] wherein, is an input vector of the time series at time step t, is a hidden state at time step t, is a cell state at time step t, , , respectively are a forget gate, an input gate and an output gate, is a candidate cell state, , , is a weight matrix, , , is a bias vector, is a Hadamard product, is a sigmoid activation function, is an output prediction vector.

[0049] The behavior of the animal is predicted according to the prediction vector output by the LSTM model.

[0050] As an implementable manner, it further comprises behavior anomaly detection on the identified behavior of the animal:

[0051] A Z-score of the behavior of the animal is calculated:

[0052]

[0053] wherein, x is an observation value of the animal behavior score, is a population mean, is a standard deviation.

[0054] The Z-score of the behavior of the animal is corrected to obtain a corrected score:

[0055]

[0056] wherein, is a median, is a median absolute deviation.

[0057] A behavior anomaly score S is obtained by correcting the score:

[0058]

[0059] wherein, weight for the i-th action, correction for the i-th action score.

[0060] If S < 1 is normal behavior, if 1≤S < 3 is mild abnormality, if 3≤S < 6 is moderate abnormality, and if S≥6 is severe abnormality.

[0061] As an implementable manner, it also includes analyzing the spatial behavior of the animal:

[0062] Calculate the kernel density of the animal estimate:

[0063]

[0064] Where n is the number of observation positions, h is the bandwidth parameter, K is the kernel function, x i The i-th observation point.

[0065] The path curvature of the animal Calculate:

[0066]

[0067] Where x', y' is the first-order derivative of the path parameter equation with respect to the parameter, and x'', y'' is the second-order derivative of the path parameter equation with respect to the parameter.

[0068] Calculate the path efficiency index E:

[0069]

[0070] Where, The actual path length, The straight-line distance from the starting point to the end point.

[0071] If E = 1 is a completely straight path, the highest efficiency, if 1 < E < 1.5 is a high-efficiency path, slight detours, if 1.5≤E < 2.0 is a medium-efficiency path, obvious detours, and if E≥2.0 is a low-efficiency path, significant detours or wandering.

[0072] In another aspect, the present application provides a revealing system for zoo animal behavior, comprising a collection module, a posture model construction module, a sound model construction module, a physiological parameter feature model construction module, a data processing module, a multi-modal fusion module and a behavior recognition module.

[0073] The collection module is used to collect video data, audio data, physiological data and environmental data of animals in the zoo.

[0074] The posture model construction module is configured to establish a behavior classification system according to the behavior of the animal, the behavior classification system including a behavior state and a posture, and to construct a posture model based on the relationship between the posture of the animal and the key points of the skeleton of the animal.

[0075] The sound model construction module is configured to analyze a sound spectrogram according to a CNN or a Transformer model, to establish a sound model, and to analyze the relationship between sound, emotion and behavior through the sound model.

[0076] The physiological parameter feature model construction module is configured to establish a physiological parameter feature model under different behavior states, and to identify the relationship between the physiological parameter and the emotion and the behavior.

[0077] The data processing module is configured to extract a sequence of key points of a skeleton in video data through an LSTM or a GRU network, to obtain posture data of the animal based on the posture model, and to obtain motion features and facial features of the animal in the video data as video features, to extract a call type and a sound parameter of the animal in audio data as audio features, and to extract a heart rate pattern, a body temperature change and a breathing pattern in physiological data as physiological features.

[0078] The multi-modal fusion module is configured to obtain video features, audio features, physiological features and environmental data of the animal at the same time or in the same time period, to perform feature extraction and multi-modal fusion.

[0079] The behavior recognition module is configured to recognize the behavior of the animal based on multi-modal fusion features through a multi-modal Transformer model.

[0080] The present application has the beneficial effects that the video features, the audio features, the physiological features and the environmental data are subjected to multi-modal fusion, the behavior of the animal is recognized based on multi-modal fusion features through a multi-modal Transformer model, and the behavior of the animal in the zoo is analyzed, thereby providing a decision basis for the management of the animals in the zoo. BRIEF DESCRIPTION OF DRAWINGS

[0081] Figure 1 A flow chart of a method for revealing the behavior of animals in a zoo is provided.

[0082] Figure 2 A schematic diagram of a system for revealing the behavior of animals in a zoo is provided. DETAILED DESCRIPTION

[0083] The present application will be further described in detail below with reference to specific embodiments.

[0084] It should be noted that these examples are only used to illustrate the present application, and are not limiting on the present application, simple improvements of the present method under the concept of the present application are within the scope of the present application.

[0085] Referring to Figure 1 For a method for revealing the behavior of zoo animals, comprising:

[0086] S100, collect video data, audio data, physiological data and environmental data of animals in the zoo.

[0087] Use algorithms such as YOLOv5 to detect animals in the video, use algorithms such as DeepSORT to track the movement trajectory of the animal, handle the case of multiple animals appearing at the same time, and maintain ID consistency. Identify individual animals in the video and continuously track them.

[0088] Video capture devices include:

[0089] High-definition camera: installed in the animal activity area, collecting animal behavior video.

[0090] Infrared camera: for night behavior monitoring.

[0091] High-speed camera: for capturing fast behavior details.

[0092] Panoramic camera: provides a monitoring view of the overall activity area.

[0093] Audio capture devices include:

[0094] Directional microphone: collects the sound of a specific animal.

[0095] Environmental microphone array: collects the sound distribution in the area.

[0096] Ultrasonic detector: collects ultrasonic sounds that cannot be heard by the human ear.

[0097] Sound quality analyzer: assesses the frequency, intensity, and other characteristics of the sound.

[0098] Physiological data monitoring devices include:

[0099] Non-contact body temperature monitor: monitors the surface temperature of the animal.

[0100] Heart rate monitoring device: collects heart rate through special collar or remote monitoring technology.

[0101] Respiratory monitoring device: monitors respiratory rate and depth.

[0102] Activity monitor: records the intensity and frequency of animal activity.

[0103] Environmental data collection devices include:

[0104] Temperature and humidity sensor: monitors environmental conditions.

[0105] Light sensor: monitors light intensity and changes.

[0106] Odor sensor: monitors specific odors in the environment.

[0107] Noise monitor: monitors environmental noise levels.

[0108] S200, establish a behavior classification system according to the behavior of the animal, the behavior classification system includes behavior state and posture, and a posture model is constructed based on the relationship between the posture of the animal and the key points of the skeleton of the animal.

[0109] Based on the position of the key points of the skeleton of the animal in a specific posture, a behavior classification system is established, such as:

[0110] Motor behavior: walking, running, jumping, swimming, climbing, etc.

[0111] Rest behavior: standing, lying, sleeping, etc.

[0112] Feeding behavior: foraging, hunting, drinking, chewing, etc.

[0113] Social behavior: close, sniff, groom, play, fight, etc.

[0114] Marking behavior: urine marking, claw marking, rubbing marking, etc.

[0115] Self-maintenance: licking, scratching, stretching, etc.

[0116] Exploration behavior: patrol, sniff, observe, etc.

[0117] Abnormal behavior: stereotyped behavior, self-injury, excessive vigilance, etc.

[0118] According to the relationship between the key points of the skeleton of the animal and the posture, a posture model is obtained, that is, the position of the key points of the skeleton in a certain posture or the posture corresponding to the key points of the skeleton can be obtained.

[0119] S300, analyze the sound spectrogram according to the CNN or Transformer model, establish a sound model, and analyze the relationship between sound, emotion and behavior through the sound model.

[0120] For example, identify different types of vocalization analysis behavior (threat, courtship, territory declaration, etc.), analyze the emotional characteristics of the sound (tension, relaxation, excitement, anger, etc.). It is convenient to analyze the behavior change before and after the sound appears, and the sound can also be used to predict the behavior that will occur.

[0121] S400, establish physiological parameter feature model in different behavior state, identify the relationship between physiological parameter and emotion and behavior.

[0122] Identifying the relationship between physiological parameter change and behavior transition can predict potential behavior change through physiological parameter. Stress level can be analyzed based on heart rate variability (HRV), combined with body temperature distribution to evaluate emotional state, and identify abnormal physiological parameter fluctuation to warn potential health problems.

[0123] S500, extract the sequence of skeletal key points in video data through LSTM or GRU network, obtain the posture data of the animal based on the posture model, and obtain the motion features and facial features of the animal in the video data as video features; extract the call type and sound parameters of the animal in the audio data as audio features; extract the heart rate pattern, body temperature change and breathing pattern in the physiological data as physiological features.

[0124] Use DeepLabCut and other tools to extract animal skeletal key points in video, extract the sequence of skeletal key points in video data through LSTM or GRU network, and obtain the posture data of the animal based on the posture model through the sequence of skeletal key points.

[0125] The structure of LSTM or GRU network includes:

[0126] Input layer: used to process sequence data of 60 frames x 17 key points x 2 (x, y coordinates).

[0127] Spatial feature extraction layer: used to process spatial relationship between skeletal points by using graph convolution network (GCN).

[0128] Time sequence feature extraction layer: double-layer bidirectional LSTM / GRU network, with hidden unit number 256 and 128.

[0129] Attention mechanism: apply attention to time step to highlight key action frames.

[0130] Output module: full connection layer followed by Softmax classifier.

[0131] Based on the extracted sequence of skeletal key points, the included angle of key points is calculated :

[0132]

[0133] wherein, 、 are two directional vectors of any two key points, the directional vector of the key point can be determined based on the center of gravity or the center point, is the dot product of the two directional vectors, is the length of the two directional vectors.

[0134] According to the included angle of the calculated key points Constructing a pose feature vector :

[0135]

[0136] Where N is the number of included angles of key points, d is the key distance of the key points corresponding to the included angle, and M is the number of key distances.

[0137] Based on the pose feature vector Calculate the similarity through the pose model and match the corresponding pose.

[0138] S600, Obtain video features, audio features, physiological features and environmental data of animals at the same time or time period, perform feature extraction and multi-modal fusion.

[0139] Multi-modal fusion includes:

[0140] Feature selection algorithm is used to screen feature combinations from video features, audio features and physiological features of animals at the same time or time period.

[0141] Based on the screened feature combinations, multi-modal fusion is performed based on weighted voting and adaptive weight adjustment using environmental data. For example:

[0142] Basic weight configuration: video modality (Wv) = 0.6, sound modality (Wa) = 0.3, and physiological parameter modality (Wp) = 0.1

[0143] Environmental condition adaptive weight adjustment:

[0144] Light condition adjustment: when the environmental light (L) is lower than the threshold (L_thresh = 50 lux), the video weight is reduced:

[0145] W'v = Wv × min(1.0, L / L_thresh).

[0146] W'a = Wa + (Wv - W'v) × 0.7. 70% of the weight is transferred to the sound modality.

[0147] W'p = Wp + (Wv - W'v) × 0.3. 30% of the weight is transferred to the physiological parameter modality.

[0148] Noise condition adjustment: when the environmental noise (N) is higher than the threshold (N_thresh = 70 dB), the sound weight is reduced:

[0149] W'a = Wa × max(0.3, 1.0 - (N-N_thresh) / 30).

[0150] W'v = Wv + (Wa - W'a) x 0.8. 80% weight transfer to video modality.

[0151] W'p = Wp + (Wa - W'a) x 0.2. 20% weight transfer to physiological parameter modality.

[0152] The performance weight of each modality is calculated by the historical recognition accuracy of each modality, and the final weight of each modality is obtained through the performance weight of each modality, so as to optimize the weight of each modality in multi-modal fusion. For example:

[0153] Calculate the accuracy Acc_v, Acc_a, Acc_p of each modality in the last N (N=50) recognitions.

[0154] Performance weight calculation:

[0155] P_v = Acc_v / (Acc_v + Acc_a + Acc_p).

[0156] P_a = Acc_a / (Acc_v + Acc_a + Acc_p).

[0157] P_p = Acc_p / (Acc_v + Acc_a + Acc_p).

[0158] Final weight calculation (β=0.7 is the historical performance weight factor):

[0159] W''v = (1-β) x W'v + β x P_v.

[0160] W''a = (1-β) x W'a + β x P_a.

[0161] W''p = (1-β) x W'p + β x P_p.

[0162] Normalization to ensure that the sum of weights is 1:

[0163] sum = W''v + W''a + W''p.

[0164] W''v = W''v / sum.

[0165] W''a = W''a / sum.

[0166] W''p = W''p / sum.

[0167] S700, identifying the behavior of the animal based on the multi-modal fusion feature by using the multi-modal Transformer model.

[0168] The method further comprises S800, analyzing the emotional state of the animal based on the multi-modal fusion feature

[0169]

[0170] wherein, is the weight of the feature, is the normalized value of the feature, and N is the total number of features.

[0171] For example, calculate the alertness score:

[0172] Known features and weights:

[0173] Ear position (w1=0.3): Ears up = 1.0 (fully erect = 1.0, fully flat = 0.0).

[0174] Eye state (w2=0.3): Pupil dilation = 0.8 (max dilation = 1.0, min dilation = 0.0).

[0175] Body posture (w3=0.4): Tense = 0.9 (most tense = 1.0, most relaxed = 0.0).

[0176] Calculate the alertness score:

[0177] E = w1×f1+ w2×f2+ w3×f3

[0178] = 0.3×1.0 + 0.3×0.8 + 0.4×0.9

[0179] = 0.3 + 0.24 + 0.36

[0180] = 0.90.

[0181] Conclusion: The alertness score of the animal is 0.90 (range 0-1), indicating a high alertness state.

[0182] Scoring criteria:

[0183] 0.0-0.3: relaxed state; 0.3-0.6: mild alertness; 0.6-0.8: moderate alertness; 0.8-1.0: high alertness.

[0184] The method further comprises S900, predicting the behavior of the animal based on the multi-modal fusion feature, comprising:

[0185] Constructing a time series based on the multi-modal fusion feature.​

[0186] The LSTM model is used to process the time series, and the LSTM model includes:

[0187]

[0188] wherein, is the input vector of the time series at time step t, is the hidden state at time step t, is the cell state at time step t, , , are the forget gate, input gate and output gate, respectively, is the candidate cell state, , , is the weight matrix, , , is the bias vector, is the Hadamard product, is the sigmoid activation function, is the output prediction vector.

[0189] The behavior of the animal is predicted according to the prediction vector output by the LSTM model.

[0190] Behavior probability is calculated as:

[0191]

[0192] wherein, is the current behavior, is the context condition, is the next behavior, is the prediction score for the next behavior , is the set of all possible behaviors.

[0193] For example, the next behavior probability is predicted as:

[0194] Current state: Latency.

[0195] Model output score.

[0196] Attack: s = 2.5.

[0197] Continue latency: s = 1.0.

[0198] Retreat: s = 0.2.

[0199] Calculate the probability of each behavior:

[0200] Denominator = exp(2.5) + exp(1.0) + exp(0.2) = 12.18 + 2.72 + 1.22 = 16.12.

[0201] P(aggression | latency) = exp(2.5) / 16.12 = 12.18 / 16.12 ≈ 0.756 = 75.6%.

[0202] P(continue latency | latency) = exp(1.0) / 16.12 = 2.72 / 16.12 ≈ 0.169 = 16.9%.

[0203] P(retreat | latency) = exp(0.2) / 16.12 = 1.22 / 16.12 ≈ 0.076 = 7.6%.

[0204] Conclusion: Based on the current state of latency, the animal is most likely to take the next step of aggression (75.6%), followed by continuing latency (16.9%), and the lowest probability of retreat (7.6%).

[0205] The method of the present application further comprises S1000, performing behavior anomaly detection on the behavior of the identified animal:

[0206] Calculate the Z-score of the animal's behavior:

[0207]

[0208] where x is the observed value of the animal's behavior score, is the overall mean of the animal's behavior score, is the standard deviation of the animal's behavior score.

[0209] The Z-score of the animal's behavior is corrected to obtain the corrected score:

[0210]

[0211] where, is the median, is the absolute deviation from the median.

[0212] The behavior anomaly score S is obtained by correcting the score:

[0213]

[0214] where, is the weight of the i-th action, is the corrected score of the i-th action;

[0215] If S < 1 is normal behavior, if 1≤S < 3 is mild abnormality, if 3≤S < 6 is moderate abnormality, and if S≥ 6 is severe abnormality.

[0216] For example, the degree of behavioral abnormality is calculated:

[0217] Known observed abnormal indicators:

[0218] Activity frequency: Z1 = 2.5, w1 = 0.4 (weight 40%).

[0219] Movement speed: Z2 = 1.8, w2 = 0.3 (weight 30%).

[0220] Sound frequency: Z3 = 3.0, w3 = 0.3 (weight 30%).

[0221] Calculate the comprehensive abnormality score:

[0222] S = w1×Z1² + w2×Z2² + w3×Z3²

[0223] = 0.4×2.5² + 0.3×1.8² + 0.3×3.0²

[0224] = 0.4×6.25 + 0.3×3.24 + 0.3×9.0

[0225] = 2.5 + 0.972 + 2.7

[0226] = 6.172.

[0227] Conclusion: The comprehensive abnormality score is 6.172, indicating that the current behavior of the animal is in a state of severe abnormality, and further observation and possible intervention measures are needed.

[0228] The method of the application further comprises S1100 analyzing the spatial behavior of the animal:

[0229] Calculate the kernel density of the animal Estimate:

[0230]

[0231] Where n is the number of observation positions, h is the bandwidth parameter, K is the kernel function, and x i is the ith observation point.

[0232] Calculate the path curvature of the animal :

[0233]

[0234] where x', y' are first derivatives of the path parameter equation with respect to the parameter, and x'', y'' are second derivatives of the path parameter equation with respect to the parameter.

[0235] For example, compute the activity region usage intensity:

[0236] Position data points: [(2,3), (2,4), (3,3), (2,3), (4,5)].

[0237] Use a Gaussian kernel with bandwidth h = 1.0.

[0238] Compute the density estimate at the point (2,3):

[0239] First, compute the distance of each data point to (2,3):

[0240] d1 = √((2-2)² + (3-3)²) = 0.

[0241] d2 = √((2-2)² + (4-3)²) = 1.

[0242] d3 = √((3-2)² + (3-3)²) = 1.

[0243] d4 = √((2-2)² + (3-3)²) = 0.

[0244] d5 = √((4-2)² + (5-3)²) = √(4+4) = 2√2 ≈ 2.83.

[0245] Apply the Gaussian kernel to each distance:

[0246] K(d1 / h) = K(0 / 1) = K(0) = 1 / √(2π) × e^(-0² / 2) = 0.399.

[0247] K(d2 / h) = K(1 / 1) = K(1) = 1 / √(2π) × e^(-1² / 2) = 0.242.

[0248] K(d3 / h) = K(1 / 1) = K(1) = 0.242.

[0249] K(d4 / h) = K(0 / 1) = K(0) = 0.399.

[0250] K(d5 / h) = K(2.83 / 1) = K(2.83) = 1 / √(2π) × e^(-2.83² / 2) = 0.008.

[0251] Compute density estimate:

[0252] f(2,3) = (1 / 5×1) × (0.399 + 0.242 + 0.242 + 0.399 + 0.008)

[0253] = 0.2 × 1.29

[0254] = 0.258.

[0255] Conclusion: The value 0.258 represents the activity density at the coordinate (2,3). This point has a higher intensity of use relative to other locations because there are two data points that lie exactly at this point, and two more that are close by.

[0256] Compute path efficiency index E:

[0257]

[0258] where, L_actual is the actual path length, L_direct is the straight-line distance from start to end;

[0259] If E = 1, the path is a perfect straight line, highest efficiency. If 1 < E < 1.5, the path is highly efficient with slight detours. If 1.5≤ E < 2.0, the path is of moderate efficiency with noticeable detours. If E ≥ 2.0, the path is low efficiency with significant detours or meandering.

[0260] For example, analyze the efficiency of a patrol path:

[0261] Given: Start point A (0,0), end point B (10,8), actual path length: 25 meters.

[0262] Compute straight-line distance:

[0263] L_direct = √(10² + 8²) = √(100 + 64) = √164 ≈ 12.8 meters.

[0264] Compute path efficiency index:

[0265] E = L_actual / L_direct = 25 / 12.8 ≈ 1.95.

[0266] Conclusion: An efficiency index of 1.95 indicates that the animal's patrol path has noticeable detours, but has not yet reached the level of significant meandering. This could indicate that the animal has multiple points of interest to check during its patrol.

[0267] Compute interaction intensity

[0268]

[0269] where, N_ij is the interaction strength between individual i and individual j, N_ij is the number of interactions, D_ij is the average interaction duration (in minutes), W_t is the time decay weight (between 0 and 1, with closer interactions weighted more heavily), and T is the total observation time (in minutes).

[0270] For example, calculate the social strength between two animals:

[0271] Number of interactions (N_ij): 20 times.

[0272] Average interaction duration (D_ij): 3 minutes per interaction.

[0273] Observation time (T): 24 hours = 1440 minutes.

[0274] Time decay weight (W_t): 0.8 (indicating a relatively even distribution of interactions over time).

[0275] Calculate social strength:

[0276] I_ij = (N_ij × D_ij × W_t) / T

[0277] = (20 × 3 × 0.8) / 1440

[0278] = 48 / 1440

[0279] ≈ 0.033.

[0280] Social strength evaluation criteria:

[0281] I_ij < 0.01: Weak relationship.

[0282] 0.01 ≤ I_ij < 0.05: Moderate relationship.

[0283] 0.05 ≤ I_ij < 0.1: Close relationship.

[0284] I_ij ≥ 0.1: Strong relationship.

[0285] Conclusion: The social strength value of 0.033 indicates that there is a moderate level of social relationship between the two animals, with interaction frequency and duration at a medium level.

[0286] Group structure analysis:

[0287]

[0288] where, is the eigenvector centrality of animal i, is an element of the adjacency matrix, is the largest eigenvalue of the adjacency matrix, and N is the total number of animals.

[0289] For example, calculate the degree centrality:

[0290] Given the adjacency matrix (1 indicates two animals have interacted, 0 indicates no interaction):

[0291] A = [[0,1,1,0], [1,0,1,1], [1,1,0,0], [0,1,0,0]].

[0292] where the matrix rows correspond to animals 1, 2, 3, 4, respectively.

[0293] Calculate the degree centrality:

[0294] C_D(1) = ∑X1ⱼ = 0+1+1+0 = 2.

[0295] C_D(2) = ∑X2ⱼ = 1+0+1+1 = 3.

[0296] C_D(3) = ∑X3ⱼ = 1+1+0+0 = 2.

[0297] C_D(4) = ∑X4ⱼ = 0+1+0+0 = 1.

[0298] Normalize the degree centrality (divide by the maximum possible degree n-1 = 3):

[0299] C'_D(1) = 2 / 3 ≈ 0.67.

[0300] C'_D(2) = 3 / 3 = 1.00.

[0301] C'_D(3) = 2 / 3 ≈ 0.67.

[0302] C'_D(4) = 1 / 3 ≈ 0.33.

[0303] Conclusion: Animal 2 has the highest degree centrality, indicating that it has the most interactions with other individuals in the group and is centrally located in the social network. Animal 4 has the lowest degree centrality, indicating that it is more peripheral.

[0304] See Figure 2The application discloses a revealing system for zoo animal behavior, comprising a collection module 100, a posture model construction module 200, a sound model construction module 300, a physiological parameter feature model construction module 400, a data processing module 500, a multi-modal fusion module 600 and a behavior recognition module 700.

[0305] The collection module 100 is used for collecting video data, audio data, physiological data and environmental data of animals in a zoo.

[0306] The posture model construction module 200 is used for establishing a behavior classification system according to the behavior of an animal, the behavior classification system comprising a behavior state and a posture, and a posture model is constructed based on the relationship between the posture of the animal and the skeleton key points of the animal.

[0307] The sound model construction module 300 is used for analyzing a sound spectrogram according to a CNN or a Transformer model, establishing a sound model, and analyzing the relationship between sound, emotion and behavior through the sound model.

[0308] The physiological parameter feature model 400 construction module is used for establishing a physiological parameter feature model under different behavior states, and identifying the relationship between physiological parameters and emotion and behavior.

[0309] The data processing module 500 is used for extracting a skeleton key point sequence in video data through an LSTM or a GRU network, obtaining posture data of an animal based on the posture model, and obtaining motion features and facial features of the animal in the video data as video features; extracting a call type and a sound parameter of the animal in audio data as audio features; and extracting a heart rate mode, a body temperature change and a breathing mode in physiological data as physiological features.

[0310] The multi-modal fusion module 600 is used for obtaining video features, audio features, physiological features and environmental data of an animal at the same time or in the same time period, performing feature extraction and multi-modal fusion.

[0311] The behavior recognition module 700 is used for recognizing the behavior of an animal based on multi-modal fusion features by adopting a multi-modal Transformer model.

[0312] The behavior classification system comprises:

[0313] Motion behavior: walking, running, jumping, swimming, climbing and the like.

[0314] Rest behavior: standing, lying, sleeping and the like.

[0315] Feeding behavior: foraging, hunting, drinking water, chewing and the like.

[0316] Social behavior: approaching, sniffing, grooming, playing and fighting and the like.

[0317] Marking behavior: urine marking, scratch marking, rubbing marking, etc.

[0318] Self-maintenance: licking, scratching, stretching, etc.

[0319] Exploratory behavior: patrolling, sniffing, observing, etc.

[0320] Abnormal behavior: stereotypy, self-injury, excessive vigilance, etc.

[0321] The structure of the LSTM or GRU network includes:

[0322] Input layer: used to process 60 frames x 17 key points x 2 (x, y coordinates) sequence data.

[0323] Spatial feature extraction layer: used to process the spatial relationship between bone points using a graph convolution network (GCN).

[0324] Temporal feature extraction layer: double-layer bidirectional LSTM / GRU network, hidden unit number 256 and 128.

[0325] Attention mechanism: apply attention to time steps to highlight key action frames.

[0326] Output module: full connection layer followed by a Softmax classifier.

[0327] The posture data of the animal obtained based on the posture model includes:

[0328] Based on the extracted bone key point sequence, the included angle of the key point is calculated :

[0329]

[0330] wherein, , are two directional vectors of any two key points, is the dot product of the two directional vectors, is the length of the two directional vectors.

[0331] According to the calculated included angle of the key point a posture feature vector is constructed :

[0332]

[0333] wherein, N is the number of included angles of the key points, d is the key distance of the key points corresponding to the included angle, and M is the number of key distances.

[0334] Based on the posture feature vector The similarity is calculated through the posture model, and the corresponding posture is matched.

[0335] The feature extraction and the multi-modal fusion are performed, and the feature extraction includes:

[0336] The feature selection algorithm is used to screen the feature combination from the video features, the audio features and the physiological features of the animals at the same time or time period.

[0337] Based on the screened feature combination, the weighted voting is performed, and the environmental data is used for adaptive weight adjustment to perform the multi-modal fusion.

[0338] The performance weight of each modality is calculated by using the historical recognition accuracy of each modality, the final weight of each modality is obtained through the performance weight of each modality, and the weight of each modality in the multi-modal fusion is optimized.

[0339] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application but not limit the present application. Although the present application has been described by referring to the preferred embodiments thereof, those skilled in the art will understand that various changes in form and detail can be made without departing from the spirit and scope of the application as defined by the appended claims.

Claims

1. A method for revealing the behavior of zoo animals, characterized in that: include: Collect video data, audio data, physiological data and environmental data of animals in the zoo; Establish a behavior classification system based on animal behavior. The behavior classification system includes behavior state and posture. Construct a posture model based on the relationship between the animal's posture and the key points of the animal's skeleton. Analyze the spectrogram using CNN or Transformer models, build a sound model, and analyze the relationship between sound, emotion, and behavior through the sound model; Establish physiological parameter characteristic models under different behavioral states and identify the relationship between physiological parameters and emotions and behaviors; Extract skeleton key point sequences from video data through LSTM or GRU networks, obtain animal posture data based on posture models, and obtain animal motion features and facial features from video data as video features; extract animal call types and sound parameters from audio data as audio features; Extracting heart rate patterns, body temperature changes, and breathing patterns from physiological data as physiological features; Obtain video features, audio features, physiological features, and environmental data of animals at the same moment or time period, perform feature extraction, and perform multimodal fusion; A multimodal Transformer model is used to identify animal behaviors based on multimodal fusion features.

2. The method for revealing zoo animal behavior according to claim 1, characterized in that: The behavior classification system includes: Locomotor behavior: walking, running, jumping, swimming, and climbing postures; Resting behavior: standing, sitting, lying, and sleeping postures; Feeding behavior: foraging, catching food, drinking, and chewing postures; Social behaviors: approach, sniffing, grooming, play, and fighting postures; Marking behaviors: urine marking, claw marking, and rub marking gestures; Self-maintenance: licking, scratching, and stretching; Exploratory behavior: patrolling, sniffing, and observing postures; Abnormal behavior: stereotypic behavior, self-injury, and hypervigilant posture.

3. The method for revealing zoo animal behavior according to claim 2, characterized in that: The structure of the LSTM or GRU network includes: Input layer: used to process sequence data of 60 frames × 17 key points × 2 (x, y coordinates); Spatial feature extraction layer: used to process the spatial relationship between skeleton points using graph convolutional network (GCN); Temporal feature extraction layer: two-layer bidirectional LSTM / GRU network with 256 and 128 hidden units; Attention mechanism: Apply attention to time steps to highlight key action frames; Output module: fully connected layer followed by Softmax classifier.

4. The method for revealing zoo animal behavior according to claim 3, characterized in that: The step of obtaining the posture data of the animal based on the posture model includes: Calculate the angle of key points based on the extracted skeleton key point sequence : in, 、 are two direction vectors of any two key points, is the dot product of two direction vectors, is the modulus of the two direction vectors; According to the calculated angle of the key points Constructing posture feature vector : Among them, N is the number of angles of key points, d is the key distance of the key points corresponding to the angle, and M is the number of key distances; Based on posture feature vector The similarity is calculated through the posture model to match the corresponding posture.

5. The method for revealing zoo animal behavior according to claim 4, characterized in that: The feature extraction and multimodal fusion includes: A feature selection algorithm is used to select a feature combination from the video features, audio features, and physiological features of animals at the same moment or time period; Based on the combination of screened features, multimodal fusion is performed based on weighted voting and adaptive weight adjustment using environmental data; The historical recognition accuracy of each modality is used to calculate the performance weight of each modality. The final weight of each modality is obtained through the performance weight of each modality, and the weight of each modality in multimodal fusion is optimized.

6. The method for revealing zoo animal behavior according to claim 5, characterized in that: Also includes: Emotional state of animals based on multimodal fusion features Perform analysis: in, is the weight of the feature, is the normalized value of the feature, and N is the total number of features.

7. The method for revealing zoo animal behavior according to claim 6, characterized in that: It also includes prediction of animal behavior based on multimodal fusion features, including: Construct time series based on multimodal fusion features; The LSTM model is used to process time series. The LSTM model includes: in, is the input vector of the time series at time step t, is the hidden state at time step t, is the cell state at time step t, 、 、 They are forget gate, input gate and output gate respectively. is the candidate cell state, 、 、 is the weight matrix, 、 、 is the bias vector, is the Hadamard product, is the sigmoid activation function, is the output prediction vector; The animal's behavior is predicted based on the prediction vector output by the LSTM model.

8. The method for revealing zoo animal behavior according to claim 7, characterized in that: It also includes behavioral anomaly detection of the behavior of identified animals: Calculate the Z score of the animal's behavior: Where x is the observed value of the animal behavior score, is the overall mean of the animal behavior scores, the standard deviation for scoring animal behavior; Correct the Z score of the animal's behavior and get the correction Fraction: in, is the median, is the median absolute deviation; By amendment Score for behavioral abnormality score S: in, is the weight of the ith action, is the correction of the i-th action Fraction; If S < 1, it is normal behavior; if 1≤S < 3, it is mild abnormality; if 3≤S < 6, it is moderate abnormality; if S ≥ 6, it is severe abnormality.

9. The method for revealing zoo animal behavior according to claim 8, characterized in that: It also includes analysis of the spatial behavior of animals: Calculate the kernel density of animals estimate: Where n is the number of observation positions, h is the bandwidth parameter, K is the kernel function, and x i is the i-th observation point; Path curvature for animals Perform the calculation: Among them, x' and y' are the first-order derivatives of the path parameter equation with respect to the parameter, and x'' and y'' are the second-order derivatives of the path parameter equation with respect to the parameter; Calculate the path efficiency index E: in, is the actual path length, is the straight-line distance from the starting point to the end point; If E = 1, it is a completely straight path with the highest efficiency. If 1 < E < 1.5, it is an efficient path with slight detours. If 1.5 ≤ E < 2.0, it is a moderately efficient path with obvious detours. If E ≥ 2.0, it is an inefficient path with significant detours or wandering.

10. A system for revealing the behavior of zoo animals, characterized in that: It includes acquisition module, posture model building module, sound model building module, physiological parameter feature model building module, data processing module, multimodal fusion module and behavior recognition module; The acquisition module is used to collect video data, audio data, physiological data and environmental data of animals in the zoo; The posture model building module is used to establish a behavior classification system according to the behavior of the animal, the behavior classification system includes behavior state and posture, and builds a posture model based on the relationship between the posture of the animal and the key points of the animal's skeleton; The sound model building module is used to analyze the spectrogram according to the CNN or Transformer model, establish a sound model, and analyze the relationship between sound, emotion and behavior through the sound model; The physiological parameter characteristic model building module is used to establish physiological parameter characteristic models under different behavioral states and identify the relationship between physiological parameters and emotions and behaviors; The data processing module is used to extract the skeleton key point sequence in the video data through the LSTM or GRU network, obtain the posture data of the animal based on the posture model, and obtain the movement characteristics and facial features of the animal in the video data as video features; extract the animal's call type and sound parameters in the audio data as audio features; Extracting heart rate patterns, body temperature changes, and breathing patterns from physiological data as physiological features; The multimodal fusion module is used to obtain video features, audio features, physiological features and environmental data of animals at the same time or time period, perform feature extraction and multimodal fusion; The behavior recognition module is used to recognize the behavior of animals based on multimodal fusion features using a multimodal Transformer model.

Images